CN115965634A - Image processing method, image processing device, computer equipment and storage medium - Google Patents

Image processing method, image processing device, computer equipment and storage medium Download PDF

Info

Publication number
CN115965634A
CN115965634A CN202111170599.XA CN202111170599A CN115965634A CN 115965634 A CN115965634 A CN 115965634A CN 202111170599 A CN202111170599 A CN 202111170599A CN 115965634 A CN115965634 A CN 115965634A
Authority
CN
China
Prior art keywords
image
background
target
filling
block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111170599.XA
Other languages
Chinese (zh)
Inventor
高语阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202111170599.XA priority Critical patent/CN115965634A/en
Publication of CN115965634A publication Critical patent/CN115965634A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Image Processing (AREA)

Abstract

The application relates to an image processing method, an image processing device, a computer device and a storage medium. The method comprises the following steps: acquiring a target image; carrying out foreground object identification on the target image to obtain an object area; removing an object area image from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image of the object area and a background image of the target image; acquiring a background image block matched with the blank image from the background image, and filling the background to be used as a filled image; performing style transformation and plane transformation on the filling image to obtain a transformed image, and determining a target view plane when performing plane transformation so as to enable the transformed image to be located on the target view plane; and fusing the transformed image and the object region image to obtain a processed image. By adopting the method, the filled image background can be transited naturally and completely and has a corresponding style, and the foreground and the background of the processed image have a three-dimensional relationship, so that the image processing effect is improved.

Description

Image processing method, image processing device, computer equipment and storage medium
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image processing method and apparatus, a computer device, and a storage medium.
Background
With the development of image processing technology, image processing is required in many cases, such as making it possible to beautify a person in an image or adding an ornament. For example, a user may import a picture into a picture processing program and process the image by clicking a function control in the image processing program. Alternatively, the image may be input to an image generation model and subjected to a genre conversion to obtain a genre-converted image
However, the conventional image processing method has a problem that the processed image is poor in effect.
Disclosure of Invention
In view of the foregoing, it is necessary to provide an image processing method, an apparatus, a computer device, and a storage medium for solving the above technical problems.
A method of image processing, the method comprising:
acquiring a target image to be processed;
carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image;
removing an image corresponding to the object area from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object area and a background image corresponding to the target image;
acquiring a background image block matched with the blank image from the background image, performing background filling on the blank image in the intermediate image according to the background image block, and taking the filled intermediate image as a filled image;
performing style transformation and plane transformation on the filling image to obtain a transformed image, wherein when the plane transformation is performed, plane transformation information corresponding to a target view plane is determined, and the filling image is subjected to plane transformation according to the plane transformation information, so that the transformed image is an image positioned on the target view plane;
and fusing the transformed image and the image corresponding to the object region to obtain a processed image corresponding to the target image.
An image processing apparatus, the apparatus comprising:
the target image acquisition module is used for acquiring a target image to be processed;
the foreground object identification module is used for carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image;
an intermediate image obtaining module, configured to remove an image corresponding to the object area from the target image to obtain an intermediate image, where the intermediate image includes a blank image corresponding to the object area and a background image corresponding to the target image;
a filling image obtaining module, configured to obtain a background image block matched with the blank image from the background image, perform background filling on the blank image in the intermediate image according to the background image block, and use the filled intermediate image as a filling image;
the conversion image acquisition module is used for carrying out style conversion and plane conversion on the filling image to obtain a conversion image, wherein when the plane conversion is carried out, plane conversion information corresponding to a target view plane is determined, and the filling image is subjected to plane conversion according to the plane conversion information, so that the conversion image is an image positioned on the target view plane;
and the processed image acquisition module is used for fusing the converted image and the image corresponding to the object area to obtain a processed image corresponding to the target image.
In an embodiment, the filler image obtaining module is configured to perform region division on the object region to obtain a plurality of divided regions; for the current pixel point in the blank image, determining a division region to which the current pixel point belongs, and using the division region as a reference division region; acquiring a background image block matched with the reference division area, and acquiring a pixel point corresponding to the current pixel point in the background image block as a reference background pixel point; and obtaining a filling pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point, and performing background filling on the blank image in the intermediate image according to the filling pixel value.
In one embodiment, the filler image is obtained through multiple rounds of background filling, and the filler image obtaining module is configured to select one or more candidate background image blocks corresponding to the reference partition area in a current background filling round based on an image block selection policy; acquiring a filling image block obtained after the reference divided area is filled in the previous round of background, and acquiring a matched background image block corresponding to the reference divided area in the previous round of background filling; calculating the similarity of the candidate background image blocks and the filling image blocks to serve as a first similarity; calculating the similarity of the matched background image block and the filled image block to serve as a second similarity; based on the first similarity and the second similarity, obtaining an image block similar to the filling image block from the matching background image block and the candidate background image block, and using the image block as a background image block matched with the reference division area in the current background filling turn.
In an embodiment, the filled image obtaining module is configured to, when a plurality of reference divided areas corresponding to the current pixel point are provided, obtain a similarity between a background image block matched with the reference divided area and a corresponding filled image block in a current background filling turn; obtaining a pixel weight corresponding to the reference background pixel point in the current background filling turn based on the similarity, wherein the pixel weight and the similarity form a positive correlation relationship; and performing weighted calculation based on the pixel value corresponding to the reference background pixel point and the pixel weight corresponding to the reference background pixel point to obtain a filling pixel value corresponding to the current pixel point.
In one embodiment, the filler image obtaining module is configured to determine a reference divided region corresponding to the reference divided region in the plurality of divided regions; acquiring the position of a background image block matched with the standard division area in the last round of background filling as a reference position; acquiring a relative position relationship between the reference divided region and the reference divided region; obtaining a first image block selection position according to the reference position and the relative position relation; and taking the background image block corresponding to the selected position of the first image block as the candidate background image block.
In one embodiment, the filler image acquisition module is configured to acquire a randomly generated random offset distance; acquiring a corresponding offset distance contraction value based on the image block ordering of the candidate background image block, wherein the contraction degree corresponding to the offset distance contraction value and the image block ordering form a positive correlation; shrinking the random offset distance based on the offset distance shrinking value to obtain a target offset distance; obtaining a second image block selection position according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance; and taking the background image block corresponding to the selected position of the second image block as the candidate background image block.
In an embodiment, the transformed image obtaining module is configured to perform style transformation on the filler image to obtain an initial style image; acquiring a background object image corresponding to a target background object in the initial style image, and removing the background object image from the initial style image to obtain an image to be processed in the initial style image; carrying out color clustering on the image to be processed to obtain a color clustering image; and fusing the color clustering image and the background object image, and obtaining a target style image based on the image obtained by fusing.
In one embodiment, the transformed image obtaining module is configured to obtain a color parameter threshold range corresponding to a target background object, where the target background object is a background object whose color consistency is greater than a consistency threshold; screening out pixel points with color parameter values within the color parameter threshold range from the initial style image, and taking the pixel points as target pixel points; and taking the continuous image blocks corresponding to the target pixel points in the initial style image as background object images corresponding to the target background object.
In an embodiment, the transformed image obtaining module is configured to fuse the color cluster image with the background object image to obtain a fused image; performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and removing the edge image from the fused image to obtain a target style image.
In an embodiment, the transformed image obtaining module is configured to perform edge extraction on the initial style image by using an edge extraction operator to obtain a first edge image corresponding to the initial style image; utilizing an edge extraction model to carry out edge extraction on the initial style image to obtain a second edge image corresponding to the initial style image; and acquiring an image intersection between the first edge image and the second edge image, and taking the image intersection as an edge image corresponding to the initial style image.
In one embodiment, the target image acquisition module is further configured to receive an image processing request, where the image processing request is triggered by the terminal in response to a stereoscopic image generation operation; the transformed image acquisition module is used for determining the image stereoscopic effect displayed by the stereoscopic image generation operation, and acquiring a view plane corresponding to the image stereoscopic effect as the target view plane; and determining plane transformation information corresponding to the target view plane, and performing plane transformation on the filling image according to the plane transformation information so as to enable the transformed image to be an image located in the target view plane.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the image processing method described above when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the image processing method described above.
A computer program product comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned image processing method.
The image processing method, the image processing device, the computer equipment and the storage medium acquire a target image to be processed; carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image; removing an image corresponding to the object area from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object area and a background image corresponding to the target image; acquiring a background image block matched with the blank image from the background image, performing background filling on the blank image in the intermediate image according to the background image block, and taking the filled intermediate image as a filled image; performing style transformation and plane transformation on the filling image to obtain a transformed image, wherein when the plane transformation is performed, plane transformation information corresponding to a target view plane is determined, and the plane transformation is performed according to the plane transformation information to enable the transformed image to be an image located in the target view plane; and fusing the transformed image and the image corresponding to the object area to obtain a processed image corresponding to the target image. According to the method and the device, the target image is identified by the foreground object, the region of the foreground object is removed from the target image, the blank image region obtained after removal is filled by the background of the target image, the filled image background can be in natural and complete transition, the filled image is in the target view plane by performing style transformation and plane transformation on the filled image, and the filled image and the image corresponding to the target region are fused, so that the image can have a corresponding style, a three-dimensional relationship can be formed between the foreground and the background of the processed image, and the image processing effect is improved.
Drawings
FIG. 1A is a diagram of an exemplary environment in which an image processing method may be implemented;
FIG. 1B is a flowchart illustrating an exemplary method of image processing;
FIG. 2 is a diagram illustrating an intermediate image obtained by removing an object region from a target image according to an embodiment;
FIG. 3 is a diagram illustrating filling of an object region of an intermediate image to obtain a filled image in one embodiment;
FIG. 4 is a schematic diagram showing processing of images in one embodiment;
FIG. 5 is a flow diagram illustrating background filling of a blank image in an intermediate image according to an embodiment;
FIG. 6 is a flowchart illustrating a process of obtaining a background image block matched with a reference partition area according to an embodiment;
FIG. 7 is a diagram illustrating a position relationship between a current pixel point and a reference division area in one embodiment;
FIG. 8 is a schematic flow chart diagram that illustrates stylistic transforming a filler image, under an embodiment;
FIG. 9 is a diagram illustrating an interface presented by the image processing terminal in one embodiment;
FIG. 10 is a flowchart illustrating an image processing method according to another embodiment;
FIG. 11 is a schematic flow chart illustrating the processing of a street view pixel fan filter system in one embodiment;
FIG. 12 is a flowchart illustrating an exemplary process for image fill;
FIG. 13 is a block diagram showing a configuration of an image processing apparatus according to an embodiment;
FIG. 14 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for presentation, analyzed data, etc.) referred to in this application are information and data authorized by the user or sufficiently authorized by each party; correspondingly, the application also provides a corresponding user authorization entrance for the user to select authorization or to select denial.
The image processing method provided by the application can be applied to the application environment shown in fig. 1A. Wherein the terminal 102 communicates with the server 104 via a network. The terminal 102 may send an image processing request to the server, where the image processing request may carry a target image to be processed or may be an image identifier of the target image, and the server may execute the image processing method provided in this application and return the processed image to the terminal. The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, vehicle-mounted devices, and portable wearable devices, and the server 104 may be implemented by an independent server or a server cluster formed by a plurality of servers. It is understood that the terminal 102 may also execute the image processing method of the present application.
In an embodiment, as shown in fig. 1B, an image processing method is provided, and this embodiment is illustrated by applying the method to a terminal, and it is to be understood that the method may also be applied to a server, and may also be applied to a system including a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the steps of:
step S102, a target image to be processed is acquired.
The target image is an image material that needs to be subjected to image processing, and may be an image input by a user in an application program for implementing an image processing process, or an image stored in advance in an image database, and the image processing may be an image processing process for generating a stereoscopic image corresponding to the target image. The processing procedure for the target image may be implemented by a user terminal such as various personal computers, notebook computers, smart phones, tablet computers, vehicle-mounted devices, and portable wearable devices, or may be implemented by a background server.
If the image processing flow is realized by the terminal, when a user needs to perform image processing on a certain image, the image can be input into the terminal as a target image to be processed, the terminal directly executes the image processing process aiming at the target image so as to output a corresponding processed image after the image processing, and if the image processing flow is realized by the server, the terminal can send the target image to the server after receiving the target image input by the user, and the server executes the image processing process so as to obtain the corresponding processed image after the image processing and return the processed image to the terminal for outputting.
And step S104, performing foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image.
The foreground object refers to an object corresponding to a foreground portion in a target image, and generally speaking, the target image may be composed of a foreground portion and a background portion, where the foreground portion generally refers to an object located in front of a subject or close to a lens of a camera in the target image, and may be used to represent a certain spatial relationship or a human relationship, and the background generally refers to a scene located behind the subject and far from the camera, and belongs to an important component of an environment. Taking the example of capturing a landscape image of a person, in general, the person in the image can be taken as a foreground object and the landscape around the person can be taken as a corresponding background image, while if the image is a landscape image capturing an animal, the animal in the image can be taken as a foreground object and the landscape around the animal is taken as a background image. The object region corresponding to the foreground object refers to an image region occupied by the foreground object in the target image.
Specifically, after the target image is obtained, the terminal may identify an object in a foreground portion of the target image, for example, when the uploaded target image is an image in which a person is photographed, the terminal may use the person as a foreground object, so that by identifying the person in the image, the person in the target image may be extracted as a foreground object based on a person identification algorithm, and an area occupied by the person in the target image may be obtained as an object area corresponding to the foreground object.
And S106, removing the image corresponding to the object area from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object area and a background image corresponding to the target image.
The intermediate image refers to an image of a background area reserved after a target area is removed from the target image, in the intermediate image, a blank area appears after the target area is removed, the image corresponding to the blank area can be used as a blank image, and the background image refers to an image corresponding to the background area in the target image.
Referring to fig. 2, a portion 201 in fig. 2 indicates an input target image, where the target image may include a person region and a background region, and a portion 202 in fig. 2 indicates an intermediate image, where the intermediate image is an image obtained by removing an image corresponding to the person region in the target image, and thus the intermediate image still carries an image corresponding to the background region in the target image, and the person region originally located in the target image is filled with a corresponding blank image.
And S108, acquiring a background image block matched with the blank image from the background image, performing background filling on the blank image in the intermediate image according to the background image block, and taking the filled intermediate image as a filled image.
The background image block refers to an image block in a background area in the intermediate image, and the intermediate image may be composed of a plurality of image blocks, where a plurality refers to at least two. That is, the intermediate image may include an image block located in the blank image and an image block located in the background area, where the background image block is the image block in the background area, and the filled image refers to an image obtained by filling the blank image in the intermediate image. Because the intermediate image is an image obtained by removing the object region from the target image and carries a blank image corresponding to the object region, in order to achieve the effect of picture harmony, the object region missing in the intermediate image can be filled, for example, a background image block in the background region can be used to fill the blank image, so that the blank image can display corresponding image colors, and thus the harmony of the image picture can be ensured.
Specifically, the blank image filling process may be to find a background image block matching the blank image from a background image, for example, each image block included in the blank image may be searched for a background image block with the highest matching degree in the background, then each background image block obtained by the search may be used as a background image block matching the blank image, then each matched background image block may be used to perform background filling on the blank image, or each image block of the blank image may be filled with a background image block matching the blank image, so that background filling of the blank image is achieved, and the blank image is used as a filled image.
Reference may be made to fig. 3, where a portion 301 in fig. 3 represents an intermediate image obtained after removing an object area from a target image, where the intermediate image may include a blank image and a background image, where an image block a in the blank image matches an image block B in the background image, when the blank image is filled, the image block B may be filled into a corresponding position of the image block a, and the image block filling process in other positions may be similar to the above process until all image blocks in the blank image are filled, so as to obtain a filled image, as shown in a portion 302 in fig. 3.
And step S110, performing style transformation and plane transformation on the filling image to obtain a transformed image, wherein when the plane transformation is performed, plane transformation information corresponding to a target view plane is determined, and the filling image is subjected to plane transformation according to the plane transformation information, so that the transformed image is an image located in the target view plane.
The style conversion may refer to a process of performing stylized conversion on the filled image obtained in step S104, and may be used to change the image background style of the filled image, where the stylized conversion process may be to perform cartoon processing on the filled image and convert the filled image into an expression form such as a cartoon, or perform black-and-white processing on the filled image and convert the filled image into a black-and-white background. The process of plane transformation refers to a process of converting a current located view plane, i.e., a horizontal plane of a viewpoint, corresponding to the filled image into a certain preset view plane or a view plane selected by a user, i.e., a target view plane, where the target view plane is set by the user according to an image processing effect that the user needs to realize on the target image, for example, when the user needs to generate a stereo image corresponding to the target image, and the background is located at the bottom of the stereo image, the view plane corresponding to the bottom of the stereo image may be used as the target view plane. By performing the view plane transformation on the background, for example, the background may be processed by translation, rotation, and the like, and the transformed image may have different visual effects, where the plane transformation information refers to transformation information required by the view plane transformation, for example, a matrix for implementing the plane transformation, and the like, and may be implemented by multiplying the filler image by the plane transformation matrix.
Meanwhile, the style conversion process and the plane conversion process are processes of further image processing on the obtained filling image respectively, and the sequence of the process can be that the style conversion is firstly carried out on the filling image, then the plane conversion is carried out on the filling image after the style conversion, the filling image is converted into a conversion image displayed in a target view plane, or the sequence can be that the plane conversion is firstly carried out on the filling image, the filling image is firstly converted into the conversion image displayed on the target view plane, then the style conversion processing is carried out on the filling image, and the style conversion is carried out on the filling image, so that the conversion image is obtained.
And step S112, fusing the converted image and the image corresponding to the object area to obtain a processed image corresponding to the target image.
The processed image is an image obtained by performing complete image processing on the target image, and may be a stereoscopic image specific to the target image, where the stereoscopic image may be a stereoscopic image in which a foreground object in the target image is in a standing state, and an image obtained by performing style conversion on a background portion of the target image is used as the bottom of the stereoscopic image, so as to present a visual effect that the foreground object is separated from a background, and the foreground object comes out from the background, for example, an effect that a person comes out from a cartoon can be achieved. The effect of the resulting processed image can be seen in fig. 4.
In the image processing method, a target image to be processed is obtained; carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image; removing an image corresponding to the object area from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object area and a background image corresponding to the target image; acquiring a background image block matched with the blank image from the background image, performing background filling on the blank image in the intermediate image according to the background image block, and taking the filled intermediate image as a filled image; performing style transformation and plane transformation on the filling image to obtain a transformed image, wherein when the plane transformation is performed, plane transformation information corresponding to a target view plane is determined, and the plane transformation is performed according to the plane transformation information to enable the transformed image to be an image located in the target view plane; and fusing the transformed image and the image corresponding to the object area to obtain a processed image corresponding to the target image. According to the method and the device, the target image is identified by the foreground object, the region of the foreground object is removed from the target image, the blank image region obtained after removal is filled by the background of the target image, the filled image background can be in natural and complete transition, the filled image is in the target view plane by performing style transformation and plane transformation on the filled image, and the filled image and the image corresponding to the target region are fused, so that the image can have a corresponding style, a three-dimensional relationship can be formed between the foreground and the background of the processed image, and the image processing effect is improved.
In one embodiment, as shown in fig. 5, step S108 may further include:
in step S502, the target region is divided into a plurality of divided regions.
The divided regions may be a plurality of regions obtained by dividing the object region in the intermediate image, and may be a plurality of image blocks in a blank image in the object region.
Step S504, for the current pixel point in the blank image, determining the partition region to which the current pixel point belongs, as a reference partition region.
The current pixel point refers to any one pixel point included in an object region of the blank image, the blank image is composed of a plurality of pixel points, a divided region obtained by dividing the blank image can also include a plurality of pixel points, the current pixel point can be one pixel point in one divided region of the blank image, and the divided region where the current pixel point is located can be used as a reference divided region corresponding to the current pixel point.
For example, the object region corresponding to the blank image may be divided into a division region a, a division region B and a division region C, where each division region may include 3 pixel points, and the division region a may include a pixel point 1, a pixel point 2 and a pixel point 3, the division region B includes a pixel point 4, a pixel point 5 and a pixel point 6, and the division region C includes a pixel point 7, a pixel point 8 and a pixel point 9, and the current pixel point may be any one of the above-mentioned pixel points, for example, the pixel point 4, and since the pixel point 4 is located in the division region B, the division region B may serve as a reference division region corresponding to the current pixel point 4.
Step S506, a background image block matched with the reference partition area is obtained, and a pixel corresponding to the current pixel in the background image block is obtained and used as a reference background pixel.
The reference background pixel point refers to a pixel point corresponding to the current pixel point in the background image block corresponding to the reference division area, and may be a pixel point corresponding to the position of the current pixel point in the reference division area in the matched background image block.
For example, the reference division region B corresponding to the current pixel point, which is the pixel point 4, may include a pixel point 4, a pixel point 5, and a pixel point 6, and the background image block B ' matched with the reference division region B may include a pixel point 4', a pixel point 5', and a pixel point 6', and pixel distributions of the pixel point 4', the pixel point 5', and the pixel point 6' are the same as those of the pixel point 4, the pixel point 5, and the pixel point 6, so that the pixel point 4' in the background image block B ' may be used as the current pixel point, that is, the reference background pixel point corresponding to the pixel point 4.
Step S508, obtaining a filling pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point, and performing background filling on the blank image in the intermediate image according to the filling pixel value.
The filling pixel value refers to a pixel value that needs to be filled into a current pixel point, and since the current pixel point is a pixel point in a blank image, when the current pixel point is not filled, the pixel value of the pixel point is 0.
In this embodiment, the object region of the blank image may be divided to obtain a reference division region corresponding to each pixel point in the blank image, and a background image block matched with the reference division region may be used to find a reference background pixel point corresponding to each pixel point, and then the pixel value is used to fill the pixel point, so that accuracy of filling the background of the blank image may be improved.
In one embodiment, the filling image is obtained through multiple rounds of background filling, as shown in fig. 6, step S506 may further include:
step S602, selecting one or more candidate background image blocks corresponding to the reference division area in the current background filling turn based on the image block selection strategy.
In this embodiment, the process of performing background filling on the blank image may be composed of multiple iterations, where the current background filling iteration refers to the current iteration, and the candidate background image block refers to a background image block that is searched in the background image in the current iteration process and may be matched with the reference divided region. In this embodiment, each iteration selects one or more background image blocks that may match the reference divided area from the background image as candidate background image blocks, and the selection is performed by a predetermined image block selection strategy, for example, the divided areas adjacent to the reference divided area, the image blocks near the matched background image block as the candidate background image blocks of the reference divided area, and so on.
Step S604, acquiring a filled image block obtained after the previous background filling of the reference divided area, and acquiring a matched background image block corresponding to the reference divided area in the previous background filling.
The previous background filling round refers to a previous background filling round of the current background filling round, for example, the current background filling round is a third background filling round, the previous background filling round may be a second background filling round, the filled image block refers to an image block corresponding to the reference divided region after background filling is performed on the reference divided region in the previous iteration, that is, an image block displayed in the current reference divided region, and the matched background image block refers to an image block determined in the previous background filling round and best matched with the reference divided region. If the current background filling round is the first round, the terminal may further randomly set a background image block corresponding to each reference divided area in the background image as a matching background image block corresponding to the reference divided area.
Step S606, calculating the similarity of the candidate background image block and the filling image block as a first similarity.
The first similarity refers to a candidate background image block corresponding to a reference division region screened in a current background filling round, and the candidate background image block and an image block currently displayed in the reference division region, that is, the similarity of a filled image block obtained after the previous background filling round, where the similarity can be obtained by calculating pixel values of pixel points included in the candidate background image block and pixel values of pixel points included in the filled image block, for example, by calculating a distance between the pixel points, so that the similarity between the image blocks is represented by the distance, and the greater the distance, the smaller the similarity is represented.
Step S608, calculating the similarity between the matched background image block and the filled image block as a second similarity.
The second similarity refers to the similarity between the background image block determined in the previous background filling round and matched with the reference divided area and the filling image block, where the number of the matched background image blocks may be one or multiple, and then the second similarity may be calculated in a manner similar to that in step S606.
Step S610, based on the first similarity and the second similarity, an image block similar to the filling image block is obtained from the matching background image block and the candidate background image block and is used as the background image block matched with the reference divided area in the current background filling round.
And finally, finding an image block similar to the filling image block from the matching background image block and the candidate background image block according to the determined first similarity and the determined second similarity, wherein a similarity threshold is determined according to the first similarity and the second similarity, and when the similarity threshold is greater than a certain value, the background image block is used as the image block similar to the filling image block, namely the background image block matched with the reference division area in the current background filling round. If only one background image block is required to be screened as the background image block matched with the reference division area, the background image block with the maximum similarity to the filling image block can be found from the matched background image block and the candidate background image blocks as the background image block matched with the reference division area, and after the background image block matched with the reference division area in the current background filling round is obtained, the background image block can be used as the matched background image block corresponding to the reference division area in the next background filling round.
Taking the example of screening only one background image block in each round as the background image block matched with the reference divided area, assuming that the current background filling round is the third time, the candidate background image blocks corresponding to the reference divided area can be obtained according to the image block selection strategy, such as the image block B, the image block C and the image block D, and can also determine the filled image block obtained after the last background filling of the reference divided area, i.e., the image block currently shown with reference to the partitioned area, may be image block a, and in the previous round of background filling, i.e., the image block found in the second background fill that matches the reference divided area, may be image block E, then image block a and the candidate background image block may be computed separately, i.e. the similarity between the image block a and the image block B, the image block C and the image block D, respectively, and the similarity between the image block a and the matching background image block, i.e. the similarity between the image block a and the image block E, and find the image block with the greatest similarity, e.g. the image block C has the greatest similarity with the image block a, then the image block C may be used as the background image block matched with the reference division area in the current background filling round, and simultaneously as the next background filling round, i.e. the matching background image block corresponding to the reference divided area in the fourth round of background filling, and the terminal may further background fill the reference divided area with the image block C, the image block C screened in the current round can be used for filling the reference divided area, the image block obtained after the filling can be an image block F which is used as the image block displayed after the background filling in the current background filling round, and meanwhile, the image block is used as the filling image block corresponding to the reference division area in the next background filling round, namely the fourth background filling round.
In this embodiment, there may be a plurality of rounds of background filling for the blank image, and each filling round may determine, according to the similarity, an image block similar to the filled image block for each reference divided region, and use the image block as a background image block matched with the reference divided region in the round, so that the matching degree of the screened background image blocks may be improved.
In one embodiment, step S508 may further include: when a plurality of reference division areas corresponding to the current pixel point are available, acquiring the similarity between the background image block matched with the reference division areas and the corresponding filling image block in the current background filling turn; obtaining a pixel weight corresponding to a reference background pixel point in the current background filling round based on the similarity, wherein the pixel weight and the similarity form a positive correlation; and performing weighted calculation based on the pixel value corresponding to the reference background pixel point and the pixel weight corresponding to the reference background pixel point to obtain a filling pixel value corresponding to the current pixel point.
The pixel weight may refer to an influence weight of a pixel value of each reference background pixel point corresponding to the current pixel point on a filling pixel value of the current pixel point, and the size of the pixel weight may form a positive correlation with the similarity between a background image block matched with each reference division area and a corresponding filling image block thereof, that is, when the similarity is larger, the pixel weight may also be correspondingly increased, the terminal may determine the pixel weight corresponding to each reference background pixel point according to the size relationship of each similarity, and then may perform a weighting operation on the pixel value corresponding to each reference background pixel point by using the pixel weight, thereby obtaining the filling pixel value corresponding to the current pixel point.
In this embodiment, as shown in fig. 7, overlapping image regions exist in a plurality of divided regions obtained by performing region division on an object region, when a current pixel point falls into the plurality of divided regions and overlapping image regions exist, a plurality of reference divided regions may exist in the current pixel point, that is, a plurality of reference divided regions corresponding to the current pixel point are respectively a divided region a and a divided region B, and at the same time, a background image block a 'and a background image block B' which are matched with the divided regions a and B respectively exist, and the background image block a 'and the background image block B' respectively carry reference background pixel points corresponding to the current pixel point, then, the similarities between the divided regions a and the background image block a 'filled in a current background filling round, and the similarities between the divided regions B and the background image block B' filled in the current background filling round can be respectively obtained, and pixel weights corresponding to the reference pixel points respectively carried by the background image block a 'and the image block B' are determined according to the similarities, and the pixel weights of the current pixel points of the background image block a 'and the background image block B' are finally calculated, so as to obtain pixel weight values of the pixel points.
In this embodiment, the current pixel may correspond to multiple reference partition regions, and after the reference background pixel corresponding to each reference partition region is determined, the corresponding pixel weight may also be determined according to the similarity between each reference partition region and the matched background image block, so that the matched block with a large similarity has a higher reliability, and the accuracy of the obtained filled pixel value is further improved.
In one embodiment, step S502 may further include: determining a reference divided area corresponding to a reference divided area in the plurality of divided areas; acquiring the position of a background image block matched with the standard division area in the last round of background filling as a reference position; acquiring a relative position relation between a reference divided region and a reference divided region; obtaining a first image block selection position according to the reference position and the relative position relation; and taking the background image block corresponding to the selected position of the first image block as a candidate background image block.
In this embodiment, the reference divided region refers to a preset divided region matched with the reference divided region, for example, a divided region adjacent to the reference divided region may be used as the reference divided region, or a divided region having a certain positional relationship with the reference divided region, for example, a divided region on an oblique angle thereof is used as the reference divided region, the relative positional relationship refers to a positional relationship between the reference divided region and the reference divided region, which may be adjacent up and down or adjacent left and right, and the like, the reference position refers to a position of a background image block matched with the reference divided region in the previous background filling round, the first image block selection position is a position of a background image block determined according to a relationship between the reference position and the relative position, and the first image block selection position may be a position of a candidate background image block corresponding to the reference divided region.
Since the background image block best matching the reference divided region is generally located in the adjacent divided region, that is, near the background image block best matching the reference divided region, the image block position of the candidate background image block corresponding to the reference divided region, that is, the first image block selection position, can be determined by the position of the matched background image block found by the reference divided region and the relative position relationship between the reference divided region and the reference divided region, and then the background image block at the first image block selection position can be used as the candidate background image block.
For example, when the reference divided area a is above the reference divided area B, after determining the position of the background image block a 'of the reference divided area, which is matched with the previous background filling turn, that is, after determining the reference position, the first image block selection position may be obtained according to the reference position and the position relationship between the reference divided area and the reference divided area, that is, the position below the reference position is used as the first image block selection position, and the background image block corresponding to the first image block selection position is used as the candidate background image block corresponding to the reference divided area, that is, the image block below a' is used as one of the candidate background image blocks of the reference divided area B.
In this embodiment, the candidate background image block of the reference divided region may also be obtained by using the position relationship between the reference divided region and the position of the background image block matched with the reference divided region in the previous background filling, so that the complexity of search may be greatly reduced by using the continuity of the image, and the acquisition efficiency of the candidate background image block may be improved.
In one embodiment, step S502 may further include: acquiring a randomly generated random offset distance; acquiring a corresponding offset distance contraction value based on the image block ordering of the candidate background image blocks, wherein the contraction degree corresponding to the offset distance contraction value and the image block ordering form a positive correlation; shrinking the random offset distance based on the offset distance shrinking value to obtain a target offset distance; obtaining a second image block selection position according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance; and taking the background image block corresponding to the selected position of the second image block as a candidate background image block.
The random offset distance is an offset distance between a preset candidate background image block in the current background filling round and a background image block matched with the previous round of the reference division area, and the distance may be positively correlated with the size of the intermediate image, and the larger the size of the intermediate image is, the larger the set offset distance may be. The offset contraction value refers to a value for contracting the random offset distance, and the value can be used to characterize the degree of contraction of the random offset distance, i.e., the magnitude of the contraction, for example, the contraction value can be a contraction ratio for the random offset distance. The contraction amplitude and the candidate background image blocks to be screened have positive correlation, that is, the larger the serial number of the screened candidate background image blocks is, the larger the contraction proportion thereof is, so that the smaller the target offset distance obtained by contracting the random offset distance according to the offset distance contraction value is. The second image block candidate position refers to an image block position obtained by shifting the position of the background image block matched in one round on the reference divided area according to the target shift distance, and the background image block corresponding to the position can be used as a candidate background image block of the reference divided area in the current background filling round.
If the best matching image block is determined only from the way of neighboring matches, it can easily fall into local optimality. For this reason, a random search step is also required. The random search step generates a series of candidate locations based on the exponentially distributed samples. The position of the candidate background image block obtained by random search can be calculated by the following formula:
Figure BDA0003293009400000171
wherein x is i X-axis position coordinates, y, representing the ith filtered candidate background image block i Representing the y-axis position coordinate, x, of the ith filtered candidate background image block 0 X-axis position coordinates, y, representing the position of the background image block matched by the reference division area in the last round of background filling 0 Y-axis position coordinate, wp, representing the position of the background image block matched by the reference division area in the last round of background filling 1 And wp 2 Denotes the random offset distance of the x-axis and y-axis, respectively, where w is the maximum diameter of the image, p 1 ,p 2 Is [ -1,1 [ ]]Of the random number(s) in (b),
Figure BDA0003293009400000172
representing an offset distance contraction value, wherein i represents the image block ordering of the screened candidate background image block, so that when i is larger, namely the image block ordering is larger, the contraction amplitude of the random offset is also larger, and then the target offset distance, namely ^ based on the target offset distance and the offset distance contraction value can be obtained>
Figure BDA0003293009400000181
And &>
Figure BDA0003293009400000182
Thus, the final second image block selection position corresponding to the candidate background image block is obtained according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance
In this embodiment, the selected position of the second image block may be searched for by the target offset distance obtained by the random offset distance and the offset distance shrinkage value, so as to avoid falling into local optimality, and since the shrinkage amplitude of the offset distance shrinkage value and the ordering of the image blocks form a positive correlation, it may be avoided that when the random offset distances are close, each candidate background image block is in the same close position, so as to further ensure the hierarchy of the candidate background image block, thereby improving the accuracy of screening the candidate background image blocks.
In one embodiment, as shown in fig. 8, step S510 may further include:
and S802, performing style transformation on the filled image to obtain an initial style image.
The initial style image refers to a style image directly obtained after the filled image is subjected to style transformation, wherein the style transformation process can be obtained by performing cartoonization processing on the filled image, specifically, after the filled image is obtained, the filled image can be input into a cartoonized style conversion model trained in advance, and the initial style image is output by the model. The model can divide a real image into three representations of appearance, structure and texture, and is obtained by carrying out generative confrontation network training by using an unpaired real image and a cartoon style image.
Step S804, a background object image corresponding to the target background object in the initial style image is obtained, and the background object image is removed from the initial style image to obtain an image to be processed in the initial style image.
The target background object refers to a background object that is not subjected to color clustering in a preset initial style image, and may be, for example, a sky portion in the initial style image, and the background object image refers to an image region corresponding to the target background object, and may be, for example, a sky region image in the initial style image. And the image to be processed refers to an image corresponding to an area of the initial style image which needs to be subjected to color clustering processing. In this embodiment, since the color clustering process may cause the sky color in the initial style image to be layered, and an unrealistic effect is generated, the sky region in the initial style image is used as a target background object, an image corresponding to the sky region is found as a background object image, and the background object image is removed from the initial style image, so as to obtain a corresponding to-be-processed image which needs to be subjected to color clustering.
And step S806, performing color clustering on the image to be processed to obtain a color clustering image.
Specifically, the color clustering may be performed on the images to be processed by a preset clustering algorithm, for example, a kmeans clustering algorithm, so as to obtain a color-clustered image, for example, the color-clustered image may be quantized into an 8-bit image.
And step S808, fusing the color clustering image and the background object image, and obtaining a target style image based on the image obtained by fusion.
Specifically, after the background object image corresponding to the target background object in the initial style image is obtained in step S804, and the color cluster image from which the background object image is removed in the initial style image is obtained in step S806, the background object image and the color cluster image may be fused, the fused image may be obtained by a weighted summation of a similar attention mechanism, and the final target style image is obtained based on the fused image, for example, the fused image may be further pixilated, and the size of the mosaic color block area is set, so that the size of the pixel in each mosaic color block area is equal to the pixel value of the area center point, thereby achieving the pixilated processing of the fused image to obtain the final target style image.
In this embodiment, the original style image obtained by performing the style conversion on the filler image may be used to obtain a background object image corresponding to a target background object such as sky, and the background object image is removed from the original style image to obtain a to-be-processed image.
In one embodiment, step S804 may further include: acquiring a color parameter threshold range corresponding to a target background object, wherein the target background object is a background object with color consistency greater than a consistency threshold; screening out pixel points with color parameter values within a color parameter threshold range from the initial style image, and taking the pixel points as target pixel points; and taking continuous image blocks corresponding to the target pixel points in the initial style image as background object images corresponding to the target background objects.
The color consistency can be used to represent whether the colors of the objects are consistent, the target background object screened in this embodiment is an object whose color needs to be consistent, for example, the target background object to be screened may be sky, the area screened may be a background object whose color needs to be consistent in blue, and if the target background object to be screened is sand, the area screened may be an area whose color needs to be consistent in yellow. Meanwhile, the color screening may be performed by setting a color parameter threshold range corresponding to the target background object, if the screening is performed on the sky, the color parameter threshold range corresponding to the blue color may be used as the color parameter threshold range corresponding to the target background object, and if the screening is performed on the beach, the color parameter threshold range corresponding to the yellow color may be used as the color parameter threshold range corresponding to the target background object. And the target pixel points refer to pixel points of which the color parameter values in the initial style image meet the color parameter threshold range.
Specifically, when a target background object in the initial-style image needs to be screened, a color parameter threshold range corresponding to the target background object may be preset, for example, a sky area of the initial-style image needs to be screened, then the color parameter threshold range corresponding to blue may be used as the color parameter threshold range corresponding to the target background object, then pixel points whose color channel values meet the color parameter threshold range may be screened from the initial-style image as target pixel points, that is, pixel points displayed as blue are screened as target pixel points, and continuous image blocks corresponding to the target pixel points, that is, image blocks continuously displaying blue are used as background object images.
In this embodiment, the corresponding target background object may be screened from the initial style image by setting a color parameter threshold range corresponding to the target background object, so that the accuracy of screening the background object image may be improved.
In one embodiment, step S808 may further include: fusing the color clustering image with the background object image to obtain a fused image; performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and removing the edge image from the fused image to obtain a target style image.
The fused image refers to an image obtained by fusing the color cluster image and the background object image, and may be obtained by performing weighted summation on the color cluster image and the background object image through a similar attention mechanism, and the edge image refers to an image obtained by performing edge extraction on the initial style image, and may be obtained by extracting an image edge corresponding to the initial style image by applying an edge detection algorithm to the initial style image.
Specifically, after obtaining the color cluster image and the background object image, the color cluster image and the background object image may be weighted and summed by a similar attention mechanism to obtain a corresponding fused image, and if the background object image is an image of a sky region part in the initial style image, the color cluster image may be obtained by color clustering only a non-sky region in the initial style image, so that the sky region of the fused image obtained by fusing may maintain an image in the initial style image that is not color clustered, and the non-sky region part may be an image in the initial style image that is color clustered. Meanwhile, edge extraction can be performed on the initial style image through an edge detection algorithm, such as a Canny edge detection algorithm, so as to obtain an edge image corresponding to the initial style image, and the edge image can be represented in the form of an edge gray map. Then, the edge gray-scale map can be subtracted from the obtained fusion image to achieve the effect of removing the edge image from the fusion image, and a final target style image is obtained.
In this embodiment, after the fused image is obtained, the edge image obtained by edge extraction may be removed from the fused image, and then the final target style image may be obtained, so that the smoothness of the target style image may be improved.
In one embodiment, performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image, includes: utilizing an edge extraction operator to perform edge extraction on the initial style image to obtain a first edge image corresponding to the initial style image; performing edge extraction on the initial style image by using an edge extraction model to obtain a second edge image corresponding to the initial style image; and acquiring an image intersection between the first edge image and the second edge image, and taking the image intersection as an edge image corresponding to the initial style image.
In this embodiment, the first edge image and the second edge image respectively refer to edge images obtained by performing edge extraction on an initial style image in different edge extraction manners, where the first edge image is an edge image obtained by performing edge extraction through a preset edge extraction operator, for example, an edge image obtained by performing edge extraction through a Canny operator, and the second edge image is an edge image obtained by performing edge extraction through a pre-trained edge extraction model, for example, an edge image obtained by performing edge extraction through an HED model. The image intersection refers to an intersection between the first edge image and the second edge image, and may be a normalized image corresponding to the first edge image and the second edge image, respectively, and the product of the normalized images is used as the edge image corresponding to the initial style image.
Specifically, the first edge image extracted by the Canny operator does not conform to the human visual system and has no semantic meaning. While the second edge image extracted by the HED (aided-edge detection) model is relatively consistent with the vision of human eyes, the extracted edge is relatively thick, and a large gray scale area is generated at the edge. Therefore, in order to have a better edge extraction effect, in this embodiment, canny operators are respectively used as edge operators to perform edge extraction to obtain a first edge image, and an HED model is used as an edge extraction model to perform edge extraction to obtain a second edge image, and an intersection of the first edge image and the second edge image, that is, a product of normalized images corresponding to the first edge image and the second edge image respectively is used as an edge image corresponding to an initial style image, so that the edge image can be refined, and the fineness of the obtained edge image is improved.
In this embodiment, edge extraction of the initial style image may be implemented by an edge extraction operator and an edge extraction model, and an image intersection of the obtained first edge image and the second edge image is used as an edge image corresponding to the initial style image, which may refine the edge image, thereby improving fineness of the obtained edge image.
In one embodiment, step S102 may further include: receiving an image processing request, wherein the image processing request is triggered by a terminal responding to a stereoscopic image generation operation; step S112 may further include: determining an image stereoscopic effect displayed by stereoscopic image generation operation, and acquiring a view plane corresponding to the image stereoscopic effect as a target view plane; and determining plane transformation information corresponding to the target view plane, and performing plane transformation on the filling image according to the plane transformation information so as to transform the image into an image positioned on the target view plane.
The image processing request refers to a request for performing image processing on a target image, and the request can be generated after responding to an operation triggered by a user and used for generating a stereoscopic image corresponding to the target image through a terminal. For example, when a user needs to perform image processing on a target image to generate a stereoscopic image corresponding to the target image, the user may enter an application program installed on the terminal for image processing, the terminal may display an image processing interface of the application program, the image processing interface may have options corresponding to a plurality of different image processing functions, the user may trigger a stereoscopic image generation operation by clicking the option for generating the stereoscopic image, and at this time, the terminal may generate a corresponding image processing request according to the operation.
The image stereoscopic effect refers to a stereoscopic effect of a stereoscopic image that a user needs to generate, and for the same target picture, the generation manner of the corresponding stereoscopic picture may also be different, and thus the stereoscopic effect of the generated stereoscopic picture may also be different, for example, the stereoscopic effect may include that a background object in the target picture is taken as a bottom, and a foreground object may be an effect that a person in the target picture is placed above a background, i.e., the person goes out of the background image, or the background object in the target picture is taken as a top, and the foreground object is placed below the background, i.e., an effect that the person goes out of the background image is presented, and the image stereoscopic effect may be realized by changing a viewing plane, so that after the user triggers the stereoscopic image generation operation, the terminal may determine a corresponding viewing plane based on the stereoscopic effect that the operation needs to realize, as the target viewing plane, and then perform plane transformation on the filled background image, i.e., the filled image, using plane transformation information corresponding to the target viewing plane, so as to obtain a transformed image that the transformed image is located on the target viewing plane.
As shown in fig. 9, a portion 901 in fig. 9 shows an interface diagram of an image processing application program displayed by a terminal, and in an application program a interface, an option window corresponding to a plurality of different image processing manners may be included, and among them, a plurality of stereoscopic image processing manners, that is, stereoscopic effect processing 1 and stereoscopic effect processing 2 may also be included, when the terminal clicks the option window of stereoscopic effect processing 1, a processing effect detail page of stereoscopic effect processing 1 may be shown as a portion 902 in fig. 9, and an immediately-made button may be displayed below the detail page, and by clicking the button, a stereoscopic image generation operation corresponding to stereoscopic effect processing 1 may be triggered, and the terminal may respond to the operation, so as to receive an image processing request for a target image, and respond to the request, and by displaying a target picture input interface through a portion 903 in fig. 9, the user may perform target image input in the target picture input interface, and the target image input process may be performed by clicking a real-time button in the picture taking screen, and may be used as a target image, or may be performed by clicking a target picture selected from the terminal, and the target picture stored in advance in the terminal, and the target picture is obtained by selecting button. After the target image is determined, the process may proceed to part 904 in fig. 9, and after the uploaded target image is processed, such as image cropping, and the immediate creation button in part 904 is clicked, the stereoscopic image creation for the stereoscopic effect 1 corresponding to the target image may be performed, so as to obtain a stereoscopic image as shown in part 905 in fig. 9.
In this embodiment, the image processing request may be obtained based on the triggered stereoscopic image generation operation, and the target view plane and the plane transformation information corresponding to the target view plane may also be determined based on the image stereoscopic effect corresponding to the stereoscopic image generation operation, so that it is realized that the transformed image is located on the target view plane, that is, a user may generate stereoscopic images with different stereoscopic effects by triggering different stereoscopic image generation operations, and the diversity of the stereoscopic effects of the stereoscopic images is improved.
In one embodiment, as shown in fig. 10, there is provided an image processing method, which may include the steps of:
step S1002, receiving an image processing request, the image processing request being triggered by the terminal in response to the stereoscopic image generation operation.
The image processing request refers to a request for performing image processing on a target image, the stereoscopic image generation operation refers to an operation triggered by a user for generating a stereoscopic image, the user can trigger the stereoscopic image generation operation by clicking an option for generating the stereoscopic image displayed on an application program installed on the terminal for image processing, and the terminal can respond to the operation and receive a corresponding image processing request.
Step S1004, acquiring a target image to be processed; carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image; and removing the image corresponding to the object area from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object area and a background image corresponding to the target image.
The target image refers to an image material which needs to be subjected to image processing, the foreground object refers to an object corresponding to a foreground part in the target image, and can be a person in the target image, and an object area corresponding to the foreground object can be a person area in the target image. The intermediate image refers to an image in which only the background area is reserved after the foreground object area is removed from the target image, and may be an image in which only the foreground part area in the target image is reserved, and the portrait area in the target image is removed and presented as a blank image.
Specifically, after the terminal responds to the image processing request, the terminal may further acquire an image material corresponding to the request as a target image, extract a foreground object, such as a human object, in the target image, obtain an object region corresponding to the foreground object, and remove the object region from the target image, thereby obtaining an intermediate image including a blank image corresponding to the object region and a background image corresponding to the target image.
Step S1006, performing area division on the object area to obtain a plurality of divided areas; and for a current pixel point in the blank image, determining a plurality of divided areas to which the current pixel point belongs as a plurality of reference divided areas.
The division regions refer to a plurality of regions obtained by dividing an object region in the intermediate image, and may be a plurality of image blocks in a blank image in the object region, the plurality of division regions may have an overlapping portion, the current pixel point refers to any one pixel point included in the object region, and the reference division region refers to a plurality of division regions in which the current pixel point is located. Because the plurality of divided regions have overlapping parts, the current pixel point can be in different divided regions, so that a plurality of reference divided regions are obtained.
Step S1008, determining a reference divided area corresponding to the reference divided area in the plurality of divided areas; acquiring the position of a background image block matched with the standard division area in the last round of background filling as a reference position; acquiring a relative position relation between a reference divided region and a reference divided region; obtaining a first image block selection position according to the reference position and the relative position relation; and taking the background image block corresponding to the selected position of the first image block as a candidate background image block.
The process of background filling for the blank image may be composed of multiple iterations, where the current background filling iteration refers to the current iteration, the candidate background image block refers to a background image block that is searched in the background image in the current iteration process and may be matched with a reference division region, and the reference division region refers to a preset division region that is matched with the reference division region, and may be, for example, a division region adjacent to the reference division region. The relative position relationship refers to a position relationship between the reference divided area and the standard divided area, and the reference divided area may be adjacent up and down or adjacent left and right, and the reference position refers to a position where the background image block matched with the standard divided area in the last background filling round is located, and the first image block selection position refers to a position where the background image block is located determined according to the reference position and the relative position relationship.
In this step, the candidate background image block matched with the reference divided region may be selected based on the position of the best matched background image block in the previous round of the adjacent reference divided regions, that is, the reference position, and the relative position relationship between the reference divided region and the reference divided region, that is, the image block adjacent to the reference position is used as the candidate background image block matched with the reference divided region in the current round.
Step S1010, acquiring a randomly generated random offset distance; acquiring a corresponding offset distance contraction value based on the image block ordering of the candidate background image blocks, wherein the contraction degree corresponding to the offset distance contraction value and the image block ordering form a positive correlation; shrinking the random offset distance based on the offset distance shrinkage value to obtain a target offset distance; obtaining a second image block selection position according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance; and taking the background image block corresponding to the selected position of the second image block as a candidate background image block.
The selected position of the second image block can be calculated by the following formula:
Figure BDA0003293009400000251
/>
wherein x is i X-axis position coordinates, y, representing the ith filtered candidate background image block i Representing the y-axis position coordinate, x, of the ith filtered candidate background image block 0 X-axis position coordinates, y, representing the position of the background image block matched by the reference division area in the last round of background filling 0 Y-axis position coordinate, wp, representing the position of the background image block matched by the reference division area in the last round of background filling 1 And wp 2 Denotes the random offset distance of the x-axis and y-axis, respectively, where w is the maximum diameter of the image, p 1 ,p 2 Is [ -1,1 [ ]]The random number in the group of random numbers,
Figure BDA0003293009400000261
representing an offset distance contraction value, wherein i represents the image block ordering of the screened candidate background image block, so that when i is larger, namely the image block ordering is larger, the contraction amplitude of the random offset is also larger, and then the target offset distance, namely ^ based on the target offset distance and the offset distance contraction value can be obtained>
Figure BDA0003293009400000262
And &>
Figure BDA0003293009400000263
And finally obtaining a second image block selection position corresponding to the candidate background image block according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance.
Step S1012, obtaining a filled image block obtained after the previous background filling of the reference divided region, and obtaining a matched background image block corresponding to the reference divided region in the previous background filling; calculating the similarity of the candidate background image block and the filling image block as a first similarity; calculating the similarity of the matched background image block and the filled image block as a second similarity; and acquiring an image block similar to the filling image block from the matching background image block and the candidate background image block based on the first similarity and the second similarity, wherein the image block is used as the background image block matched with the reference division area in the current background filling turn.
The filling image block refers to an image block corresponding to the reference divided area after background filling is performed on the reference divided area in the previous iteration, that is, the displayed image block of the current reference divided area, and the matching background image block refers to an image block which is determined in the previous background filling and is best matched with the reference divided area. The first similarity refers to the similarity between the candidate background image block screened in the current background filling round and the filling image block of the previous round, and the second similarity refers to the similarity between the matching background image block and the filling image block of the previous round.
Specifically, after determining a plurality of candidate background image blocks of the reference divided area, the terminal may calculate a similarity of a filled image block after background filling of the reference divided area after the previous background filling as a first similarity, further calculate a similarity between a matching background image block determined in the previous background filling and best matched with the reference divided area and the filled image block as a second similarity, and find an image block with the largest similarity based on the first similarity and the second similarity as a background image block best matched with the reference divided area in the current background filling pass.
Step 1014, acquiring the similarity between the background image blocks matched with the multiple reference division areas and the corresponding filling image blocks in the current background filling round; obtaining a pixel weight corresponding to a reference background pixel point in the current background filling round based on the similarity, wherein the pixel weight and the similarity form a positive correlation; and performing weighted calculation based on the pixel value corresponding to the reference background pixel point and the pixel weight corresponding to the reference background pixel point to obtain a filling pixel value corresponding to the current pixel point, and performing background filling on a blank image in the intermediate image according to the filling pixel value.
The pixel weight may refer to an influence weight of a pixel value of each reference background pixel point on a fill pixel value of a current pixel point, and the size of the pixel weight may form a positive correlation with the similarity between a background image block matched with each reference division region and a corresponding fill image block thereof, that is, the pixel weight may be correspondingly increased when the similarity is larger. The terminal can determine the pixel weight corresponding to each reference background pixel point according to the size relationship of each similarity, and then can perform weighting operation on the pixel value corresponding to each reference background pixel point by using the pixel weight, so as to obtain the filling pixel value corresponding to the current pixel point. And then, the background filling of the object region can be realized by using the filling pixel value corresponding to each current pixel point in the blank image.
Step S1016, carrying out style transformation on the filled image to obtain an initial style image; acquiring a color parameter threshold range corresponding to a target background object, wherein the target background object is a background object with color consistency greater than a consistency threshold; screening out pixel points with color parameter values within a color parameter threshold range from the initial style image, and taking the pixel points as target pixel points; and taking continuous image blocks corresponding to the target pixel points in the initial style image as background object images corresponding to the target background objects.
The initial style image refers to a style image directly obtained after performing style transformation on the filling image, and may be a cartoon image obtained after cartoonizing the filling image, and the target background object is a background object whose color consistency is greater than a consistency threshold, that is, an object that needs to satisfy color consistency, and may be a sky area of the cartoon image. The color parameter threshold range corresponding to the target background object refers to a preset parameter range meeting the requirement of color consistency, and if the parameter range is a sky area, the color parameter threshold range corresponding to the target background object refers to a color parameter threshold range corresponding to blue.
Specifically, the terminal can screen out pixel points which meet the color channel value and meet the set blue parameter threshold range in the cartoon image as target pixel points, and use continuous image blocks corresponding to the target pixel points, namely image blocks which are to continuously display blue, as background object images, so as to screen out a blue sky area as a target background object and obtain a sky area image corresponding to the sky area as a background object image.
Step S1018, removing the background object image from the initial style image to obtain an image to be processed in the initial style image, and performing color clustering on the image to be processed to obtain a color clustered image; and fusing the color clustering image and the background object image to obtain a fused image.
The image to be processed refers to an image corresponding to an area of the initial style image, which needs to be subjected to color clustering processing, and may be a non-sky area image of the remaining part of the image of the sky area after the sky area image is removed from the cartoon image, the color clustering refers to a method for clustering according to channel values of colors, and is used for realizing a quantization process of the image, and the color clustering image refers to an 8-bit image quantized through color clustering.
Specifically, in order to avoid the phenomenon that the color clustering process may cause the layering of sky colors in the initial style image, which causes an unrealistic effect, the screened sky area is removed from the initial style image, so that a corresponding to-be-processed image which needs to be subjected to color clustering processing is obtained, and only the non-sky area image is subjected to color clustering through a kmeans clustering algorithm, so that a color clustering image is obtained. Then, the clustered non-sky region image and the non-clustered sky region image may be fused, and the fused image may be obtained by a weighted summation manner of a similar attention mechanism.
Step S1020, performing edge extraction on the initial style image by using an edge extraction operator to obtain a first edge image corresponding to the initial style image; performing edge extraction on the initial style image by using an edge extraction model to obtain a second edge image corresponding to the initial style image; acquiring an image intersection between the first edge image and the second edge image, and taking the image intersection as an edge image corresponding to the initial style image; and removing the edge image from the fused image to obtain a target style image.
The first edge image and the second edge image respectively refer to edge images obtained by performing edge extraction on the initial style image in different edge extraction modes, wherein the first edge image is an edge image obtained by performing edge extraction through a preset edge extraction operator, for example, an edge image realized by a Canny operator, and the second edge image is an edge image obtained by performing edge extraction through a pre-trained edge extraction model, for example, an edge image realized by an HED model. The image intersection may then be the intersection between the first edge image and the second edge image. The target style image is an image obtained by performing final style conversion processing on the filler image.
Specifically, in this step, edge extraction may be performed on the initial style image by using a Canny operator and an HED model respectively to obtain a first edge image and a second edge image, and an image intersection between the first edge image and the second edge image is obtained, where a product of normalized images corresponding to the first edge image and the second edge image is used as an edge image corresponding to the initial style image, and the edge image is removed from the fused image obtained in step S1018, so as to obtain a final target style image.
Step S1022, determining the image stereoscopic effect displayed by the stereoscopic image generation operation, and acquiring a view plane corresponding to the image stereoscopic effect as a target view plane; determining plane transformation information corresponding to the target view plane, and performing plane transformation on the target style image according to the plane transformation information so as to transform the image into an image positioned on the target view plane; and fusing the transformed image and the image corresponding to the object area to obtain a processed image corresponding to the target image.
The image stereoscopic effect refers to a stereoscopic effect of a stereoscopic image that a user needs to generate, the stereoscopic effect may correspond to a triggered stereoscopic image generation operation, the target view plane refers to a view plane corresponding to the image stereoscopic effect, the plane transformation information is transformation information that is needed by view plane transformation, for example, a matrix or the like for implementing plane transformation, the processed image refers to a stereoscopic image obtained by performing complete image processing on the target image, the stereoscopic image may be a stereoscopic image in which a foreground object in the target image is in a standing state, and an image obtained by performing style transformation on a background portion of the target image is used as the bottom of the stereoscopic image, so that a visual effect in which the foreground object is separated from the background is presented.
Specifically, the terminal may determine a stereoscopic effect of a stereoscopic image to be generated according to a triggered stereoscopic image generation operation, thereby determining a target view plane where the target style image is located, thereby obtaining a corresponding plane transformation matrix, and may obtain a transformed image after plane transformation of the target style image by multiplying the target style image by the plane transformation matrix, and may further fuse the transformed image with an image corresponding to the foreground object region, thereby obtaining a final processed image.
According to the image processing method, the filled image background can be in natural and complete transition, meanwhile, the image can have a corresponding style, and the foreground and the background of the processed image have a three-dimensional relationship, so that the image processing effect is improved. In the filling process, the background image blocks matched with the reference division regions are utilized, the reference background pixel points corresponding to each pixel point are found, the pixel values are utilized to fill the pixel points, the accuracy of filling the blank image background can be improved, meanwhile, a plurality of rounds of background filling can be performed, the image blocks similar to the filled image blocks can be determined according to the similarity of the reference division regions in each filling round, the image blocks are used as the background image blocks matched with the reference division regions in the rounds, and the matching degree of the screened background image blocks can be improved. Meanwhile, in order to improve the screening efficiency and accuracy of the candidate background image blocks, the candidate background image blocks of the reference divided regions are obtained through the position relation between the reference divided regions and the positions of the background image blocks matched with the reference divided regions in the last background filling round, and the candidate image blocks are searched through the target offset distance obtained through the random offset distance and the offset distance contraction value, so that the screening efficiency and accuracy of the candidate background image blocks can be further improved. And for the pixel value filling of the current pixel point, the corresponding pixel weight can be determined according to the similarity of each reference division region and the matched background image block, so that the matched block with high similarity has higher reliability, and the accuracy of the obtained filled pixel value is further improved. In addition, in the style conversion process, only color clustering is performed on the removed image to be processed, and layering of colors of the target background object caused by quantization can be avoided, so that the reality of the filled image after the style conversion is improved, wherein the target background object is performed in a mode of setting a color parameter threshold range corresponding to the target background object, and the accuracy of background object image screening can be improved. Meanwhile, after the fused image is obtained, the edge image obtained through edge extraction can be removed from the fused image, and then the final target style image is obtained, so that the smoothness of the target style image can be improved, the edge extraction is achieved through an edge extraction operator and an edge extraction model respectively, the intersection of the extraction results is used as the edge image corresponding to the initial style image, the edge image can be refined, and the fineness of the obtained edge image is improved. The process of plane transformation is to determine a target view plane based on the image stereoscopic effect corresponding to the stereoscopic image generation operation, so that a user can generate stereoscopic images with different stereoscopic effects by triggering different stereoscopic image generation operations, and the diversity of the stereoscopic effects of the stereoscopic images is improved.
In one embodiment, the method splits the picture into the front view and the back view, folds the foreground portrait, completes the background, combines a brand new street view pixel style filter effect, increases the image hierarchy, and converts the image style when the picture is changed into the stereo effect.
A user can click on a material by entering a small program home page, a material detail page can be displayed, the user can select/shoot pictures from an album by clicking 'instant making', the pictures are transmitted to a background, and a video result is generated and returned to the user through an algorithm model.
The specific process can comprise the following steps:
detecting a human face, namely detecting the human face, not detecting the human face, returning the picture, and reselecting the picture;
face detection-monitoring face-image segmentation-complementing background-street view pixel air filter processing (no filter processing is performed in the image area) -image folding.
(1) Street view pixel wind filter;
the flow of the street view pixel wind filter system is shown in fig. 11. The street view image is firstly subjected to cartoon processing to generate a cartoon street view image, the cartoon street view image is subjected to edge extraction and thinning operation to obtain an edge gray-scale image, a sky is segmented to obtain a binary segmentation graph, and color clustering is carried out to obtain an 8-bit street view image; then, with the segmentation graph as a mask, reserving a sky area of the original cartoon street view and other areas of the clustered 8-bit street view to obtain a quantized street view; and respectively performing pixelation processing on the edge gray level image and the quantitative street view image, and subtracting the weighted edge pixel image from the street view pixel image to obtain a final result.
(a) Cartoon type
The White box cartonification is used for cartoonizing the original street view image, the model divides the real image into three representations of appearance, structure and texture, and the non-paired real image and the cartoon style image are used for training.
(b) Sky segmentation
To avoid quantified sky color layering, a large slice of blue is treated separately as the sky. Converting the original image into HSV space, selecting blue part, and performing threshold segmentation. And (4) only reserving a large-area blue area through noise reduction and graphic opening operation to obtain a binary segmentation map of the sky area and other areas.
(c) Edge extraction and refinement
And carrying out cooperative edge detection operation on the cartoon street view image by using a Canny operator and an HED model. The threshold is adjusted so that Canny operators extract dense edges, and HED is used to retain edges with semantics. The Canny operator extraction result for edge detection in the traditional image processing does not conform to a human visual system and has no semantic meaning. The result of HED is relatively consistent with human vision, but has the disadvantage of relatively coarse extracted edges and large areas of gray scale at the edges. The product of the normalized images of the two is used as a refined edge map to better delineate the street view pixel image.
(d) Color clustering
And carrying out quantization operation on the cartoon street view image by using a kmeans clustering algorithm to generate an 8-bit street view image. In order to avoid layering after clustering of gradual change sky colors, a binary sky segmentation graph is used for carrying out weighted summation of a similar attention mechanism on an 8-bit street view image and an original cartoon street view image to synthesize a quantized street view image so as to reserve a sky area of the original image.
(e) Pixelization
The quantized street view image and the edge image are respectively subjected to pixelization processing, the size of the mosaic is determined according to the image size, and the processing effect that the mosaic size is 3 is shown in the figure. The specific operation is to make the pixel in each mosaic color block region equal to the pixel value of the central point of the region. And finally, subtracting the weighted edge pixel map from the street view pixel map to obtain a final street view pixel wind image.
(2) Image completion
The invention uses the image completion technology and uses the background texture to fill the missing portrait position in the background, thereby achieving the effect of harmony of the whole picture. As shown in fig. 12, the basic assumption of the algorithm is that the background region and the region to be filled have a certain correlation. During calculation, a series of image blocks which are most similar to the image blocks in the area to be filled are found out from the background, and then the images are subjected to proper weighted average to complete the content filling of the area to be filled.
In this process, the main amount of computation is in the matching of image blocks. Suppose that the foreground region contains N f Each pixel, the background region comprises N b The complexity of direct violent search of each pixel can reach O (N) f N b ). Here we use a random hill-climbing strategy to search the most similar image blocks, which can reduce the complexity of the block matching algorithm to O (N) f logN b )。
(a) Brief introduction to image Block matching principles
The image block matching firstly randomly assigns a matched image block to the image block corresponding to each position of the whole image. Then, the matching blocks are continuously improved, and the similarity is improved. There are two ways of improvement, one is propagation and the other is random search.
The propagation step exploits the continuity in the image structure that the best match for blocks in the vicinity of any image block is most likely to occur in the vicinity of the best match for that image block. By utilizing the continuity of the images, the searching cost can be greatly reduced. We can use the known matching block information of the neighboring image blocks to improve the matching block of the current position. If the matching block similarity corresponding to any one of the positions is higher, the matching block similarity is improved correspondingly. This process needs to be repeated several times, with odd steps going down right and even steps going up left. Experiments show that a good approximation of the best matching block can be obtained after several iterations.
If there is only a propagation step, it is easy to get into local optimality. For this reason, a random search step is also required. The random search step generates a series of candidate locations based on the exponentially distributed samples. Similarly, if the block similarity corresponding to any one of the positions is larger, the result is improved accordingly. The positions of the candidate points are generated as follows:
Figure BDA0003293009400000331
where w is the maximum diameter of the image, (x) 0 ,y 0 ) Is the current matching block position, p 1 ,p 2 Is [ -1,1]The random number in (c). The convergence speed of the whole algorithm is related to the position of the initial random matching block. Thus, at initialization, the trial may be repeated multiple times (e.g., 15 times) to improve the matching block. Typically, the iterative process described above converges after 2-5 steps.
(b) Introduction to image content filling principles
With the nearest neighbor field, image information in the region to be filled is generated next from the best matching block. The entire filling process is similar to the EM iterative algorithm. For any one pixel in the area to be filled, there are many image blocks containing that pixel. These blocks all have respective best matching blocks. The pixel values corresponding to the original pixels on these best matching blocks are now weighted averaged by similarity. The greater the similarity of the matching blocks, the higher the confidence level, and the higher the assigned weight.
(3) Image folding
After the filled image is available, affine transformation is carried out on the filled image, and the filled image and the foreground are fused together to form a separation effect. In addition, a large radius gaussian blur result is used across the large background.
By the image processing method, three technologies of image completion, image folding and image stylization are combined to achieve the whole playing effect, wherein the image completion is to fill the background after the portrait is segmented, the image stylization is to convert the style of the picture background, and the image folding is to stand the static picture, so that the effect of a dynamic three-dimensional image is achieved.
It should be understood that, although the steps in the flowcharts of the present application are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least some of the steps in the figures may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the steps or stages is not necessarily sequential, but may be performed alternately or alternatively with other steps or at least some of the steps or stages in other steps.
In one embodiment, as shown in fig. 13, there is provided an image processing apparatus including: a target image acquisition module 1302, a foreground object identification module 1304, an intermediate image acquisition module 1306, a fill image acquisition module 1308, a transform image acquisition module 1310, and a process image acquisition module 1312, wherein:
a target image obtaining module 1302, configured to obtain a target image to be processed;
a foreground object identification module 1304, configured to perform foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image;
an intermediate image obtaining module 1306, configured to remove an image corresponding to the object area from the target image to obtain an intermediate image, where the intermediate image includes a blank image corresponding to the object area and a background image corresponding to the target image;
a filled image obtaining module 1308, configured to obtain a background image block matched with the blank image from the background image, perform background filling on the blank image in the intermediate image according to the background image block, and use the filled intermediate image as a filled image;
a transformed image obtaining module 1310, configured to perform style transformation and plane transformation on the filler image to obtain a transformed image, where in the process of performing plane transformation, plane transformation information corresponding to a target view plane is determined, and the filler image is subjected to plane transformation according to the plane transformation information, so that the transformed image is an image located on the target view plane;
a processed image obtaining module 1312, configured to fuse the transformed image with the image corresponding to the target area, so as to obtain a processed image corresponding to the target image.
In one embodiment, the filling image obtaining module 1308 is configured to perform area division on the object area to obtain a plurality of divided areas; for a current pixel point in a blank image, determining a division region to which the current pixel point belongs, and using the division region as a reference division region; acquiring a background image block matched with the reference division area, and acquiring a pixel point corresponding to the current pixel point in the background image block as a reference background pixel point; and obtaining a filling pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point, and filling the background of the blank image in the intermediate image according to the filling pixel value.
In an embodiment, the filling image is obtained through multiple background filling rounds, and the filling image obtaining module 1308 is configured to select one or more candidate background image blocks corresponding to the reference division area in the current background filling round based on an image block selection policy; acquiring a filling image block of the reference division area obtained after the last round of background filling, and acquiring a matching background image block corresponding to the reference division area in the last round of background filling; calculating the similarity of the candidate background image blocks and the filling image blocks to serve as a first similarity; calculating the similarity of the matched background image block and the filled image block as a second similarity; and acquiring an image block similar to the filling image block from the matching background image block and the candidate background image block based on the first similarity and the second similarity, and taking the image block as the background image block matched with the reference division area in the current background filling turn.
In an embodiment, the filling image obtaining module 1308 is configured to, when a plurality of reference divided areas corresponding to the current pixel point are provided, obtain a similarity between a background image block matched with a reference divided area and a corresponding filling image block in the current background filling turn; obtaining a pixel weight corresponding to a reference background pixel point in the current background filling round based on the similarity, wherein the pixel weight and the similarity form a positive correlation; and performing weighted calculation based on the pixel value corresponding to the reference background pixel point and the pixel weight corresponding to the reference background pixel point to obtain a filling pixel value corresponding to the current pixel point.
In one embodiment, the padded image obtaining module 1308 is configured to determine a reference divided region corresponding to a reference divided region in the plurality of divided regions; acquiring the position of a background image block matched with the standard division area in the last round of background filling as a reference position; acquiring a relative position relation between a reference divided region and a reference divided region; obtaining a first image block selection position according to the reference position and the relative position relation; and taking the background image block corresponding to the selected position of the first image block as a candidate background image block.
In one embodiment, the padded image acquisition module 1308 is configured to acquire a randomly generated random offset distance; acquiring a corresponding offset distance contraction value based on the image block ordering of the candidate background image blocks, wherein the contraction degree corresponding to the offset distance contraction value and the image block ordering form a positive correlation; shrinking the random offset distance based on the offset distance shrinkage value to obtain a target offset distance; obtaining a second image block selection position according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance; and taking the background image block corresponding to the selected position of the second image block as a candidate background image block.
In one embodiment, the transformed image obtaining module 1310 is configured to perform a style transformation on the filler image to obtain an initial style image; acquiring a background object image corresponding to a target background object in the initial style image, and removing the background object image from the initial style image to obtain an image to be processed in the initial style image; carrying out color clustering on the image to be processed to obtain a color clustering image; and fusing the color clustering image and the background object image, and obtaining a target style image based on the image obtained by fusing.
In one embodiment, the transformed image obtaining module 1310 is configured to obtain a color parameter threshold range corresponding to a target background object, where the target background object is a background object whose color consistency is greater than a consistency threshold; screening out pixel points with color parameter values within a color parameter threshold range from the initial style image, and taking the pixel points as target pixel points; and taking continuous image blocks corresponding to the target pixel points in the initial style image as background object images corresponding to the target background object.
In an embodiment, the transformed image obtaining module 1310 is configured to fuse the color cluster image with the background object image to obtain a fused image; performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image; and removing the edge image from the fused image to obtain a target style image.
In an embodiment, the transformed image obtaining module 1310 is configured to perform edge extraction on the initial style image by using an edge extraction operator to obtain a first edge image corresponding to the initial style image; performing edge extraction on the initial style image by using an edge extraction model to obtain a second edge image corresponding to the initial style image; and acquiring an image intersection between the first edge image and the second edge image, and taking the image intersection as an edge image corresponding to the initial style image.
In one embodiment, the target image obtaining module 1302 is further configured to receive an image processing request, where the image processing request is triggered by the terminal in response to the stereoscopic image generation operation; a transformed image obtaining module 1310 for determining an image stereoscopic effect shown by the stereoscopic image generating operation, and obtaining a view plane corresponding to the image stereoscopic effect as a target view plane; and determining plane transformation information corresponding to the target view plane, and performing plane transformation on the filling image according to the plane transformation information so as to transform the image into an image positioned on the target view plane.
For specific limitations of the image processing apparatus, reference may be made to the above limitations of the image processing method, which are not described herein again. The respective modules in the image processing apparatus described above may be wholly or partially implemented by software, hardware, and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in fig. 14. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for communicating with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, an operator network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement an image processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the configuration shown in fig. 14 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, carries out the steps in the above-described method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database or other medium used in the embodiments provided herein can include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), for example.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (15)

1. An image processing method, characterized in that the method comprises:
acquiring a target image to be processed;
performing foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image;
removing an image corresponding to the object region from the target image to obtain an intermediate image, wherein the intermediate image comprises a blank image corresponding to the object region and a background image corresponding to the target image;
acquiring a background image block matched with the blank image from the background image, performing background filling on the blank image in the intermediate image according to the background image block, and taking the filled intermediate image as a filled image;
performing style transformation and plane transformation on the filling image to obtain a transformed image, wherein when the plane transformation is performed, plane transformation information corresponding to a target view plane is determined, and the filling image is subjected to plane transformation according to the plane transformation information, so that the transformed image is an image positioned on the target view plane;
and fusing the transformed image and the image corresponding to the object region to obtain a processed image corresponding to the target image.
2. The method according to claim 1, wherein the obtaining a background image block matching the blank image from the background image, and performing background filling on the blank image in the intermediate image according to the background image block comprises:
carrying out region division on the object region to obtain a plurality of divided regions;
for the current pixel point in the blank image, determining a division region to which the current pixel point belongs, and using the division region as a reference division region;
acquiring a background image block matched with the reference division area, and acquiring a pixel point corresponding to the current pixel point in the background image block as a reference background pixel point;
and obtaining a filling pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point, and performing background filling on the blank image in the intermediate image according to the filling pixel value.
3. The method according to claim 2, wherein the filler image is obtained through multiple rounds of background filling, and the obtaining the background image block matched with the reference partition area comprises:
selecting one or more candidate background image blocks corresponding to the reference division area in the current background filling turn based on an image block selection strategy;
acquiring a filling image block obtained after the reference division area is filled in the last round of background, and acquiring a matched background image block corresponding to the reference division area in the last round of background filling;
calculating the similarity of the candidate background image blocks and the filling image blocks to serve as a first similarity;
calculating the similarity of the matched background image block and the filling image block as a second similarity;
and acquiring an image block similar to the filling image block from the matching background image block and the candidate background image block based on the first similarity and the second similarity, and using the image block as the background image block matched with the reference divided area in the current background filling round.
4. The method according to claim 3, wherein obtaining the filling pixel value corresponding to the current pixel point based on the pixel value corresponding to the reference background pixel point comprises:
when a plurality of reference division areas corresponding to the current pixel point are available, acquiring the similarity between the background image block matched with the reference division area and the corresponding filling image block in the current background filling round;
obtaining a pixel weight corresponding to the reference background pixel point in the current background filling turn based on the similarity, wherein the pixel weight and the similarity form a positive correlation relationship;
and performing weighted calculation based on the pixel value corresponding to the reference background pixel point and the pixel weight corresponding to the reference background pixel point to obtain a filling pixel value corresponding to the current pixel point.
5. The method as claimed in claim 3, wherein the selecting one or more candidate background image blocks corresponding to the reference division area in the current background filling round based on the image block selection strategy comprises:
determining a reference divided region corresponding to the reference divided region in the plurality of divided regions;
acquiring the position of a background image block matched with the standard division area in the last round of background filling as a reference position;
acquiring a relative positional relationship between the reference divided region and the reference divided region;
obtaining a first image block selection position according to the reference position and the relative position relation;
and taking the background image block corresponding to the selected position of the first image block as the candidate background image block.
6. The method as claimed in claim 3, wherein the selecting one or more candidate background image blocks corresponding to the reference division area in the current background filling round based on the image block selection strategy comprises:
acquiring a randomly generated random offset distance;
acquiring a corresponding offset distance contraction value based on the image block ordering of the candidate background image block, wherein the contraction degree corresponding to the offset distance contraction value and the image block ordering form a positive correlation;
shrinking the random offset distance based on the offset distance shrinking value to obtain a target offset distance;
obtaining a second image block selection position according to the position of the background image block matched with the reference division area in the last round of background filling and the target offset distance;
and taking the background image block corresponding to the selected position of the second image block as the candidate background image block.
7. The method of claim 1, wherein the step of performing a style transformation on the filler image comprises:
carrying out style transformation on the filled image to obtain an initial style image;
acquiring a background object image corresponding to a target background object in the initial style image, and removing the background object image from the initial style image to obtain an image to be processed in the initial style image;
carrying out color clustering on the image to be processed to obtain a color clustering image;
and fusing the color clustering image and the background object image, and obtaining a target style image based on the image obtained by fusing.
8. The method of claim 7, wherein the obtaining of the background object image corresponding to the target background object in the initial style image comprises:
acquiring a color parameter threshold range corresponding to a target background object, wherein the target background object is a background object with a color consistency degree greater than a consistency degree threshold;
screening out pixel points with color parameter values within the color parameter threshold range from the initial style image, and taking the pixel points as target pixel points;
and taking the continuous image blocks corresponding to the target pixel points in the initial style image as background object images corresponding to the target background objects.
9. The method according to claim 7, wherein the fusing the color cluster image with the background object image, and obtaining a target style image based on the fused image comprises:
fusing the color clustering image with the background object image to obtain a fused image;
performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image;
and removing the edge image from the fused image to obtain a target style image.
10. The method according to claim 9, wherein the performing edge extraction on the initial style image to obtain an edge image corresponding to the initial style image comprises:
utilizing an edge extraction operator to perform edge extraction on the initial style image to obtain a first edge image corresponding to the initial style image;
utilizing an edge extraction model to carry out edge extraction on the initial style image to obtain a second edge image corresponding to the initial style image;
and acquiring an image intersection between the first edge image and the second edge image, and taking the image intersection as an edge image corresponding to the initial style image.
11. The method of claim 1, wherein the acquiring a target image to be processed further comprises:
receiving an image processing request, wherein the image processing request is triggered by a terminal responding to a stereoscopic image generation operation;
performing plane transformation on the filling image to obtain a transformed image, wherein the step of obtaining the transformed image comprises the following steps:
determining an image stereoscopic effect displayed by the stereoscopic image generation operation, and acquiring a view plane corresponding to the image stereoscopic effect as the target view plane;
and determining plane transformation information corresponding to the target view plane, and performing plane transformation on the filling image according to the plane transformation information so as to enable the transformed image to be an image located in the target view plane.
12. An image processing apparatus, characterized in that the apparatus comprises:
the target image acquisition module is used for acquiring a target image to be processed;
the foreground object identification module is used for carrying out foreground object identification on the target image to obtain an object area corresponding to a foreground object in the target image;
an intermediate image obtaining module, configured to remove an image corresponding to the object region from the target image to obtain an intermediate image, where the intermediate image includes a blank image corresponding to the object region and a background image corresponding to the target image;
a filling image obtaining module, configured to obtain a background image block matched with the blank image from the background image, perform background filling on the blank image in the intermediate image according to the background image block, and use the filled intermediate image as a filling image;
the conversion image acquisition module is used for carrying out style conversion and plane conversion on the filling image to obtain a conversion image, wherein when the plane conversion is carried out, plane conversion information corresponding to a target view plane is determined, and the filling image is subjected to plane conversion according to the plane conversion information, so that the conversion image is an image positioned on the target view plane;
and the processed image acquisition module is used for fusing the transformed image and the image corresponding to the object area to obtain a processed image corresponding to the target image.
13. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 11 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 11.
15. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1-11 when executed by a processor.
CN202111170599.XA 2021-10-08 2021-10-08 Image processing method, image processing device, computer equipment and storage medium Pending CN115965634A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111170599.XA CN115965634A (en) 2021-10-08 2021-10-08 Image processing method, image processing device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111170599.XA CN115965634A (en) 2021-10-08 2021-10-08 Image processing method, image processing device, computer equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115965634A true CN115965634A (en) 2023-04-14

Family

ID=87357237

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111170599.XA Pending CN115965634A (en) 2021-10-08 2021-10-08 Image processing method, image processing device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115965634A (en)

Similar Documents

Publication Publication Date Title
Monroy et al. Salnet360: Saliency maps for omni-directional images with cnn
CN107945282B (en) Rapid multi-view three-dimensional synthesis and display method and device based on countermeasure network
CN110300264B (en) Image processing method, image processing device, mobile terminal and storage medium
JP6438403B2 (en) Generation of depth maps from planar images based on combined depth cues
US20120163672A1 (en) Depth Estimate Determination, Systems and Methods
CN111696028A (en) Method and device for processing cartoon of real scene image, computer equipment and storage medium
JP7387202B2 (en) 3D face model generation method, apparatus, computer device and computer program
CN110288534B (en) Image processing method, device, electronic equipment and storage medium
CN111008935B (en) Face image enhancement method, device, system and storage medium
Luo et al. A disocclusion inpainting framework for depth-based view synthesis
CN109886223B (en) Face recognition method, bottom library input method and device and electronic equipment
CN110852940A (en) Image processing method and related equipment
KR100927734B1 (en) Multi-view image generating device and method
US20230274400A1 (en) Automatically removing moving objects from video streams
Fujii et al. RGB-D image inpainting using generative adversarial network with a late fusion approach
CN111836058B (en) Method, device and equipment for playing real-time video and storage medium
CN116740261A (en) Image reconstruction method and device and training method and device of image reconstruction model
CN111402118B (en) Image replacement method and device, computer equipment and storage medium
CN113642359B (en) Face image generation method and device, electronic equipment and storage medium
CN113822965A (en) Image rendering processing method, device and equipment and computer storage medium
CN115965634A (en) Image processing method, image processing device, computer equipment and storage medium
CN110177216A (en) Image processing method, device, mobile terminal and storage medium
Zhu et al. Multi-feature fusion algorithm in VR panoramic image detail enhancement processing
CN114387326A (en) Video generation method, device, equipment and storage medium
CN113643357A (en) AR portrait photographing method and system based on 3D positioning information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40084256

Country of ref document: HK