CN111353946A

CN111353946A - Image restoration method, device, equipment and storage medium

Info

Publication number: CN111353946A
Application number: CN201811573981.3A
Authority: CN
Inventors: 田野; 王志斌
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2018-12-21
Filing date: 2018-12-21
Publication date: 2020-06-30
Anticipated expiration: 2038-12-21
Also published as: CN111353946B; WO2020125739A1; US11908105B2; US20210248721A1

Abstract

The embodiment of the invention discloses an image restoration method, an image restoration device, image restoration equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: displaying an original image to be restored, and determining a first original area to be restored and a second original area except the first original area in the original image; acquiring a target search region in the second original region according to the target pixel points and the target pixel blocks containing the target pixel points, wherein the area of the target search region is smaller than that of the second original region; searching a reference pixel block matched with the target pixel block in the target search area, repairing the target pixel block according to the reference pixel block, displaying the repaired image, and searching in the target search area with a smaller area, so that the search area is reduced, the search time is shortened, and the search speed is improved.

Description

Image restoration method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the technical field of image processing, in particular to an image restoration method, an image restoration device, image restoration equipment and a storage medium.

Background

With the continuous improvement of the photographing function and the gradual improvement of the requirement of the user on the photo display effect, more and more users are interested in repairing the photographed photo, removing characters or articles in the photo and achieving the effect of beautifying the photo. Therefore, image restoration is a common image processing method.

When image restoration is needed, acquiring a target image to be restored, wherein the target image comprises a first original region to be restored and a second original region which does not need to be restored, and determining target pixel points on a boundary and target pixel blocks containing the target pixel points according to the boundary of the first original region and the second original region; and searching a reference pixel block matched with the target pixel point in the second original region, and repairing the target pixel block according to the reference pixel block. After repairing, according to the boundary between the current unrepaired region and other regions except the region, selecting a target pixel point on the boundary, and continuing to execute the repairing step aiming at the selected target pixel point until the first original region is repaired.

When the first original region is repaired, a reference pixel block matched with the target pixel block needs to be searched in a second original region which does not need to be repaired, and the area of the second original region is large, so that the searching process needs to consume too long time, and the searching speed is slow.

Disclosure of Invention

The embodiment of the invention provides an image restoration method, an image restoration device, image restoration equipment and a storage medium, which can solve the problems in the related art. The technical scheme is as follows:

in one aspect, an image inpainting method is provided, the method including:

displaying an original image to be restored, and determining a first original area to be restored and a second original area except the first original area in the original image;

acquiring a target search region in the second original region according to a target pixel point and a target pixel block containing the target pixel point, wherein the area of the target search region is smaller than that of the second original region;

searching a reference pixel block matched with the target pixel block in the target searching area, repairing the target pixel block according to the reference pixel block, and displaying a repaired image.

In another aspect, an image inpainting method is provided, the method including:

acquiring a target image to be restored, wherein the target image comprises a first original area to be restored and a second original area except the first original area;

acquiring a current repairing boundary, wherein the repairing boundary is a boundary between an area which is not repaired currently and other areas except the area;

determining target pixel points on the repair boundary and target pixel blocks containing the target pixel points;

acquiring a target search region in the second original region, wherein the area of the target search region is smaller than that of the second original region;

and searching a reference pixel block matched with the target pixel block in the target search area, and repairing the target pixel block according to the reference pixel block.

In another aspect, there is provided an image repair apparatus, the apparatus including:

the display module is used for displaying an original image to be restored;

the determining module is used for determining a first original area to be restored and a second original area except the first original area in the original image;

a region obtaining module, configured to obtain a target search region in the second original region according to a target pixel point and a target pixel block including the target pixel point, where an area of the target search region is smaller than an area of the second original region;

a searching module for searching a reference pixel block matched with the target pixel block in the target searching region;

and the restoration module is used for restoring the target pixel block according to the reference pixel block and displaying the restored image.

the image acquisition module is used for acquiring a target image to be restored, wherein the target image comprises a first original area to be restored and a second original area except the first original area;

a boundary acquisition module, configured to acquire a current repair boundary, where the repair boundary is a boundary between an area that is not currently repaired and another area other than the area;

the determining module is used for determining target pixel points on the restoration boundary and target pixel blocks containing the target pixel points;

a region obtaining module, configured to obtain a target search region in the second original region, where an area of the target search region is smaller than an area of the second original region;

and the repairing module is used for repairing the target pixel block according to the reference pixel block.

In another aspect, there is provided an apparatus for repairing an image, the apparatus including: a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, the instruction, the program, the set of codes, or the set of instructions being loaded and executed by the processor to implement the operations performed in the image inpainting method as described.

In another aspect, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a set of codes, or a set of instructions is stored, which is loaded and executed by a processor to implement the operations performed in the image inpainting method as described.

According to the image restoration method, the image restoration device, the image restoration equipment and the storage medium, the original image to be restored is displayed, the first original region to be restored and the second original region except the first original region in the original image are determined, the target search region is obtained in the second original region, the reference pixel block matched with the target pixel block is searched in the target search region, the target pixel block is restored according to the reference pixel block, the area of the target search region is smaller than that of the second original region, compared with the search in the second original region, the search is carried out in the target search region with the smaller area, the search region is reduced, the search time is shortened, and the search speed is improved.

In addition, the first original area to be repaired is determined by detecting the selection operation in the original image, so that the user can set the area to be repaired only by performing the selection operation in the original area, the operation is simple, convenient and quick, and the flexible setting of the user is convenient.

In addition, in the process of selecting the area by the user, the dynamic effect of the repairing process is displayed, a dynamic repairing interface is displayed for the user, the display effect is improved, and the interestingness is enhanced.

The image restoration method, device, equipment and storage medium provided by the embodiments of the present invention, by obtaining a target image to be restored, the target image including a first original region to be restored and a second original region except the first original region, obtaining a restoration boundary between a region not currently restored and another region except the region, determining a target pixel point on the restoration boundary and a target pixel block including the target pixel point, obtaining a target search region in the second original region, searching for a reference pixel block matching the target pixel block in the target search region, restoring the target pixel block according to the reference pixel block, the area of the target search region being smaller than that of the second original region, searching in the target search region having a smaller area than that of the second original region, and reducing the search region, the search time is shortened, and the search speed is improved.

In addition, the target search area with the target pixel point as the circle center is acquired in the second original area, so that the target search area can be guaranteed to contain image information in the original image, pixel blocks searched in the target search area can be used for repairing, the pixel blocks can be searched nearby the target pixel blocks, the accuracy rate is improved, and the repairing effect is improved.

And for the pixel point of the target pixel block, the pixel value of the pixel point is fused with the pixel value of the corresponding pixel point of the pixel point in the reference pixel block, and the pixel value obtained by fusion is determined as the pixel value of the pixel point after the pixel point is repaired, so that the target pixel block and the surrounding area are in smooth transition visually, and obvious artificial repair traces in the repair area are avoided.

Moreover, the second region is extracted, so that only the second region is processed subsequently, other regions except the second region in the target image are not considered any more, the area of the region is reduced, the target search region is obtained in a region with a smaller area, and the search time is shortened.

Moreover, by acquiring the target search area with the target pixel point as the center of a circle in the intersection area of the second area and the second original area, the target search area can be ensured to contain the image information in the original image, the pixel block searched in the target search area can be used for repairing, and the pixel block can be searched near the target pixel block, so that the accuracy is improved, the repairing effect is improved, the area range adopted when the target search area is reduced is acquired, the search time is shortened, and the search speed is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a schematic illustration of an implementation environment provided by an embodiment of the invention;

FIG. 2 is a flowchart of an image restoration method according to an embodiment of the present invention;

FIG. 3 is a diagram of an original image according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a marker image provided by an embodiment of the present invention;

FIG. 5 is a schematic diagram of a target image provided by an embodiment of the invention;

FIG. 6 is a schematic illustration of a repaired boundary provided by an embodiment of the invention;

FIG. 7 is a schematic illustration of a repaired boundary provided by an embodiment of the invention;

fig. 8 is a schematic diagram of repairing a pixel point on a boundary according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a repaired image according to an embodiment of the invention;

FIG. 10 is a flowchart of an image restoration method according to an embodiment of the present invention;

FIG. 11 is a diagram illustrating a region partition according to an embodiment of the present invention;

FIG. 12 is a schematic flow chart of an operation provided by an embodiment of the present invention;

FIG. 13 is a flowchart of an image restoration method according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of an image restoration apparatus according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of an image restoration apparatus according to an embodiment of the present invention;

fig. 16 is a schematic structural diagram of a terminal according to an embodiment of the present invention;

fig. 17 is a block diagram of a server according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings.

Fig. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention, and referring to fig. 1, the implementation environment includes: a terminal 101.

The terminal 101 may be a mobile phone, a computer, a tablet computer, a smart television, or other various devices. The terminal 101 has an image restoration function, and can restore a certain area in a target image after acquiring the target image.

In a possible implementation, the implementation environment may further include a server 102, and the terminal 101 and the server 102 are connected through a network. The server 102 may be a server, a server cluster composed of several servers, or a cloud computing service center.

The terminal 101 acquires a target image, transmits the target image to the server 102, and the server 102 receives the target image, repairs a certain area in the target image, and transmits the repaired image to the terminal 101. The terminal 101 receives the restored image and presents it to the user.

The embodiment of the invention is applied to any scene for repairing a certain area in a certain image.

For example, a user takes a photo of a person, but other strangers exist in the photo except for the user, and at this time, the area where the stranger exists in the photo can be used as the area to be repaired.

For another example, a user has shot a section of food video, but there are dishes in the video besides food, and at this time, the area where the dishes are located in the video can be used as the area to be repaired.

Fig. 2 is a flowchart of an image restoration method according to an embodiment of the present invention. The execution subject of the embodiment of the present invention is an image restoration device, which is the terminal 101 or the server 102 in the embodiment shown in fig. 1 described above. Referring to fig. 2, the method includes:

201. and acquiring a target image to be repaired.

The method and the device are applied to the scene of image restoration, and for any one obtained original image, the target image to be restored is obtained by determining the area to be restored in the original image, wherein the target image comprises a first original area to be restored and a second original area except the first original area.

The difference between the target image and the original image is: in the original image, the pixel value of each pixel point in the first original region and the second original region is an original pixel value, and in the target image, the pixel value of each pixel point in the second original region is an original pixel value, and the pixel value of each pixel point in the first original region is a designated pixel value, so that the first original region displays the color corresponding to the designated pixel value.

The designated pixel value may be a lighter pixel value, such as a white pixel value. The pixel value of each pixel point of the first original region is modified into the designated pixel value, so that the first original region in the target image can be displayed in a specific color, and the interference caused by the original color of the first original region can be avoided in the process of repairing the first original region.

In one possible implementation, the first original area may be determined by a user through a selection operation. The method comprises the steps that an original image is displayed by an image repairing device, a user triggers selection operation in the original image, when the image repairing device detects the selection operation, a region corresponding to the selection operation is determined as a first original region, an unselected region is determined as a second original region, the pixel value of each pixel point in the first original region is modified into a designated pixel value, and the pixel value of each pixel point in the second original region is kept unchanged, so that a target image to be repaired is obtained. The selection operation may be a sliding operation, a clicking operation, a long-press operation, or the like. For example: the user may trigger a slide operation in the original image to determine the slide region as the first original region, or the user may click on any position in the original image to determine a region of a preset size centered on the position as the first original region, or the user may press any position in the original image to determine a region of a preset size centered on the position as the first original region.

For example, the image repairing device displays an image management interface, the image management interface includes an image calling option, when a user performs a confirmation operation on the image calling option, an image calling instruction may be triggered, and after receiving the image calling instruction, the image repairing device calls an image database to obtain an original image selected by the user from the image database. The image restoration device displays the original image on an image management interface, a user carries out selection operation in the original image, when the image restoration device detects the selection operation, the image restoration device determines an area corresponding to the selection operation as a first original area, and determines an area except the first original area in the original image as a second original area. And modifying the pixel value of the pixel point in the first original region into a specified pixel value, and keeping the pixel value of the pixel point in the second original region unchanged to obtain a target image. At this time, the target image can be displayed for the user to view, or the target image can not be displayed, and the original image is directly switched to display the repaired image subsequently.

In another possible implementation manner, in order to obtain the target image while retaining the original image, the image repairing device displays the original image, determines a region corresponding to the selection operation according to the detected selection operation, creates a marker image according to the region corresponding to the selection operation, where the size of the marker image is equal to the size of the original image, and a pixel value of each pixel point in the region corresponding to the selection operation in the marker image is a first specified pixel value, and a pixel value of each pixel point in the unselected region is a second specified pixel value. Acquiring a boundary between two areas in the marked image, performing area division on the original image according to the position of the boundary, determining a first original area and a second original area in the original image, modifying the pixel value of a pixel point in the first original area into a third specified pixel value, and keeping the pixel value of the pixel point in the second original area unchanged to obtain the target image.

For example, fig. 3 is an original image, a user performs a sliding operation on an area where a person is located in the original image, the image repairing apparatus determines, according to the sliding operation of the user, that an area selected by the user is the area where the person is located, creates a mask image as shown in fig. 4, further determines a first original area and a second original area in the original image according to a position where a boundary between the two areas in the mask image is located, and obtains a target image as shown in fig. 5 by modifying pixel values of pixel points in the first original area.

202. And acquiring a current repair boundary.

The repair boundary is a boundary between an area that is not currently repaired and an area other than the area, and includes, but is not limited to, the following two cases:

in the first case: after the image restoration device acquires the target image and before the target image is restored, the area that is not restored at present is the first original area, and the restoration boundary is the boundary between the first original area and the second original area, as shown in fig. 6 (different lines in fig. 6 represent different colors).

In the second case: after the image repair apparatus has repaired a part of the first original region, the region that is not currently repaired is the remaining region of the first original region, except the repaired region, which is not currently repaired, and the region that is not currently repaired is the second original region and the repaired region of the first original region, and the repair boundary is the boundary between the region that is not currently repaired and the region other than the region, as shown in fig. 7 (different lines in fig. 7 represent different colors).

203. And determining target pixel points on the repair boundary and target pixel blocks containing the target pixel points.

After the image restoration device obtains the current restoration boundary, a certain pixel point on the restoration boundary is determined as a target pixel point, and a pixel block containing the target pixel point is determined as a target pixel block to be restored.

In a possible implementation manner, one pixel point may be randomly selected from the repair boundary line as a target pixel point, or the repair priorities of a plurality of pixel points on the repair boundary line may also be obtained, and the pixel point with the highest repair priority is used as the target pixel point.

For any pixel point on the repair boundary, the repair priority can be obtained by adopting the following formula:

P(p)＝C(p)*D(p)

as shown in fig. 8 (different lines in fig. 8 represent different colors), I represents the target image, Ω represents a region that has not been currently repaired, Φ represents a region other than the region, δ Ω represents a repair boundary, ψ_pRepresenting a block of pixels comprising a pixel point p, psi_p∩ omega for a block of pixels psi_pRegion to be repaired of psi_p∩ phi denotes a block of pixels psi_pOther than the area to be repaired. P (p) represents the repair priority of the pixel p, c (p) represents the confidence of the pixel p, and d (p) represents the data item of the pixel p. C (q) is the confidence coefficient of the pixel point q, and the initial value of C (q) is 1. Phi_pI represents a block of pixels ψ_pThe area of (a) is,

orthogonal vector, n, representing the gradient vector of pixel point p_pThe unit vector with the pixel point p as the vertex, β represents the normalization factor, and when the target image is a grayscale image, β is 225.

Or, for any pixel point on the repair boundary, the repair priority can be obtained by adopting any one of the following formulas:

P(p)＝C(p)+D(p)

P(p)＝aC(p)+bD(p)

P(p)=aR_c(p)+bD(p)，R_c(p)＝(1-w)*C(p)+w

wherein p (p) represents the repair priority of the pixel p, c (p) represents the confidence of the pixel p, d (p) represents the data item of the pixel p, a and b represent weights, and a + b is 1. w is a fixed constant between 0 and 1.

In another possible implementation manner, determining a pixel block including a target pixel point as a target pixel block to be repaired includes acquiring a pixel block with a size of a preset size and taking the target pixel point as a center as the target pixel block, or acquiring a pixel block with a size of a preset size and taking the target pixel point as a vertex as the target pixel block, and intersecting both the first original region and the second original region as the target pixel block, where the preset size is 9 × 9, 15 × 15, or other sizes.

204. A target search area is acquired in the second original area.

The image restoration device acquires a target search region in the second original region so as to search a pixel block matched with the target pixel block in the target search region, wherein the target search region belongs to the second original region, the area of the target search region is smaller than that of the second original region, and the area of the region refers to the total number of pixel points contained in the region.

Wherein, regarding the shape of the target search area, the shape may be any shape, such as a circle, a rectangle, or an irregular figure, and regarding the manner of obtaining the target search area, the image inpainting apparatus may use the intersection area of the neighborhood of the target pixel block and the second original area as the target search area. The neighborhood is a region centered on the target pixel block, and the size or shape of the neighborhood can be set arbitrarily.

Compared with the search in the second original region, the search is carried out in the target search region with smaller area, so that the search region is reduced, the search time is shortened, and the search speed is improved.

In a possible implementation manner, the image inpainting device acquires a circular area with the target pixel point as a circle center, and determines an intersection area of the circular area and the second original area as a target search area. Wherein the radius of the circular area may be a fixed value.

In another possible implementation manner, the radius of the circular region is in a positive correlation with the area of the target search region, and the area of the target search region is in a positive correlation with the time duration for searching the pixel block in the target search region, so that the radius of the circular region affects the search time duration, and further affects the image restoration efficiency.

Table 1 is a relation diagram between the radius of the circular area and the search time duration provided in the embodiment of the present invention, and it can be known from table 1 that the smaller the radius of the circular area is, the smaller the search time duration is.

TABLE 1

Radius of circular area (Pixel)	Search duration (seconds)
		200	47.81
150	37.05
		120	25

Therefore, the target search area may be obtained according to the radius of the circular area, including the following steps 2041-2043:

2041. and acquiring a circular area which takes the target pixel point as the center of a circle and the search distance as the radius.

The search distance is equal to the product of the area of the first original region and a preset coefficient, and the preset coefficient is larger than 0. For example, the predetermined coefficient may be 0.5, 0.45, 0.4, or may be other values greater than 0.

For example: when the target pixel point is p, the area of the first original region is S, and the preset coefficient is 0.45, the image restoration device acquires a circular region with the p point as the center and the 0.45S as the radius.

After the image restoration device acquires the circular region, it will also determine whether the circular region intersects with the second original region, and according to the determination result, execute the following steps 2042 or 2043:

2042. and when the circular area intersects with the second original area, determining the intersection area of the circular area and the second original area as the target search area.

When the circular area is intersected with the second original area, the intersected area of the circular area and the second original area is directly determined as the target searching area, so that searching in the area which is near the target pixel point and belongs to the second original area can be guaranteed, the area of the target searching area cannot be too large, and searching time is shortened as much as possible.

2043. And when the circular area and the second original area are not intersected, expanding the search distance, acquiring the circular area which takes the target pixel point as the center of a circle and takes the expanded search distance as the radius, and determining the intersected area of the currently acquired circular area and the second original area as the target search area until the currently acquired circular area is intersected with the second original area.

Taking the current search distance as the first search distance as an example, when the target pixel point is taken as the circle center, and the circular area taking the first search distance as the radius is not intersected with the second original area, the search distance is expanded to obtain the second search distance, the circular area taking the target pixel point as the circle center and taking the second search distance as the radius is obtained, whether the circular area is intersected with the second original area is continuously judged, when the currently obtained circular area is not intersected with the second original area, the search distance is continuously expanded to obtain the third search distance, and the like, until the currently obtained circular area is intersected with the second original area, the intersected area of the currently obtained circular area and the second original area is determined as the target search area.

When the search distance is enlarged, the search distance can be enlarged according to a preset step length, wherein the preset step length can be 5 pixels, 9 pixels or other number of pixels; alternatively, the search distance may be enlarged by a preset multiple, which may be 0.5, 1, or other multiple.

For example: and if the circular region taking the point p as the center of the circle and the radius R as the radius does not intersect with the second original region, the expanded search distance is 2R. And if the circular area taking the point p as the center of the circle and the radius of 2R as the center of the circle is intersected with the second original area, determining the intersected area as a target search area. Or if the circular area with the p point as the center of the circle and the radius of 2R does not intersect with the second original area, the search distance is extended to 3R, and so on, until the currently acquired circular area intersects with the second original area, the intersection area of the currently acquired circular area and the second original area is determined as the target search area.

It should be noted that, when the image restoration device restores the target image for the first time, the determined target pixel point is located on the boundary between the first original region and the second original region, and therefore, the obtained circular region taking the target pixel point as the center and the search distance as the radius is inevitably intersected with the second original region. At this time, the intersection region of the circular region and the second original region may be determined as the target search region by using step 2042.

And along with the gradual restoration of the first original region, the restoration boundary line gradually moves towards the inside of the first original region, the image restoration device gradually selects a target pixel point along the direction of moving towards the inside of the first original region, so that the distance between the target pixel point and the boundary between the first original region and the second original region gradually increases, when the distance is too large, the circular region where the target pixel point is located may not intersect with the second original region, and the target search region belonging to the second original region cannot be acquired, at this time, the step 2043 may be adopted, and the circular region is reconstructed in a manner of expanding the search distance, so that the target search region belonging to the second original region is acquired.

By the method for obtaining the target search area, the target search area can be guaranteed to belong to the second original area and can contain image information in the original image, the pixel blocks searched in the target search area can be used for repairing, and the target search area can be guaranteed to search the pixel blocks nearby the target pixel blocks by taking the target pixel points as the circle center, so that the accuracy rate is improved, and the repairing effect is improved.

In a possible implementation manner, the pixel value of each pixel point in the second original region may be obtained, so that the second original region is divided into a plurality of regions according to the difference of the pixel values, so that the pixel values of the pixel points in each region belong to the same pixel range. When the target pixel block is obtained, at least one pixel value contained in the target pixel block is determined according to the pixel value of each pixel point in the target pixel block, at this time, the target search area is only required to be obtained in the second original area and the area corresponding to the at least one pixel value, and other areas in the second original area are not considered.

The second original area may be directly subjected to color segmentation, or may be subjected to gray segmentation after being converted into a gray map.

205. In the target search region, a reference pixel block matching the target pixel block is searched.

In one possible implementation, the process of searching for the reference pixel block may include the following steps 2051-2052:

2051. in the target search area, a plurality of pixel blocks identical to the target pixel block size are acquired.

After the image restoration device acquires the target pixel block, a search window with the same size as the target pixel block can be set, the search window is moved for multiple times in the target search area, the pixel block in the search window is acquired after the position of the search window is determined each time, then the search window is moved to the next position, the pixel block in the search window is continuously acquired, and a plurality of pixel blocks with the same size as the target pixel block are acquired after the search window is moved in the target search area.

When the moving step of the search window is 1, representing each time of moving, the search window is moved by a pixel point in both the transverse direction and the longitudinal direction. Table 2 is a relationship diagram between the moving step length of the search window and the search duration provided in the embodiment of the present invention, and it can be known from table 2 that, in the same area, the larger the moving step length of the search window is, the smaller the search duration is.

TABLE 2

Step size Step	Search duration (seconds)
		Step＝1	52.95
Step＝2	13.53
		Step＝3	6.26
Step＝4	3.74

However, if the moving step size of the search window is set too large, the search window may not search for a pixel block matching the target pixel block. Therefore, considering the search duration and the requirement of searching for a matching pixel block together, the following formula can be used to set a suitable moving step size:

wherein L represents the moving step length, S₁Representing the area of the first original region.

2052. And obtaining the similarity between the plurality of pixel blocks and the target pixel block, and determining the pixel block with the maximum similarity between the target pixel block and the pixel blocks in the plurality of pixel blocks as a reference pixel block.

The similarity between any two pixel blocks represents the similarity between the two pixel blocks, and the higher the similarity is, the more similar the two pixel blocks are represented.

After the image restoration device acquires the plurality of pixel blocks, the similarity between each pixel block of the plurality of pixel blocks and the target pixel block is acquired, and the pixel block with the maximum similarity between the plurality of pixel blocks and the target pixel block is determined as the reference pixel block, so that the target pixel block can be restored by adopting the pixel block most similar to the target pixel block, and the restoration effect is improved.

The similarity can be expressed by Euclidean distance, the Euclidean distance is larger, the expression similarity is smaller, the Euclidean distance between a plurality of pixel blocks and a target pixel block is obtained, and the pixel block with the minimum Euclidean distance between the pixel blocks and the target pixel block is determined as a reference pixel block. The euclidean distance may be L1, L2, or L3.

The Euclidean distance is calculated according to the color space of the target image color map, which may be RGB color space, LAB color space, or other forms of color space.

In one possible implementation, in blocks of pixels

And block of pixels psi_qFor example, when the color space of the pixel block is the RGB color space, the following formula is adopted to respectively obtain the pixel blocks

And block of pixels psi_qL1 euro distance between:

wherein the content of the first and second substances,

and

respectively representing pixel blocks

Red, green and blue component pixel values, R, at pixel point u in (1)_q(v)、G_q(v) And B_q(v) Respectively representing blocks of pixels psi_qAnd the red component pixel value, the green component pixel value and the blue component pixel value of the pixel point v corresponding to the pixel point u.

The calculation method of the euclidean distance L2 and the euclidean distance L3 is similar to the calculation method of the euclidean distance L1, and is not described herein again.

It should be noted that, in the above-mentioned steps 2051-2052, only after a plurality of pixel blocks with the same size as the target pixel block are obtained, the similarity between the pixel blocks and the target pixel block is respectively obtained, and in another possible implementation manner, the similarity between the pixel block and the target pixel block is obtained after a pixel block with the same size as the target pixel block is obtained each time.

In another possible implementation, the process of searching for the reference pixel block may further include steps 2053-2054:

2053. a plurality of pixel blocks matching the target pixel block are acquired in the target search area.

In the embodiment of the invention, the image restoration device acquires the similarity between each pixel block in the target search area and the target pixel block, and sorts the similarity according to the descending order to acquire a plurality of pixel blocks with higher similarity, or acquires a plurality of pixel blocks with the similarity not less than a preset threshold.

The preset threshold may be determined by the similarity between two pixel blocks that typically match each other. The manner of obtaining the similarity is the same as the manner of obtaining the similarity in step 2052, and is not described herein again.

For example: and calculating an L1 Euclidean distance between the target pixel block and the searched pixel block, sequencing the L1 Euclidean distances from small to large, and acquiring a plurality of pixel blocks with small L1 Euclidean distances or acquiring a plurality of pixel blocks with L1 Euclidean distances smaller than a preset threshold value.

2054. And performing weighted fusion on the plurality of pixel blocks according to the similarity between each pixel block in the plurality of pixel blocks and the target pixel block to obtain a reference pixel block.

The image restoration device obtains the weight of each pixel block according to the similarity between each pixel block in the plurality of pixel blocks and the target pixel block, determines the pixel points with the same position in the plurality of pixel blocks, performs weighted fusion on the pixel values of the determined plurality of pixel points according to the weight of the pixel blocks, and takes the fused pixel values as the pixel values of the pixel points with the same position in the reference pixel block. By adopting the method, the pixel value of the pixel point at each position in the reference pixel block can be obtained, so that the reference pixel block is obtained.

Wherein, the weight of each pixel block can be determined by the similarity between each pixel block and the target pixel block, and the similarity and the weight have positive correlation. Then, by obtaining the similarity between each pixel block and the target pixel block, the weight corresponding to the pixel block can be obtained. The similarity can be expressed in percentage, decimal, or other expression modes.

In one possible implementation, the similarity may be expressed by using an L1 euclidean distance, and after a plurality of pixel blocks with smaller L1 euclidean distances are acquired, the reciprocal of the L1 euclidean distance between each pixel block and the target pixel block is used as the weight of the pixel block, so that the smaller the L1 euclidean distance between a certain pixel block and the target pixel block, the greater the weight of the pixel block.

206. And repairing the target pixel block according to the reference pixel block.

When repairing the target pixel block according to the reference pixel block, any one of the following items is included:

2061. and for a first pixel point in the target pixel block, determining the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block as the pixel value repaired by the first pixel point.

When the position of the first pixel point in the target pixel block is the same as the position of another pixel point in the reference pixel block, the first pixel point and the another pixel point are corresponding pixel points.

When the image restoration device restores the target pixel block according to the reference pixel block, a first pixel point in the target pixel block and the position of the first pixel point in the target pixel block are obtained, the pixel value of the pixel point at the corresponding position in the reference pixel block is obtained, and the pixel value is determined as the pixel value after the first pixel point is restored. And then, repairing other pixel points in the target pixel block by adopting the method so as to finish repairing the target pixel block.

2062. And for a first pixel point in the target pixel block, fusing the pixel value of the first pixel point in the reference pixel block and the pixel value of the first pixel point, and determining the pixel value obtained by fusion as the pixel value after the first pixel point is repaired.

The fusion mode can also be weighted fusion, and the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block and the pixel value of the first pixel point are weighted fusion according to the weight of the pixel block to which the first pixel point belongs by setting the weights of the target pixel block and the reference pixel block.

Wherein the sum of the weights of the target pixel block and the reference pixel block is 1. For example, the weights of the target pixel block and the reference pixel block are set to 0.4 and 0.6, or 0.65 and 0.35, or 0.5 and 0.5, respectively, or the weights of both may also be determined in other ways.

In a possible implementation manner, if the first pixel point is located in an unrepaired region in the target pixel block, the pixel value of the first pixel point is not considered, and the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block is determined as the pixel value of the first pixel point after restoration. And if the first pixel point is positioned in other areas except the area which is not repaired in the target pixel block, fusing the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block with the pixel value of the first pixel point, and determining the pixel value obtained by fusion as the pixel value after the first pixel point is repaired.

In another possible implementation manner, the repair of the area which is not repaired and the other areas respectively in the above manner may cause jaggy of the boundary between the two areas after the repair, which affects the image display effect. Therefore, before repairing, a smooth boundary may be selected in the target pixel block, and the target pixel block may be divided into two designated regions by the boundary. If the first pixel point is located in a designated area with a larger area of an intersection area with an area which is not repaired yet, the pixel value of the first pixel point is not considered, the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block is determined as the pixel value of the repaired first pixel point, if the first pixel point is located in another designated area, the pixel value of the corresponding pixel point of the first pixel point in the reference pixel block is fused with the pixel value of the first pixel point, and the pixel value obtained by fusion is determined as the pixel value of the repaired first pixel point. The mode of selecting the smooth boundary line may be a dynamic programming mode, or may also be another mode.

Therefore, after the target pixel block is repaired, the target pixel block and the surrounding area are in smooth transition visually, and obvious artificial repair traces in the repaired area are avoided.

It should be noted that, after the repair of the current target pixel block is completed, the region to be repaired may change, and the repair boundary may also change, and then the above step 202 and step 206 are executed again according to the changed repair boundary, and the repair is continued until the repair of the first original region is completed, so as to obtain the repaired image. At this time, the restored image can be displayed for the user to check, and the restored image is stored in the gallery according to the storage operation triggered by the user, or the restored image can be displayed for the user to check, and the restored image is automatically stored in the gallery.

For example, the original image is the image shown in fig. 3, and the image shown in fig. 9 can be obtained by performing the restoration based on the target image shown in fig. 5.

It should be noted that, in the case that the first original area is determined by detecting the selection operation of the user, in the process of the selection operation performed by the user, the area corresponding to the selection operation is gradually increased, and then the above-mentioned step 202 and step 206 are performed for the gradually increased area to perform the repairing. And moreover, the dynamic effect that the pixel value of each pixel point in the region corresponding to the selection operation is switched from the original pixel value to the repaired pixel value can be displayed, until the selection operation is stopped, each pixel point in the first original region is repaired, and at this moment, the image repairing equipment displays the repaired image. The dynamic effect of the repairing process can be displayed in the selection process of the user, the dynamic repairing interface is displayed for the user, the display effect is improved, and the interestingness is enhanced.

The method provided by the embodiment of the invention comprises the steps of obtaining a target image to be repaired, wherein the target image comprises a first original region to be repaired and a second original region except the first original region, obtaining a repair boundary between a region which is not repaired at present and other regions except the region, determining a target pixel point on the repair boundary and a target pixel block comprising the target pixel point, obtaining a target search region in the second original region, searching a reference pixel block matched with the target pixel block in the target search region, repairing the target pixel block according to the reference pixel block, searching in the target search region with a smaller area compared with the second original region because the area of the target search region is smaller than that of the second original region, reducing the search region and shortening the search time, the search speed is improved.

And for the pixel point of the target pixel block, the pixel value of the pixel point is fused with the pixel value of the corresponding pixel point of the pixel point in the reference pixel block, and the pixel value obtained by fusion is determined as the pixel value of the pixel point after the pixel point is repaired, so that the target pixel block and the surrounding area are in smooth transition visually, and obvious artificial repair marks in the repair area are avoided.

On the basis of the method embodiment, another image restoration method is also provided. Fig. 10 is a flowchart of an image restoration method according to an embodiment of the present invention, and referring to fig. 10, the method includes:

1001. and acquiring a target image to be restored, wherein the target image comprises a first original area to be restored and a second original area except the first original area.

1002. And acquiring a current repair boundary, wherein the repair boundary is a boundary between an area which is not repaired currently and other areas except the area.

1003. And determining target pixel points on the repair boundary and target pixel blocks containing the target pixel points.

Wherein, the steps 1001-1003 are the same as the steps 201-203, and are not described herein again.

1004. The method comprises the steps of obtaining a first area, wherein the first area comprises a first original area, and the first original area is inscribed in the first area.

The first region may be of any shape, such as: square, circular, etc. it is only necessary to ensure that the first original region is inscribed in the first region.

1005. And extracting a second region including the first region from the target image, reducing the second region to obtain a third region, and acquiring a first target search region in a region except the reduced region of the first original region in the third region.

In the embodiment of the present disclosure, after acquiring the first region, the image repair apparatus may acquire a second region including the first region, and acquire the target search region in a region other than the first original region in the second region, where an area of the second region is larger than an area of the first region and smaller than an area of the target image.

The process of acquiring the target search area is similar to step 204, except that the area range adopted in acquiring the target search area is changed from the second original area to an area other than the first original area in the second area, i.e. the intersection area of the second area and the second original area.

In a possible implementation manner, when the image repair device acquires the second region, a boundary between the first region and another region except the first region may be acquired, where the boundary includes a plurality of pixel points, and the plurality of pixel points located outside the boundary and having a preset distance from the plurality of pixel points form a boundary between the second region and another region, so as to obtain the second region including the first region and having an area larger than that of the first region. The preset distance may be 100, 500 or other distances. And the preset distance may include a horizontal preset distance and a vertical preset distance.

Fig. 11 is a schematic diagram of region division according to an embodiment of the present invention, and referring to fig. 11, the image repairing apparatus acquires a circumscribed rectangular region of the first original region as the first region, and acquires a larger-sized rectangular region including the first region as the second region.

After the image repairing device acquires the second region, the step 1005 may be executed, and only the second region is subsequently processed by extracting the second region and performing reduction, without considering other regions except the second region in the target image, so as to reduce the area of the region, and acquire the target search region in a region with a smaller area, so as to shorten the search time.

The process of obtaining the first target search area is similar to that in step 204, and the area is only that the area range adopted when obtaining the target search area is changed from the second original area to the area other than the area of the third area after the first original area is reduced, which is not described herein again.

1006. In the first target search area, a first reference pixel block matching the reduced pixel block of the target pixel block is searched, and step 1007 or 1008 is performed.

The process of searching for the first reference pixel block in the first target search area is similar to step 205, except that the pixel block and the target search area are both reduced, and the searched first reference pixel block is also a reduced pixel block.

1007. Repairing the pixel block after the target pixel block is reduced according to the first reference pixel block; and amplifying the repaired third area to obtain a fourth area with the area equal to that of the second area, and replacing the second area in the target image with the fourth area.

In the embodiment of the present invention, after the first reference pixel block is obtained, the pixel block after the target pixel block is reduced may be directly repaired according to the first reference pixel block, the repairing process is similar to step 206, and only the difference is that the target pixel block and the reference pixel block involved in the repairing process are both reduced pixel blocks, which is not described herein again.

And after the target pixel block is repaired, obtaining a repaired third region, amplifying the third region to obtain a fourth region, wherein the fourth region has the same area as the second region in the target image, and the fourth region is the repaired second region, and replacing the second region with the fourth region to obtain the repaired target image.

1008. And acquiring a second reference pixel block corresponding to the first reference pixel block in a second region in the target image, and repairing the target pixel block according to the second reference pixel block.

In the embodiment of the present invention, after the first reference pixel block is obtained, a second reference pixel block corresponding to the first reference pixel block in a second region in the target image may also be obtained, and the target pixel block is repaired according to the second reference pixel block, where the repairing process is similar to step 206 and is not described herein again.

The first reference pixel block is a pixel block in the third area after the second area is reduced, the second reference pixel block is a pixel block in the original second area, and the position of the first reference pixel block in the third area is the same as the position of the second reference pixel block in the second area.

In addition to the above steps 1005-1008, the target search area may be obtained in an area other than the first original area in the second area in other manners.

For example, the above step 1005-1008 can be replaced by the following step 1009:

1009. and acquiring a circular area with the target pixel point as the center of a circle, and determining the intersection area of the circular area, the second area and the second original area as a target search area. The acquisition process comprises the following steps:

and acquiring a circular area which takes the target pixel point as the center of a circle and the search distance as the radius, wherein the search distance is equal to the product of the area of the first original area and a preset coefficient, and the preset coefficient is larger than 0.

When the circular area intersects with the intersection area of the second area and the second original area, determining the intersection area of the circular area, the second area and the intersection area of the second original area as a target search area; alternatively, the first and second electrodes may be,

and when the circular area does not intersect with the intersection area of the second area and the second original area, expanding the search distance, acquiring the circular area which takes the target pixel point as the center of a circle and takes the expanded search distance as the radius, and determining the intersection area of the currently acquired circular area and the intersection area of the second area and the second original area as the target search area until the currently acquired circular area intersects with the intersection area of the second area and the second original area.

Here, the search distance is expanded in a similar manner to the search distance expansion in step 2043.

In one possible implementation, which of the above-mentioned ways to repair may be determined according to the area of the first original region. If the size of the first original region is smaller than the first preset threshold, step 204-.

Wherein the first preset threshold and the second preset threshold may be determined according to a requirement for a calculation amount and a requirement for a search accuracy.

Fig. 12 is a schematic diagram of an operation flow provided by an embodiment of the present invention, and referring to fig. 12, the operation flow includes:

the method comprises the steps that an original image input by a user is obtained by an image repairing device, after the user selects a region to be repaired in the original image, the image repairing device obtains a target image to be repaired and a current repairing boundary in the target image, after target pixel points on the repairing boundary and target pixel blocks containing the target pixel points are determined, pixel blocks matched with the target pixel blocks are searched in other regions except the region, the target pixel blocks are repaired according to the matched pixel blocks, and then whether the first original region is repaired or not is judged. If yes, ending; if not, re-determining the repair boundary according to the changed area to be repaired, and re-executing the steps according to the changed repair boundary until the area selected by the user is repaired.

According to the method provided by the embodiment of the invention, only the second region is processed subsequently by extracting the second region, and other regions except the second region in the target image are not considered any more, so that the region area is reduced, the target search region is obtained in a region with a smaller area, and the search time is shortened.

By acquiring the target search area with the target pixel point as the center of a circle in the intersection area of the second area and the second original area, the target search area can be ensured to contain the image information in the original image, the pixel block searched in the target search area can be used for repairing, the pixel block can be ensured to be searched near the target pixel block, the accuracy is improved, the repairing effect is improved, in addition, the area range adopted when the target search area is reduced is acquired, the searching time is shortened, and the searching speed is improved.

On the basis of the foregoing embodiment, an image repairing method is further provided, fig. 13 is a flowchart of an image repairing method provided in an embodiment of the present invention, and referring to fig. 13, the method includes:

1301. displaying an original image to be restored, and determining a first original area to be restored and a second original area except the first original area in the original image.

In one possible implementation manner, when a selection operation in an original image is detected, a region corresponding to the selection operation is determined as a first original region; and determining the area of the original image except the first original area as a second original area.

1302. And acquiring a target search region in the second original region according to the target pixel points and the target pixel blocks containing the target pixel points, wherein the area of the target search region is smaller than that of the second original region.

1303. And searching a reference pixel block matched with the target pixel block in the target search area, repairing the target pixel block according to the reference pixel block, and displaying the repaired image.

In a possible implementation manner, in the process that the region corresponding to the selection operation is gradually increased, a dynamic effect that each pixel point in the region corresponding to the selection operation is switched from an original pixel value to a repaired pixel value is displayed, and until the selection operation is stopped, the step of displaying the repaired image is performed.

It should be noted that the

steps

1301 and 1303 are similar to the embodiment shown in fig. 2, and are not described again here.

The image restoration method provided by the embodiment of the invention determines a first original region to be restored and a second original region except the first original region in the original image by displaying the original image to be restored, acquires a target search region in the second original region, searches a reference pixel block matched with the target pixel block in the target search region, restores the target pixel block according to the reference pixel block, the area of the target search region is smaller than that of the second original region, and compared with the search in the second original region, the search is performed in the target search region with a smaller area, so that the search region is reduced, the search time is shortened, and the search speed is improved.

Fig. 14 is a schematic structural diagram of an image repairing apparatus according to an embodiment of the present invention. Referring to fig. 14, the apparatus includes:

a display module 1401 for displaying an original image to be restored;

a determining module 1402, configured to determine a first original region to be restored and a second original region except the first original region in an original image;

a region obtaining module 1403, configured to obtain a target search region in the second original region according to the target pixel point and the target pixel block including the target pixel point, where an area of the target search region is smaller than an area of the second original region;

a searching module 1404 for searching, in the target search region, for a reference pixel block matching the target pixel block;

and a repairing module 1405, configured to repair the target pixel block according to the reference pixel block, and display a repaired image.

The image restoration device provided by the embodiment of the invention determines a first original region to be restored and a second original region except the first original region in the original image by displaying the original image to be restored, acquires a target search region in the second original region, searches a reference pixel block matched with the target pixel block in the target search region, restores the target pixel block according to the reference pixel block, the area of the target search region is smaller than that of the second original region, and compared with the search in the second original region, the search is performed in the target search region with a smaller area, so that the search region is reduced, the search time is shortened, and the search speed is improved.

Optionally, the determining module 1402 includes:

a first determining unit configured to determine, when a selection operation in an original image is detected, an area corresponding to the selection operation as a first original area;

and a second determining unit configured to determine an area other than the first original area in the original image as a second original area.

Optionally, the apparatus further comprises:

and the display module is used for displaying the dynamic effect that each pixel point in the area corresponding to the selection operation is switched from the original pixel value to the repaired pixel value in the process that the area corresponding to the selection operation is gradually increased until the selection operation is stopped, and executing the step of displaying the repaired image.

Optionally, the region acquiring module 1403 includes:

and the first area acquisition unit is used for acquiring a circular area with the target pixel point as the center of a circle and determining the intersection area of the circular area and the second original area as a target search area.

Optionally, the first area obtaining unit includes:

the first acquisition subunit is used for acquiring a circular area which takes the target pixel point as the center of a circle and takes the search distance as the radius;

the second acquisition subunit is used for determining the intersection area of the circular area and the second original area as a target search area when the circular area and the second original area intersect;

and the third obtaining subunit is configured to, when the circular region does not intersect with the second original region, expand the search distance, obtain a circular region with the target pixel point as a center of a circle and the expanded search distance as a radius, and determine, until the currently obtained circular region intersects with the second original region, an intersection region of the currently obtained circular region and the second original region as the target search region.

Optionally, the region acquiring module 1403 includes:

and a second area acquisition unit configured to acquire the first area, acquire a second area, and acquire the target search area in an area other than the first original area in the second area.

Optionally, the second area obtaining unit includes:

and the first acquisition subunit is used for acquiring the second area, acquiring a circular area with the target pixel point as the center of a circle, and determining the intersection area of the circular area, the second area and the second original area as a target search area.

Optionally, the second area obtaining unit includes:

a second obtaining subunit, configured to extract a second region from the target image, reduce the second region to obtain a third region, and obtain the first target search region in a region of the third region other than the region in which the first original region is reduced;

a searching module 1404, configured to search, in the first target search region, for a first reference pixel block that matches the reduced pixel block of the target pixel block;

repair module 1405, comprising: a replacement unit or a repair unit;

the replacing unit is used for repairing the pixel block after the target pixel block is reduced according to the first reference pixel block; amplifying the repaired third area to obtain a fourth area with the same area as the second area, and replacing the second area in the target image with the fourth area;

and the repairing unit is used for acquiring a second reference pixel block corresponding to the first reference pixel block in a second area in the target image and repairing the target pixel block according to the second reference pixel block.

Optionally, the search module 1404 includes:

a first pixel block acquisition unit configured to acquire, in a target search region, a plurality of pixel blocks having the same size as a target pixel block;

and the searching unit is used for acquiring the similarity between the plurality of pixel blocks and the target pixel block and determining the pixel block with the maximum similarity between the target pixel block and the pixel blocks as the reference pixel block.

Optionally, the search unit is further configured to:

acquiring Euclidean distances between a plurality of pixel blocks and a target pixel block;

and determining a pixel block with the minimum Euclidean distance to the target pixel block in the plurality of pixel blocks as a reference pixel block.

Optionally, the search module 1404 includes:

a second pixel block acquisition unit configured to acquire a plurality of pixel blocks matched with the target pixel block in the target search region;

and the fusion unit is used for performing weighted fusion on the plurality of pixel blocks according to the similarity between each pixel block in the plurality of pixel blocks and the target pixel block to obtain a reference pixel block.

Optionally, the repair module 1405 further comprises:

the determining unit is used for determining the pixel value of a corresponding pixel point of a first pixel point in a reference pixel block as the repaired pixel value of the first pixel point for the first pixel point in the target pixel block; alternatively, the first and second electrodes may be,

and the fusion unit is used for fusing the pixel value of the first pixel point in the reference pixel block with the pixel value of the first pixel point, and determining the pixel value obtained by fusion as the repaired pixel value of the first pixel point.

Fig. 15 is a schematic structural diagram of an image restoration apparatus according to an embodiment of the present invention. Referring to fig. 15, the apparatus includes:

an image obtaining module 1501, configured to obtain a target image to be repaired;

a boundary obtaining module 1502, configured to obtain a current repair boundary;

a determining module 1503, configured to determine a target pixel point and a target pixel block;

an area obtaining module 1504, configured to obtain a target search area in the second original area;

a search module 1505 for searching a reference pixel block matching the target pixel block in the target search region;

and the repairing module 1506 is configured to repair the target pixel block according to the reference pixel block.

The device provided by the embodiment of the invention obtains the target image to be repaired, the target image comprises a first original area to be repaired and a second original area except the first original area, obtains the repair boundary between the area which is not repaired at present and other areas except the area, determines the target pixel point on the repair boundary and the target pixel block containing the target pixel point, acquiring a target search area in the second original area, searching a reference pixel block matched with the target pixel block in the target search area, repairing the target pixel block according to the reference pixel block, the area of the target search area is smaller than the area of the second original area, compared to searching in the second original area, the searching is carried out in the target searching area with smaller area, so that the searching area is reduced, the searching time is shortened, and the searching speed is improved.

Optionally, the area obtaining module 1504 includes:

Optionally, the first area obtaining unit includes:

Optionally, the area obtaining module 1504 includes:

Optionally, the second area obtaining unit includes:

a searching module 1505 for searching a first reference pixel block matched with the reduced pixel block of the target pixel block in the first target search region;

the repair module 1506 includes: a replacement unit or a repair unit;

Optionally, the search module 1505 includes:

Optionally, the search unit is further configured to:

Optionally, the search module 1505 includes:

Optionally, the repair module 1506 further comprises:

It should be noted that: the image restoration device provided in the above embodiment is only illustrated by the division of the above functional modules when restoring an image, and in practical applications, the above function allocation may be completed by different functional modules according to needs, that is, the internal structure of the image restoration device is divided into different functional modules to complete all or part of the above described functions. In addition, the image restoration device and the image restoration method provided by the above embodiment belong to the same concept, and the specific implementation process thereof is described in the method embodiment, which is not described herein again.

Fig. 16 is a schematic structural diagram of a terminal 1600 according to an embodiment of the present invention. The terminal 1600 may be a portable mobile terminal such as: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts group Audio Layer III, motion Picture Experts compression standard Audio Layer 3), an MP4 player (Moving Picture Experts compression standard Audio Layer IV, motion Picture Experts compression standard Audio Layer 4), a notebook computer, a desktop computer, a head-mounted device, or any other intelligent terminal. Terminal 1600 may also be referred to by other names such as user equipment, portable terminal, laptop terminal, desktop terminal, etc.

Generally, terminal 1600 includes: a processor 1601, and a memory 1602.

Processor 1601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 1601 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 1601 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also referred to as a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 1601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that the display screen needs to display. In some embodiments, the processor 1601 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

Memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 1602 is used to store at least one instruction for being possessed by processor 1601 to implement the image inpainting methods provided by method embodiments herein.

In some embodiments, the terminal 1600 may also optionally include: peripheral interface 1603 and at least one peripheral. Processor 1601, memory 1602 and peripheral interface 1603 may be connected by buses or signal lines. Various peripheral devices may be connected to peripheral interface 1603 via buses, signal lines, or circuit boards. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1604, a touch screen display 1605, a camera 1606, audio circuitry 1607, a positioning component 1608, and a power supply 1609.

Peripheral interface 1603 can be used to connect at least one I/O (Input/Output) related peripheral to processor 1601 and memory 1602. In some embodiments, processor 1601, memory 1602, and peripheral interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1601, the memory 1602 and the peripheral device interface 1603 may be implemented on a separate chip or circuit board, which is not limited by this embodiment.

The Radio Frequency circuit 1604 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuitry 1604 communicates with communication networks and other communication devices via electromagnetic signals. The rf circuit 1604 converts the electrical signal into an electromagnetic signal to be transmitted, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuit 1604 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: metropolitan area networks, various generation mobile communication networks (2G, 3G, 4G, and 16G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the rf circuit 1604 may also include NFC (Near Field Communication) related circuits, which are not limited in this application.

The display 1605 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 1605 is a touch display screen, the display screen 1605 also has the ability to capture touch signals on or over the surface of the display screen 1605. The touch signal may be input to the processor 1601 as a control signal for processing. At this point, the display 1605 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 1605 may be one, providing the front panel of the terminal 1600; in other embodiments, the display screens 1605 can be at least two, respectively disposed on different surfaces of the terminal 1600 or in a folded design; in still other embodiments, display 1605 can be a flexible display disposed on a curved surface or a folded surface of terminal 1600. Even further, the display 1605 may be arranged in a non-rectangular irregular pattern, i.e., a shaped screen. The Display 1605 may be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other materials.

The camera assembly 1606 is used to capture images or video. Optionally, camera assembly 1606 includes a front camera and a rear camera. Generally, a front camera is disposed at a front panel of the terminal, and a rear camera is disposed at a rear surface of the terminal. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, and the main camera and the wide-angle camera are fused to realize panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, camera assembly 1606 can also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

The audio circuitry 1607 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 1601 for processing or inputting the electric signals to the radio frequency circuit 1604 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, the microphones may be multiple and disposed at different locations of terminal 1600. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 1601 or the radio frequency circuit 1604 into sound waves. The loudspeaker can be a traditional film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, the audio circuit 1607 may also include a headphone jack.

The positioning component 1608 is configured to locate a current geographic location of the terminal 1600 for navigation or LBS (location based Service). The positioning component 1608 may be a positioning component based on the united states GPS (global positioning System), the chinese beidou System, the russian graves System, or the european union galileo System.

Power supply 1609 is used to provide power to the various components of terminal 1600. Power supply 1609 may be alternating current, direct current, disposable or rechargeable. When power supply 1609 includes a rechargeable battery, the rechargeable battery may support wired or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, terminal 1600 also includes one or more sensors 1610. The one or more sensors 1610 include, but are not limited to: acceleration sensor 1611, gyro sensor 1612, pressure sensor 1613, fingerprint sensor 1614, optical sensor 1615, and proximity sensor 1616.

Acceleration sensor 1611 may detect acceleration in three coordinate axes of a coordinate system established with terminal 1600. For example, the acceleration sensor 1611 may be used to detect components of the gravitational acceleration in three coordinate axes. The processor 1601 may control the touch display screen 1605 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1611. The acceleration sensor 1611 may also be used for acquisition of motion data of a game or a user.

Gyroscope sensor 1612 can detect the organism direction and the turned angle of terminal 1600, and gyroscope sensor 1612 can gather the 3D action of user to terminal 1600 with acceleration sensor 1611 in coordination. From the data collected by the gyro sensor 1612, the processor 1601 may perform the following functions: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

Pressure sensors 1613 may be disposed on a side bezel of terminal 1600 and/or underlying touch display 1605. When the pressure sensor 1613 is disposed on the side frame of the terminal 1600, a user's holding signal of the terminal 1600 can be detected, and the processor 1601 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1613. When the pressure sensor 1613 is disposed at the lower layer of the touch display 1605, the processor 1601 controls the operability control on the UI interface according to the pressure operation of the user on the touch display 1605. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 1614 is configured to collect a fingerprint of the user, and the processor 1601 is configured to identify the user based on the fingerprint collected by the fingerprint sensor 1614, or the fingerprint sensor 1614 is configured to identify the user based on the collected fingerprint. Upon recognizing that the user's identity is a trusted identity, the user is authorized by processor 1601 to have relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying for and changing settings, etc. The fingerprint sensor 1614 may be disposed on the front, back, or side of the terminal 1600. When a physical key or vendor Logo is provided on the terminal 1600, the fingerprint sensor 1614 may be integrated with the physical key or vendor Logo.

The optical sensor 1615 is used to collect ambient light intensity. In one embodiment, the processor 1601 may control the display brightness of the touch display screen 1605 based on the ambient light intensity collected by the optical sensor 1615. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1605 is increased; when the ambient light intensity is low, the display brightness of the touch display 1605 is turned down. In another embodiment, the processor 1601 may also dynamically adjust the shooting parameters of the camera assembly 1606 based on the ambient light intensity collected by the optical sensor 1615.

A proximity sensor 1616, also referred to as a distance sensor, is typically disposed on the front panel of terminal 1600. The proximity sensor 1616 is used to collect the distance between the user and the front surface of the terminal 1600. In one embodiment, the processor 1601 controls the touch display 1605 to switch from the light screen state to the rest screen state when the proximity sensor 1616 detects that the distance between the user and the front surface of the terminal 1600 is gradually decreased; when the proximity sensor 1616 detects that the distance between the user and the front surface of the terminal 1600 is gradually increased, the touch display 1605 is controlled by the processor 1601 to switch from the breath screen state to the bright screen state.

Those skilled in the art will appreciate that the configuration shown in fig. 16 is not intended to be limiting of terminal 1600, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components may be employed.

Fig. 17 is a schematic structural diagram of a server according to an embodiment of the present invention, where the server 1700 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 1701 and one or more memories 1702, where the memory 1702 stores at least one instruction, and the at least one instruction is loaded and executed by the processors 1701 to implement the methods provided by the foregoing method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input/output interface, so as to perform input/output, and the server may also include other components for implementing the functions of the device, which are not described herein again.

The server 1700 may be configured to perform the steps performed by the image inpainting apparatus in the image inpainting method described above.

An embodiment of the present invention further provides an apparatus for repairing an image, where the apparatus includes a processor and a memory, where the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the instruction, the program, the code set, or the instruction set is loaded by the processor and has an operation to implement the image repairing method of the foregoing embodiment.

An embodiment of the present invention further provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set, or a set of instructions is stored, and the instruction, the program, the code set, or the set of instructions is loaded by a processor and has an operation to implement the image inpainting method of the above-described embodiment.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only a preferred embodiment of the present invention, and should not be taken as limiting the invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image inpainting method, comprising:

2. The method of claim 1, wherein the determining a first original region to be restored and a second original region other than the first original region in the original image comprises:

when a selection operation in the original image is detected, determining a region corresponding to the selection operation as the first original region;

determining an area of the original image other than the first original area as the second original area.

3. The method of claim 2, further comprising:

and displaying the dynamic effect of switching each pixel point in the area corresponding to the selection operation from the original pixel value to the repaired pixel value in the process of gradually increasing the area corresponding to the selection operation until the selection operation is stopped, and executing the step of displaying the repaired image.

4. An image inpainting method, comprising:

5. The method of claim 1 or 4, wherein the obtaining a target search area in the second original area comprises:

and acquiring a circular area with the target pixel point as the center of a circle, and determining the intersection area of the circular area and the second original area as the target search area.

6. The method according to claim 5, wherein the obtaining a circular region with the target pixel point as a center, and determining an intersection region of the circular region and the second original region as the target search region comprises:

acquiring a circular area which takes the target pixel point as a circle center and a search distance as a radius, wherein the search distance is equal to the product of the area of the first original area and a preset coefficient, and the preset coefficient is greater than 0;

when the circular area intersects with the second original area, determining an intersection area of the circular area and the second original area as the target search area; alternatively, the first and second electrodes may be,

and when the circular area does not intersect with the second original area, expanding the search distance, acquiring the circular area which takes the target pixel point as the center of a circle and the expanded search distance as the radius, and determining the intersection area of the currently acquired circular area and the second original area as the target search area until the currently acquired circular area intersects with the second original area.

7. The method of claim 1 or 4, wherein the obtaining a target search area in the second original area comprises:

acquiring a first area, wherein the first area comprises the first original area, and the first original area is internally tangent to the first area;

and acquiring a second region, wherein the target search region is acquired in a region except the first original region in the second region, the second region comprises the first region, and the area of the second region is smaller than that of the target image.

8. The method according to claim 7, wherein the obtaining the second area, and obtaining the target search area in an area other than the first original area in the second area comprises:

and acquiring the second area, acquiring a circular area with the target pixel point as the center of a circle, and determining the intersection area of the circular area, the second area and the second original area as the target search area.

9. The method according to claim 7, wherein the obtaining the second area, and obtaining the target search area in an area other than the first original area in the second area comprises:

extracting the second region from the target image, reducing the second region to obtain a third region, and acquiring a first target search region in a region of the third region except the reduced region of the first original region;

searching a reference pixel block matched with the target pixel block in the target search area, and repairing the target pixel block according to the reference pixel block, wherein the method comprises the following steps:

searching a first reference pixel block matched with the reduced pixel block of the target pixel block in the first target search region;

repairing the pixel block after the target pixel block is reduced according to the first reference pixel block; amplifying the repaired third region to obtain a fourth region with the same area as the second region, and replacing the second region in the target image with the fourth region; alternatively, the first and second electrodes may be,

and acquiring a second reference pixel block corresponding to the first reference pixel block in the second region of the target image, and repairing the target pixel block according to the second reference pixel block.

10. The method according to claim 1 or 4, wherein searching for the reference pixel block matching the target pixel block in the target search region comprises:

acquiring a plurality of pixel blocks with the same size as the target pixel block in the target search area;

and acquiring the similarity between the plurality of pixel blocks and the target pixel block, and determining the pixel block with the highest similarity between the plurality of pixel blocks and the target pixel block as the reference pixel block.

11. The method according to claim 1 or 4, wherein searching for the reference pixel block matching the target pixel block in the target search region comprises:

acquiring a plurality of pixel blocks matched with the target pixel block in the target search area;

and performing weighted fusion on the plurality of pixel blocks according to the similarity between each pixel block in the plurality of pixel blocks and the target pixel block to obtain the reference pixel block.

12. An image restoration apparatus, characterized in that the apparatus comprises:

the display module is used for displaying an original image to be restored;

an image obtaining module, configured to obtain a target image according to the first original region and the second original region, where a pixel value of each pixel point in the first original region in the target image is an assigned pixel value;

13. An image restoration apparatus, characterized in that the apparatus comprises:

14. An apparatus for inpainting an image, the apparatus comprising: a processor and a memory, the memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, the instruction, the program, the set of codes, or the set of instructions being loaded and executed by the processor to carry out operations performed in the image inpainting method according to any one of claims 1 to 3, or operations performed in the image inpainting method according to any one of claims 4 to 11.

15. A computer-readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to perform the operations performed in the image inpainting method of any one of claims 1 to 3 or the operations performed in the image inpainting method of any one of claims 4 to 11.