WO2020140617A1

WO2020140617A1 - Image processing method, terminal apparatus, and computer readable medium

Info

Publication number: WO2020140617A1
Application number: PCT/CN2019/117576
Authority: WO
Inventors: 陈颖聪
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-01-04
Filing date: 2019-11-12
Publication date: 2020-07-09
Also published as: CN109785256A

Abstract

An image processing method, a terminal apparatus, and a computer readable medium, pertaining to image match processing. The method comprises: determining an image region to be repaired in an image to be repaired (101); selecting a first image region surrounding the image region to be repaired from the image (102); selecting a target reference image from a reference image set, the target reference image containing a second image region matching the first image region (103); and repairing the image region to be repaired by using the second image region and a region surrounded by the second image region in the target reference image as a reference (104). In the method, the first image region neighboring a defective region in the image to be repaired is used to search for a second image region having a similarity level to the first image region that reaches a threshold; and accordingly, the image is repaired by using the image containing the second image region.

Description

Image processing method, terminal equipment and computer readable medium

This application requires the priority of the Chinese patent application submitted to the China Patent Office on January 4, 2019, with the application number of 201910013962.3 and the application name of "An Image Processing Method, Terminal Equipment, and Computer-readable Medium". The reference is incorporated in this application.

Technical field

The present application relates to the field of image processing technology, and in particular, to an image processing method, terminal device, and computer-readable medium.

Background technique

With the continuous development of smart terminal technology, smart terminals are becoming more and more personalized and humanized. In daily life, users often use smart terminals to shoot, and the camera function of smart terminals is becoming more and more intelligent and humanized. However, when taking pictures with a smart terminal, users often have defects (such as closed eyes, blurry, etc.) due to various reasons (such as terminal shaking, subject blinking, etc.). At present, the commonly used solution is to use continuous shooting to solve the above problems, and then select one of the best photos from the summary of more continuous shooting photos.

However, in the continuous shooting process, all the photos taken continuously may have defects, but the defect areas are different. Therefore, how to deal with the defects of photos quickly and effectively is a problem to be solved at present.

Summary of the invention

Embodiments of the present application provide an image processing method that can quickly and effectively repair defects in photos.

In a first aspect, an embodiment of the present application provides an image processing method. The method includes:

Determine the image area to be repaired in the image to be repaired;

Selecting the first image area surrounding the image area to be repaired in the image to be repaired;

Selecting a target reference image from the reference image set, where a second image area matching the first image area exists in the target reference image;

Using the second image area in the target reference image and the area surrounded by the second image area as references, the image area to be repaired is repaired.

In a second aspect, an embodiment of the present application provides a terminal device. The terminal device includes a unit for performing the method of the first aspect.

In a third aspect, an embodiment of the present application provides another terminal device, including a processor, an input device, an output device, and a memory, where the processor, input device, output device, and memory are connected to each other, wherein the memory is used for A computer program supporting the terminal device to execute the above method is stored, the computer program includes program instructions, and the processor is configured to call the program instructions to perform the method of the first aspect.

According to a fourth aspect, an embodiment of the present application provides a computer non-volatile readable storage medium, the computer storage medium stores a computer program, the computer program includes program instructions, and the program instructions when executed by a processor Causing the processor to perform the method of the first aspect described above.

In an embodiment of the present application, a second image area that has a similarity to the first image area that reaches a threshold can be found through the first image area adjacent to the defective area in the image to be repaired, so as to utilize the image where the second image area is located To repair the image to be repaired.

BRIEF DESCRIPTION

In order to explain the technical solutions of the embodiments of the present application more clearly, the drawings required in the description of the embodiments will be briefly introduced below.

1 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

2 is a schematic flowchart of another image processing method provided by an embodiment of the present application;

3 is a schematic flowchart of yet another image processing method provided by an embodiment of the present application;

4 is a block diagram of a possible functional unit of a terminal device provided by an embodiment of the present application;

FIG. 5 is a schematic block diagram of a terminal according to another embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

It should be understood that when used in this specification and the appended claims, the terms "including" and "comprising" indicate the presence of described features, wholes, steps, operations, elements, and/or components, but do not exclude one or The presence or addition of multiple other features, wholes, steps, operations, elements, components, and/or collections thereof.

It should also be understood that the terminology used in the description of this application is for the purpose of describing particular embodiments only and is not intended to limit this application. As used in the specification of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms.

It should also be further understood that the term "and/or" used in the specification of the present application and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes these combinations .

As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "once" or "in response to determination" or "in response to detection" depending on the context . Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted in the context to mean "once determined" or "in response to a determination" or "once detected [described condition or event ]" or "In response to detection of [the described condition or event]".

In a specific implementation, the terminal devices described in the embodiments of the present application include but are not limited to other portable devices such as mobile phones, laptop computers, or tablet computers with touch-sensitive surfaces (eg, touch screen displays and/or touch pads) . It should also be understood that, in some embodiments, the device is not a portable communication device, but a desktop computer with a touch-sensitive surface (eg, touch screen display and/or touch pad).

In the discussion that follows, terminal devices including displays and touch-sensitive surfaces are described. However, it should be understood that the terminal device may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.

The terminal device supports various applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk burning applications, spreadsheet applications, game applications, phones Applications, video conferencing applications, email applications, instant messaging applications, exercise support applications, image management applications, digital camera applications, digital camera applications, web browsing applications, digital music player applications And/or digital video player applications.

Various applications that can be executed on the terminal device can use at least one common physical user interface device such as a touch-sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the terminal device may be adjusted and/or changed between applications and/or within the corresponding applications. In this way, the common physical architecture of the terminal device (eg, touch-sensitive surface) can support various applications with a user interface that is intuitive and transparent to the user.

Referring to FIG. 1, FIG. 1 is a schematic flowchart of an image processing method according to an embodiment of the present application. As shown in the figure, the method may include:

101: The terminal device acquires the image to be repaired, and determines the area of the image to be repaired in the image to be repaired.

In the embodiments of the present application, the terminal devices may include various terminal devices such as mobile phones, tablet computers, personal digital assistants (Personal Digital Assistant (PDA), mobile Internet devices (MID), etc., and the embodiments of the present application are not limited.

The above-mentioned image to be repaired may be a photo selected by the user from the terminal device that needs to be repaired with a certain defect, or may be a picture downloaded from the Internet with a certain defect. The image area to be repaired may be an area selected by the user to be repaired, or the terminal device may detect whether a defect corresponding to the repair type exists in the image according to a preset repair type; when a defect is detected, the area where the defect is located is determined It is the above-mentioned image area to be repaired, wherein the above-mentioned image area to be repaired may be an image area included in a regular set image, or may be an image area included in an irregular closed curve, that is, contains only a defective area.

As an optional embodiment, after the user selects the image to be repaired from the gallery, the terminal device displays the image to be repaired on the screen of the terminal device, and the user can select the image repair tool from the toolbar next to the image After the terminal device receives the operation of the user clicking the image repair tool, the terminal device displays a repair image area selection box on the image to be repaired. Then, the terminal device receives the user's operation instruction on the above-mentioned repair image area selection frame, and completes the operation corresponding to the above operation instruction. Then, after receiving the confirmation operation instruction, the user terminal determines the image area in the repair image area selection frame as the image area to be repaired.

Wherein, the operation instructions of the user on the selection frame of the repaired image area may include a drag operation instruction, a zoom-in operation instruction, a zoom-out operation instruction, a stretch operation instruction, a deformation operation instruction, and the like. The confirmation operation instruction may be a confirmation option that the user clicks on the screen, or a specific operation of the user on the repair image area selection box, for example, double-clicking the repair image area selection box.

As another optional embodiment, after the terminal device displays the image to be repaired, the terminal device receives the first repair type selected by the user from the preset repair types, and then the terminal device automatically detects the first repair type according to the first repair type Whether there is a defect corresponding to the first repair type in the image to be repaired, and if so, the area where the defect is located is determined to be the image area to be repaired. Wherein, the above-mentioned repair types may include at least one of portrait closed-eye repair, blurry area repair, shadow area repair, distortion area repair, designated area repair, and group photo repair.

102: The terminal device selects the first image area surrounding the image area to be repaired among the image to be repaired.

In the embodiment of the present application, after the terminal device determines the image area to be repaired in the image to be repaired, the terminal device automatically determines a first surrounding the image area to be repaired from the image to be repaired according to the image area to be repaired An image area, so that a second image area that is the same as or similar to the first image area can be searched for based on the first image area, so that the second image area and the area surrounded by the second image area can be used to repair the to-be-repaired Image area.

As an optional embodiment, after the terminal device determines the image area to be repaired in the image to be repaired, the terminal device calculates the maximum radiation radius of the image area to be repaired, and then according to the maximum radiation radius of the image area to be repaired Determine the position of the first image area in the image to be repaired. Specifically, the positional relationship between the first image area and the image area to be repaired may be: the maximum radiation radius of the image area to be repaired is R, then the first image area may be selected with the center of the image area to be repaired as the center of the circle and 2R as the radius Remove the area where the image area to be repaired is located in the circular area; the positional relationship between the first image area and the image area to be repaired may also be: the maximum radiation radius of the image area to be repaired is R, and the first image area is selected as described above The center of the image area to be repaired is taken as the center, and the area where the image area to be repaired is located is removed from the area of the square whose four sides are 2R away from the center.

It can be understood that the shape of the first image area may be other sizes or shapes than those shown in FIG. 3, and only needs to satisfy that the first image area surrounds the image area to be repaired.

103: The terminal device selects a target reference image from the reference image set, and there is a second image area matching the first image area in the target reference image.

The above-mentioned reference image set may be a picture existing in the gallery of the terminal device; or may be a picture that the terminal device selects from the terminal device according to the above-mentioned image to be repaired or downloaded from the Internet that meets the preset conditions, such as the above-mentioned to-be-repaired When the image is a photo, the above reference image set may be a photo taken at the same time period as the photo to be repaired, or a photo with the same background as the photo to be repaired (for example, photos taken continuously with the photo to be repaired) , Can also be a picture manually selected by the user from the terminal device gallery.

In the embodiment of the present application, after the terminal device determines the first image area, the terminal device searches for a target reference with a second image area matching the first image area from the reference image set according to the first image area Image to use the target reference image to repair the image to be repaired. The matching between the first image area and the second image area may be that the first image area and the second image area are the same or the similarity between the first image area and the second image area is greater than the threshold. In addition, the shape of the border outline of the first image area and the second image area are the same, and the size of the border outline of the first image area and the second image area may be the same or different, that is, the second graphics area may be the first image area The enlarged or reduced version of.

As an optional implementation manner, the terminal device may use the image of the first image area as a template image, and then sequentially match each image in the reference image set with the template image. When matching, first extract the feature points in the template image and the reference image to be matched, and then match the feature points in the template image and the reference image to be matched, and then determine the reference image to be matched by the matched feature points Whether there is an area similar to the template image in the image, if it exists, the similarity between the template image and the similar area similar to the template image in the image to be matched is calculated, and if the similarity is greater than the threshold, the image to be matched is used as the target reference image , The aforementioned similar area is used as the aforementioned second image area.

As an optional implementation manner, after determining that the similarity between the first image area and the similar area is greater than the threshold, the terminal device determines whether the image area surrounded by the similar area is defective, and if the image area surrounded by the similar area is determined If there is no defect, the similar area is determined as the second image area, and the reference image corresponding to the similar area is determined as the target reference image.

Further, in the above process of searching the image area matching the template image from the reference image set, one or more reference images may be searched from the reference image set to satisfy the existence and similarity of the image area similar to the template image Degree is greater than the threshold. In the above case, in order to make the effect of subsequent image restoration better, when the terminal device monitors two or more reference images from the reference image set to meet the above target reference image conditions, the terminal device selects the one with the largest similarity The reference image serves as the target reference image that is ultimately used to repair the image to be repaired. Here, the similarity here refers to the similarity of the similar area in the template image and the reference image.

As an optional implementation manner, when the above image using the first image region as a template image matches similar regions from the above reference image set, it may be that the image range of the first image region is too small and the extracted feature image points If there are too few results in all reference image sets in the reference image set that have no similar areas matching the template image, the terminal device expands the range of the first image area according to preset conditions (for example, expanding the first image area to the original Twice), and then continue to perform the above 103 steps.

104: The terminal device uses the second image area in the target reference image and the area surrounded by the second image area as references to repair the image area to be repaired.

In the embodiment of the present application, after the terminal device determines the target reference image, there is a second image area matching the above-mentioned first image area in the target reference image, and the terminal device according to the second image area in the target reference image And the image area surrounded by the second image area repairs the image area to be repaired in the image to be repaired.

As an optional implementation, when the second image area in the target reference image is the same as the boundary outline size of the first image area in the image to be repaired, the terminal device directly sets the second image in the target reference image The image of the image area surrounded by the area and the second image area is copied as the image to be stitched, and then the image with the stitching is spliced onto the first image area of the image to be repaired, so that the second image area in the target reference image And the image of the image area surrounded by the second image area covers the first image area and the image area to be repaired on the image to be repaired.

As another optional embodiment, when the boundary contour size of the second image area in the target reference image and the first graphic area in the phase to be repaired is different, that is, the image of the second image area may be different from the above If the enlarged or reduced version of the image of the first image area is similar, the terminal device needs to process the copied image after copying the second image area in the target reference image and the image area surrounded by the second image area Create an image with the same size as the outer boundary of the first image area as the image to be stitched, and then stitch the image to be stitched onto the first image area of the image to be repaired, so that the second image area in the target reference image And the image of the image area surrounded by the second image area covers the first image area and the image area to be repaired on the image to be repaired.

Image stitching technology is mainly divided into three main steps: image preprocessing, image registration, image fusion and boundary smoothing. Image preprocessing mainly refers to geometric distortion correction and noise point suppression on the image, so that the reference image and the image to be stitched There is no obvious geometric distortion. If the image quality is not ideal, the image stitching can easily cause some mismatches if the image is not preprocessed.

Image preprocessing is mainly to prepare for the next image registration, so that the image quality can meet the requirements of image registration. Image registration mainly refers to extracting the matching information in the reference image and the image to be stitched, finding the best match in the extracted information, and completing the alignment between the images.

The success of image stitching is mainly the registration of images. Between the images to be stitched, there may be multiple transformations such as translation, rotation, scaling, or large areas of the same color area that are difficult to match. A good image registration algorithm should be able to accurately find the inter-image Corresponding information, align the image.

Image fusion refers to stitching the image after the image matching is completed, and smoothing the stitched boundary to let the stitching transition naturally. Since the acquisition conditions of any two adjacent images cannot be exactly the same, for some image characteristics that should be the same, such as the illumination characteristics of the image, the two images will not behave exactly the same. The image stitching gap is caused by the transition of some relevant characteristics in the image when it transitions from the image area of one image to the image area of another image. The purpose of image fusion is to make the stitching gap between the images less obvious and the stitching more natural.

Therefore, in the embodiment of the present application, the splicing of the image to be spliced onto the first image area of the image to be repaired may include: the terminal device performing image correction processing on the image to be spliced; and then correcting the corrected image to be spliced with Image registration is performed on the first image area; according to the image registration result, the image to be stitched in the corrected image is merged with the first image area to complete image stitching.

First, the terminal device needs to perform image correction processing on the image to be stitched. When the image is formed, when the lens or camera of the camera system is not directly facing the scene to be photographed, then the scene image captured will be deformed to some extent. This is the most common case of geometric distortion. In addition, pincushion or barrel distortion caused by the limitation of optical imaging system or electronic scanning system is also a typical case of geometric distortion. Geometric distortion will cause great problems for image stitching. Originally the same objects in two images will become mismatched due to distortion, which will cause great problems for image registration. Therefore, it is very important to solve the problem of geometric distortion. In the embodiment of the present application, the image correction may be to perform barrel transformation on the image to be stitched. The main purpose of the transformation is to reduce the distortion of the stitched picture.

Then, the terminal device needs to perform image registration on the corrected image to be stitched and the first image area. The terminal device separately extracts feature points from the first image area in the image to be repaired and the image to be spliced, and then performs feature point matching to find the corresponding positions of the feature points in the image to be spliced in the first image area, and Determine the transformation relationship between the two images.

Finally, according to the image registration result, the terminal device performs image fusion on the area to be merged with the first image area in the corrected image to be stitched to complete image stitching. The terminal device calculates each parameter value in the mathematical model according to the corresponding relationship between the image to be stitched and the first image region obtained in the image registration step, thereby establishing a mathematical transformation model of the two images. Then, according to the established mathematical conversion model, the image to be spliced is converted into the coordinate system in the image of the first image area to complete the unified coordinate transformation. Finally, the overlapping areas of the images to be stitched (that is, the first image area and the corresponding second image area in the image to be stitched) are fused to obtain a stitched and reconstructed smooth and seamless panoramic image.

In the embodiment of the present application, by determining the image area to be repaired in the image to be repaired; selecting the first image area surrounding the image area to be repaired in the image to be repaired; selecting the target reference image from the reference image set, the target reference image There is a second image area matching the first image area; the second image area in the target reference image and the area surrounded by the second image area are used as a reference to repair the image area to be repaired. In an embodiment of the present application, a second image area that has a similarity to the first image area that reaches a threshold can be found through the first image area adjacent to the defective area in the image to be repaired, so as to utilize the image where the second image area is located To repair the image to be repaired.

Referring to FIG. 2, FIG. 2 is a schematic flowchart of another image processing method provided by an embodiment of the present application. As shown in the figure, the method may include:

201: The terminal device starts the photo processing application, and displays the photo to be repaired selected by the user from the gallery on the screen of the terminal device.

Among them, the above photo processing application is an application that can repair photos.

In the embodiment of the present application, when the user needs to repair a defective photo, the user can start the photo processing application on the terminal. Specifically, after the terminal device starts the photo processing application, the display interface of the photo processing application is displayed. The above display interface includes a photo display area and a toolbar, and the toolbar includes an option to repair photos. Then, after receiving the user's instruction to open the photo to be repaired in the gallery, the terminal device displays the photo to be repaired on the photo display area of the photo processing application.

202: The terminal device receives the repair operation instruction for the above photo, and displays a selection frame of the image area to be repaired on the photo to be repaired.

In the embodiment of the present application, when the terminal device then displays the photo to be repaired in the photo display area of the photo processing application, after receiving the user clicks the photo repair option in the toolbar, the terminal device displays the repaired image area on the photo to be repaired A check box, so that the user can select the area to be repaired on the photo to be repaired by selecting the check box for the repaired image area. The shape of the selection frame of the repaired image area is a rectangular frame.

203: The terminal device receives the operation instruction on the selection box of the image region to be repaired, and executes the operation corresponding to the operation instruction on the selection box of the image region to be repaired.

The operation instructions of the selection frame of the image area to be repaired include a drag operation instruction, a zoom-in operation instruction, a zoom-out operation instruction, a stretch operation instruction, a deformation operation instruction, and the like.

After the terminal device receives the operation instruction of the selection frame of the image area to be repaired, the terminal device responds to the operation instruction and implements operations such as dragging, enlarging, reducing, stretching, and deforming the selection frame of the repaired image area.

204: The terminal device then determines the image area to be repaired of the photo to be repaired according to the current position and size of the selection frame of the image area to be repaired.

After using the terminal device to complete the user's operation instruction for the repair image area selection box, that is, after the user determines the defective area of the photo to be repaired through the repair image area selection box, the terminal device receives the confirmation instruction to repair the image area, the terminal device will The area selected by the frame of the image area to be repaired on the photo to be repaired is taken as the image area to be repaired. Wherein, the confirmation operation instruction may be a confirmation option that the user clicks on the screen, or a specific operation of the user on the repair image area selection box, such as double-clicking the repair image area selection box.

205: The terminal device determines the first image area surrounding the image area to be repaired according to the image area to be repaired.

In the embodiment of the present application, the first image area is the same center as the image area to be repaired, and the length and width are areas where the length and width of the image area to be repaired are twice out of the partial area of the image area to be repaired, that is, the above The first image area is an adjacent area surrounding the image area to be repaired.

206: The terminal device searches the gallery for a target reference photo that has a second image area matching the first image area according to the first image area.

207: The terminal device copies the image of the second image area and the image area surrounded by the second image area in the target reference photo as the image to be stitched.

208: The terminal device stitches the image to be spliced onto the graphics area where the first image area of the image to be repaired is located, so as to cover the image area to be repaired.

For the above steps 206 to 208, reference may be made to the specific descriptions in steps 103 to 104 in the embodiment shown in FIG. 1, which will not be repeated here.

In the embodiment of the present application, the photo to be repaired is selected from the terminal device, and the photo to be repaired is displayed on the screen of the terminal device, and then the image area to be repaired is determined by the image area to be repaired selection box, and according to the image to be repaired The image area determines the first image area; then the target reference photo is determined from the gallery according to the image area to be repaired; finally, the image area to be repaired of the photo to be repaired is repaired according to the target reference photo. In the embodiment of the present application, a second image area that has a similarity to the first image area that reaches a threshold can be found through the first image area adjacent to the defective area in the photo to be repaired, so as to utilize the image where the second image area is located To repair the image to be repaired.

Referring to FIG. 3, FIG. 3 is a schematic flowchart of yet another image processing method provided by an embodiment of the present application. As shown in the figure, the method may include:

301: The terminal device starts the photo processing application, and displays the photo to be repaired selected by the user from the gallery on the screen of the terminal device.

In the embodiment of the present application, when the user needs to repair a defective photo, the user can start the photo processing application on the terminal. Specifically, after the terminal device starts the photo processing application, the display interface of the photo processing application is displayed. Then, after receiving the user's instruction to open the photo to be repaired in the gallery, the terminal device displays the photo to be repaired on the photo display area of the photo processing application.

302: The terminal device receives the repair type selection operation instruction, and determines the repair type of the photo to be repaired according to the repair type selection operation instruction.

Among them, the above display interface includes a photo display area and a toolbar, and the toolbar includes a photo repair type selection option, and the repair type selection options include portrait closed eye repair, blur area repair, shadow area repair, distortion area repair, designated Regional restoration, at least one of the group photo restoration.

In the embodiment of the present application, after the terminal device displays the photo to be repaired on the screen of the terminal device, it receives the repair type selection instruction input by the user in the toolbar, and the terminal device determines the above-mentioned to be repaired according to the repair type selection instruction The type of photo repair.

303: The terminal device then detects, according to the repair type, the image area to be repaired in which the defect corresponding to the repair type is present in the photo to be repaired.

In the embodiment of the present application, after the terminal device determines the repair type of the photo to be repaired, the terminal device detects whether the above defect exists in the photo to be repaired according to the defect corresponding to the repair type, and if so, determines the area where the defect is located It is the image area to be repaired above.

304: The terminal device determines the first image area surrounding the image area to be repaired according to the image area to be repaired.

305: The terminal device searches for a target reference photo in the second image area matching the first image area from the gallery according to the first image area.

306: The terminal device copies the image of the second image area and the image area surrounded by the second image area in the target reference photo as an image to be stitched.

307: The terminal device stitches the image to be spliced onto the graphics area where the first image area of the image to be repaired is located, so as to cover the image area to be repaired.

For the above steps 304-307, reference may be made to the specific descriptions in steps 102-104 of the embodiment shown in FIG. 1, which will not be repeated here.

For this step, reference may be made to the specific description in step 103 of the embodiment shown in FIG. 1, and details are not described herein again.

In the embodiment of the present application, the photos to be repaired are selected from the terminal device, and the photos to be repaired are displayed on the screen of the terminal device, and then the type of the photo to be repaired needs to be repaired, and the photo and repair are detected according to the repair type Defect area corresponding to the type, the defect area is determined as the image area to be repaired; then the first image area is determined according to the image area to be repaired, and the target reference photo is determined from the gallery according to the image area to be repaired; finally, the target photo is repaired according to the target reference photo Repair the image area of the photo to be repaired. In the embodiment of the present application, a second image area that has a similarity to the first image area that reaches a threshold can be found through the first image area adjacent to the defective area in the photo to be repaired, so as to utilize the image where the second image area is located To repair the image to be repaired.

Please refer to FIG. 4. FIG. 4 is a block diagram of a possible functional unit composition of a terminal device provided by an embodiment of the present application. The terminal device includes: a first determining unit 410, a first selecting unit 420, and a second selecting unit 430 、 Repair unit 440.

The first determining unit 410 is used to determine the image area to be repaired in the image to be repaired;

The first selection unit 420 is configured to select a first image area surrounding the image area to be repaired in the image to be repaired;

The second selecting unit 430 is configured to select a target reference image from the reference image set, and there is a second image area matching the first image area in the target reference image;

The repair unit 440 is configured to repair the image area to be repaired using the second image area and the area surrounded by the second image area in the target reference image as a reference.

Optionally, the above terminal device further includes:

The receiving unit 450 is configured to receive the first repair type selected by the user from the preset repair types for the image to be repaired, the repair type includes closed-eye repair, blurred area repair, repair from the shadow area, and distortion deformation area repair, At least one of the repairs in the designated area and the collective photo repair;

The first determining unit 410 is configured to detect a defect area where a defect corresponding to the first repair type in the image to be repaired is located; and determine the defect area in the image to be repaired as the image area to be repaired .

Optionally, the first determining unit 410 is configured to identify a user's selection operation on the image to be repaired; and determine an area corresponding to the selection operation as the image to be repaired region.

Optionally, the second selection unit 430 includes:

The searching unit 431 is configured to search for the similar region with the highest similarity to the first image region from the reference image, and the boundary contour shape of the similar region and the first image region is the same;

The third determining unit 432 is configured to determine the reference image corresponding to the similar area as the target reference image and determine the similar area as the second image area if the similarity between the first image area and the similar area is greater than a threshold .

Optionally, the third determining unit 422 is configured to determine whether the image area surrounded by the similar area is defective; if not, determine the reference image corresponding to the similar area as the target reference image and determine the similar area as the above The second image area.

Optionally, the repair unit 440 includes:

The copying unit 441 is configured to copy the images of the second image area and the area surrounded by the second image area in the target reference image, and use the copied image as the image to be stitched;

The stitching unit 442 is configured to stitch the above-mentioned image to be spliced into the above-mentioned image to be repaired to cover the region of the image to be repaired.

Optionally, the above-mentioned stitching unit 442 is specifically configured to perform image correction processing on the above-mentioned image to be stitched; perform image registration on the corrected image to be stitched with the above-mentioned first image area; After the image to be stitched is merged with one of the first image regions to complete the image stitching.

According to the specific implementation of the embodiments of the present application, the steps 101 to 104, 201 to 208, and 301 to 307 involved in the image processing method shown in FIGS. 1, 2, and 3 may be performed by the terminal device shown in FIG. Various modules to execute.

Referring to FIG. 5, FIG. 5 is a schematic block diagram of a terminal according to another embodiment of the present application. As shown in the figure, the terminal in this embodiment may include: one or more processors 501; one or more input devices 502, one or more output devices 503, and a memory 504. The processor 501, the input device 502, the output device 503, and the memory 504 are connected via a bus 505. The memory 502 is used to store a computer program. The computer program includes program instructions, and the processor 501 is used to execute the program instructions stored in the memory 502. The processor 501 is configured to call the above program instructions to execute: determine the image area to be repaired in the image to be repaired; select the first image area surrounding the image area to be repaired in the image to be repaired; select the target from the reference image set Reference image, there is a second image area matching the first image area in the target reference image; the second image area in the target reference image and the area surrounded by the second image area are used as references to repair the to-be-repaired Image area.

It should be understood that in the embodiment of the present application, the so-called processor 501 may be a central processing unit (Central Processing Unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP) , Application Specific Integrated Circuit (Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The input device 502 may include a touchpad, a fingerprint sensor (for collecting user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 503 may include a display (LCD, etc.), a speaker, and the like.

The memory 504 may include a read-only memory and a random access memory, and provide instructions and data to the processor 501. A portion of storage 504 may also include non-volatile random access memory. For example, the memory 504 may also store device type information.

In a specific implementation, the processor 501, the input device 502, and the output device 503 described in the embodiments of the present application may execute the first and second embodiments of an image processing method provided in the embodiments of the present application. The implementation manner can also implement the implementation manner of the terminal described in the embodiments of the present application, which will not be repeated here.

In another embodiment of the present application, a computer non-volatile readable storage medium is provided. The computer non-volatile readable storage medium stores a computer program. The computer program includes program instructions. The program instructions are processed by a processor. Achieved during execution: determining the image area to be repaired in the image to be repaired; selecting the first image area surrounding the image area to be repaired in the image to be repaired; selecting the target reference image from the reference image set The second image area matching the first image area; using the second image area in the target reference image and the area surrounded by the second image area as a reference to repair the image area to be repaired.

The non-volatile computer-readable storage medium may be the internal storage unit of the terminal described in any of the foregoing embodiments, such as the hard disk or the memory of the terminal. The non-volatile computer-readable storage medium may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a smart memory card (Smart Media, Card, SMC), and secure digital (SD) Cards, flash cards, etc. Further, the computer non-volatile readable storage medium may also include both the internal storage unit of the terminal and the external storage device. The computer non-volatile readable storage medium is used to store the computer program and other programs and data required by the terminal. The computer non-volatile readable storage medium may also be used to temporarily store data that has been output or will be output.

Those of ordinary skill in the art may realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two, in order to clearly explain the hardware and software. Interchangeability, in the above description, the composition and steps of each example have been generally described according to function. Whether these functions are executed in hardware or software depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working processes of the above-described end devices and units can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed terminal device and method may be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above-mentioned units is only a division of logical functions. In actual implementation, there may be other division modes, for example, multiple units or components may be combined or may Integration into another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections through some interfaces, devices, or units, and may also be electrical, mechanical, or other forms of connection.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The above integrated unit can be implemented in the form of hardware or software function unit.

If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application essentially or part of the contribution to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, the computer software product is stored in a storage medium In it, several instructions are included to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above methods in various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code .

The above is only the specific implementation of this application, but the scope of protection of this application is not limited to this, any person skilled in the art can easily think of various equivalents within the technical scope disclosed in this application Modifications or replacements, these modifications or replacements should be covered within the scope of protection of this application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

An image processing method, which includes:

Determine the image area to be repaired in the image to be repaired;

Selecting the first image area surrounding the image area to be repaired in the image to be repaired;

Selecting a target reference image from the reference image set, where a second image area matching the first image area exists in the target reference image;

Using the second image area in the target reference image and the image area surrounded by the second image area as references, the image area to be repaired is repaired.
The method according to claim 1, wherein before the determining the area of the image to be repaired in the image to be repaired, the method further comprises:

Receiving the first repair type selected by the user from the preset repair types for the image to be repaired, the preset repair type includes portrait closed-eye repair, blurred area repair, shadow area repair, warped deformation area repair, and designated area Repair, at least one of the group photo repairs;

The determining of the image area to be repaired in the image to be repaired includes:

Detecting a defect area where a defect corresponding to the first repair type in the image to be repaired is located;

The defect area in the image to be repaired is determined as the image area to be repaired.
The method according to claim 1, wherein the determining the area of the image to be repaired in the image to be repaired comprises:

Identify the user's selection operation on the image to be repaired;

The area corresponding to the selection operation is determined as the image area to be repaired.
The method according to any one of claims 1 to 3, wherein the target reference image is selected from the reference image set, and there is a second image area matching the first image area in the target reference image, include:

Searching for a similar region with the highest similarity to the first image region from the reference image, and the similar region has the same boundary contour shape as the first image region;

If the similarity between the first image area and the similar area is greater than a threshold, the reference image corresponding to the similar area is determined as the target reference image, and the similar area is determined as the second image area.
The method according to claim 4, wherein after determining that the similarity between the first image area and the similar area is greater than a threshold, the method further comprises:

Determine whether the image area surrounded by the similar area has defects;

If not, the reference image corresponding to the similar area is determined as the target reference image, and the similar area is determined as the second image area.
The method according to claim 5, wherein the repairing the image area to be repaired with reference to the second image area in the target reference image and the image area surrounded by the second image area includes :

Copy the images of the second image area and the area surrounded by the second image area in the target reference image, and use the copied image as the image to be stitched;

The image to be spliced is spliced into the image to be repaired to cover the image area to be repaired.
The method according to claim 6, wherein the splicing the image to be spliced into the image to be repaired to cover the area of the image to be repaired includes:

Performing image correction processing on the image to be stitched;

Performing image registration on the corrected image to be stitched with the first image area;

According to the result of the image registration, the image to be stitched in the corrected image to be merged with the first image region to complete the image stitching.
The method according to claim 4, characterized in that before searching for a similar region with the greatest similarity to the first image region from the reference image, the method further comprises:

Extract feature points in the template image and the reference image respectively;

The feature points in the template image and the reference image are matched, and whether the similar region similar to the template image exists in the reference image is determined by the matched feature points.
The method according to claim 7, wherein the image correction processing on the image to be stitched comprises:

Perform barrel transformation on the image to be stitched.
A terminal device is characterized by comprising: a first determination unit, a first selection unit, a second selection unit, and a repair unit, wherein:

The first determining unit is configured to determine the image area to be repaired in the image to be repaired;

The first selection unit is configured to select a first image area surrounding the image area to be repaired in the image to be repaired;

The second selection unit is configured to select a target reference image from the reference image set, and there is a second image area matching the first image area in the target reference image;

The repair unit is configured to repair the image area to be repaired by using the second image area in the target reference image and the area surrounded by the second image area as a reference.
The terminal device according to claim 10, wherein the terminal device further comprises:

The receiving unit is configured to receive the first repair type selected by the user from the preset repair types for the image to be repaired, the repair type including portrait closed-eye repair, blurry area repair, shadow area repair, distortion area repair, At least one of the repairs in the designated area and the collective photo repair;

The first determining unit is configured to detect a defect area where a defect corresponding to the first repair type in the image to be repaired is located; and determine the defect area in the image to be repaired as the image area to be repaired .
The terminal device according to claim 10, wherein the first determining unit is configured to identify a user's selection operation on the image to be repaired; and determine an area corresponding to the selection operation as the to-be-repaired Image area.
The terminal device according to any one of claims 10-12, wherein the second selection unit includes:

A search unit, configured to search for a similar region with the highest similarity to the first image region from the reference image, and the similar region has the same boundary contour shape as the first image region;

A third determining unit, configured to determine the reference image corresponding to the similar area as the target reference image if the similarity between the first image area and the similar area is greater than a threshold, and determine the similar area as the The second image area.
The terminal device according to claim 13, wherein the third determining unit is configured to determine whether an image area surrounded by the similar area has a defect; if not, determine a reference image corresponding to the similar area For the target reference image, the similar area is determined as the second image area.
The terminal device according to claim 14, wherein the repair unit comprises:

A copying unit, for copying images of the second image area and the area surrounded by the second image area in the target reference image, and using the copied image as the image to be stitched;

A stitching unit is used to stitch the image to be spliced into the image to be repaired to cover the image area to be repaired.
The terminal device according to claim 15, wherein the stitching unit is specifically configured to perform image correction processing on the image to be spliced; and perform image matching on the first image area after the corrected image to be spliced According to the result of the image registration, the image to be stitched in the corrected image is merged with the first image region to complete the image stitching.
The terminal device according to claim 13, wherein the second selection unit is further configured to:

Extract feature points in the template image and the reference image respectively;

The search unit is specifically configured to match feature points in the template image and the reference image, and determine whether the similar region similar to the template image exists in the reference image by the matched feature points.
The terminal device according to claim 16, wherein the stitching unit is further specifically configured to perform barrel transformation on the image to be stitched.
A terminal device, comprising a processor, a memory and a bus; the processor and the memory are connected to each other through the bus; wherein the memory is used to store a computer program, and the computer program includes program instructions The processor is configured to call the program instructions to perform the method according to any one of claims 1 to 9.
A computer non-volatile readable storage medium, characterized in that the computer non-volatile readable storage medium stores a computer program, the computer program includes program instructions, when the program instructions are executed by a processor So that the processor executes the method according to any one of claims 1 to 9.