CN111899184B

CN111899184B - Image defect repair and neural network training method, device, equipment and system

Info

Publication number: CN111899184B
Application number: CN202010244020.9A
Authority: CN
Inventors: 肖全之; 闫玉凤; 黄荣均; 方桂萍
Original assignee: Zhuhai Jieli Technology Co Ltd
Current assignee: Zhuhai Jieli Technology Co Ltd
Priority date: 2020-03-31
Filing date: 2020-03-31
Publication date: 2023-11-28
Anticipated expiration: 2040-03-31
Also published as: CN111899184A

Abstract

The invention discloses an image defect repairing and neural network training method, device and system, wherein the repairing method comprises the following steps: determining initial boundaries of a first direction and a second direction of the defect image block, wherein the first direction and the second direction are opposite directions; extracting image data of a region with a preset size close to a first direction initial boundary to obtain first image data; extracting image data of a region with a preset size close to the initial boundary of the second direction to obtain second image data; respectively inputting a first repair model and a second repair model of the neural network, so that the first repair model repairs the defective image block along a first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along a second direction based on the second image data to obtain second repair data; and repairing the defective image block based on the first repair data and the second repair data. The repairing efficiency is improved, and the repairing time consumption is reduced; the repair accuracy can be improved.

Description

Image defect repair and neural network training method, device, equipment and system

Technical Field

The invention relates to the technical field of image data processing, in particular to an image defect repairing and neural network training method, device and system.

Background

With the popularization of electronic products such as digital cameras, mobile phones, video monitoring and the like in life, more and more digital images and videos are presented to people. However, because the photographed object itself has defects, such as damage or missing of some antiques, art works or cultural relics due to the past ages; or shooting an area which is disliked or wanted to fade away on the surface of the object, such as spots on the face and flaws on the surface of the object; or partial data of the image, such as watermark and caption of the image, which is wanted to be changed; or the pictures in the video need to be repaired due to the fact that the images or the video cause partial data damage in the storage or transmission process and the need to be recovered.

The traditional image restoration model mainly adopts an image translation or small block filling method, features are analyzed and extracted through samples near the area to be restored, matched textures are selected and combined into the area to be restored, but the filled image is not natural enough due to the fact that different areas of the image have a plurality of differences, or the image loss part does not exist in the nearby area at all, so that the traditional filling method is invalid. The traditional filling method also depends on the size of the selected block, and if the large block is filled, the repairing effect is rough, and the small block is filled and repaired for a long time.

In order to reduce repair time consumption, in the prior art, in repairing a missing image block, a window is generally selected in a vicinity of the missing image block, the selected window includes the missing image block, and then the missing image block is repaired through the selected window, so that an image is perfected. Although the repairing method can reduce the repairing time consumption, the filled image is not natural enough due to the fact that a plurality of differences exist in different areas of the image, particularly when the missing image block is larger, a larger selection window needs to be configured, and the missing image block is repaired by taking the data of the edge of the selection window as a reference, so that the phenomenon that the repairing effect is unnatural can be increased.

Therefore, how to reduce the time consumption of repairing the image defect and improve the repairing precision is a technical problem to be solved.

Disclosure of Invention

Based on the above-mentioned current situation, the main objective of the present invention is to provide an image defect repairing method, a neural network training device and a neural network training system, so as to reduce the time consumption of repairing the image defect and improve the repairing precision.

In order to achieve the above purpose, the technical scheme adopted by the invention is as follows:

according to a first aspect, an embodiment of the present invention discloses a neural network-based image defect repair method, including:

Step S101, obtaining an image to be repaired, wherein the image has a defective image block; step S103, determining the initial boundary of a first direction and a second direction of the defect image block, wherein the first direction and the second direction are opposite directions; step S105, extracting image data of a region with a preset size close to a first direction initial boundary to obtain first image data; extracting image data of a region with a preset size close to the initial boundary of the second direction to obtain second image data; step S107, inputting the first image data and the second image data into a first repair model and a second repair model of the neural network respectively, so that the first repair model repairs the defective image block along a first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along a second direction based on the second image data to obtain second repair data; and step S109, repairing the defective image block based on the first repair data and the second repair data.

Optionally, step S103 includes: step S1031, searching a plurality of groups of relative boundary pairs of the defect image block; step S1032, determining a pair of boundary pairs with shortest boundary-to-boundary distances as target boundary pairs among a plurality of sets of relative boundary pairs; and step S1033, one boundary of the object boundary pair faces the other boundary to be the initial boundary determined as the first direction, and the other boundary faces the one boundary to be the initial boundary determined as the second direction.

Optionally, the plurality of sets of relative boundary pairs include: in a Cartesian coordinate system, a boundary pair consisting of a boundary obtained in the positive direction along the X-axis and a boundary obtained in the negative direction along the X-axis, a boundary pair consisting of a boundary obtained in the positive direction along the Y-axis and a boundary obtained in the negative direction along the Y-axis, a boundary pair consisting of a boundary obtained in a direction toward the first quadrant at a preset angle and a boundary obtained in a direction toward the third quadrant at a preset angle, and a boundary pair consisting of a boundary obtained in a direction toward the second quadrant at a preset angle and a boundary obtained in a direction toward the fourth quadrant at a preset angle.

Optionally, the size of the preset angle is 45 °.

Optionally, between step S105 and step S107, further includes: step S106, converting the data format of the first image data and the second image data into YUV format; in step S107 and step S109, the defective image block is repaired using the image data in YUV format.

Optionally, in step S107, the sum of the areas occupied by the first repair data and the second repair data in the defective image block is smaller than the area size of the defective image block; step S109 includes: step S1091, repairing corresponding areas in the defect image block respectively through the first repair data and the second repair data; step S1092, repairing unrepaired image data in the defective image block by the first repair data and the second repair data.

Optionally, step S1092 includes: inputting the first repair data and the second repair data into a third repair model of the neural network; in the third repair model, obtaining third repair data based on the first repair data and the second repair data; and repairing unrepaired image data in the defective image block by the third repair data.

Optionally, in the repair of the defective image block in step S109: when the repair data is outside the boundary of the defect image block, adopting corresponding original image data in the image to be repaired to replace the repair data; the repair data comprise first repair data, second repair data and third repair data, wherein the third repair data is obtained based on the first repair data and the second repair data; when the repair data is within the boundary of the defective image block, repairing a corresponding region within the defective image block using the repair data.

Optionally, repairing a corresponding region within the defective image block using the repair data includes: and presetting a transition region with a size in the defect image block, and filling the transition region by adopting the repairing data of the transition region and the weighting of the original image data.

Optionally, the following formula is used to fill each pixel point in the transition region: u=0.5a+0.5b, where u is a pixel value of a single pixel point in the transition region after repair, a is a pixel value of a single pixel point in the repair data, and b is a pixel value of a single pixel point in the original image data.

According to a second aspect, an embodiment of the present invention discloses an image defect repair apparatus based on a neural network, including:

the image acquisition module is used for acquiring an image to be repaired, wherein the image has a defective image block; the initial boundary determining module is used for determining initial boundaries of a first direction and a second direction of the defect image block, wherein the first direction and the second direction are opposite directions; the image extraction module is used for extracting image data of a region with a preset size close to the initial boundary of the first direction to obtain first image data; extracting image data of a region with a preset size close to the initial boundary of the second direction to obtain second image data; the repair data acquisition module is used for inputting the first image data and the second image data into a first repair model and a second repair model of the neural network respectively, so that the first repair model repairs the defective image block along a first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along a second direction based on the second image data to obtain second repair data; and a repair module for repairing the defective image block based on the first repair data and the second repair data.

Optionally, the starting boundary determining module includes: the boundary searching unit is used for searching a plurality of groups of relative boundary pairs of the defect image block; a target determining unit configured to determine, as a target boundary pair, a pair of boundary pairs having shortest boundary-to-boundary distances among a plurality of sets of relative boundary pairs; and a start determining unit configured to take one boundary of the pair of target boundaries toward the other boundary as a start boundary determined as a first direction, and the other boundary toward the one boundary as a start boundary determined as a second direction.

Optionally, the size of the preset angle is 45 °.

Optionally, the method further comprises: the format conversion module is used for converting the data format of the first image data and the second image data into a YUV format; and in the repair data acquisition module and the repair module, repairing the defective image block by adopting the image data in the YUV format.

Optionally, in the repair data acquisition module, a sum of areas occupied by the first repair data and the second repair data in the defect image block is smaller than an area size of the defect image block; the repair module includes: the direct repair unit is used for repairing corresponding areas in the defect image block through the first repair data and the second repair data respectively; and an indirect repair unit for repairing unrepaired image data in the defective image block by the first repair data and the second repair data.

Optionally, the indirect repair unit comprises: an input subunit for inputting the first repair data and the second repair data into a third repair model of the neural network; the data obtaining subunit is used for obtaining third repair data based on the first repair data and the second repair data in the third repair model; and a repair subunit for repairing the unrepaired image data in the defective image block by the third repair data.

Optionally, during the repair of the defective image block by the repair module: when the repair data is outside the boundary of the defect image block, adopting corresponding original image data in the image to be repaired to replace the repair data; the repair data comprise first repair data, second repair data and third repair data, wherein the third repair data is obtained based on the first repair data and the second repair data; when the repair data is within the boundary of the defective image block, repairing a corresponding region within the defective image block using the repair data.

According to a third aspect, an embodiment of the present invention discloses a terminal device, including: a processor configured to implement the method of any of the above first aspects.

Optionally, the terminal device is a recorder, a monitoring device, a mobile terminal or a camera with image processing capabilities.

According to a fourth aspect, an image interaction system according to an embodiment of the present invention includes: a first device and a second device; the first device sends the image to the second device; the second device is configured to implement the method disclosed in any of the first aspects above.

Optionally, the first device is an image capturing device, and the second device is an intelligent device with image processing capability.

According to a fifth aspect, an embodiment of the present invention discloses a neural network training method for repairing an image defect, where a model of a neural network includes a first repair model and a second repair model, and the neural network training method includes:

step S201, obtaining an image sample to be learned, wherein a plurality of mutually independent images are obtained; step S203, eliminating image blocks in a preset area in an image frame according to respective preset rules to obtain an input sample with defective image blocks; step S205, inputting an input sample frame into a neural network, and training a first repair model and a second repair model; the first repair model is used for repairing the defect image block along a first direction to obtain first repair data, and the second repair model is used for repairing the defect image block along a second direction to obtain second repair data; extracting image data of a region with a preset size close to a first direction initial boundary to obtain first image data training a first repair model; extracting image data of a region with a preset size close to the initial boundary in the second direction to obtain second image data, and training a second repair model; step S207, outputting the trained first parameter and second parameter to solidify the first repair model and the second repair model; the first parameters are parameters of the first repair model after training, and the second parameters are parameters of the second repair model after training.

Alternatively, in step S205, the first direction and the second direction are opposite directions.

Optionally, the image block from which the preset area is eliminated has a plurality of boundaries, and the distance from the start boundary in the first direction to the start boundary in the second direction is the shortest from the respective boundaries to the opposite boundary distances.

Optionally, the model of the neural network further comprises a third repair model; in step S205, the sum of the areas occupied by the first repair data and the second repair data in the image block of the removed preset area is smaller than the area size of the image block of the removed preset area; after step S205, further includes: step S206, inputting the first repair data and the second repair data into a third repair model, and training the third repair model; the third repair model is used for obtaining third repair data based on the first repair data and the second repair data, and the third repair data is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block.

Optionally, when the repair data is located outside the boundary of the image block of the removed preset area in the image frame, replacing the repair data with the corresponding original image data in the image frame; the repair data includes first repair data, second repair data, and third repair data; and when the repair data is within the boundary of the image block from which the preset area is removed, repairing the corresponding area in the defective image block by adopting the repair data.

Optionally, training the first repair model, the second repair model, and the third repair model in YUV format.

According to a sixth aspect, an embodiment of the present invention discloses a neural network training device for repairing an image defect, the model of the neural network including a first repair model and a second repair model, the neural network training device including:

the sample acquisition module is used for acquiring an image sample to be learned, and a plurality of mutually independent images are acquired by the image sample; the image block removing module is used for removing the image blocks in the preset area from the image frame according to respective preset rules to obtain an input sample with defective image blocks; the sample input module is used for inputting an input sample frame into the neural network and training a first repair model and a second repair model; the first repair model is used for repairing the defect image block along a first direction to obtain first repair data, and the second repair model is used for repairing the defect image block along a second direction to obtain second repair data; extracting image data of a region with a preset size close to a first direction initial boundary to obtain first image data training a first repair model; extracting image data of a region with a preset size close to the initial boundary in the second direction to obtain second image data, and training a second repair model; the first model solidifying module and the second model solidifying module are used for outputting the first parameter and the second parameter after training so as to solidify the first repairing model and the second repairing model; the first parameters are parameters of the first repair model after training, and the second parameters are parameters of the second repair model after training.

Optionally, in the sample input module, the first direction and the second direction are opposite directions.

Optionally, the model of the neural network further comprises a third repair model; in the sample input module, the sum of the areas occupied by the first repair data and the second repair data in the image blocks of the eliminated preset area is smaller than the area size of the image blocks of the eliminated preset area; the neural network training device further includes: the third model solidifying module inputs the first repair data and the second repair data into a third repair model to train the third repair model; the third repair model is used for obtaining third repair data based on the first repair data and the second repair data, and the third repair data is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block.

According to a seventh aspect, an embodiment of the present invention discloses a neural network structure for image defect repair, including:

the data acquisition module is used for acquiring first image data and second image data, wherein the first image data is image data of a region with a preset size extracted along a first direction in an image to be repaired of which the defect image block exists, the second image data is image data of a region with a preset size extracted along a second direction in the image to be repaired of which the defect image block exists, and the first direction and the second direction are opposite directions; the first repair model is used for repairing the defective image block along a first direction based on the first image data to obtain first repair data; the second repair model is used for repairing the defective image block along a second direction based on the second image data to obtain second repair data; the first repair data and the second repair data are used to repair the defective image block.

Optionally, the sum of the areas occupied by the first repair data and the second repair data in the defect image block is smaller than the area size of the defect image block; the neural network structure further includes: the third repair model is used for obtaining third repair data based on the first repair data and the second repair data; the third repair data is used to repair the unrepaired image data in the defective image block.

According to an eighth aspect, an embodiment of the present invention discloses a neural network training system for image defect repair, including: the image data acquisition device is used for acquiring an image sample to be learned, and a plurality of mutually independent images are acquired by the image sample; a memory for storing a program; a processor that receives a sample of images to be learned for executing a program to implement the method of any of the fifth aspects disclosed above.

According to a ninth aspect, an embodiment of the present invention discloses a computer readable storage medium having stored thereon a computer program, characterized in that the computer program stored in the storage medium is adapted to be executed to implement the method of any of the above-mentioned first aspects; alternatively, a computer program stored in a storage medium is adapted to be executed to implement the method of any of the above fifth aspects.

According to a tenth aspect, an embodiment of the invention discloses a chip of an image device having an integrated circuit thereon, the integrated circuit being designed for implementing the method of any of the above-described first aspects; or for implementing the method of any of the above fifth aspects.

According to an eleventh aspect, an embodiment of the present invention discloses a server having stored thereon a computer program for being executed to implement the method of any of the above-mentioned first aspects; alternatively, a stored computer program is adapted to be executed to implement the method of any of the above fifth aspects.

According to a twelfth aspect, an embodiment of the present invention discloses a platform server, including:

the request receiving module is used for receiving the data request; a data issuing module for providing a computer program and/or a computer program link to a user according to a data request, the computer program being adapted to be executed to implement the method disclosed in any of the first aspects above; alternatively, the computer program is for being executed to implement the method of any of the above fifth aspects.

According to the image defect repairing and neural network training method, device and system disclosed by the embodiment of the invention, after the images to be repaired, which are provided with the defect image blocks, are obtained, the initial boundaries of the first direction and the second direction of the defect image blocks are determined, then the first repairing model repairs the defect image blocks along the first direction based on the first image data to obtain first repairing data, and the second repairing model repairs the defect image blocks along the second direction based on the second image data to obtain second repairing data, so that the defect image blocks can be repaired by the first repairing data and the second repairing data; in the embodiment of the invention, the first direction and the second direction are opposite directions, so that the first repair model and the second repair model repair image data in opposite directions, namely, the first repair model and the second repair model can repair different area data of the defective image block by different work, thereby improving repair efficiency and reducing repair time consumption; in addition, the first repair model and the second repair model repair image data in different directions respectively, so that repair objects of the first repair model and the second repair model are clear, and compared with a mode of repairing a defect image block by adopting a large window in a general way in the prior art, in the scheme of the embodiment of the invention, the first repair model and the second repair model repair data have pertinence, and therefore, the repair precision can be improved.

As an alternative, among a plurality of sets of opposite boundary pairs, a pair of boundary pairs with the shortest boundary-to-boundary distance is determined as a target boundary pair, one boundary of the target boundary pairs faces the other boundary to be the initial boundary determined as a first direction, and the other boundary faces the one boundary to be the initial boundary determined as a second direction, so that when repairing the defective image block along the first direction and the second direction, on one hand, the repairing distance can be reduced, and the repairing efficiency is improved; on the other hand, since the target boundary pair is the boundary with the shortest distance, in other words, the boundary of the target boundary pair is the longest, more original image data can be extracted when extracting the data of the area with the preset size of the initial boundary, thereby providing more original image data for the first repair model and the second repair model, that is, providing more effective input data, and improving the repair accuracy of the first repair model and the second repair model.

As an alternative, in the cartesian coordinate system, different types of boundary pairs are distinguished, so that different shapes of the defective image block can be adapted when the target boundary pairs are found, and thus, the target boundary pairs can be rapidly located.

As an alternative scheme, the defective image block is repaired by adopting the image data in the YUV format, so that the operand can be reduced, and the data repair efficiency is improved.

As an alternative scheme, the sum of the areas occupied by the first repair data and the second repair data in the defect image block is smaller than the size of the area of the defect image block, the corresponding areas in the defect image block are repaired respectively through the first repair data and the second repair data, and the unrepaired image data in the defect image block are repaired through the first repair data and the second repair data, so that on one hand, the image data of different areas can be repaired separately, and the repair efficiency is improved; on the other hand, the unrepaired image data in the defective image block is repaired by the first repair data and the second repair data, so that the first repair data and the second repair data can be naturally transited in the unrepaired image data, the image transition smoothness of different areas of the defective image block is improved, and the unnatural manifestation of image mutation of different areas is reduced.

As an alternative scheme, in the third repair model, third repair data is obtained based on the first repair data and the second repair data, and the unrepaired image data in the defective image block is repaired by the third repair data, so that the first repair data and the second repair data can be further fused in the third repair model, the image transition smoothness of different areas of the defective image block is further improved, and the unnatural manifestation of image mutation of different areas is reduced.

Alternatively, when the repair data is located outside the boundary of the defective image block, the repair data is replaced with the corresponding original image data in the image to be repaired, that is, the data of the original image outside the boundary of the defective image block is maintained, and the pixel value of the original image outside the boundary is prevented from being erroneously changed, thereby maintaining the authenticity of the image data outside the boundary of the defective image block.

As an alternative scheme, a transition region with a preset size is arranged in the defect image block, and the transition region is filled by adopting the restoration data of the transition region and the weighting of the original image data, so that the image data of the transition region can be prevented from jumping greatly, and the transition naturalness and smoothness of the image data of the transition region are improved.

Other advantages of the present invention will be set forth in the description of specific technical features and solutions, by which those skilled in the art should understand the advantages that the technical features and solutions bring.

Drawings

Embodiments according to the present invention will be described below with reference to the accompanying drawings. In the figure:

fig. 1 is a flowchart of an image defect repairing method based on a neural network according to the disclosure of the present embodiment;

Fig. 2A, 2B, and 2C are schematic diagrams of a comparative example of a defective image block disclosed in the present embodiment, wherein fig. 2A is a schematic diagram of a complete image example disclosed in the present embodiment; FIG. 2B is a schematic diagram of an example of artificially adding an occlusion subtitle disclosed in this embodiment; fig. 2C is a schematic diagram showing an example of data loss of a certain image block of an image disclosed in the present embodiment;

fig. 3 is a schematic diagram illustrating an exemplary defective image block repairing process according to the present embodiment;

FIG. 4 is a flow chart of a method for determining the starting boundaries of the first and second directions according to the present embodiment;

FIG. 5 is a schematic diagram illustrating an example of a process for determining a boundary of a target according to the present embodiment;

FIG. 6 is a schematic diagram of an example of searching for multiple sets of relative boundary pairs according to the disclosure;

fig. 7 is a schematic diagram showing an example of a process of repairing a defective image block by first repair data and second repair data according to the present embodiment;

FIG. 8 is a flowchart of a method for repairing a defective image block by first repair data and second repair data according to the present embodiment;

fig. 9 is a schematic diagram of a transition region of a defective image block according to the present embodiment;

fig. 10 is a schematic structural diagram of an image defect repairing device based on a neural network according to the present embodiment;

FIG. 11 is a flowchart of a neural network training method for image defect repair according to the present embodiment;

FIG. 12 is a schematic diagram of an image frame in an input sample according to the present embodiment;

FIG. 13 is a schematic diagram showing an example of a process of training a neural network by first repair data and second repair data according to the present embodiment;

fig. 14 is a schematic structural diagram of a neural network training device for repairing image defects according to the present embodiment;

fig. 15 is a schematic structural diagram of a neural network structure for repairing image defects according to the present embodiment.

Detailed Description

In order to reduce the time consumption of repairing the image defects and improve the repairing precision, the embodiment discloses an image defect repairing method based on a neural network, please refer to fig. 1, which is a flowchart of an image defect repairing method based on a neural network disclosed in the embodiment, the image defect repairing method based on the neural network comprises:

step S101, an image to be repaired, in which a defective image block exists, is acquired. Specifically, the defect image block may have a defect existing in the image of the target object, or may be an artificially increased occlusion for a certain area block of the image, or may be a data loss of the image itself. Specifically, the defect of the target object itself may be, for example, a defect on the body of the target object, for example, a spot, a flaw, or the like on the body of the target object; the artificially increased occlusion for a certain region block of the image may be e.g. subtitles, watermarks etc. added in the image; the data loss of the image itself may be the data loss of a certain image block or a plurality of image blocks of the image, and the data loss may be the data loss in the process of transmitting the image data, or the data loss of the acquired image already exists. The digital image is generally stored after compression, so that in the implementation process, if the image is in a compressed format, a corresponding decoder is required to restore the image to image data represented in YUV or RGB format, and of course, in the implementation process, the image may also be in other non-compressed formats.

Referring to fig. 2A, 2B, and 2C, fig. 2A is a schematic diagram of a complete image example disclosed in the present embodiment; fig. 2B is a schematic diagram of an example of artificially adding an occlusion subtitle disclosed in this embodiment, and fig. 2B illustrates a word of "wireless mouse"; fig. 2C is a schematic diagram illustrating an example of data loss of a certain image block of an image disclosed in the present embodiment, and an oval area in fig. 2C is a defective image block of lost data.

Step S103, determining a start boundary of the defective image block in the first direction and the second direction. In this embodiment, the first direction and the second direction are opposite directions, for example, the first direction is from left to right, and the second direction is from right to left; for another example, the first direction is from top to bottom, and the second direction is from bottom to top. In this embodiment, the start boundary refers to a boundary of a start position in the first direction or the second direction among boundaries of defective image blocks. It will be appreciated that the starting boundary is typically the outline of a defective image block, for example, in the "wireless mouse" defective image block shown in fig. 2B, each side of the rectangular outline thereof may be the starting boundary; for another example, in the defective image block of the missing data shown in fig. 2C, a tangent line at any position of the ellipse may be the starting boundary.

Step S105, extracting the image data of the area with the preset size near the initial boundary in the first direction to obtain the first image data, and extracting the image data of the area with the preset size near the initial boundary in the second direction to obtain the second image data. In this embodiment, referring to fig. 2C for an example, please refer to fig. 3, which is a schematic diagram illustrating a defect image block repairing process disclosed in this embodiment, dashed lines in fig. 3 respectively illustrate a first direction starting boundary and a second direction starting boundary, wherein a downward arrow is a first direction, and an upward arrow is a second direction. In a specific implementation process, image data of a region with a preset size can be extracted from a region near a starting boundary in a first direction to obtain first image data 11; image data of a region with a preset size can be extracted from the adjacent region of the initial boundary in the second direction to obtain second image data 12; in this embodiment, the specific size of the preset size area is not limited, and may be determined empirically; in an implementation, the first image data 11 and the second image data 12 may each contain image data outside the starting boundary. In addition, as an example of size selection of the first and second image data (11, 12), in the implementation, the first and second image data (11, 12) are selected such that, after being translated in the respective directions, the areas covered by the translation tracks of the first and second image data (11, 12) should be capable of covering at least the defective image blocks.

Step S107, inputting the first image data and the second image data into a first repair model and a second repair model of the neural network, respectively. In this embodiment, please refer to fig. 3, the first repair model is used for repairing the defective image block along the first direction, and the second repair model is used for repairing the defective image block along the second direction. In a specific embodiment, the first image data and the second image data are respectively input into a first repair model and a second repair model of the neural network, so that the first repair model can repair the defective image block along a first direction based on the first image data to obtain first repair data, and the second repair model can repair the defective image block along a second direction based on the second image data to obtain second repair data. In the implementation process, the first repair model and the second repair model can repair the data of the corresponding areas gradually along the first direction and the second direction respectively, and the data obtained by each repair can be used as the data input next time so as to repair the subsequent areas. In a specific implementation process, the first repair model and the second repair model can be obtained by training a neural network.

Step S109, repairing the defective image block based on the first repair data and the second repair data. Referring to fig. 3, since the first and second repair models repair the defective image block along the first and second directions, the first and second repair models repair the defective image block to obtain first and second repair data, and then the first and second repair data can cover the defective image block to repair the defective image block.

In one embodiment, the first and second repair models repair the upper and lower half parts of the defective image block in opposite directions, and the defective image block is obtained by splicing the upper and lower half parts.

In another embodiment, the first and second repair models repair a small portion of the defective image block, respectively, in opposite directions, for example, repair 1/3, 2/5 of the defective image block, respectively, i.e., the first and second repair models aggregate to repair, for example, 2/3, 4/5 of the defective image block, and then repair the remaining portion (for example, 1/3, 1/5) of the first and second models that is not repaired based on the repaired small portion (for example, 2/3, 4/5). It should be noted that the foregoing numerical values are merely examples for facilitating understanding by those skilled in the art, and are not to be construed as limiting the present application.

In the scheme of the embodiment, after an image to be repaired, which has a defective image block, is acquired, an initial boundary of the defective image block in a first direction and a second direction is determined, then, a first repair model repairs the defective image block along the first direction based on first image data to obtain first repair data, and a second repair model repairs the defective image block along the second direction based on second image data to obtain second repair data, so that the defective image block can be repaired by the first repair data and the second repair data; in the embodiment of the invention, the first direction and the second direction are opposite directions, so that the first repair model and the second repair model repair image data in opposite directions, namely, the first repair model and the second repair model can repair different area data of the defective image block by different work, thereby improving repair efficiency and reducing repair time consumption; in addition, the first repair model and the second repair model repair image data in different directions respectively, so that repair objects of the first repair model and the second repair model are clear, and compared with a mode of repairing a defect image block by adopting a large window in a general way in the prior art, in the scheme of the embodiment of the invention, the first repair model and the second repair model repair data have pertinence, so that repair precision can be improved.

In order to further reduce the distance of repair, improve the efficiency of repair, and further improve the accuracy of repair, in an alternative embodiment, a pair of boundary pairs with the shortest distance may be determined as the starting boundary of the first direction and the second direction. Specifically, referring to fig. 4, a flowchart of a method for determining a first direction start boundary and a second direction start boundary in the present embodiment is disclosed, and when executing step S103, the method includes:

step S1031, a plurality of sets of relative boundary pairs of the defective image block are found. Referring to fig. 5, an exemplary schematic diagram of a determination process of a target boundary disclosed in this embodiment is shown, in the image shown in fig. 5, a defective image block 1 exists, and the defective image block 1 has a plurality of sets of opposite boundary pairs A1, A2, A3 … …, which are all opposite in position and are tangent lines of the outline of the defective image block 1.

In step S1032, among the plurality of sets of the relative boundary pairs, a pair of boundary pairs having the shortest boundary-to-boundary distance is determined as the target boundary pair. After the multiple sets of relative boundary pairs of the defective image block are found, the boundary-to-boundary distances of the relative boundary pairs, for example, the boundary-to-boundary distances of the relative boundary pair A1, may be respectively set; and a pair of boundary pairs having the shortest boundary-to-boundary distance is taken as the target boundary pair. Of the sets of opposing boundary pairs A1, A2, A3 … … illustrated in fig. 5, the boundary-to-boundary distance of the opposing boundary pair A1 is the shortest, and thus the set of opposing boundary pairs A1 can be regarded as the target boundary pair.

In step S1033, one boundary in the target boundary pair is oriented toward the other boundary as a start boundary determined as the first direction, and the other boundary is oriented toward the one boundary as a start boundary determined as the second direction. Specifically, in the example of fig. 5, after determining the set of relative boundary pairs A1 as the target boundary pairs, the start boundaries of the first and second directions in the target boundary pairs may be determined. Specifically, the object boundary pair A1 includes a boundary a11 and a boundary a12, the boundary a11 may be determined to be in a first direction toward the boundary a12, and accordingly, the boundary a12 may be determined to be in a second direction toward the boundary a11, that is, the boundary a11 is a start boundary in the first direction and the boundary a12 is a start boundary in the second direction; of course, it is also possible to determine the direction of the boundary a11 toward the boundary a12 as the second direction, and accordingly, the direction of the boundary a12 toward the boundary a11 is determined as the first direction, that is, the boundary a11 is the start boundary of the second direction and the boundary a12 is the start boundary of the first direction.

In order to reduce the number of searches for relative boundary pairs, in an alternative embodiment, a plurality of sets of directions may be preset, and then the boundary pairs may be searched for in the plurality of sets of directions, specifically, the plurality of sets of directions may be preset by a cartesian coordinate system, specifically:

Referring to fig. 6, an exemplary diagram of searching for multiple sets of opposite boundary pairs according to the present embodiment is disclosed, where the multiple sets of opposite boundary pairs include four types:

(1) Direction of X axis

In the cartesian coordinate system, a boundary pair consisting of a boundary obtained in the positive direction along the X-axis and a boundary obtained in the negative direction along the X-axis is a boundary pair consisting of directions shown as B1 in fig. 6.

(2) Y-axis direction

In the cartesian coordinate system, a boundary pair consisting of a boundary obtained in the positive direction along the Y-axis and a boundary obtained in the negative direction along the Y-axis, such as a boundary pair consisting of directions shown as B2 in fig. 6.

(3) First, third quadrant direction

In the cartesian coordinate system, a boundary pair constituted by a boundary obtained toward the first quadrant along the preset angle and a boundary obtained toward the third quadrant along the preset angle, such as a boundary pair constituted by directions shown as B3 in fig. 6. In an alternative embodiment, the preset angle is of a magnitude of 45 °, i.e. 45 ° in a direction positive to the X-axis and 45 ° in a direction negative to the X-axis.

(4) Second, fourth quadrant direction

In the cartesian coordinate system, a boundary pair constituted by a boundary obtained toward the second quadrant along the preset angle and a boundary obtained toward the fourth quadrant along the preset angle, such as a boundary pair constituted by directions shown as B4 in fig. 6. In an alternative embodiment, the preset angle is of a magnitude of 45 °, i.e. 45 ° with respect to the positive Y-axis direction and 45 ° with respect to the negative Y-axis direction.

In this embodiment, in the cartesian coordinate system, multiple sets of directions are preset to distinguish different types of boundary pairs, so that when the target boundary pairs are searched, different shapes of the defect image block can be adapted, thereby, the target boundary pairs can be rapidly positioned, and the searching efficiency is improved.

In order to reduce the amount of computation and improve the data repair efficiency, in an alternative embodiment, between step S105 and step S107, the method further includes:

step S106, converting the data format of the first image data and the second image data into YUV format. In step S107 and step S109, the defective image block is repaired using the image data in YUV format.

Specifically, it is known that one pixel of an image can be represented by RGB or YUV, and in the present application, the repair process can be performed in an RGB format or in a YUV format, and an image with a resolution of 80×60 is taken as an example, and total pixels are 4800, which includes 80 rows and 60 columns. When processed in RGB format, each pixel is represented by three signals of RGB, and the image can be decomposed into three 80×60 matrices, which represent R, G, and B matrices, respectively.

Considering that human eyes are sensitive to the brightness (i.e. gray level, Y) of an image, but not to the color and saturation (i.e. U, V) of the image, after converting the RGB format into the YUV format, the U, V matrix can reduce the rank and column correspondence by one time, thereby reducing the operation amount.

In an alternative embodiment, please refer to fig. 7 and 8, fig. 7 is an exemplary schematic diagram of a process of repairing a defective image block by using first repair data and second repair data disclosed in this embodiment, and fig. 8 is a flowchart of a method of repairing a defective image block by using first repair data and second repair data disclosed in this embodiment.

In step S107, the sum of the areas occupied by the first repair data and the second repair data in the defective image block is smaller than the area size of the defective image block. Referring to fig. 7, the first repair data C1 is a rectangular frame of a lower left region in the image to be repaired, the second repair data C2 is a rectangular frame of an upper right region in the image to be repaired, the first repair data C1 and the second repair data C2 do not cover all regions of the defect image block 1, and the uncovered regions of the defect image block 1 are indicated by a symbol C3, that is, the sum of the regions occupied by the first repair data C1 and the second repair data C2 in the defect image block 1 is smaller than the region size of the defect image block 1.

Referring to fig. 8, step S109 includes:

step S1091, repairing corresponding areas in the defect image block respectively through the first repair data and the second repair data; step S1092, repairing unrepaired image data in the defective image block by the first repair data and the second repair data. Specifically, referring to fig. 7, after the first and second repair data (C1, C2) are obtained through the first and second repair models, the third repair data C3 may be obtained through repair of the first and second repair data (C1, C2), and the image data not repaired in the defective image block may be repaired through the third repair data C3, so that the defective image block is completely repaired.

In order to improve the image transition smoothness of different areas of the defective image block and reduce the unnatural performance of image abrupt changes of the different areas, in an alternative embodiment, the first repair data and the second repair data may be fused through a third repair model, and repair is performed to obtain third repair data C3. Specifically, when step S1092 is performed, it includes: inputting the first repair data and the second repair data into a third repair model of the neural network; in the third repair model, obtaining third repair data based on the first repair data and the second repair data; and repairing unrepaired image data in the defective image block by the third repair data. In a specific implementation process, the third repair model can also be obtained through training samples.

In order to maintain the authenticity of the image data outside the boundary of the defective image block, the original image pixel values outside the boundary are prevented from being erroneously changed, in an alternative embodiment, during the repair of the defective image block in step S109:

when the repair data is outside the boundary of the defect image block, adopting corresponding original image data in the image to be repaired to replace the repair data; the repair data comprise first repair data, second repair data and third repair data, wherein the third repair data is obtained based on the first repair data and the second repair data; when the repair data is within the boundary of the defective image block, repairing a corresponding region within the defective image block using the repair data.

Specifically, referring to fig. 7, taking the first repair model as an example, after the first repair data C1 is obtained by repairing the first repair model, the first repair data C1 includes data outside and inside the boundary of the defective image block: wherein, the area of the grid line of the rectangular frame of the first repair data C1 is the repair data within the boundary of the defect image block; the white area of the rectangular frame of the first repair data C1 is repair data outside the boundary of the defective image block, that is, this part of data is original image data in the image to be repaired. In the specific implementation process, for the area of the rectangular frame grid line of the first repair data C1, repairing the corresponding area in the defect image block by adopting the first repair data C1; for the white area of the rectangular frame of the first repair data C1, adopting the corresponding original image data in the image to be repaired to replace the data of the white area of the rectangular frame of the first repair data C1, so that the white area of the rectangular frame of the first repair data C1 is also the corresponding original image data in the image to be repaired. For the second and third repair models, similar processing methods are also adopted after the second and third repair data (C2 and C3) are obtained by repair, and are not described herein.

In order to avoid larger image data jump in the transition area and improve the naturalness and smoothness of the transition of the image data in the transition area, please refer to fig. 9, which is a schematic diagram of the transition area of the defect image block disclosed in this embodiment, in an alternative embodiment, repair data is used to repair a corresponding area in the defect image block, including: and presetting a transition region with a size in the defect image block, and filling the transition region by adopting the repairing data of the transition region and the weighting of the original image data. In this embodiment, the transition area in the defective image block refers to an area in the defective image block that transitions from the boundary of the defective image block to the inside of the defective image block, and the transition area can be simply understood as an outer ring of a predetermined size in the defective image block, as shown by a white area within the boundary in fig. 9.

In an alternative embodiment, the following formula is used to fill each pixel point in the transition region:

u=0.5a+0.5b, where u is a pixel value of a single pixel point in the transition region after repair, a is a pixel value of a single pixel point in the repair data, and b is a pixel value of a single pixel point in the original image data.

The embodiment also discloses an image defect repairing device based on a neural network, please refer to fig. 10, which is a schematic structural diagram of the image defect repairing device based on the neural network disclosed in the embodiment, the image defect repairing device based on the neural network includes: an image acquisition module 101, a starting boundary determination module 103, an image extraction module 105, a repair data acquisition module 107, and a repair module 109, wherein:

The image acquisition module 101 is used for acquiring an image to be repaired, wherein the image has a defective image block; the initial boundary determining module 103 is configured to determine an initial boundary of a first direction and a second direction of the defective image block, where the first direction and the second direction are opposite directions; the image extraction module 105 is configured to extract image data of an area with a preset size adjacent to a first direction start boundary to obtain first image data; extracting image data of a region with a preset size close to the initial boundary of the second direction to obtain second image data; the repair data acquisition module 107 is configured to input the first image data and the second image data into a first repair model and a second repair model of the neural network, so that the first repair model repairs the defective image block along a first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along a second direction based on the second image data to obtain second repair data; and a repair module 109 for repairing the defective image block based on the first repair data and the second repair data.

Optionally, the starting boundary determining module includes: the boundary searching unit is used for searching a plurality of groups of relative boundary pairs of the defect image block; a target determining unit configured to determine, as a target boundary pair, a pair of boundary pairs having shortest boundary-to-boundary distances among a plurality of sets of relative boundary pairs; and a start determining unit for taking one boundary of the object boundary pair towards the other boundary as a start boundary determined as a first direction and the other boundary towards the one boundary as a start boundary determined as a second direction.

Optionally, the size of the preset angle is 45 °.

The embodiment also discloses a terminal device, which is a device with image processing capability, and may be, for example, a recorder, a monitoring device, a mobile terminal, a camera, or the like, where the recorder may be, for example, a vehicle recorder, a law enforcement recorder, a miner's lamp recorder, or the like, and the camera may be, for example, an aerial camera, a motion DV, a portable IPC (network camera), or the like.

In this embodiment, the terminal device includes: and the processor is used for realizing any image defect repairing method based on the neural network.

The embodiment also discloses an image interaction system, which comprises: a first device and a second device; the first device sends the image to the second device; the second device is used for implementing any of the neural network-based image defect repairing methods.

In a specific embodiment, the first device is an image acquisition device, and the second device is an intelligent device with image processing capability. Specifically, the first device may be a simple image capturing device, for example, a camera, or may be a terminal device disclosed in the foregoing embodiment, and the second device may be a terminal device disclosed in the foregoing embodiment, or may be a computer, a mobile terminal, a server, or the like.

The embodiment also discloses a neural network training method for repairing image defects, wherein the neural network model comprises a first repairing model and a second repairing model, please refer to fig. 11, which is a flowchart of the neural network training method for repairing image defects, the neural network training method comprises:

step S201, an image sample to be learned is acquired. In this embodiment, the image sample is a plurality of mutually independent images. Please refer to fig. 2A, which is a schematic diagram illustrating a complete image disclosed in the present embodiment.

Step S203, eliminating the image blocks in the preset area in the image frame according to the respective preset rules to obtain an input sample with the defective image blocks. Referring to fig. 12, a schematic diagram of an image frame in an input sample disclosed in this embodiment is shown, after an image block in a preset area (shown as a grid area) is removed from the image frame to be learned, an image frame to be learned with a defective image block 1 is obtained, and the image frame after removing the preset area can be used as an image frame to be learned of the input sample. It should be noted that, in the implementation process, the shape of the preset area may be regular or irregular; the size of the preset area can be determined according to actual needs.

In this embodiment, the "eliminating the image block of the preset area" may be to block the image block of the preset area; it is also possible to delete the image data of the image block of the preset area directly, i.e. the data of the image block is lost. Taking fig. 2A as an example: referring to fig. 2B, an exemplary schematic diagram of artificially adding an occlusion subtitle is disclosed in this embodiment, and fig. 2B illustrates a defective image block 1 in which a word of "wireless mouse" is artificially added; fig. 2C is a schematic diagram showing an example of data loss of a certain image block of an image disclosed in the present embodiment, and an oval area in fig. 2C is a defective image block 1 in which data is deleted directly.

Step S205, inputting the input sample frame into the neural network, and training the first repair model and the second repair model. In this embodiment, the first repair model is used for repairing the defective image block along the first direction to obtain first repair data, and the second repair model is used for repairing the defective image block along the second direction to obtain second repair data. In this embodiment, the first direction and the second direction are opposite directions, for example, the first direction is from left to right, and the second direction is from right to left; for another example, the first direction is from top to bottom, and the second direction is from bottom to top.

In a specific embodiment, image data of a region with a preset size near the initial boundary in the first direction can be extracted to obtain first image data for training a first repair model, and image data of a region with a preset size near the initial boundary in the second direction can be extracted to obtain second image data for training a second repair model. In this embodiment, the start boundary refers to a boundary of a start position in the first direction or the second direction among boundaries of defective image blocks. It will be appreciated that the starting boundary is typically the outline of a defective image block, for example, in the "wireless mouse" defective image block shown in fig. 2B, each side of the rectangular outline thereof may be the starting boundary; for another example, in the defective image block of the missing data shown in fig. 2C, a tangent line at any position of the ellipse may be the starting boundary.

In this embodiment, referring to fig. 2C for an example, please refer to fig. 3, which is a schematic diagram illustrating a defect image block repairing process disclosed in this embodiment, dashed lines in fig. 3 respectively illustrate a first direction starting boundary and a second direction starting boundary, wherein a downward arrow is a first direction, and an upward arrow is a second direction. In a specific implementation process, image data of a region with a preset size can be extracted from a region near a starting boundary in a first direction to obtain first image data 11; image data of a region with a preset size can be extracted from the adjacent region of the initial boundary in the second direction to obtain second image data 12; in this embodiment, the specific size of the preset size area is not limited, and may be determined empirically; in an implementation, the first image data 11 and the second image data 12 may each contain image data outside the starting boundary. In addition, as an example of size selection of the first and second image data (11, 12), in the implementation, the first and second image data (11, 12) are selected such that, after being translated in the respective directions, the areas covered by the translation tracks of the first and second image data (11, 12) should be capable of covering at least the defective image blocks.

In this embodiment, the training sequence of the first repair model and the second repair model is not limited, and the first repair model and the second repair model may be separately trained, or may be synchronously trained at the same PC end.

In a specific implementation, the first repair model and/or the second repair model may be trained in an iterative manner, for example, and of course, other manners may be used to train the first repair model and the second repair model, respectively.

When the first repair model and/or the second repair model are trained respectively in a repeated iterative manner:

iteratively training the first repair model iteratively includes: obtaining ith repair data after the ith iteration, wherein i is a positive integer; judging whether a first error between the ith repair data and the pixel value of the corresponding position in the removed preset area is in a preset range or not; outputting model parameters obtained by the ith iteration to solidify the first repair model if the first error is in a preset range; in this embodiment, the model parameter may be, for example, a weight, a coefficient, or the like in the first repair model.

Iteratively training the second repair model iteratively includes: obtaining jth repair data after the jth iteration, wherein j is a positive integer; judging whether a second error between the j-th repair data and the pixel value of the corresponding position in the removed preset area is in a preset range or not; and if the second error is within the preset range, outputting model parameters obtained by the jth iteration to solidify the second repair model. In this embodiment, the model parameters may be, for example, weights, coefficients, etc. in the second repair model.

Step S207, outputting the trained first parameter and second parameter to solidify the first repair model and the second repair model. In this embodiment, the first parameter is a parameter of the first repair model after training, and the second parameter is a parameter of the second repair model after training. In the implementation process, after the first repair model and the second repair model are trained and cured, the first repair model and the second repair model can be stored in a storage device, so that the first repair model and the second repair model can be directly called when the image frame of the defect image block is repaired.

In an alternative embodiment, the image block from which the preset area is eliminated has a plurality of boundaries, and the distance from the start boundary in the first direction to the start boundary in the second direction is the shortest among the distances from each boundary to the opposite boundaries. Referring to fig. 12, an exemplary process for determining the initial boundaries of the first and second directions is shown in fig. 12, where a defective image block 1 exists in the image shown in fig. 12, and the defective image block 1 has multiple sets of opposite boundary pairs A1, A2, A3 and … …, where the opposite boundary pairs are all opposite in position, for example, the opposite boundary pair A1 includes boundaries a11 and a12, and is a tangent line to the contour of the defective image block 1. Of the plural boundaries illustrated in fig. 12, the distance between the boundaries a11 and a12 is shortest, and therefore, the boundaries a11 and a12 can be determined as the start boundary of the first direction to the start boundary of the second direction.

In the training process, the input sample is obtained by manually removing the preset area, so that in the specific implementation process, the boundary pair with the shortest distance (for example, two boundaries in the X-axis direction or two boundaries in the Y-axis direction) can be set when the preset area is removed, and therefore, the distance judgment is not required after the sample is input into the first repair model and the second repair model. Such embodiments should be considered equivalents of the inventive arrangements.

In an alternative embodiment, the model of the neural network further comprises a third repair model;

in step S205, the sum of the areas occupied by the first repair data and the second repair data in the image block of the culled preset area is smaller than the area size of the image block of the culled preset area. Referring to fig. 13, an exemplary process of training a neural network by using first repair data and second repair data is disclosed in this embodiment, wherein the first repair data C1 is a rectangular frame of a left area in the image to be repaired, the second repair data C2 is a rectangular frame of a right area in the image to be repaired, the first repair data C1 and the second repair data C2 do not cover all areas of the defect image block 1, and the uncovered areas in the defect image block 1 are indicated by a symbol C3, that is, the sum of the areas occupied by the first repair data C1 and the second repair data C2 in the defect image block 1 is smaller than the area size of the defect image block 1.

After step S205, further includes:

step S206, inputting the first repair data and the second repair data into the third repair model, and training the third repair model. The third repair model is used for obtaining third repair data C3 based on the first repair data and the second repair data, and the third repair data C3 is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block. In a specific embodiment, the third repair model may also be trained in an iterative manner, which is not described herein.

In this embodiment, the third repair model repairs the first repair data C1 and the second repair data C2 to obtain the third repair data C3, and the first repair data C1 and the second repair data C2 can be fused, so that the image transition smoothness of different areas of the defective image block is improved, and the unnatural performance of image mutation of different areas is reduced.

In order to maintain the authenticity of the image data outside the boundary of the defective image block, avoiding false changes to the original image pixel values outside the boundary, in an alternative embodiment, when the repair data is outside the boundary of the image block in the image frame from which the preset area is removed, the repair data is replaced with the corresponding original image data in the image frame; the repair data includes first repair data, second repair data, and third repair data; and when the repair data is within the boundary of the image block from which the preset area is removed, repairing the corresponding area in the defective image block by adopting the repair data.

Specifically, referring to fig. 13, taking the first repair model as an example, after the first repair data C1 is obtained by repairing the first repair model, the first repair data C1 includes data outside and inside the boundary of the image block (i.e., the defect image block 1) of the removed preset area in the image frame: wherein, the area of the grid line of the rectangular frame of the first repair data C1 is the repair data within the boundary of the defect image block 1; the white area of the rectangular frame of the first repair data C1 is repair data outside the boundary of the defective image block 1, that is, this part of data is original image data in the image to be learned. In the specific implementation process, for the area of the rectangular frame grid line of the first repair data C1, repairing the corresponding area in the defect image block by adopting the first repair data C1; for the white area of the rectangular frame of the first repair data C1, adopting the corresponding original image data in the image to be learned to replace the data of the white area of the rectangular frame of the first repair data C1, so that the white area of the rectangular frame of the first repair data C1 is also the corresponding original image data in the image to be learned. For the second and third repair models, similar processing methods are also adopted after the second and third repair data (C2 and C3) are obtained by repair, and are not described herein.

In a specific embodiment, the first repair model, the second repair model, and the third repair model may be trained in RGB format, or the first repair model, the second repair model, and the third repair model may be trained in YUV format.

In a preferred embodiment, the first, second, and third repair models are trained in YUV format. Considering that human eyes are sensitive to the brightness (i.e. gray level, Y) of an image, but not to the color and saturation (i.e. U, V) of the image, after converting the RGB format into the YUV format, the U, V matrix can reduce the rank and column correspondence by one time, thereby reducing the operation amount.

It should be noted that, in this embodiment, it is only preferable to recommend that the first repair model, the second repair model and the third repair model be trained in YUV format, and image data in other formats is not excluded, so those skilled in the art may select an appropriate image format (e.g. RGB) to train the first repair model, the second repair model and the third repair model according to the description of the present application, and should be considered as being within the scope of the present application.

The embodiment also discloses a neural network training device for repairing image defects, the model of the neural network includes a first repairing model and a second repairing model, please refer to fig. 14, which is a schematic structural diagram of the neural network training device for repairing image defects disclosed in the embodiment, the neural network training device includes: a sample acquisition module 201, an image block culling module 203, a sample input module 205, and a first and second model curing module 207, wherein:

A sample acquiring module 201, configured to acquire an image sample to be learned, where the image sample is a plurality of mutually independent images; the image block removing module 203 is configured to remove image blocks in a preset area from an image frame according to respective preset rules, so as to obtain an input sample with defective image blocks; a sample input module 205, configured to input an input sample frame to the neural network, and train the first repair model and the second repair model; the first repair model is used for repairing the defect image block along a first direction to obtain first repair data, and the second repair model is used for repairing the defect image block along a second direction to obtain second repair data; extracting image data of a region with a preset size close to a first direction initial boundary to obtain first image data training a first repair model; extracting image data of a region with a preset size close to the initial boundary in the second direction to obtain second image data, and training a second repair model; a first and second model curing module 207 for outputting the trained first and second parameters to cure the first and second repair models; the first parameters are parameters of the first repair model after training, and the second parameters are parameters of the second repair model after training.

In an alternative embodiment, in the sample input module 205, the first direction and the second direction are opposite directions.

In an alternative embodiment, the image block from which the preset area is eliminated has a plurality of boundaries, and the distance from the start boundary in the first direction to the start boundary in the second direction is the shortest among the distances from each boundary to the opposite boundaries.

In an alternative embodiment, the model of the neural network further comprises a third repair model; in the sample input module 205, the sum of the areas occupied by the first repair data and the second repair data in the image blocks of the eliminated preset area is smaller than the area size of the image blocks of the eliminated preset area; the neural network training device further includes: the third model curing module 206 inputs the first repair data and the second repair data into a third repair model to train the third repair model; the third repair model is used for obtaining third repair data based on the first repair data and the second repair data, and the third repair data is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block.

In an alternative embodiment, when the repair data is outside the boundary of the image block of the removed preset area in the image frame, the repair data is replaced by the corresponding original image data in the image frame; the repair data includes first repair data, second repair data, and third repair data; and when the repair data is within the boundary of the image block from which the preset area is removed, repairing the corresponding area in the defective image block by adopting the repair data.

In an alternative embodiment, the first, second, and third repair models are trained in YUV format.

The embodiment also discloses a neural network structure for repairing image defects, please refer to fig. 15, which is a schematic structural diagram of the neural network structure for repairing image defects disclosed in the embodiment, the neural network structure for repairing image defects includes: a data acquisition module 151, a first repair model 152, a second repair model 153, wherein:

a data obtaining module 151, configured to obtain first image data and second image data, where the first image data is image data of a region of a preset size extracted along a first direction in an image to be repaired in which a defective image block exists, the second image data is image data of a region of a preset size extracted along a second direction in the image to be repaired in which the defective image block exists, and the first direction and the second direction are opposite directions; a first repair model 152, configured to repair the defective image block along a first direction based on the first image data to obtain first repair data; a second repair model 153, configured to repair the defective image block along a second direction based on the second image data to obtain second repair data; the first repair data and the second repair data are used to repair the defective image block.

Specifically, please refer to the description of the above embodiments, which is not repeated herein.

In an alternative embodiment, the sum of the areas occupied by the first repair data and the second repair data in the defective image block is smaller than the area size of the defective image block;

the neural network structure further includes: a third repair model 154, the third repair model 154 being configured to obtain third repair data based on the first repair data and the second repair data; the third repair data is used to repair the unrepaired image data in the defective image block.

The embodiment also discloses a neural network training system for repairing image defects, which comprises: image data acquisition device, memory and processor, wherein:

the image data acquisition device is used for acquiring an image sample to be learned, and a plurality of mutually independent images are acquired by the image sample; the memory is used for storing programs; and the processor is used for receiving the voice signal sample data to be learned and executing a program to realize the neural network training method for repairing the image defects disclosed in any embodiment.

In this embodiment, the neural network training system may be implemented by a computer or a server.

The present embodiment also discloses a computer-readable storage medium having stored thereon a computer program, the computer program stored in the storage medium being for being executed to implement the neural network-based image defect repair method disclosed in any of the above embodiments.

The present embodiment also discloses a computer-readable storage medium having stored thereon a computer program, the computer program stored in the storage medium being for being executed to implement the neural network training method for image defect repair disclosed in any of the above embodiments.

The embodiment also discloses a chip of the image device, and the chip is provided with an integrated circuit, and the integrated circuit is designed to realize the neural network-based image defect repairing method disclosed in any embodiment.

The embodiment also discloses a chip of the image device, and the chip is provided with an integrated circuit, and the integrated circuit is designed to realize the neural network training method for repairing the image defects disclosed in any embodiment.

The embodiment also discloses a server, on which a computer program is stored, the stored computer program being used to be executed to implement the neural network-based image defect repair method disclosed in any of the above embodiments.

The embodiment also discloses a server, on which a computer program is stored, the stored computer program being used to be executed to implement the neural network training method for repairing image defects disclosed in any of the above embodiments.

The embodiment also discloses a platform server, comprising: a request receiving module and a data issuing module, wherein: the request receiving module is used for receiving a data request; the data issuing module is used for providing a computer program and/or a computer program link for a user according to the data request, and the computer program is used for being executed to realize the neural network-based image defect repairing method disclosed in any embodiment.

The embodiment also discloses a platform server, comprising: a request receiving module and a data issuing module, wherein: the request receiving module is used for receiving a data request; the data issuing module is used for providing a computer program and/or a computer program link for a user according to the data request, and the computer program is used for being executed to realize the neural network training method for repairing the image defects disclosed in any embodiment.

According to the image defect repairing and neural network training method, device and system disclosed by the embodiment, after the image to be repaired, which has the defect image block, is obtained, the initial boundaries of the first direction and the second direction of the defect image block are determined, then the first repairing model repairs the defect image block along the first direction based on the first image data to obtain first repairing data, and the second repairing model repairs the defect image block along the second direction based on the second image data to obtain second repairing data, so that the defect image block can be repaired through the first repairing data and the second repairing data; in the embodiment of the invention, the first direction and the second direction are opposite directions, so that the first repair model and the second repair model repair image data in opposite directions, namely, the first repair model and the second repair model can repair different area data of the defective image block by different work, thereby improving repair efficiency and reducing repair time consumption; in addition, the first repair model and the second repair model repair image data in different directions respectively, so that repair objects of the first repair model and the second repair model are clear, and compared with a mode of repairing a defect image block by adopting a large window in a general way in the prior art, in the scheme of the embodiment of the invention, the first repair model and the second repair model repair data have pertinence, and therefore, the repair precision can be improved.

Those skilled in the art will appreciate that the above-described preferred embodiments can be freely combined and stacked without conflict.

It will be understood that the above-described embodiments are merely illustrative and not restrictive, and that all obvious or equivalent modifications and substitutions to the details given above may be made by those skilled in the art without departing from the underlying principles of the invention, are intended to be included within the scope of the appended claims.

Claims

1. An image defect repairing method based on a neural network, which is characterized by comprising the following steps:

step S101, obtaining an image to be repaired, wherein the image has a defective image block;

step S103, determining the initial boundary of the first direction and the second direction of the defect image block, wherein the first direction and the second direction are opposite directions;

step S105, extracting image data of a region with a preset size adjacent to the first direction initial boundary to obtain first image data; extracting image data of a region with a preset size adjacent to the initial boundary of the second direction to obtain second image data;

step S107, inputting the first image data and the second image data into a first repair model and a second repair model of a neural network respectively, so that the first repair model repairs the defective image block along the first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along the second direction based on the second image data to obtain second repair data; a kind of electronic device with high-pressure air-conditioning system

And step S109, repairing the defective image block based on the first repair data and the second repair data.

2. The image defect repair method of claim 1, wherein the step S103 includes:

step S1031, searching a plurality of groups of relative boundary pairs of the defect image block;

step S1032, determining a pair of boundary pairs with shortest boundary-to-boundary distances as target boundary pairs among the plurality of sets of relative boundary pairs; a kind of electronic device with high-pressure air-conditioning system

Step S1033, setting one boundary of the pair of target boundaries toward the other boundary as a start boundary in the first direction, and setting the other boundary toward the one boundary as a start boundary in the second direction.

3. The image defect repair method of claim 2 wherein the plurality of sets of relative boundary pairs comprise: in a Cartesian coordinate system, a boundary pair consisting of a boundary obtained in the positive direction along the X-axis and a boundary obtained in the negative direction along the X-axis, a boundary pair consisting of a boundary obtained in the positive direction along the Y-axis and a boundary obtained in the negative direction along the Y-axis, a boundary pair consisting of a boundary obtained in a direction toward the first quadrant at a preset angle and a boundary obtained in a direction toward the third quadrant at a preset angle, and a boundary pair consisting of a boundary obtained in a direction toward the second quadrant at a preset angle and a boundary obtained in a direction toward the fourth quadrant at a preset angle.

4. The image defect repair method of claim 3, wherein the magnitude of the preset angle is 45 °.

5. The image defect repair method of any one of claims 1-4, further comprising, between the step S105 and the step S107:

step S106, converting the data format of the first image data and the second image data into YUV format;

in the steps S107 and S109, the defective image block is repaired using the image data in YUV format.

6. The method for repairing an image defect according to any one of claims 1 to 4,

in the step S107, the sum of the areas occupied by the first repair data and the second repair data in the defective image block is smaller than the area size of the defective image block;

the step S109 includes:

step S1091, repairing the corresponding area in the defective image block by the first repair data and the second repair data, respectively;

and step S1092, repairing unrepaired image data in the defective image block through the first repairing data and the second repairing data.

7. The image defect repair method of claim 6, wherein the step S1092 comprises:

Inputting the first repair data and the second repair data into a third repair model of the neural network;

in the third repair model, third repair data is obtained based on the first repair data and the second repair data;

and repairing unrepaired image data in the defective image block through the third repairing data.

8. The image defect repair method of any one of claims 1-4, wherein during repair of the defective image block in step S109:

when the repair data is outside the boundary of the defect image block, adopting corresponding original image data in the image to be repaired to replace the repair data; the repair data comprises the first repair data, the second repair data and third repair data, wherein the third repair data is obtained based on the first repair data and the second repair data;

and when the repair data is within the boundary of the defect image block, repairing the corresponding area in the defect image block by adopting the repair data.

9. The image defect repair method of claim 8, wherein repairing the corresponding region within the defective image block using the repair data comprises:

And presetting a transition region with a size in the defect image block, and filling the transition region by adopting the repairing data of the transition region and the weighting of the original image data.

10. The image defect repair method of claim 9 wherein each pixel point in the transition region is filled using the formula:

u=0.5a+0.5b, where u is a pixel value of the single pixel point in the transition region after repair, a is a pixel value of the single pixel point in the repair data, and b is a pixel value of the single pixel point in the original image data.

11. An image defect repair apparatus based on a neural network, comprising:

an image acquisition module (101) for acquiring an image to be repaired, of a defective image block;

a start boundary determining module (103) configured to determine a start boundary of a first direction and a second direction of the defective image block, where the first direction and the second direction are opposite directions;

the image extraction module (105) is used for extracting image data of an area with a preset size adjacent to the first direction starting boundary to obtain first image data; extracting image data of a region with a preset size adjacent to the initial boundary of the second direction to obtain second image data;

A repair data acquisition module (107) configured to input the first image data and the second image data into a first repair model and a second repair model of a neural network, respectively, so that the first repair model repairs the defective image block along the first direction based on the first image data to obtain first repair data, and the second repair model repairs the defective image block along the second direction based on the second image data to obtain second repair data; a kind of electronic device with high-pressure air-conditioning system

A repair module (109) for repairing the defective image block based on the first repair data and the second repair data.

12. The image defect repair apparatus of claim 11, wherein the start boundary determination module comprises:

the boundary searching unit is used for searching a plurality of groups of relative boundary pairs of the defect image block;

a target determining unit configured to determine, as a target boundary pair, a pair of boundary pairs having a shortest boundary-to-boundary distance among the plurality of sets of relative boundary pairs; a kind of electronic device with high-pressure air-conditioning system

And the start determining unit is used for taking one boundary of the target boundary pair towards the other boundary as a start boundary of a first direction and taking the other boundary towards the one boundary as a start boundary of a second direction.

13. The image defect repair apparatus of claim 12, wherein the plurality of sets of opposing boundary pairs comprise: in a Cartesian coordinate system, a boundary pair consisting of a boundary obtained in the positive direction along the X-axis and a boundary obtained in the negative direction along the X-axis, a boundary pair consisting of a boundary obtained in the positive direction along the Y-axis and a boundary obtained in the negative direction along the Y-axis, a boundary pair consisting of a boundary obtained in a direction toward the first quadrant at a preset angle and a boundary obtained in a direction toward the third quadrant at a preset angle, and a boundary pair consisting of a boundary obtained in a direction toward the second quadrant at a preset angle and a boundary obtained in a direction toward the fourth quadrant at a preset angle.

14. The image defect repair apparatus of claim 13, wherein the magnitude of the predetermined angle is 45 °.

15. The image defect repair apparatus of claim 11, further comprising:

the format conversion module is used for converting the data format of the first image data and the second image data into a YUV format;

and in the repair data acquisition module and the repair module, repairing the defective image block by adopting image data in a YUV format.

16. The image defect repair apparatus according to any one of claims 11-15, wherein,

In the repair data acquisition module, the sum of the areas occupied by the first repair data and the second repair data in the defect image block is smaller than the area size of the defect image block;

the repair module includes:

a direct repair unit, configured to repair corresponding areas in the defective image block through the first repair data and the second repair data, respectively;

and an indirect repair unit configured to repair the unrepaired image data in the defective image block by the first repair data and the second repair data.

17. The image defect repair apparatus of claim 16, wherein the indirect repair unit comprises:

an input subunit configured to input the first repair data and the second repair data into a third repair model of the neural network;

a data obtaining subunit, configured to obtain, in the third repair model, third repair data based on the first repair data and the second repair data;

and a repair subunit for repairing unrepaired image data in the defective image block by the third repair data.

18. The image defect repair apparatus of any one of claims 11-15, wherein, during repair of the defective image block by the repair module:

19. The image defect repair apparatus of claim 18, wherein repairing a corresponding region within the defective image block using the repair data comprises:

20. The image defect repair apparatus of claim 19, wherein each pixel point in the transition region is filled using the formula:

21. A terminal device, comprising:

a processor for implementing the method of any of claims 1-10.

22. The terminal device of claim 21, wherein the terminal device is a recorder, a monitoring device, a mobile terminal or a camera with image processing capabilities.

23. An image interaction system, comprising: a first device and a second device;

the first device sends an image to the second device;

the second device being adapted to implement the method of any of claims 1-10.

24. The image interaction system of claim 23, wherein the first device is an image acquisition device and the second device is a smart device having image processing capabilities.

25. A neural network training method for image defect repair, the model of the neural network comprising a first repair model and a second repair model, the neural network training method comprising:

step S201, obtaining an image sample to be learned, wherein a plurality of mutually independent images are obtained from the image sample;

step S203, eliminating image blocks in a preset area in an image frame according to respective preset rules to obtain an input sample with defective image blocks;

Step S205, inputting the input sample frame into the neural network, and training the first repair model and the second repair model; the first repair model is used for repairing the defective image block along a first direction to obtain first repair data, and the second repair model is used for repairing the defective image block along a second direction to obtain second repair data; extracting image data of a region with a preset size adjacent to the initial boundary of the first direction to obtain first image data, and training the first repair model; extracting image data of a region with a preset size adjacent to the initial boundary of the second direction to obtain second image data, and training the second repair model;

step S207, outputting the trained first parameter and second parameter to solidify the first repair model and the second repair model; the first parameters are parameters of the first repair model after training, and the second parameters are parameters of the second repair model after training.

26. The neural network training method of claim 25, wherein in step S205, the first direction and the second direction are opposite directions.

27. The neural network training method of claim 26, wherein the image block from which the predetermined region is culled has a plurality of boundaries, each boundary being at a shortest distance from a start boundary in the first direction to a start boundary in the second direction among the distances from the opposite boundaries.

28. The neural network training method of any of claims 25-27, wherein the model of the neural network further comprises a third repair model;

in the step S205, the sum of the areas occupied by the first repair data and the second repair data in the image block of the removed preset area is smaller than the area size of the image block of the removed preset area;

after the step S205, further includes:

step S206, inputting the first repair data and the second repair data into the third repair model, and training the third repair model; the third repair model is used for obtaining third repair data based on the first repair data and the second repair data, and the third repair data is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block.

29. The neural network training method of claim 28, wherein when the repair data is outside the boundaries of the image blocks in the image frame from which the predetermined region is removed, the repair data is replaced with the corresponding original image data in the image frame; the repair data includes the first repair data, the second repair data, and third repair data;

and when the repair data is within the boundary of the image block of the eliminated preset area, repairing the corresponding area in the defective image block by adopting the repair data.

30. The neural network training method of claim 28, wherein the first repair model, the second repair model, and the third repair model are trained in YUV format.

31. A neural network training device for image defect repair, the model of the neural network comprising a first repair model and a second repair model, the neural network training device comprising:

a sample acquisition module (201) for acquiring an image sample to be learned, the image sample being a plurality of mutually independent images;

the image block removing module (203) is used for removing the image blocks in the preset area from the image frame according to respective preset rules to obtain an input sample with defective image blocks;

A sample input module (205) for inputting the input sample frame to the neural network, training the first repair model and the second repair model; the first repair model is used for repairing the defective image block along a first direction to obtain first repair data, and the second repair model is used for repairing the defective image block along a second direction to obtain second repair data; extracting image data of a region with a preset size adjacent to the initial boundary of the first direction to obtain first image data, and training the first repair model; extracting image data of a region with a preset size adjacent to the initial boundary of the second direction to obtain second image data, and training the second repair model;

a first and second model curing module (207) for outputting the trained first and second parameters to cure the first and second repair models; the first parameters are parameters of the first repair model after training, and the second parameters are parameters of the second repair model after training.

32. The neural network training device of claim 31, characterized in that in the sample input module (205), the first direction and the second direction are opposite directions.

33. The neural network training device of claim 32, wherein the image block from which the predetermined region is culled has a plurality of boundaries, each boundary being at a shortest distance from a starting boundary in the first direction to a starting boundary in the second direction from among the facing boundary distances.

34. The neural network training device of any of claims 31-33, wherein the model of the neural network further comprises a third repair model;

in the sample input module (205), the sum of the areas occupied by the first repair data and the second repair data in the image blocks of the eliminated preset area is smaller than the area size of the image blocks of the eliminated preset area;

the neural network training device further includes:

a third model consolidation module (206) that inputs the first repair data and the second repair data into the third repair model, training the third repair model; the third repair model is used for obtaining third repair data based on the first repair data and the second repair data, and the third repair data is used for repairing image data, which is not repaired by the first repair model and the second repair model, in the defect image block.

35. The neural network training device of claim 34, wherein when the repair data is outside the boundaries of the image blocks in the image frame from which the predetermined region was removed, the repair data is replaced with the corresponding raw image data in the image frame; the repair data includes the first repair data, the second repair data, and third repair data;

36. The neural network training device of claim 34, wherein the first repair model, the second repair model, and the third repair model are trained in YUV format.

37. A neural network structure for image defect repair, comprising:

the device comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring first image data and second image data, the first image data is image data of a preset-size area extracted along a first direction in an image to be repaired of a defect image block, the second image data is image data of a preset-size area extracted along a second direction in the image to be repaired of the defect image block, and the first direction and the second direction are opposite directions;

The first repair model is used for repairing the defect image block along the first direction based on the first image data to obtain first repair data;

the second repair model is used for repairing the defective image block along the second direction based on the second image data to obtain second repair data;

the first repair data and the second repair data are used to repair the defective image block.

38. The neural network structure of claim 37, wherein a sum of areas occupied by the first repair data and the second repair data in the defective image block is smaller than an area size of the defective image block;

the neural network structure further includes:

the third repair model is used for obtaining third repair data based on the first repair data and the second repair data; the third repair data is used for repairing unrepaired image data in the defective image block.

39. A neural network training system for image defect repair, comprising:

an image data acquisition device for acquiring an image sample to be learned, wherein the image sample is a plurality of mutually independent images;

A memory for storing a program;

a processor receiving the image samples to be learned for executing the program to implement the method of any one of claims 25-30.

40. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program stored in the storage medium is for being executed to implement the method according to any of claims 1-10; alternatively, a computer program stored in a storage medium for being executed to implement the method of any of claims 25-35.

41. A chip of an image device having an integrated circuit thereon, wherein the integrated circuit is designed for implementing the method of any of claims 1-10; or for implementing the method of any one of claims 25-35.

42. A server, characterized in that it has stored thereon a computer program for being executed to implement the method according to any of claims 1-10; alternatively, a stored computer program is adapted to be executed to implement the method of any of claims 25-35.

43. A platform server, comprising:

The request receiving module is used for receiving the data request;

a data issuing module for providing a computer program and/or a computer program link to a user in accordance with the data request, the computer program being adapted to be executed to implement the method of any of claims 1-10; alternatively, the computer program is adapted to be executed to implement the method of any of claims 25-35.