CN117671229A - Image correction method, apparatus, computer device, and computer-readable storage medium - Google Patents

Abstract

The application relates to an image correction method, an image correction device, a computer device and a computer readable storage medium. The method comprises the following steps: acquiring an image to be processed; processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the method comprises the steps of identifying the region type of a target region, the region position of the target region and the offset of each edge corner point of the target region; and if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image. By adopting the method, the accuracy of image correction can be improved.

Description

Image correction method, apparatus, computer device, and computer-readable storage medium

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image correction method, an image correction device, a computer device, and a computer readable storage medium.

Background

When a certain shape of an object in an image is identified by image detection, the identified object is deformed to different degrees or is blocked due to different shooting angles, so that the image needs to be corrected.

The image correction method in the conventional technology is to process an image containing a target object through an edge detection algorithm to obtain an edge of the target object, and correct the image by using the edge of the target object. However, the image containing the object is deformed or blocked, so that the obtained edge of the object is incomplete and unclear, and the incomplete and unclear edge of the object is used for correcting the image, so that the problem of inaccurate image correction exists.

Disclosure of Invention

In view of the foregoing, it is desirable to provide an image correction method, apparatus, computer device, and computer-readable storage medium capable of improving image correction accuracy.

In a first aspect, the present application provides a method of image correction. The method comprises the following steps:

acquiring an image to be processed;

processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the method comprises the steps of identifying the region type of a target region, the region position of the target region and the offset of each edge corner point of the target region;

and if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

In one embodiment, the correcting the target area according to the offset of each edge corner point to obtain a corrected target image includes:

acquiring a center point of a preset detection frame corresponding to the region position;

decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner;

and correcting the target area according to the corner coordinates of each edge corner, so as to obtain a corrected target image.

In one embodiment, the decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner includes:

the preset detection frame is rectangular, and products obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame are respectively added with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

In one embodiment, the correcting the target area according to the corner coordinates of each edge corner to obtain a corrected target image includes:

determining the vertex coordinates of the target shape of the target area;

Calculating angular point coordinates of the edge angular points, and transforming the angular point coordinates into a projection transformation matrix of the target shape vertex coordinates;

and performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

In one embodiment, the training the manner of obtaining the target detection model includes:

acquiring a sample data set, wherein each sample data set comprises a sample image, a region type label of a region type of a target region corresponding to each sample image, a region position of the marked target region and a position coordinate of each edge corner point of the marked target region;

processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge;

calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, and determining model loss, wherein the model loss comprises loss between the region type label and the predicted region type, loss between the region position and the predicted region position, and loss between the position coordinates and the predicted position coordinates, and updating the initial target detection model according to the model loss until reaching a training ending condition, so as to obtain the trained target detection model.

In one embodiment, after the acquiring the sample dataset, before training the initial target detection model using the sample dataset, the method further comprises:

carrying out data amplification treatment on the sample image to obtain a sample image after the data amplification treatment;

marking a target area, a position area of the target area and position coordinates of each edge corner point of the target area with the area type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

training an initial target detection model by adopting the sample data set to obtain a trained target detection model, wherein the training comprises the following steps:

and training an initial target detection model by adopting the sample data set and the sample data amplified by the data to obtain the trained target detection model.

In one embodiment, the target detection model includes a feature extraction network, a feature fusion enhancement network, and a prediction branch network connected in sequence, the prediction branch network including a classification branch network, a target region prediction branch network, and an offset prediction branch network, and the target region prediction branch network and the offset prediction branch network sharing part of network parameters;

The feature extraction network is used for extracting image features of the image to be processed;

the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

the classified branch network is used for determining the area type of the identified target area based on the feature map;

the target area prediction branch network is used for obtaining the area position of the target area based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

In a second aspect, the present application also provides an image correction device. The device comprises:

the image acquisition module is used for acquiring an image to be processed;

the image processing module is used for processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, and the image detection result comprises: the method comprises the steps of identifying the region type of a target region, the region position of the target region and the offset of each edge corner point of the target region;

and the image correction module is used for correcting the target area according to the area position and the offset of each edge corner point if the area type of the target area is the target type, so as to obtain a corrected target image.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of any of the methods described above when the processor executes the computer program.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of any of the methods described above.

In a fifth aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the methods described above.

The image correction method, the image correction device, the computer equipment and the computer readable storage medium are used for processing the image to be processed by utilizing the target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the identified region type of the target region, the region position of the target region and the offset of each corner point of the target region, and when the region type of the target region is the target type, the image correction is carried out according to the region position and the offset of each corner point. Compared with the prior art that the image is corrected inaccurately due to unclear edges obtained by processing the image by using an edge detection algorithm, in the embodiment, the image to be processed is processed by using the target detection model, so that the accurate region position of the target region and the offset of each corner point of the target region can be obtained, the image correction is performed according to the region position and the offset of each corner point on the basis, the accurate image correction result can be obtained, and the problem of inaccurate image correction due to unclear edges detected is avoided.

Drawings

FIG. 1 is a schematic diagram of an image of a target object provided in an embodiment of the present application;

fig. 2 is a flowchart of an image correction method provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of an image to be processed according to an embodiment of the present application;

FIG. 4 is a flow chart of a training method for obtaining a target detection model in one embodiment;

FIG. 5 is a schematic illustration of an image of an artificial mark sample provided in an embodiment of the present application;

FIG. 6 is a schematic flow chart of training an initial target detection model after data amplification in one embodiment;

FIG. 7 is a schematic diagram of an initial object detection model provided in an embodiment of the present application;

FIG. 8 is a flow chart illustrating image correction according to the position of the region and the offset of each edge corner in one embodiment;

fig. 9 is a schematic diagram of a positional relationship between a preset detection frame and edge corner points provided in an embodiment of the present application;

FIG. 10 is a schematic flow chart of image correction according to corner coordinates of each edge corner in one embodiment;

FIG. 11 is a schematic diagram of image rectification based on perspective transformation provided in an embodiment of the present application;

FIG. 12 is a block diagram of an image correction device provided in an embodiment of the present application;

FIG. 13 is an internal block diagram of a computer device provided in an embodiment of the present application;

fig. 14 is an internal structural diagram of another computer device provided in an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

Before describing a specific embodiment of the present application, an application scenario of the image correction method proposed in the present application is described. In order to identify a certain shape of an object through an image, the image containing the object is processed, the object in the image is deformed or blocked due to the shooting angle, and the shot image containing the object can be corrected by using the image correction method. The shape of the object may be rectangular, triangular, or other convex polygon, without limitation. When the object is rectangular in shape, the object is, for example, a whiteboard, a blackboard, or a computer display screen. Taking a whiteboard as an example, the photographed image is shown in fig. 1, four points Q1, Q2, Q3 and Q4 in fig. 1 are four corner points of the whiteboard, and it is obvious that the whiteboard region formed by Q1, Q2, Q3 and Q4 in fig. 1 is not rectangular. Therefore, the shape of the object in the image including the object is distorted, and may be as shown in fig. 1 or may be another convex quadrangle, so that the image needs to be corrected. The dashed box in fig. 1 is the circumscribed rectangular box of the whiteboard, containing some background information. When the object is triangular, the object is, for example, a set square or a set iron for teaching; when the object is in the shape of a convex polygon, the object is, for example, a five-pointed star, pentagonal flower pot.

In the present embodiment, an image correction method is provided, which can be applied to a terminal device. The terminal equipment acquires an image to be processed; and processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, and carrying out image correction based on the image detection result.

The image correction method can also be applied to a server, and the server acquires an image to be processed; and processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, and carrying out image correction based on the image detection result. The server may obtain the image to be processed from the terminal device, or may obtain the image to be processed from a database or other servers in other manners, for example.

The image correction method can be applied to a system comprising the terminal equipment and the server and is realized through interaction of the terminal equipment and the server. The target detection model obtained by the pre-training can be obtained by training a terminal device or by training a server.

Fig. 2 is a flowchart of an image correction method provided in an embodiment of the present application, and the method is applied to a terminal device or a server for example, and includes the following steps:

S201, acquiring an image to be processed.

The image to be processed is an image containing a target area, and the image is obtained through shooting by a camera or other equipment capable of shooting the image. An example of an image to be processed is seen in fig. 3, where area a in fig. 3 is a whiteboard, and areas B and C are tiles on a ceiling.

S202, processing an image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the identified region type of the target region, the region position of the target region, and the offset of each edge corner point of the target region.

The target detection model may be a neural network model, and the region position of the target region refers to a circumscribed rectangular frame of the target region.

Taking the target type as a whiteboard as an example, in the example of fig. 3, 3 convex quadrilateral regions, namely, a region, B region and C region, are detected, but only the a region is the whiteboard region finally detected, and the B region and the C region belong to false detection. Therefore, the target detection model obtained through pre-training is used for processing the image to be processed, the target detection model can be used for carrying out semantic recognition, so that the whiteboard region is detected, and compared with the situation that 3 regions, namely an A region, a B region and a C region, cannot be distinguished in image detection in the traditional technology, a large number of false detections can be caused, and the efficiency and the accuracy of detecting the image are higher in the embodiment.

And S203, if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

According to the image correction method provided by the embodiment, the image to be processed is obtained, the image to be processed is processed by utilizing the target detection model obtained through pre-training, the image detection result is obtained, the image detection result comprises the identified region type of the target region, the region position of the target region and the offset of each corner point of the target region, and when the region type of the target region is the target type, the image correction is carried out according to the region position and the offset of each corner point. Compared with the prior art that the image is corrected inaccurately due to unclear edges obtained by processing the image by utilizing an edge detection algorithm, in the embodiment, the image to be processed is processed by utilizing the target detection model, so that the accurate region position of the target region and the offset of each corner point of the target region can be obtained.

The target detection model obtained by the pre-training in S202 may be obtained by training a network model. As shown in FIG. 4, the training to obtain the target detection model in one embodiment may include the following:

s401, acquiring a sample data set, wherein each sample data in the sample data set comprises a sample image, and whether the region type of the target region corresponding to each sample image is a region type label of the target type, the region position of the marked target region and the position coordinates of each edge corner point of the marked target region.

The region type tag of the target type refers to whether the region type of the marked target region in the sample image is the target type.

The sample data set may be obtained by various possible ways, taking the target type as a whiteboard as an example, the sample data set obtained in one example may be a certain number of manually marked sample images, for example, 2 ten thousand sample images, where the sample images may be images containing the whiteboard or images not containing the whiteboard, and the region type label corresponding to the sample image containing the whiteboard is that the sample image contains the whiteboard, and meanwhile, the position coordinates of the position region of the whiteboard region in the sample image and the position coordinates of each corner point of the whiteboard region may be manually marked, and the manually marked way may be as shown in fig. 5, where left in fig. 5 represents the distance between the left boundary of the external rectangular frame and the left boundary of the sample image, top represents the distance between the upper boundary of the external rectangular frame and the upper boundary of the sample image, width and height respectively represent the width and height of the external rectangular frame, and width and height of the external rectangular frame are determined by left, top, height, respectively. The circumscribed rectangular frame of the whiteboard in each sample image is marked by a marking mode shown in fig. 5, and the position coordinates of the four corner points of the whiteboard are marked as (x 1, y1, x2, y2, x3, y3, x4, y 4).

S402, processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted corner point offset of each edge.

S403, calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, determining model loss, wherein the model loss comprises loss between a region type label and a predicted region type, loss between a region position and a predicted region position and loss between a position coordinate and a predicted position coordinate, updating an initial target detection model according to the model loss until a training ending condition is reached, and obtaining a trained target detection model.

In this embodiment, the sample data set is used to train the initial target detection model, so that accuracy of the target detection model in detecting the image can be improved.

In some embodiments, after the sample data set is acquired, the sample data set may be further subjected to a data amplification process to increase the abundance of the samples in the sample data set before training the initial target detection model with the sample data set.

Referring to fig. 6, taking training of the initial target detection model after data amplification as an example, after the sample data set, the method of obtaining the target detection model in step S402 includes the following steps:

S601, performing data amplification processing on the sample image to obtain a sample image after the data amplification processing.

The sample image may be subjected to the data amplification process in a variety of possible ways, for example, the sample image may be rotated, flipped or cropped, or a combination of at least two of the rotation, flipping or cropping may be performed to further expand the sample data set.

S602, marking the position coordinates of the target region, the position region of the target region and each edge corner point of the target region with the region type being the target type in the sample image after the data amplification processing, and obtaining sample data after the data amplification.

In this embodiment, the labeling method for the sample image in S401 may be referred to as the method for labeling the target region, the position region of the target region, and the position coordinates of each edge corner point of the target region having the region type of the target region in the sample image after the labeling data amplification processing in S602.

S603, training the initial target detection model by using the sample data set and the sample data after data amplification to obtain a trained target detection model.

In this embodiment, the initial target detection model is trained by using the sample data set and the sample data after data amplification, so that the sample types during training are richer, and the accuracy of the target detection model after training is higher.

As shown in fig. 7, the structure of the target detection model in one embodiment includes a feature extraction network, a feature fusion enhancement network, and a prediction branch network that are sequentially connected, the prediction branch network includes a classification branch network, a target area prediction branch network, and an offset prediction branch network, and the target area prediction branch network and the offset prediction branch network share part of network parameters.

The feature extraction network, which may also be referred to as a backbone network, is used for extracting image features of an image to be processed;

the feature fusion enhancement network can also be called a feature pyramid and is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

a classification branch network for determining a region type of the identified target region based on the feature map;

a target region prediction branch network for obtaining a region position of a target region based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

Taking the network structure shown in fig. 7 as an example, in a specific training process, a sample image is input during each training, and the feature extraction network performs feature extraction on the sample image, where feature extraction may be performed by a downsampling manner, so as to obtain image features corresponding to images with different resolutions of the sample image. And then, carrying out feature fusion and enhancement on the features of each image through a feature fusion enhancement network to obtain a plurality of feature images, and finally, accessing each obtained feature image into a prediction branch network to predict, and outputting three branches in total. The classification branch is used for distinguishing whether the background or the whiteboard and outputting a result of whether the type area of the target area is the target type or not; the target region prediction branch network is used for predicting an circumscribed rectangular frame of the target region and outputting the region position of the target region (namely the position of the circumscribed rectangular frame); the offset prediction branch is used for predicting the offset of each edge corner point of the target area and outputting the offset of each edge corner point of the target area. The target area prediction branch and the offset prediction branch are regression tasks and share part of parameters. Calculating the regional position loss of the regional position output by the target region prediction branch and the regional position loss of the position region corresponding to the marked sample image, predicting the position of each edge corner point according to the offset of each edge corner point of the target region, calculating the corner point offset loss of the position coordinates of each predicted edge corner point and each marked edge corner point, updating the model by combining the calculated regional position loss and corner point offset loss, iterating on the network structure for a plurality of times by using a sample data set until reaching the training end condition, and if the mapping result of the sample image after passing through the network structure approaches the manual marking result, using the model after the last updating as a trained target detection model. By using the trained target detection model, semantic information can be learned, and false detection of the shape analogue can be effectively inhibited.

Referring to fig. 8, fig. 8 is a schematic flow chart of image correction according to the offset of each edge corner in an embodiment, which relates to an implementation manner of image correction according to the offset of each edge corner in this embodiment. On the basis of the above embodiment, S203 described above includes:

s801, a center point of a preset detection frame corresponding to the position of the area is acquired.

The preset detection frame is a detection window used in the detection process of the target detection model, and can be set according to the resolution of an image input into the target detection model. Referring to fig. 9, the identified region positions are rectangular frames formed by B1, B2, B3, and B4, the dashed frame is a preset detection frame corresponding to the region position, and (cx, cy) is the position coordinate of the center point of the preset detection frame. The positions of the edge corner points to be determined are A1, A2, A3 and A4 respectively.

S802, decoding offset of each edge corner according to the center point to obtain corner coordinates of each edge corner.

The offset of each edge corner point refers to the offset of each edge corner point relative to the center point of a preset detection frame.

In this embodiment, the decoding manner of the offset of each edge corner point according to the center point may be that a product obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame is added to the position coordinate of the center point, or may be that a product obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame is added to the position coordinate of the center point and then multiplied by the preset error coefficient.

In some embodiments, decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner includes:

and (3) multiplying the offset of each edge corner point by the width of the preset detection frame or the product obtained by the height of the preset detection frame, and respectively adding the product with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

In the example as in fig. 9, the center point position coordinates are expressed as

center＝numpy.array([cx,cy])；

The width w and the height h of the preset detection frame are expressed as

size＝numpy.array([w,h])；

The offset of each edge corner point is expressed as

offset＝numpy.array([dx5,dy5,dx6,dy6,dx7,dy7,dx8,dy8])，

Wherein dx5, dx6, dx7, dx8 represent the offset of each edge corner in the x direction, and dy5, dy6, dy7, dy8 represent the offset of each edge corner in the y direction;

corner coordinates of four edge corners are respectively A1 (x 5, y 5), A2 (x 6, y 6), A3 (x 7, y 7) and A4 (x 8, y 8) according to the following corner formula:

corner＝offset.reshape(N,2)*size+center，

here, reshape refers to a one-dimensional vector with an original length of n×2, which is converted into a two-dimensional matrix of (N, 2), where N represents the number of edge points, so taking n=4, offset refers to the offset of each edge point, and center refers to the center point coordinates.

Specifically, the calculation process for obtaining the A1 coordinate is exemplified as follows: x5=dx5 x w+cx, y5=dy5 x h+cy.

And S803, correcting the target area according to the corner coordinates of each edge corner, and obtaining a corrected target image.

Referring to fig. 10, fig. 10 is a schematic flow chart of image correction according to the corner coordinates of each edge corner in an embodiment, which relates to an implementation manner of image correction according to the corner coordinates of each edge corner in this embodiment. On the basis of the above embodiment, S803 described above includes:

s1001, determining the vertex coordinates of the target shape of the target region.

The target shape of the target area refers to the target shape of the target area after the preset image correction. Referring to fig. 11, fig. 11 is a schematic diagram of image correction based on perspective transformation provided in the embodiment of the present application, in fig. 11-1, the EFGH region is an original target region, E, F, G, H is the coordinates of the corner points of the four corners of the original target region, which can be represented by the following formula (1), in fig. 11-2, the E 'F' G 'H' region is the target shape of the target region, and the E ', F', G ', H' are the coordinates of the four vertices (i.e., the corner points) of the target shape of the target region, which can be represented by the following formula (2).

Wherein the values of z5, z6, z7, z8, z5', z6', z7', z8' are all 0.

S1002, calculating a projection transformation matrix for transforming the corner coordinates of each edge corner point to the vertex coordinates of the target shape.

In this embodiment, the projective transformation matrix is a preset order square matrix, which may be represented as a 3×3 rotation matrix. For example, the projective transformation matrix may be calculated by the following formula (3):

wherein the projective transformation matrix is

S1003, performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

Accordingly, based on the example shown in fig. 10, in a specific example, the manner of performing image correction according to the corner coordinates of each edge corner may be specifically as follows.

First, the vertex coordinates of the target shape of the target area are determined, and, taking the target shape as a rectangular area as an example, the vertex coordinates include four vertex coordinates, which can be expressed as:

dst＝np.float32([[0,0],[400,0],[400,600],[0,600]])。

then, the corner coordinates src of each edge corner are calculated, transformed into a projective transformation matrix of the target shape vertex coordinates dst, and the projective transformation matrix M is calculated, which can be expressed as:

M＝cv.getPerspectiveTransform(src,dst)。

and then, performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain the target region after image correction. Wherein, the corresponding perspective transformation function can be determined based on the projection transformation matrix, and the perspective transformation function is utilized to perform perspective transformation on the region position of the target region. The perspective transformation function in one example may be expressed as processed=cv.warp Perselected (img, M, (400, 600)), where img is an image containing an original target region, and processed is a rectangular image of the original target region surrounded by four edge corner points, which is scaled (400, 600) after projective transformation, i.e., a rectified target region.

In this embodiment, image correction is performed according to the corner coordinates of each edge corner, so that the method for correcting the image by determining the corner coordinates of each edge corner of the object in the application can locate the object more accurately, and also avoids interference of background information contained in an external rectangular frame, so that accuracy of image correction is improved, compared with correction inaccuracy caused by unclear edges when correcting the image by using the edges of the object in the conventional technology.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides an image correction device for realizing the above-mentioned image correction method. The implementation of the solution provided by the device is similar to that described in the above method, so the specific limitation of one or more embodiments of the image correction device provided below may be referred to above for the limitation of the image correction method, which is not repeated here.

Referring to fig. 12, fig. 12 is a block diagram of an image correction device according to an embodiment of the present application, where the device 1200 includes: an image acquisition module 1201, an image processing module 1202, and an image correction module 1203, wherein:

an image acquisition module 1201, configured to acquire an image to be processed;

the image processing module 1202 is configured to process an image to be processed through a target detection model obtained through pre-training, and obtain an image detection result, where the image detection result includes: the identified region type of the target region, the region position of the target region and the offset of each edge corner point of the target region;

the image correction module 1203 is configured to correct, if the region type of the target region is a target type, the target region according to the region position and the offset of each edge corner point, to obtain a corrected target image.

According to the image correction device provided by the embodiment, the image to be processed is acquired through the image acquisition module, the image to be processed is processed through the image processing module by utilizing the target detection model obtained through pre-training, the image detection result is obtained, the image detection result comprises the identified region type of the target region, the region position of the target region and the offset of each corner point of the target region, and when the region type of the target region is the target type, the image correction is carried out through the image correction module according to the region position and the offset of each corner point. Compared with the prior art that the image is corrected inaccurately due to unclear edges obtained by processing the image by utilizing an edge detection algorithm, in the embodiment, the image to be processed is processed by utilizing the target detection model, so that the accurate region position of the target region and the offset of each corner point of the target region can be obtained.

Optionally, the image correction module 1203 includes:

the center point acquisition unit is used for acquiring the center point of the preset detection frame corresponding to the region position;

the decoding unit is used for decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner;

and the image correction unit is used for correcting the target area according to the corner coordinates of each edge corner, and obtaining a corrected target image.

Optionally, the decoding unit includes:

the angular point coordinate obtaining subunit is configured to obtain angular point coordinates of each edge angular point by adding a product obtained by multiplying an offset of each edge angular point by a width of the preset detection frame or a height of the preset detection frame to position coordinates of a center point.

Optionally, the image correction unit includes:

a vertex coordinate determination subunit, configured to determine a vertex coordinate of a target shape of the target area;

the coordinate transformation subunit is used for calculating the projection transformation matrix for transforming the corner coordinates of each edge corner into the vertex coordinates of the target shape;

and the perspective transformation subunit is used for carrying out perspective transformation on the region position of the target region based on the projection transformation matrix to obtain the target image after image correction.

Optionally, the apparatus 1200 further includes:

the sample acquisition module is used for acquiring a sample data set, wherein each sample data in the sample data set comprises a sample image, and an area type label of a type area of a target area corresponding to each sample image, an area position of a marked target area and position coordinates of each edge corner point of the marked target area;

the model training module is used for processing the sample image through the initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge; calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, determining model loss, wherein the model loss comprises loss between a region type label and a predicted region type, loss between a region position and a predicted region position and loss between a position coordinate and a predicted position coordinate, updating an initial target detection model according to the model loss until reaching a training ending condition, and obtaining a trained target detection model.

Optionally, the apparatus 1200 further comprises a data amplification module and an amplified data labeling module, wherein:

the data amplification module is used for carrying out data amplification processing on the sample image to obtain a sample image after the data amplification processing;

The amplified data marking module is used for marking a target area, a position area of the target area and position coordinates of each edge corner point of the target area with the area type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

the model training module is also used for training the initial target detection model by adopting the sample data set and the sample data after data amplification to obtain a trained target detection model.

Optionally, the target detection model includes a feature extraction network, a feature fusion enhancement network and a prediction branch network which are sequentially connected, the prediction branch network includes a classification branch network, a target area prediction branch network and an offset prediction branch network, and the target area prediction branch network and the offset prediction branch network share part of network parameters; the feature extraction network is used for extracting image features of the image to be processed; the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map; a classification branch network for determining a region type of the identified target region based on the feature map; a target region prediction branch network for obtaining a region position of a target region based on the feature map; and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

The various modules in the image correction device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 13. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used for storing images to be processed, target detection models and sample image data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an image rectification method.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 14. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement an image rectification method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structures shown in fig. 13 and 14 are block diagrams of only portions of structures associated with the present application and are not intended to limit the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, where the memory stores a computer program, and the processor implements the steps of the image correction method provided in the above embodiment when the computer program is executed.

Acquiring an image to be processed;

processing an image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the identified region type of the target region, the region position of the target region and the offset of each edge corner point of the target region;

and if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

In one embodiment, the processor when executing the computer program further performs the steps of:

Acquiring a center point of a preset detection frame corresponding to the position of the region;

decoding the offset of each edge corner point according to the center point to obtain the corner point coordinates of each edge corner point;

and correcting the target area according to the corner coordinates of each edge corner, and obtaining a corrected target image.

In one embodiment, the processor when executing the computer program further performs the steps of:

the preset detection frame is rectangular, and the product obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame is respectively added with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

In one embodiment, the processor when executing the computer program further performs the steps of:

determining the vertex coordinates of a target shape of a target area;

calculating angular point coordinates of each edge angular point, and transforming the angular point coordinates into a projection transformation matrix of the target shape vertex coordinates;

and performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

In one embodiment, the processor when executing the computer program further performs the steps of:

acquiring a sample data set, wherein each sample data in the sample data set comprises a sample image, and a region type label of a region type of a target region corresponding to each sample image, a region position of a marked target region and position coordinates of each edge corner point of the marked target region;

Processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge;

calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, determining model loss, wherein the model loss comprises loss between a region type label and a predicted region type, loss between a region position and a predicted region position and loss between a position coordinate and a predicted position coordinate, updating an initial target detection model according to the model loss until reaching a training ending condition, and obtaining the trained target detection model.

In one embodiment, the processor when executing the computer program further performs the steps of:

carrying out data amplification treatment on the sample image to obtain a sample image after the data amplification treatment;

marking a target region, a position region of the target region and position coordinates of each edge corner point of the target region with the region type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

training the initial target detection model by adopting a sample data set to obtain a trained target detection model, wherein the training comprises the following steps:

And training the initial target detection model by adopting the sample data set and the sample data after data amplification to obtain a trained target detection model.

In one embodiment, the processor when executing the computer program further performs the steps of:

the target detection model comprises a feature extraction network, a feature fusion enhancement network and a prediction branch network which are sequentially connected, the prediction branch network comprises a classification branch network, a target area prediction branch network and an offset prediction branch network, and the target area prediction branch network and the offset prediction branch network share part of network parameters;

the feature extraction network is used for extracting image features of the image to be processed;

the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

a classification branch network for determining a region type of the identified target region based on the feature map;

a target region prediction branch network for obtaining a region position of a target region based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

The implementation principle and technical effects of the above embodiment are similar to those of the above method embodiment, and are not repeated here.

In one embodiment, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image correction method provided by the above embodiments:

acquiring an image to be processed;

processing an image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the identified region type of the target region, the region position of the target region and the offset of each edge corner point of the target region;

and if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

In one embodiment, the computer program when executed by the processor further performs the steps of:

acquiring a center point of a preset detection frame corresponding to the position of the region;

decoding the offset of each edge corner point according to the center point to obtain the corner point coordinates of each edge corner point;

and correcting the target area according to the corner coordinates of each edge corner, and obtaining a corrected target image.

In one embodiment, the computer program when executed by the processor further performs the steps of:

The preset detection frame is rectangular, and the product obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame is respectively added with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

In one embodiment, the computer program when executed by the processor further performs the steps of:

determining the vertex coordinates of a target shape of a target area;

calculating angular point coordinates of each edge angular point, and transforming the angular point coordinates into a projection transformation matrix of the target shape vertex coordinates;

and performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

In one embodiment, the computer program when executed by the processor further performs the steps of:

acquiring a sample data set, wherein each sample data in the sample data set comprises a sample image, and a region type label of a region type of a target region corresponding to each sample image, a region position of a marked target region and position coordinates of each edge corner point of the marked target region;

processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge;

Calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, determining model loss, wherein the model loss comprises loss between a region type label and a predicted region type, loss between a region position and a predicted region position and loss between a position coordinate and a predicted position coordinate, updating an initial target detection model according to the model loss until reaching a training ending condition, and obtaining the trained target detection model.

In one embodiment, the computer program when executed by the processor further performs the steps of:

carrying out data amplification treatment on the sample image to obtain a sample image after the data amplification treatment;

marking a target region, a position region of the target region and position coordinates of each edge corner point of the target region with the region type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

training the initial target detection model by adopting a sample data set to obtain a trained target detection model, wherein the training comprises the following steps:

and training the initial target detection model by adopting the sample data set and the sample data after data amplification to obtain a trained target detection model.

In one embodiment, the computer program when executed by the processor further performs the steps of:

the target detection model comprises a feature extraction network, a feature fusion enhancement network and a prediction branch network which are sequentially connected, the prediction branch network comprises a classification branch network, a target area prediction branch network and an offset prediction branch network, and the target area prediction branch network and the offset prediction branch network share part of network parameters;

the feature extraction network is used for extracting image features of the image to be processed;

the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

a classification branch network for determining a region type of the identified target region based on the feature map;

a target region prediction branch network for obtaining a region position of a target region based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

The implementation principle and technical effects of the above embodiment are similar to those of the above method embodiment, and are not repeated here.

In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, performs the steps of:

Acquiring an image to be processed;

processing an image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the identified region type of the target region, the region position of the target region and the offset of each edge corner point of the target region;

and if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

In one embodiment, the computer program when executed by the processor further performs the steps of:

acquiring a center point of a preset detection frame corresponding to the position of the region;

decoding the offset of each edge corner point according to the center point to obtain the corner point coordinates of each edge corner point;

and correcting the target area according to the corner coordinates of each edge corner, and obtaining a corrected target image.

In one embodiment, the computer program when executed by the processor further performs the steps of:

the preset detection frame is rectangular, and the product obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame is respectively added with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

In one embodiment, the computer program when executed by the processor further performs the steps of:

determining the vertex coordinates of a target shape of a target area;

calculating angular point coordinates of each edge angular point, and transforming the angular point coordinates into a projection transformation matrix of the target shape vertex coordinates;

and performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

In one embodiment, the computer program when executed by the processor further performs the steps of:

acquiring a sample data set, wherein each sample data in the sample data set comprises a sample image, and a region type label of a region type of a target region corresponding to each sample image, a region position of a marked target region and position coordinates of each edge corner point of the marked target region;

processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge;

calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, determining model loss, wherein the model loss comprises loss between a region type label and a predicted region type, loss between a region position and a predicted region position and loss between a position coordinate and a predicted position coordinate, updating an initial target detection model according to the model loss until reaching a training ending condition, and obtaining the trained target detection model.

In one embodiment, the computer program when executed by the processor further performs the steps of:

carrying out data amplification treatment on the sample image to obtain a sample image after the data amplification treatment;

marking a target region, a position region of the target region and position coordinates of each edge corner point of the target region with the region type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

training the initial target detection model by adopting a sample data set to obtain a trained target detection model, wherein the training comprises the following steps:

and training the initial target detection model by adopting the sample data set and the sample data after data amplification to obtain a trained target detection model.

In one embodiment, the computer program when executed by the processor further performs the steps of:

the target detection model comprises a feature extraction network, a feature fusion enhancement network and a prediction branch network which are sequentially connected, the prediction branch network comprises a classification branch network, a target area prediction branch network and an offset prediction branch network, and the target area prediction branch network and the offset prediction branch network share part of network parameters;

The feature extraction network is used for extracting image features of the image to be processed;

the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

a classification branch network for determining a region type of the identified target region based on the feature map;

a target region prediction branch network for obtaining a region position of a target region based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

The implementation principle and technical effects of the above embodiment are similar to those of the above method embodiment, and are not repeated here.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims (10)

1. A method of image correction, the method comprising:

acquiring an image to be processed;

processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, wherein the image detection result comprises the following steps: the method comprises the steps of identifying the region type of a target region, the region position of the target region and the offset of each edge corner point of the target region;

And if the region type of the target region is the target type, correcting the target region according to the region position and the offset of each edge corner point to obtain a corrected target image.

2. The method according to claim 1, wherein the correcting the target area according to the offset of each edge corner point to obtain the corrected target image includes:

acquiring a center point of a preset detection frame corresponding to the region position;

decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner;

and correcting the target area according to the corner coordinates of each edge corner, so as to obtain a corrected target image.

3. The method according to claim 2, wherein decoding the offset of each edge corner according to the center point to obtain the corner coordinates of each edge corner includes:

the preset detection frame is rectangular, and products obtained by multiplying the offset of each edge corner point by the width of the preset detection frame or the height of the preset detection frame are respectively added with the position coordinates of the center point to obtain the corner point coordinates of each edge corner point.

4. The method according to claim 2, wherein said correcting the target area according to the corner coordinates of each of the edge corners to obtain a corrected target image includes:

determining the vertex coordinates of the target shape of the target area;

calculating angular point coordinates of the edge angular points, and transforming the angular point coordinates into a projection transformation matrix of the target shape vertex coordinates;

and performing perspective transformation on the region position of the target region based on the projection transformation matrix to obtain a target image after image correction.

5. The method of claim 1, wherein training the manner in which the target detection model is obtained comprises:

acquiring a sample data set, wherein each sample data set comprises a sample image, a region type label of a region type of a target region corresponding to each sample image, a region position of the marked target region and a position coordinate of each edge corner point of the marked target region;

processing the sample image through an initial target detection model to obtain a predicted region type, a predicted region position and predicted angular point offset of each edge;

Calculating predicted position coordinates of each edge corner according to the predicted offset of each edge corner, and determining model loss, wherein the model loss comprises loss between the region type label and the predicted region type, loss between the region position and the predicted region position, and loss between the position coordinates and the predicted position coordinates, and updating the initial target detection model according to the model loss until reaching a training ending condition, so as to obtain the trained target detection model.

6. The method of claim 5, wherein after the acquiring the sample dataset and before training the initial target detection model using the sample dataset, further comprises:

carrying out data amplification treatment on the sample image to obtain a sample image after the data amplification treatment;

marking a target area, a position area of the target area and position coordinates of each edge corner point of the target area with the area type being the target type in the sample image after the data amplification processing to obtain sample data after the data amplification;

training an initial target detection model by adopting the sample data set to obtain a trained target detection model, wherein the training comprises the following steps:

And training an initial target detection model by adopting the sample data set and the sample data amplified by the data to obtain the trained target detection model.

7. The method of claim 1, wherein the target detection model comprises a feature extraction network, a feature fusion enhancement network, and a predictive branch network connected in sequence, the predictive branch network comprising a classification branch network, a target area predictive branch network, and an offset predictive branch network, and wherein the target area predictive branch network and the offset predictive branch network share a portion of network parameters;

the feature extraction network is used for extracting image features of the image to be processed;

the feature fusion enhancement network is used for carrying out feature fusion and enhancement processing on the image features to obtain a feature map;

the classified branch network is used for determining the area type of the identified target area based on the feature map;

the target area prediction branch network is used for obtaining the area position of the target area based on the feature map;

and the offset prediction branch network is used for obtaining the offset of each edge corner point of the target area based on the feature map.

8. An image correction device, the device comprising:

the image acquisition module is used for acquiring an image to be processed;

the image processing module is used for processing the image to be processed through a target detection model obtained through pre-training to obtain an image detection result, and the image detection result comprises: the method comprises the steps of identifying the region type of a target region, the region position of the target region and the offset of each edge corner point of the target region;

and the image correction module is used for correcting the target area according to the area position and the offset of each edge corner point if the area type of the target area is the target type, so as to obtain a corrected target image.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 7 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 7.