WO2023130966A1

WO2023130966A1 - Image processing method, image processing apparatus, electronic device and storage medium

Info

Publication number: WO2023130966A1
Application number: PCT/CN2022/140852
Authority: WO
Inventors: 徐青松; 李青
Original assignee: 杭州睿胜软件有限公司
Priority date: 2022-01-10
Filing date: 2022-12-22
Publication date: 2023-07-13
Also published as: CN114387165A

Abstract

An image processing method, an image processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. The image processing method comprises: acquiring an original image; processing the original image to obtain a preprocessed image, the preprocessed image comprising at least two first lines, and the at least two first lines being sequentially arranged in parallel along the same direction; processing the preprocessed image by means of a distortion processing model to obtain an intermediate image, the intermediate image comprises at least two second lines, the at least two second lines being sequentially arranged in parallel along the same direction, and the at least two second lines corresponding one-to-one with the at least two first lines; and re-mapping the original image on the basis of a mapping relationship between the preprocessed image and the intermediate image, to obtain an output image.

Description

Image processing method, image processing device, electronic device, storage medium

technical field

Embodiments of the present disclosure relate to an image processing method, an image processing apparatus, electronic equipment, and a non-transitory computer-readable storage medium.

Background technique

With the development of digital technology, objects can be scanned or photographed to be converted into electronic images, which are easy to store and transmit on the Internet. In addition, electronic images may be identified using image recognition technology to obtain information recorded in the electronic images. However, in the process of scanning or photographing an object to obtain an electronic image, it is unavoidable that the content in the obtained electronic image will be tilted, distorted or deformed, which will affect the analysis of the electronic image. Such processing has adverse effects, for example, making the recognition results inaccurate, etc., and also affects the user's viewing experience.

Contents of the invention

At least one embodiment of the present disclosure provides an image processing method, including: acquiring an original image; processing the original image to obtain a pre-processed image, wherein the pre-processed image includes at least two first lines, the At least two first lines are arranged side by side in sequence along the same direction; the preprocessed image is processed through a distortion processing model to obtain an intermediate image, wherein the intermediate image includes at least two second lines, and the at least two The second lines are arranged side by side in sequence along the same direction, and the at least two second lines correspond to the at least two first lines one by one; based on the mapping relationship between the preprocessed image and the intermediate image, for all The original image is remapped to obtain the output image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, the mapping relationship between the preprocessed image and the intermediate image includes the mapping between the at least two first lines and the at least two second lines relationship and a mapping relationship between the area between the at least two first lines in the preprocessed image and the area between the at least two second lines in the intermediate image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, based on the mapping relationship between the preprocessed image and the intermediate image, the original image is remapped to obtain an output image, including: based on The mapping relationship between the pre-processing image and the intermediate image is to determine the pre-processing mapping information corresponding to the pre-processing image through an interpolation method, wherein the pre-processing mapping information is used to indicate that in the pre-processing image Mapping parameters of at least some of the pixels; based on the pre-processing mapping information, determine the mapping information corresponding to the area corresponding to the original image in the pre-processing image; perform mapping information on the area corresponding to the original image Scaling processing to determine mapping information corresponding to the original image; remapping the original image based on the mapping information corresponding to the original image to obtain the output image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, at least some of the pixels in the pre-processed image include pixels in the region between the at least two first lines in the pre-processed image and Pixels on the at least two first lines.

For example, in the image processing method provided in at least one embodiment of the present disclosure, processing the original image to obtain a pre-processed image includes: performing binarization processing on the original image to obtain an input image; Perform scaling processing on the input image to obtain a zoomed image; perform filling processing on the zoomed image to obtain a filled image; perform region division on the filled image to obtain the preprocessed image .

For example, in the image processing method provided in at least one embodiment of the present disclosure, the scaled image includes a first scaled image side and a second scaled image side opposite to each other, and the pre-processed image includes a first pre-scaled image side opposite to each other. A processed image side and a second pre-processed image side, the first pre-processed image side corresponds to the first scaled image side, the second pre-processed image side corresponds to the second scaled image side, and the at least Two first lines are arranged between the first pre-processed image side and the second pre-processed image side along the direction from the first pre-processed image side to the second pre-processed image side, for Filling the zoomed image to obtain the filled image includes: filling a first padding area on the side of the first zoomed image away from the side of the second zoomed image and filling the first padding area on the side of the second zoomed image The side of the side of the second scaled image away from the side of the first scaled image is filled with the second filling area to obtain the filled image, wherein the two opposite sides of the first filling area are the first The side of the scaled image and the side of the first pre-processed image, and the two sides opposite to each other of the second filled area are the side of the second scaled image and the side of the second pre-processed image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, the size of the first filled area is the same as that of the second filled area.

For example, in the image processing method provided in at least one embodiment of the present disclosure, processing the original image to obtain a pre-processed image includes: performing binarization processing on the original image to obtain an input image; performing filling processing on the input image to obtain a filled image; performing scaling processing on the filled image to obtain a scaled image; performing region division on the scaled image to obtain the preprocessed image .

For example, in the image processing method provided in at least one embodiment of the present disclosure, the at least two first lines are at least two bisector lines that bisect the preprocessed image along the same direction.

For example, in the image processing method provided in at least one embodiment of the present disclosure, the warping processing model is a model based on a neural network.

For example, in the image processing method provided by at least one embodiment of the present disclosure, image content in the original image is distorted.

For example, the image processing method provided by at least one embodiment of the present disclosure further includes: training the warp processing model, wherein training the warp processing model includes: generating a training image, wherein the training image includes at least two training lines, The at least two training lines are arranged side by side in sequence along the same direction; based on the training image, a target image corresponding to the training image is generated, wherein the target image includes at least two target training lines, and the at least two The target training lines are arranged side by side in sequence along the same direction, and the at least two target training lines correspond to the at least two training lines; based on the training image and the target image, the distortion processing model to be trained Perform training to obtain the trained warp processing model.

For example, in the image processing method provided in at least one embodiment of the present disclosure, based on the training image and the target image, the warping model to be trained is trained to obtain the trained warping model, It includes: processing the training image through the warp processing model to be trained to obtain an output training image, wherein the output training image includes at least two output lines, and the at least two output lines are sequentially along the same direction Arranged side by side, and the at least two output lines correspond to the at least two training lines; based on the output training image and the target image, adjust the parameters of the warping model to be trained; When the loss function corresponding to the warping model to be trained meets the predetermined condition, obtain the trained warping model, and when the loss function corresponding to the warping model to be trained does not meet the predetermined condition, continue to input the The training image and the target image are used to repeatedly perform the above training process.

For example, in the image processing method provided in at least one embodiment of the present disclosure, generating the training image includes: generating an input training image; performing scaling processing on the input training image to obtain a scaled input training image; The scaled input training image is filled to obtain a filled input training image; the filled input training image is distorted to obtain a distorted input training image; the distorted input training image is performing region division to obtain the training image including the at least two training lines.

For example, in the image processing method provided in at least one embodiment of the present disclosure, generating a target image corresponding to the training image based on the training image includes: based on the distortion parameter corresponding to the distortion processing, performing the training image Perform reverse warping to obtain the target image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, generating an input training image includes: acquiring an original training image; performing binarization on the original training image to obtain the input training image.

For example, in the image processing method provided in at least one embodiment of the present disclosure, the at least two training lines are at least two bisector lines that bisect the training image along the same direction.

At least one embodiment of the present disclosure also provides an image processing device, including: an image acquisition module configured to acquire an original image; a first processing module configured to process the original image to obtain a pre-processed image, wherein , the pre-processed image includes at least two first lines, and the at least two first lines are arranged side by side in sequence along the same direction; the second processing module is configured to process the pre-processed image through a distortion processing model , to obtain an intermediate image, wherein the intermediate image includes at least two second lines, the at least two second lines are arranged side by side in sequence along the same direction, and the at least two second lines and the at least two first One-to-one correspondence between lines; the mapping module is configured to remap the original image based on the mapping relationship between the preprocessed image and the intermediate image to obtain an output image.

At least one embodiment of the present disclosure further provides an electronic device, including: a memory storing computer-executable instructions in a non-transitory manner; a processor configured to run the computer-executable instructions, wherein the computer-executable instructions are The processor implements the image processing method according to any embodiment of the present disclosure when running.

At least one embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when executed by a processor, the computer-executable instructions can The image processing method according to any one of the embodiments of the present disclosure is implemented.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present disclosure more clearly, the accompanying drawings of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure .

Fig. 1 is a schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure;

Fig. 2 is a schematic diagram of an original image provided by at least one embodiment of the present disclosure;

Fig. 3 is a schematic diagram of a preprocessed image provided by at least one embodiment of the present disclosure;

Fig. 4A is a schematic diagram of a scaled image provided by at least one embodiment of the present disclosure;

Fig. 4B is a schematic diagram of a filled image provided by at least one embodiment of the present disclosure;

Fig. 5 is a schematic diagram of an intermediate image provided by at least one embodiment of the present disclosure;

Fig. 6 is a schematic diagram of an output image provided by at least one embodiment of the present disclosure;

FIG. 7 is a flowchart of a model training method provided by at least one embodiment of the present disclosure;

Fig. 8A is a schematic diagram of an original training image provided by at least one embodiment of the present disclosure;

Fig. 8B is a schematic diagram of a filled training image provided by at least one embodiment of the present disclosure;

Fig. 8C is a schematic diagram of a warped training image provided by at least one embodiment of the present disclosure;

Fig. 8D is a schematic diagram of a training image provided by at least one embodiment of the present disclosure;

Fig. 8E is a schematic diagram of a target image provided by at least one embodiment of the present disclosure;

Fig. 9 is a schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure;

Fig. 10 is a schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure;

Fig. 11 is a schematic diagram of a non-transitory computer-readable storage medium provided by at least one embodiment of the present disclosure;

Fig. 12 is a schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Apparently, the described embodiments are some of the embodiments of the present disclosure, not all of them. Based on the described embodiments of the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative effort fall within the protection scope of the present disclosure.

Unless otherwise defined, the technical terms or scientific terms used in the present disclosure shall have the usual meanings understood by those skilled in the art to which the present disclosure belongs. "First", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Comprising" or "comprising" and similar words mean that the elements or items appearing before the word include the elements or items listed after the word and their equivalents, without excluding other elements or items. Words such as "connected" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "Down", "Left", "Right" and so on are only used to indicate the relative positional relationship. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components.

Currently, neural network models can be used to identify electronic images to obtain information recorded in electronic images. The electronic image can be taken or scanned by the user. In the process of obtaining the electronic image, due to the shooting angle and other reasons, it is unavoidable that the content in the electronic image will be distorted or deformed, so that the neural network model can be recognized. The result is not accurate.

At least one embodiment of the present disclosure provides an image processing method. The image processing method includes: acquiring an original image; processing the original image to obtain a preprocessed image, wherein the preprocessed image includes at least two first lines, and the at least two first lines are arranged side by side in sequence along the same direction; The warping processing model processes the preprocessed image to obtain an intermediate image, wherein the intermediate image includes at least two second lines, the at least two second lines are arranged side by side in sequence along the same direction, and the at least two second lines are connected with the at least two second lines One-to-one correspondence; based on the mapping relationship between the preprocessed image and the intermediate image, the original image is remapped to obtain the output image.

In the image processing method provided by the embodiment of the present disclosure, firstly, the pre-processed image is processed by using the warping processing model, and then according to the mapping relationship between the input and output of the warping processing model, that is, the mapping between the pre-processing image and the intermediate image Relationship, remap the original image to obtain the output image, so as to realize the correction of the original image, effectively solve the problem of image distortion, improve the accuracy of the recognition result based on the output image, improve the efficiency of image recognition, and enhance the image The readability of the image improves the user's experience of viewing the output image.

At least one embodiment of the present disclosure also provides an image processing device, an electronic device, and a non-transitory computer-readable storage medium.

The image processing method provided by the embodiment of the present disclosure can be applied to the image processing device provided by the embodiment of the present disclosure, and the image processing device can be configured on an electronic device. The electronic device may be a personal computer, a mobile terminal, etc., and the mobile terminal may be a hardware device with various operating systems, such as a mobile phone and a tablet computer. That is to say, the execution subject of the image processing method may be a personal computer, a mobile terminal, and the like.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

Fig. 1 is a schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure, and Fig. 2 is a schematic diagram of an original image provided by at least one embodiment of the present disclosure.

As shown in FIG. 1 , the image processing method provided by the embodiment of the present disclosure includes the following steps S10 to S13.

First, in step S10, an original image is acquired.

For example, the original image is an image obtained by photographing or scanning an object, the object includes at least one of various characters, various symbols and various graphics, and the characters may include Chinese (for example, Chinese characters or pinyin), English , Japanese, French, Korean, Latin, numbers, etc. Symbols can include mathematical symbols and punctuation marks, etc. Mathematical symbols include plus sign, minus sign, greater than sign, less than sign, percent sign, etc. Punctuation marks can include periods, commas , question marks, etc., graphics can include straight lines, curves, circles, rectangles, heart shapes, various pictures, etc., as shown in Figure 2, the original image 100 can include Chinese characters, numbers, graphics of houses (for example, Xiaohongjia, schools, etc.), graphics of characters, etc.

For example, the original image can be various types of images, such as business cards, test papers, exercise sets, contracts, invoices, etc., so that the original images can be images of shopping lists, images of restaurant receipts, images of test papers, and exercise sets. images of contracts, images of contracts, etc. For example, characters, symbols and graphics can be obtained by handwriting, printing or machine.

For example, in some embodiments, the image content in the original image is distorted, that is, the object in the original image is deformed, the object in the original image is inconsistent with the actual shape of the object, for example, the object in the object Characters located on the same line produce skew, distortion, distortion, etc. For example, warping may include one or more of translation, rotation, scaling, affine transformation, perspective transformation, cylindrical transformation, and the like. For example, as shown in FIG. 2 , in some embodiments, the original image 100 may be an image obtained by photographing a page of a problem set (for example, a math problem set, etc.), and the text in the original image 100 is distorted, for example, In this page of the exercise set, the lines connecting the centers of the texts in "Integer Tens Plus One-Digit and Corresponding Subtraction" are on the same straight line. However, in the original image 100, "Integer Tens Plus One-digit number and corresponding subtraction" is distorted, and the connecting lines of the centers of each text in "Integer tens plus one-digit number and corresponding subtraction" are not on the same straight line, but on a curve (irregular or regular curve )superior.

For example, the shape of the original image may be various suitable shapes such as a rectangle. The shape and size of the original image can be set by the user according to the actual situation, which is not limited by the embodiments of the present disclosure.

For example, the original image may be an image captured by an image acquisition device (for example, a digital camera or a camera on a mobile phone, etc.), and the original image may be a grayscale image, a black and white image, or a color image. It should be noted that the original image refers to a form that presents an object in a visual manner, such as a picture of the object. For another example, the original image may also be obtained by means of scanning or the like. For example, the original image may be an image directly captured by the image acquiring device, or may be an image obtained after preprocessing the captured image. For example, in order to avoid the impact of the data quality and data imbalance of the images directly collected by the image acquisition device on the subsequent processing, before processing the original image, the image processing method may also include preprocessing the images directly collected by the image acquisition device operation. The preprocessing may include, for example, cropping, gamma (Gamma) correction, or noise reduction filtering on the image directly captured by the image acquisition device. Preprocessing can eliminate irrelevant information or noise information in the original image, so as to facilitate the subsequent processing of the original image.

Fig. 3 is a schematic diagram of a pre-processed image provided by at least one embodiment of the present disclosure. For example, the preprocessed image shown in FIG. 3 is an image obtained by processing the original image shown in FIG. 2 .

As shown in FIG. 1 , in step S11 : the original image is processed to obtain a preprocessed image.

For example, in some embodiments, step S11 includes: performing binarization processing on the original image to obtain an input image; performing scaling processing on the input image to obtain a zoomed image; performing padding processing on the zoomed image to obtain A filled image is obtained; the filled image is divided into regions to obtain a preprocessed image.

For example, in some other embodiments, step S11 includes: performing grayscale processing on the original image to obtain an input image; performing scaling processing on the input image to obtain a zoomed image; performing filling processing on the zoomed image, To obtain a filled image; perform region division on the filled image to obtain a preprocessed image.

For example, performing binarization processing or grayscale processing on the original image can reduce the data processing amount of subsequent processing and improve processing speed. Binarization or grayscale processing is used to remove the interfering pixels in the original image, and only keep the content that needs to be processed, such as characters, graphics or images.

For example, the method of binarization processing may include threshold method, bimodal method, P parameter method, big law method (OTSU method), maximum entropy method, iterative method and the like.

For example, methods of grayscale processing include component method, maximum value method, average value method, and weighted average method.

It should be noted that the order of binarization/grayscale processing, scaling processing and filling processing can be adjusted arbitrarily, and is not limited to the above description. For example, scaling processing can be performed first, then filling processing, and finally Binarization/grayscale processing.

For example, the dimensions of the input image and the original image can be the same. The size of the padded image is larger than the size of the scaled image, and the size of the padded image is equal to the size of the preprocessed image. For example, when the scaling process is a reduction process, the size of the scaled image is smaller than the size of the input image, and when the scaling process is an expansion process, the size of the scaled image is larger than the size of the input image.

In the embodiments of the present disclosure, binarization/grayscale processing can reduce the amount of data processing, thereby improving the processing speed of image processing; scaling processing can unify the size of the image to facilitate model processing; filling processing can prevent distortion after operation The content corresponding to the object in the pre-processed image exceeds the screen area of the pre-processed image, so as to avoid loss of image content and ensure the integrity of the image content.

It should be noted that, in the image processing method provided by the embodiments of the present disclosure, binarization/grayscale processing may not be performed, thereby reducing the processing flow.

The preprocessed image shown in FIG. 3 is the image after grayscale processing. As shown in Figure 3, the preprocessed image 200 includes a first preprocessed image edge PB1, a second preprocessed image edge PB2, a third preprocessed image edge PB3 and a fourth preprocessed image edge PB4, the first preprocessed image edge PB1 The side PB2 and the second pre-processing image are two sides facing each other, and the side PB3 of the third pre-processing image and the side PB4 of the fourth pre-processing image are two sides facing each other. For example, the pre-processed image 200 can be a rectangle. At this time, the first pre-processed image side PB1 and the second pre-processed image side PB2 are parallel to each other and parallel to the X1 direction; the third pre-processed image side PB3 and the fourth pre-processed image The sides PB4 are parallel to each other and to the Y1 direction; the first pre-processing image side PB1 and the third pre-processing image side PB3 are perpendicular to each other. For example, the X1 direction is the width direction of the pre-processed image 200 , and the Y1 direction is the height direction of the pre-processed image 200 .

For example, the pre-processed image includes at least two first lines, the at least two first lines are arranged side by side in sequence along the same direction, and the at least two first lines are located between the side of the first pre-processed image and the side of the second pre-processed image of the pre-processed image. The sides are arranged along a direction from the side of the first pre-processed image to the side of the second pre-processed image.

For example, in some examples, as shown in FIG. 3 , the preprocessed image 200 may include at least two first lines L1, and the at least two first lines L1 are along the same direction (for example, Y1 direction, that is, the height of the preprocessed image 200 direction) are arranged side by side. At least two first lines L1 are parallel to each other and parallel to the X1 direction. At least two first lines L1 are located between the first pre-processed image side PB1 and the second pre-processed image side PB2.

It should be noted that, in the embodiments of the present disclosure, the arrangement of the at least two first lines L1 is not limited to the one shown in FIG. direction, at this time, at least two first lines L1 are parallel to the Y1 direction, and are located between the third pre-processing image side PB3 and the fourth pre-processing image side PB4.

For example, in some embodiments, as shown in FIG. 3 , at least two first lines L1 are at least two bisector lines that equally divide the preprocessed image 200 along the same direction (for example, Y1 direction), that is to say , the distance h3 between any two adjacent first lines L1 is a fixed value.

For example, the number of at least two first lines L1 can be set according to actual conditions. For example, in some examples, as shown in FIG. 3 , the number of at least two first lines L1 can be 23. The image 200 is equally divided into 24 parts along the edge PB3 of the third pre-processed image, so as to obtain 23 first lines L1. For example, if the length of the side PB3 of the third pre-processed image is 768 pixels, the distance h3 between any two adjacent first lines L1 is 32 pixels.

It should be noted that the number of first lines L1 can be less or more, for example, the number of first lines L1 can be within the numerical range of 12-48, for example, 12 or 48, the more the number of first lines L1 , the final output image is more accurate, but the amount of data processing is more.

In FIG. 3 , in order to clearly show the first line L1 , the first line L1 is represented by a thicker line. The width of the first line L1 can be set according to actual conditions, for example, it can be 1-2 pixels.

Fig. 4A is a schematic diagram of a scaled image provided by at least one embodiment of the present disclosure; Fig. 4B is a schematic diagram of a filled image provided by at least one embodiment of the present disclosure. The filled image shown in FIG. 4B is obtained by filling the zoomed image shown in FIG. 4A .

For example, in some embodiments, as shown in FIG. 4A , the scaled image 300 includes a first scaled image side CB1 and a second scaled image side CB2 opposite to each other. The first preprocessed image side corresponds to the first scaled image side CB1, and the second preprocessed image side corresponds to the second scaled image side CB2, that is, in the preprocessed image, the first preprocessed image side corresponds to the first scaled image side The image side CB1 is located on the same side, such as the upper side shown in FIG. 4A , and the second preprocessed image side and the second scaled image side CB2 are located on the same side, such as the lower side shown in FIG. 4A .

In step S11, filling the scaled image to obtain the filled image includes: filling the first filling area on the side of the first scaled image away from the side of the second scaled image and filling the area on the side of the second scaled image The side away from the side of the first scaled image is filled with the second filled area to obtain a filled image.

For example, the filled image includes a scaled image, a first filled area and a second filled area. The preprocessed image includes the filled image and at least one first line. In the pre-processed image, the two opposite sides of the first filled area are the first scaled image side and the first pre-processed image side, and the two opposite sides of the second filled area are the second scaled image side and the second scaled image side. 2. Preprocess image edges.

For example, as shown in FIG. 4B , in some embodiments, the first padded region 310 is padded (e.g., spliced) to the side of the first scaled image side CB1 of the scaled image 300 away from the second scaled image side CB2, The second filling area 320 fills (eg, stitches) to a side of the second scaled image side CB2 of the scaled image 300 away from the first scaled image side CB1 . The filled image 2000 includes a complete area composed of the scaled image 300 , the first filled area 310 and the second filled area 320 .

For example, as shown in FIG. 4B, the filled image 2000 includes a first filled image side FB1 and a second filled image side FB2 opposite to each other. After the filled image 2000 is divided into regions to obtain a pre-processed image, the The first filling image edge FB1 is the first preprocessing image edge, and the second filling image edge FB2 is the second preprocessing image edge.

For example, as shown in FIG. 4A , the scaled image 300 further includes a third scaled image side CB3 and a fourth scaled image side CB4 opposite to each other. For example, the zoomed image 300 may be a rectangle. At this time, the first zoomed image side CB1 and the second zoomed image side CB2 are parallel to each other and parallel to the X2 direction; the third zoomed image side CB3 and the fourth zoomed image side CB4 are parallel to each other parallel to the Y2 direction; the first zoomed image side CB1 and the third zoomed image side CB3 are perpendicular to each other. For example, the X2 direction is the width direction of the zoomed image 300 , and the Y2 direction is the height direction of the zoomed image 300 .

For example, as shown in FIG. 4B , the first filling area 310 may be a rectangle, and the second filling area 320 may also be a rectangle. The length of the side parallel to the Y2 direction of the first padding area 310 can be h1, and the length of the side parallel to the Y2 direction of the second padding area 320 can be h2, as shown in FIGS. 4A and 4B, the first scaled image side The length of CB1 is w1, the length of the side parallel to the X2 direction of the first filling region 310 is w1, and the length of the side parallel to the X2 direction of the second filling region 320 is w1. For example, the size of the first filling area 310 is the same as the size of the second filling area 320 , at this time, h1 is equal to h2 .

For example, in some embodiments, h1 may be 64 pixels. For example, if the size of the image 300 after zooming may be 576 (pixels)*640 (pixels), then the size of the filled image 2000 may be 576 (pixels). *768 (pixels).

For example, the pixel value of each pixel in the first filling area 310 and the second filling area 320 can be set according to actual conditions, for example, both are 0, which is not limited in the present disclosure.

In some embodiments of the present disclosure, the scaling process may be performed first, and then the filling process may be performed. However, the present disclosure is not limited thereto. In other embodiments, the filling process may be performed first, and then the scaling process may be performed. The specific filling parameters corresponding to the filling process (that is, the sizes of the first filling area and the second filling area, etc.) can be set according to the actual situation, which is not limited in the present disclosure.

For example, the filled image may be divided into regions along the height direction of the filled image by using at least two first lines to obtain a preprocessed image.

For example, in some other embodiments, step S11 includes: performing binarization processing on the original image to obtain an input image; performing filling processing on the input image to obtain a filled image; performing scaling processing on the filled image, to obtain a zoomed image; the zoomed image is divided into regions to obtain a preprocessed image; or, step S11 includes: grayscale processing is performed on the original image to obtain an input image; filling processing is performed on the input image to obtain The filled image is obtained; the filled image is scaled to obtain a scaled image; the scaled image is divided into regions to obtain a preprocessed image.

It should be noted that the padding process can be determined according to the distortion direction of the image content in the original image, for example, if the image content in the original image is distorted in the length direction, then in the padding process, in the two sides of the length direction of the image Fill a padding area on each side of the image; if the image content in the original image is distorted in the width direction, in the padding process, fill a padding area on each of the two sides of the width direction of the image; if the original image is The content of the image is distorted in both the length direction and the width direction, then in the padding process, a padding area is filled on each of the two sides in the length direction of the image, and at the same time, a padding area is filled on each of the two sides in the width direction of the image The sides are also filled with a padding area.

Fig. 5 is a schematic diagram of an intermediate image provided by at least one embodiment of the present disclosure. The intermediate image 400 shown in FIG. 5 is an image obtained after processing the pre-processed image shown in FIG. 3 through a warping processing model.

As shown in FIG. 1 , in step S12 , the preprocessed image is processed by the warping processing model to obtain an intermediate image.

For example, the warp processing model may be implemented using machine learning technology (eg, deep learning technology). For example, in some embodiments, the warp processing model may be a model based on a neural network. The distortion processing model can use the pix2pixHD (pixel to pixel HD) model, which uses a multi-level generator (coarse-to-fine generator) and a multi-scale discriminator (multi-scale discriminator) to distort the preprocessed image Processing to generate an intermediate image after warping. The generator of the pix2pixHD model includes a global generator network (global generator network) and a local enhancer network (local enhancer network). The global generator network part adopts the U-Net structure, and the features output by the global generator network part are extracted from the local enhancement network part. The feature fusion of the local enhancement network is used as the input information of the local enhancement network part, and the warped intermediate image is output by the local enhancement network part. For example, the warp processing model can also use other models, such as U-Net model, etc., which is not limited in the present disclosure. The training process for the warping processing model is described later and will not be repeated here.

For example, the intermediate image includes at least two second lines, the at least two second lines are arranged side by side in sequence along the same direction, and the at least two second lines are in one-to-one correspondence with the at least two first lines. For example, in some examples, as shown in FIG. 5, the intermediate image 400 includes at least two second lines L2, and at least two second lines L2 are arranged side by side in sequence along the same direction (for example, the Y3 direction, that is, the height direction of the intermediate image 400). Cloth, the extending direction of the at least two second lines L2 is the X3 direction. The at least two second lines L2 shown in FIG. 5 correspond one-to-one to the at least two first lines L1 shown in FIG. 3 . The second line L2 is a twisted line of the first line L1. As shown in FIG. 5 , each second line L2 is a regular or irregular curve, and the shapes of each second line L2 are different. It should be noted that one or several second lines L2 may also be straight lines, and the present disclosure does not specifically limit the shape and other properties of the second lines L2.

It should be noted that the X1 direction, the X2 direction and the X3 direction are parallel to each other, and the Y1 direction, the Y2 direction and the Y3 direction are also parallel to each other. In some embodiments, the X1 direction, the X2 direction and the X3 direction are all width directions of the image, for example, the width direction of the image is parallel to the horizontal direction. The Y1 direction, the Y2 direction and the Y3 direction are all height directions of the image, for example, the height direction of the image is parallel to the vertical direction.

Fig. 6 is a schematic diagram of an output image provided by at least some embodiments of the present disclosure. The output image shown in FIG. 6 is an image obtained by processing the original image shown in FIG. 2 through the image processing method provided by the embodiment of the present disclosure.

As shown in FIG. 1 , in step S13 , based on the mapping relationship between the preprocessed image and the intermediate image, the original image is remapped to obtain an output image.

For example, the mapping relationship between the preprocessing image and the intermediate image includes the mapping relationship between at least two first lines and at least two second lines and the area between the at least two first lines in the preprocessing image and the intermediate image The mapping relationship between regions between at least two second lines in .

In addition, it should be noted that the region between at least two first lines in the preprocessed image and at least two regions in the intermediate image need to be determined according to the mapping relationship between at least two first lines and at least two second lines. The mapping relationship between the regions between the second lines.

For example, in some embodiments, step S13 may include: determining mapping information corresponding to the original image based on the mapping relationship between the preprocessed image and the intermediate image; remapping the original image based on the mapping information corresponding to the original image, to get the output image.

For example, in step S13, based on the mapping relationship between the preprocessed image and the intermediate image, determining the mapping information corresponding to the original image includes: based on the mapping relationship between the preprocessed image and the intermediate image, determining the processing the preprocessing mapping information corresponding to the image; determining the mapping information corresponding to the area corresponding to the original image in the preprocessing image based on the preprocessing mapping information; performing scaling processing on the mapping information of the area corresponding to the original image to determine the The mapping information corresponding to the image.

For example, the preprocessing mapping information is used to indicate mapping parameters of at least some pixels in the preprocessing image. At least some of the pixels in the pre-processed image include pixels in a region between at least two first lines and pixels on at least two first lines in the pre-processed image. As shown in FIG. 3 , the preprocessed image 200 includes an area A1 and an area A2. The area A1 and the area A2 are not located between the two first lines L1. At least some pixels in the preprocessed image 200 include the pixels in the preprocessed image. All the pixels except area A1 and area A2.

It should be noted that the preprocessing mapping information may also indicate mapping parameters of all pixels in the preprocessing image, which is not limited in the present disclosure.

In this disclosure, the original image is remapped according to the mapping relationship between the input and output of the distortion processing model (that is, based on the mapping relationship between the preprocessed image and the intermediate image), so as to realize the correction of the original image after distortion , to obtain the output image, effectively solve the problem of image distortion, improve the accuracy of the recognition result based on the output image, improve the efficiency of image recognition, enhance the readability of the image, and improve the user's experience of viewing the output image.

For example, the area between any two adjacent second lines in the intermediate image may correspond to the area between corresponding two adjacent first lines in the preprocessed image, and each second line in the intermediate image may correspond to the preprocessed image. The corresponding first line in the image is processed, so that pre-processing mapping information corresponding to the pre-processing image can be determined through an interpolation method based on the mapping relationship between the pre-processing image and the intermediate image. As shown in FIGS. 3 and 5 , the area between any two adjacent first lines L1 (for example, the first line L11 and the first line L12 ) in the preprocessed image 200 is the same as the area between the two adjacent lines in the intermediate image 400 . The areas between the two second lines L2 (for example, the second line L21 and the second line L22) respectively corresponding to the first line L1 are mapped correspondingly to each other, that is to say, the first line L11 and the first line L12 in the preprocessed image 200 The area in between needs to be mapped to the area between the second line L21 and the second line L22 in the intermediate image 400 . The first line L1 in the preprocessed image 200 and the second line L2 corresponding to the first line L1 in the intermediate image 400 are also mapped correspondingly to each other, for example, the first line L11 and the first line L12 in the preprocessed image 200 need to be mapped are the second line L21 and the second line L22 in the intermediate image 400 .

For example, interpolation methods may include methods such as nearest neighbor interpolation, bilinear interpolation, bicubic spline interpolation, bicubic interpolation, and Lanczos interpolation (lanczos), and the disclosure does not limit the interpolation methods.

For example, the mapping information corresponding to the original image may include mapping parameters corresponding to all pixels in the original image, that is, the number of mapping parameters in the mapping information corresponding to the original image may be the same as the number of all pixels in the original image. For example, the mapping parameter corresponding to a pixel may represent the coordinate value of the position to which the pixel is mapped; or, may also represent an offset between the coordinate value of the pixel and the coordinate value of the position to which the pixel is mapped.

It should be noted that the coordinate value of the pixel can represent the coordinate value in the coordinate system corresponding to the original image, and the coordinate origin of the coordinate system corresponding to the original image is a certain pixel point of the original image (for example, the center of the original image corresponds to pixel or the pixel in the upper left corner of the original image), the two coordinate axes of the coordinate system corresponding to the original image are the width and height of the original image respectively. The coordinate value of the position to which the pixel is mapped can represent the coordinate value in the coordinate system corresponding to the output image, and the coordinate origin of the coordinate system corresponding to the output image corresponds to the coordinate origin of the coordinate system corresponding to the original image in the output image The two coordinate axes of the coordinate system corresponding to the output image are the width and height of the output image respectively.

For example, based on the mapping relationship between the preprocessed image and the intermediate image as a reference, the mapping parameters corresponding to each pixel in the original image can be determined, so that the mapping information corresponding to the original image can be obtained. Based on the mapping information corresponding to the original image, the mapped position of each pixel after correcting the image distortion can be determined, thereby realizing the mapping process.

For example, in step S13, remapping the original image based on the mapping information corresponding to the original image to obtain an output image may include: calling a remapping function (ie, a remap function) in opencv based on the mapping information corresponding to the original image Remap the original image to get the output image. As shown in FIG. 6 , in the output image 500, the lines connecting the centers of the characters in "Ten plus one digit and the corresponding subtraction" are on the same straight line, so that the text can be straightened, thereby effectively correcting the original image The distorted state solves the problem of image distortion and deformation, improves the accuracy of recognition results based on the output image, improves the efficiency of image recognition, enhances the readability of the image, and improves the user's experience of viewing the output image.

For example, in some embodiments of the present disclosure, the image processing method further includes: training a warping processing model.

At least one embodiment of the present disclosure further provides a model training method for realizing the above operation of training a warping processing model. Fig. 7 is a flowchart of a model training method provided by at least one embodiment of the present disclosure.

In some embodiments, the model training method may include training a warping processing model. For example, as shown in FIG. 7 , training the warping processing model includes the following steps S20-S22.

Step S20: Generate training images. For example, the training image includes at least two training lines, and the at least two training lines are arranged side by side sequentially along the same direction.

Step S21: Based on the training image, generate a target image corresponding to the training image. For example, the target image includes at least two target training lines, the at least two target training lines are arranged side by side in sequence along the same direction, and the at least two target training lines are in one-to-one correspondence with the at least two training lines.

Step S22: Based on the training image and the target image, train the warping model to be trained to obtain a trained warping model.

For example, in some embodiments, step S20 may include: generating an input training image; performing scaling processing on the input training image to obtain a scaled input training image; performing padding processing on the scaled input training image to obtain a filled the input training image; distorting the filled input training image to obtain a distorted input training image; performing region division on the distorted input training image to obtain a training image including at least two training lines.

For example, in some other embodiments, step S20 may include: generating an input training image; filling the input training image to obtain a filled input training image; performing scaling processing on the filled input training image to obtain a scaled The input training image after the warping process is performed on the scaled input training image to obtain a warped input training image; the warped input training image is divided into regions to obtain a training image including at least two training lines.

It should be noted that, in step S20, the order of the filling process and the scaling process can be set according to actual conditions, which is not limited in the present disclosure. In the following description of the present disclosure, the scaling process is performed first and then the filling process is performed as an example for illustration.

For example, in step S20, generating the input training image may include: acquiring an original training image; performing binarization or grayscale processing on the original training image to obtain the input training image. Perform binarization or grayscale processing on the original training image, so that the interference (noise) in the original training image can be removed, and the amount of data processing in the subsequent training process can also be reduced. It should be noted that the binarization or grayscale processing is not a necessary step, and the original training image can also be directly filled, scaled and divided to obtain the training image.

Fig. 8A is a schematic diagram of an original training image provided by at least one embodiment of the present disclosure.

For example, the original training image may be an image that has not been distorted. As shown in FIG. 8A , in the original training image 810, all texts are not distorted.

Fig. 8B is a schematic diagram of a filled training image provided by at least one embodiment of the present disclosure. The filled training image shown in FIG. 8B may be an image after scaling and filling processing are performed on the original training image shown in FIG. 8A .

For example, the input training image can be scaled and filled to a fixed size, and the uniform size of the image can facilitate the processing of the image by the warping model to be trained. For example, in one embodiment, as shown in FIG. 8B , the input training image may first be scaled to obtain a scaled training image 830, the size of the scaled training image 830 may be 576*640 (pixels), and after scaling The training image 830 includes the image side CTB1 and the image side CTB2 opposite to each other, a training filling area 831 is filled on the side of the image side CTB1 of the zoomed training image 830 away from the image side CTB2, and the zoomed training image 830 The side of the image side CTB2 far away from the image side CTB1 fills a training padding area 832, thereby obtaining the training image 820 after filling, the training image 820 after filling can include the training padding area 831, the training image 830 after scaling and training Fill area 832 constitutes the area. The padding process can prevent the contents from exceeding the screen after the distortion operation. For example, the size of the training padding area 831 and the size of the training padding area 832 can be the same, and the size of the training padding area 831 can be 576*64 (pixels), so that the training image after padding The size of the 820 can be 576*768 (pixels).

It should be noted that, for a detailed description of the filling process and the scaling process, reference may be made to the description of the filling process and the scaling process in the above embodiment of the image processing method, and repeated descriptions will not be repeated.

Fig. 8C is a schematic diagram of a warped training image provided by at least one embodiment of the present disclosure. The warped training image shown in Fig. 8C may be an image after warping the filled training image shown in Fig. 8B .

For example, the way of warping processing is not limited. In some embodiments, opencv can be used to implement warping processing. For example, first, a set of offsets is randomly generated, and then Gaussian filtering is performed on the offsets to make the offsets smooth and continuous , use the offset after Gaussian filtering to generate a warped parameter matrix (for example, map), and call the remap function in opencv to remap the filled image to achieve warping processing, thereby obtaining a warped training image.

Fig. 8D is a schematic diagram of a training image provided by at least one embodiment of the present disclosure. The training image 850 shown in FIG. 8D may be an image after processing the warped image 840 shown in FIG. 8C .

For example, in some embodiments, as shown in FIG. 8D , the training image 850 may include at least two training lines TL1, and the at least two training lines TL1 are along the same direction as the training image 850 (for example, the height direction Y4 of the training image 850). ) at least two bisectors for bisecting. As shown in FIG. 8D , the training lines TL1 may be parallel to each other and extend along the width direction X4 of the training image 850 . For example, as shown in FIG. 8C and FIG. 8D , the warped image 840 can be equally divided, and an equal line can be drawn to obtain a training image 850 . For example, the warped image 840 may be equally divided along its height direction. The quantity of at least two training lines TL1 can be set according to actual conditions, as shown in Figure 8D, the quantity of at least two training lines TL1 can be 23, however, the quantity of at least two training lines TL1 can be less or more, for example , 12 to 48, etc., the more the number of at least two training lines TL1 is, the more accurate the warping processing model obtained after training is, but the amount of data processing is more.

Fig. 8E is a schematic diagram of a target image provided by at least one embodiment of the present disclosure. The target image 860 shown in FIG. 8E may be an image obtained by reverse warping the training image 850 shown in FIG. 8D .

For example, in some embodiments, step S21 includes: performing reverse warping processing on the training image based on the warping parameters corresponding to the warping process to obtain the target image.

For example, the purpose of the reverse warping process is to restore the part of the image content in the training image 850 except the training line TL1 to the state before the warping process (ie, the padded image 820 shown in FIG. 8B ). As shown in Figure 8E, the target image 860 includes at least two target training lines TL2, at least two target training lines TL2 are arranged in sequence along the height direction of the target image 860, and at least two target training lines TL2 extend along the width direction of the target image 860 . The at least two target training lines TL2 in the target image 860 are in one-to-one correspondence with the at least two training lines TL1 in the training image 850 shown in FIG. To distort the resulting lines.

For example, in some embodiments, step S21 may include: processing the training image through the warping model to be trained to obtain an output training image; based on the output training image and the target image, adjusting the parameters of the warping model to be trained ; When the loss function corresponding to the distortion processing model to be trained meets the predetermined condition, the trained distortion processing model is obtained, and when the loss function corresponding to the distortion processing model to be trained does not meet the predetermined condition, continue to input the training image and the target image to Repeat the above training process.

For example, the output training image includes at least two output lines, the at least two output lines are arranged side by side in sequence along the same direction, and the at least two output lines correspond to the at least two training lines one by one, and the at least two output lines can be The warp processing model processes the lines after at least two training lines.

For example, in step S21, the warping processing model to be trained processes the image content and the training line in the training image as a whole to obtain an output training image.

For example, in step S21, based on the output training image and the target image, adjusting the parameters of the warping model to be trained may include: based on the output training image and the target image, calculating the loss function corresponding to the warping model to be trained The loss value of the warping model; and adjust the parameters of the warping model to be trained based on the loss value.

For example, in one example, the predetermined condition corresponds to the minimization of a loss function corresponding to the warping model to be trained when a certain number of training images are input. In another example, the predetermined condition is that the number of training times or training cycles corresponding to the warping model to be trained reaches a predetermined number, and the predetermined number may be millions, as long as the number of training images used for training is large enough.

It should be noted that in the operation of repeatedly executing the training process, different training images and their corresponding target images can be used to train the warping processing model to be trained; in addition, the same training image and its corresponding target image can also be used The above training process is performed multiple times.

At least one embodiment of the present disclosure further provides an image processing device, and FIG. 9 is a schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure.

For example, as shown in FIG. 9 , in some embodiments, an image processing apparatus 900 may include an image acquisition module 901 , a first processing module 902 , a second processing module 903 and a mapping module 904 .

The image acquisition module 901 is configured to acquire original images. The image acquisition module 901 is used to implement step S10 shown in FIG. 1 . For specific descriptions of the functions implemented by the image acquisition module 901, reference may be made to the relevant description of step S10 shown in FIG. 1 in the embodiment of the above-mentioned image processing method, repeat The place will not be repeated.

For example, the image acquisition module 901 may include a camera, such as a camera of a smart phone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, or even a web camera.

The first processing module 902 is configured to process the original image to obtain a pre-processed image. For example, the preprocessed image includes at least two first lines, and the at least two first lines are sequentially arranged side by side along the same direction. The first processing module 902 is used to realize the step S11 shown in FIG. 1 . For the specific description of the functions realized by the first processing module 902, please refer to the relevant description of the step S11 shown in FIG. 1 in the embodiment of the above-mentioned image processing method , the repetitions will not be repeated.

The second processing module 903 is configured to process the pre-processed image by warping the processing model to obtain an intermediate image. For example, the intermediate image includes at least two second lines, the at least two second lines are arranged side by side in sequence along the same direction, and the at least two second lines are in one-to-one correspondence with the at least two first lines. The second processing module 903 is used to realize the step S12 shown in FIG. 1 . For the specific description of the functions realized by the second processing module 903, please refer to the relevant description of the step S12 shown in FIG. 1 in the embodiment of the above-mentioned image processing method , the repetitions will not be repeated.

The mapping module 904 is configured to remap the original image based on the mapping relationship between the preprocessed image and the intermediate image to obtain an output image. The mapping module 904 is used to implement step S13 shown in FIG. 1. For specific descriptions of the functions implemented by the mapping module 904, please refer to the relevant description of step S13 shown in FIG. 1 in the above-mentioned embodiment of the image processing method. No longer.

For example, data communication may be performed among the image acquisition module 901 , the first processing module 902 , the second processing module 903 and the mapping module 904 .

For example, in some embodiments, the image processing device 900 may further include a model training module. The model training module is configured to train the warp processing model.

For example, in some embodiments, the model training module may include an image generation submodule and a training submodule.

For example, the image generating submodule is configured to: generate training images; and generate target images corresponding to the training images based on the training images. For example, the training image includes at least two training lines, the at least two training lines are arranged side by side in sequence along the same direction, the target image includes at least two target training lines, the at least two target training lines are arranged side by side in sequence along the same direction, and at least The two target training lines are in one-to-one correspondence with at least two training lines. The image generation sub-module is used to realize step S20 and step S21 shown in FIG. 7 . For specific descriptions about the functions realized by the image generation sub-module, reference can be made to step S20 and step S20 shown in FIG. 7 in the embodiment of the above-mentioned image processing method. The related description of S21 will not be repeated here.

For example, the training submodule is configured to train the warping model to be trained based on the training image and the target image, so as to obtain a trained warping model. The training sub-module is used to realize the step S22 shown in FIG. 7. For the specific description of the functions realized by the training sub-module, please refer to the relevant description of the step S22 shown in FIG. 7 in the embodiment of the above-mentioned image processing method. No longer.

In some examples, the training submodule is configured to process the training image through the warping processing model to be trained to obtain an output training image; based on the output training image and the target image, adjust the parameters of the warping processing model to be trained; When the loss function corresponding to the warping model to be trained satisfies a predetermined condition, a trained warping model is obtained. For example, the image generation submodule is further configured to continue to generate at least one training image and a target image corresponding to the at least one training image when the loss function corresponding to the warping model to be trained does not meet the predetermined condition. At least one training image and its corresponding target image are used to repeatedly execute the above training process.

For example, the image acquisition module 901, the first processing module 902, the second processing module 903, the mapping module 904 and/or the model training module include codes and programs stored in memory; the processor can execute the codes and programs to achieve the above Some or all of the functions of the image acquisition module 901, the first processing module 902, the second processing module 903, the mapping module 904 and/or the model training module described above. For example, the image acquisition module 901, the first processing module 902, the second processing module 903, the mapping module 904 and/or the model training module may be dedicated hardware devices, which are used to implement the above-mentioned image acquisition module 901, the first processing module 902, some or all functions of the second processing module 903, the mapping module 904 and/or the model training module. For example, the image acquisition module 901 , the first processing module 902 , the second processing module 903 , the mapping module 904 and/or the model training module may be a circuit board or a combination of multiple circuit boards for realizing the functions described above. In the embodiment of the present application, the circuit board or a combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-transitory memories connected to the processors; and (3) Processor-executable firmware stored in memory.

It should be noted that the image processing apparatus can achieve technical effects similar to those of the aforementioned image processing method, which will not be repeated here.

At least one embodiment of the present disclosure further provides an electronic device, and FIG. 10 is a schematic block diagram of the electronic device provided by at least one embodiment of the present disclosure.

For example, as shown in FIG. 10 , an electronic device 1000 may include a processor 1001 and a memory 1002 . The memory 1002 non-transitory stores computer-executable instructions; the processor 1001 is configured to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor 1001, the image processing method according to any embodiment of the present disclosure can be realized. For the specific implementation of each step of the image processing method and related explanations, reference may be made to the above-mentioned embodiment of the image processing method, which will not be repeated here.

For example, as shown in FIG. 10 , the electronic device 1000 may further include a communication interface 1003 and a communication bus 1004 . The processor 1001, the memory 1002 and the communication interface 1003 communicate with each other through the communication bus 1004, and the components such as the processor 1001, the memory 1002 and the communication interface 1003 can also communicate through a network connection. The present disclosure does not limit the type and function of the network here.

For example, other implementations of the image processing method realized by the processor 1001 executing the program stored in the memory 1002 are the same as the implementations mentioned in the foregoing image processing method embodiments, and will not be repeated here.

For example, communication bus 1004 may be a Peripheral Component Interconnect Standard (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus 1004 can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

For example, the communication interface 1003 is used to implement communication between the electronic device 1000 and other devices.

For example, the processor 1001 and the memory 1002 may be set at the server (or cloud), or at the client (for example, a mobile device such as a mobile phone).

For example, the processor 1001 may control other components in the electronic device 1000 to perform desired functions. The processor 1001 can be a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable Logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture or the like. The GPU can be integrated directly on the motherboard alone, or built into the north bridge chip of the motherboard. A GPU can also be built into a central processing unit (CPU).

For example, memory 1002 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include random access memory (RAM) and/or cache memory (cache), etc., for example. Non-volatile memory may include, for example, read only memory (ROM), hard disks, erasable programmable read only memory (EPROM), compact disc read only memory (CD-ROM), USB memory, flash memory, and the like. One or more computer-executable instructions can be stored on the computer-readable storage medium, and the processor 1001 can run the computer-executable instructions to implement various functions of the electronic device 1000 . Various application programs and various data can also be stored in the memory 1002 .

It should be noted that the electronic device 1000 can achieve technical effects similar to those of the foregoing image processing method, and repeated descriptions will not be repeated here.

Fig. 11 is a schematic diagram of a non-transitory computer-readable storage medium provided by at least one embodiment of the present disclosure. For example, as shown in FIG. 11 , one or more computer-executable instructions 1101 may be non-transitory stored on a non-transitory computer-readable storage medium 1100 . For example, one or more steps in the image processing method according to any embodiment of the present disclosure may be executed when the computer-executable instructions 1101 are executed by the processor.

For example, the non-transitory computer-readable storage medium 1100 may be applied in the above-mentioned electronic device 1000 , for example, it may include the memory 1002 in the electronic device 1000 .

For example, for the description of the non-transitory computer-readable storage medium 1100, reference may be made to the description of the memory 1002 in the embodiment of the electronic device 1000, and repeated descriptions will not be repeated.

Fig. 12 is a schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. The electronic device provided by the present disclosure can be applied in the Internet system.

The functions of the image processing apparatus and/or electronic equipment involved in the present disclosure can be realized by using the computer system provided in FIG. 12 . Such computer systems can include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets, and any smart portable or wearable devices. The specific system in this embodiment illustrates a hardware platform including a user interface using functional block diagrams. Such computer equipment may be a general purpose computer equipment or a special purpose computer equipment. Both computer devices can be used to realize the image processing device and/or electronic device in this embodiment. The computer system may include any components that implement the presently described information needed to achieve image processing. For example, a computer system can be realized by a computer device through its hardware devices, software programs, firmware, and combinations thereof. For the sake of convenience, only one computer device is drawn in Fig. 12, but the relevant computer functions for realizing the information required for image processing described in this embodiment can be implemented by a group of similar platforms in a distributed manner, Distribute the processing load of a computer system.

As shown in Figure 12, the computer system can include a communication port 250, which is connected to a network for data communication ("from/to network" in Figure 12), for example, the computer system can send and receive data through the communication port 250 Information and data, that is, the communication port 250 can realize wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 220 (ie, the processor described above) for executing program instructions. The processor group 220 may consist of at least one processor (eg, CPU). The computer system may include an internal communication bus 210 . A computer system may include different forms of program storage units and data storage units (i.e., memory or storage media described above), such as hard disk 270, read-only memory (ROM) 230, random access memory (RAM) 240, which can be used to store Various data files used by the computer for processing and/or communicating, and possibly program instructions executed by the processor group 220 . The computer system may also include an input/output 260 for enabling input/output data flow between the computer system and other components (eg, user interface 280, etc.).

Typically, the following devices can be connected to input/output 260: input devices including, for example, touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication interfaces.

While FIG. 12 shows a computer system with various devices, it should be understood that the computer system is not required to have all of the devices shown and, instead, the computer system may have more or fewer devices.

For this disclosure, the following points need to be explained:

(1) The drawings of the embodiments of the present disclosure only relate to the structures involved in the embodiments of the present disclosure, and other structures may refer to general designs.

(2) For clarity, in the drawings used to describe the embodiments of the present invention, the thickness and size of layers or structures are exaggerated. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, it can be "directly on" or "under" the other element, Or intervening elements may be present.

(3) In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

Although the present disclosure has been described in detail with general descriptions and specific implementations above, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the embodiments of the present disclosure. Therefore, the modifications or improvements made on the basis of not departing from the spirit of the present disclosure all belong to the protection scope of the present disclosure.

Claims

An image processing method, comprising:

get the original image;

Processing the original image to obtain a pre-processed image, wherein the pre-processed image includes at least two first lines, and the at least two first lines are arranged side by side in sequence along the same direction;

The pre-processing image is processed by a distortion processing model to obtain an intermediate image, wherein the intermediate image includes at least two second lines, the at least two second lines are arranged side by side in sequence along the same direction, and the at least The two second lines correspond one-to-one to the at least two first lines;

Based on the mapping relationship between the preprocessed image and the intermediate image, the original image is remapped to obtain an output image.
The image processing method according to claim 1, wherein the mapping relationship between the preprocessed image and the intermediate image comprises a mapping relationship between the at least two first lines and the at least two second lines and a mapping relationship between the area between the at least two first lines in the preprocessed image and the area between the at least two second lines in the intermediate image.
The image processing method according to claim 1, wherein, based on the mapping relationship between the preprocessed image and the intermediate image, remapping the original image to obtain an output image comprises:

Based on the mapping relationship between the pre-processing image and the intermediate image, determine pre-processing mapping information corresponding to the pre-processing image by an interpolation method, where the pre-processing mapping information is used to indicate the pre-processing image Mapping parameters for at least some of the pixels in ;

determining, based on the pre-processing mapping information, mapping information corresponding to a region corresponding to the original image in the pre-processing image;

performing scaling processing on the mapping information of the area corresponding to the original image to determine the mapping information corresponding to the original image;

Remapping the original image based on the mapping information corresponding to the original image to obtain the output image.
The image processing method according to claim 3, wherein at least some of the pixels in the pre-processed image include pixels in the area between the at least two first lines in the pre-processed image and the at least Pixels on the two first lines.
The image processing method according to any one of claims 1-4, wherein processing the original image to obtain a preprocessed image comprises:

performing binarization processing on the original image to obtain an input image;

performing scaling processing on the input image to obtain a scaled image;

performing filling processing on the scaled image to obtain a filled image;

Perform region division on the filled image to obtain the preprocessed image.
The image processing method according to claim 5, wherein the scaled image comprises a first scaled image side and a second scaled image side opposite to each other, and the pre-processed image comprises a first pre-processed image side opposite to each other and the second pre-processing image side, the first pre-processing image side corresponds to the first zoomed image side, the second pre-processing image side corresponds to the second zoomed image side, and the at least two second a line is arranged between the first pre-processed image side and the second pre-processed image side along a direction from the first pre-processed image side to the second pre-processed image side,

Filling the scaled image to obtain the filled image includes:

filling a first padding area on a side of the first zoomed image away from a side of the second zoomed image and filling a second padding area on a side of the second zoomed image away from a side of the first zoomed image, To get the padded image,

Wherein, the two opposite sides of the first filled area are the first scaled image side and the first pre-processed image side, and the two opposite sides of the second filled area are the first A second scaled image edge and said second preprocessed image edge.
The image processing method according to claim 6, wherein the size of the first filled area is the same as that of the second filled area.
The image processing method according to any one of claims 1-4, wherein processing the original image to obtain a preprocessed image comprises:

performing binarization processing on the original image to obtain an input image;

performing filling processing on the input image to obtain a filled image;

Scaling the filled image to obtain a zoomed image;

Perform region division on the scaled image to obtain the preprocessed image.
The image processing method according to any one of claims 1-4, wherein the at least two first lines are at least two bisector lines that equally divide the preprocessed image along the same direction.
The image processing method according to any one of claims 1-4, wherein the distortion processing model is a model based on a neural network.
The image processing method according to any one of claims 1-4, wherein the image content in the original image is distorted.
The image processing method according to any one of claims 1-4, further comprising: training the warping processing model, wherein training the warping processing model comprises:

generating a training image, wherein the training image includes at least two training lines, and the at least two training lines are arranged side by side in sequence along the same direction;

Based on the training image, generate a target image corresponding to the training image, wherein the target image includes at least two target training lines, the at least two target training lines are arranged side by side in sequence along the same direction, and the At least two target training lines correspond to the at least two training lines;

Based on the training image and the target image, the warping model to be trained is trained to obtain the trained warping model.
The image processing method according to claim 12, wherein, based on the training image and the target image, training the warping model to be trained to obtain the trained warping model comprises:

The training image is processed by the warping processing model to be trained to obtain an output training image, wherein the output training image includes at least two output lines, and the at least two output lines are arranged side by side in sequence along the same direction Cloth, and the at least two output lines correspond to the at least two training lines one by one;

adjusting parameters of the warping model to be trained based on the output training image and the target image;

When the loss function corresponding to the warping model to be trained meets the predetermined condition, obtain the trained warping model, and when the loss function corresponding to the warping model to be trained does not meet the predetermined condition, continue to input the The training image and the target image are used to repeatedly perform the above training process.
The image processing method according to claim 12, wherein generating the training image comprises:

generate input training images;

performing scaling processing on the input training image to obtain a scaled input training image;

Filling the scaled input training image to obtain the filled input training image;

Distorting the filled input training image to obtain a distorted input training image;

performing region division on the warped input training image to obtain the training image including the at least two training lines.
The image processing method according to claim 14, wherein, based on the training image, generating a target image corresponding to the training image comprises: reversely performing the training image on the basis of the distortion parameter corresponding to the distortion processing Warp processing to get the target image.
The image processing method according to claim 14, wherein generating an input training image comprises:

Get the original training image;

Perform binarization processing on the original training image to obtain the input training image.
The image processing method according to claim 12, wherein the at least two training lines are at least two bisector lines that equally divide the training image along the same direction.
An image processing device, comprising:

An image acquisition module configured to acquire an original image;

The first processing module is configured to process the original image to obtain a pre-processed image, wherein the pre-processed image includes at least two first lines, and the at least two first lines are juxtaposed in sequence along the same direction arrangement;

The second processing module is configured to process the pre-processed image through a warping processing model to obtain an intermediate image, wherein the intermediate image includes at least two second lines, and the at least two second lines are sequentially along the same direction arranged side by side, and the at least two second lines correspond to the at least two first lines one by one;

The mapping module is configured to remap the original image based on the mapping relationship between the preprocessed image and the intermediate image to obtain an output image.
An electronic device comprising:

memory non-transitoryly storing computer-executable instructions;

a processor configured to execute said computer-executable instructions,

Wherein, when the computer-executable instructions are executed by the processor, the image processing method according to any one of claims 1-17 is realized.
A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the computer-executable instructions according to any one of claims 1-17 are implemented. The image processing method described in one item.