CN111860475B

CN111860475B - Image processing method and device, electronic equipment and storage medium

Info

Publication number: CN111860475B
Application number: CN201910348391.9A
Authority: CN
Inventors: 刘学博; 梁鼎
Original assignee: Beijing Sensetime Technology Development Co Ltd
Current assignee: Beijing Sensetime Technology Development Co Ltd
Priority date: 2019-04-28
Filing date: 2019-04-28
Publication date: 2023-12-19
Anticipated expiration: 2039-04-28
Also published as: CN111860475A

Abstract

The embodiment of the application discloses an image processing method and device, electronic equipment and a storage medium, wherein the method comprises the following steps: performing key point detection on the acquired card image to obtain predicted position information of a plurality of key points of a target card in the card image; determining the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card; determining whether to store the card image based at least in part on the integrity of the target card contained in the card image is advantageous for improving card recognition accuracy and user experience.

Description

Image processing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer vision, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

Card identification refers to a technique of processing a card image by using computer vision technology to identify information recorded on a card contained in the card image. Taking a bank card as an example, a user needs to record information of the bank card when handling a new bank card so as to bind the bank card with the user, however, the efficiency of manually inputting the information of the bank card is low and mistakes are easy to occur, the user experience is poor, a bank card photo is taken and is processed by using a bank card identification technology, so that identification of the information of the bank card is realized, and the processing efficiency and accuracy rate can be improved, thereby improving the user experience.

Disclosure of Invention

The embodiment of the application provides an image processing method and device, electronic equipment and a storage medium.

A first aspect of an embodiment of the present application provides an image processing method, including:

performing key point detection on the acquired card image to obtain predicted position information of a plurality of key points of a target card in the card image;

determining the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card;

based at least in part on the integrity of the target card contained in the card image, a determination is made as to whether to store the card image.

In an alternative embodiment, the plurality of key points of the target card includes a plurality of vertices of the target card.

In an alternative embodiment, the determining the integrity of the target card included in the card image based on the predicted position information of the plurality of key points of the target card includes:

determining whether a plurality of key points are positioned in the card image based on predicted position information of the key points of the target card and boundary position information of the card image;

And determining the integrity of the target card contained in the card image based on whether the plurality of key points are located within the card image.

In an alternative embodiment, the determining the integrity of the target card contained in the card image based on whether the plurality of keypoints are located within the card image comprises:

and determining that the target card contained in the card image is missing in response to at least one of the plurality of keypoints being not within the card image.

obtaining a minimum distance from at least one first keypoint of the plurality of keypoints to a boundary of the target card in response to the at least one first keypoint not being within the card image;

and determining that the target card contained in the card image is missing in response to the at least one first key point having a minimum distance from the boundary of the target card greater than the preset threshold value.

In an alternative embodiment, the method further comprises:

and determining that the target card contained in the card image is complete in response to the minimum distances from the at least one first key point to the boundary of the target card being smaller than or equal to the preset threshold value.

In an alternative embodiment, before the determining the integrity of the target card included in the card image based on the predicted position information of the plurality of key points of the target card, the method further includes:

carrying out correction processing on the card image to obtain a card image after the correction processing;

the determining whether to store the card image includes:

and determining whether to store the card image after the transfer processing.

In an alternative embodiment, the determining whether to store the card image based at least in part on the integrity of the target card contained in the card image comprises:

and storing the card image in response to determining that the target card contained in the card image is complete.

In an alternative embodiment, the method further comprises:

performing text recognition on the card image, and determining a text recognition result of the target card;

The determining whether to store the card image based at least in part on the integrity of the target card contained in the card image includes:

determining whether to store the card image based at least in part on the integrity of the target card contained in the card image and the text recognition result of the target card.

In an alternative embodiment, the method further comprises:

and responding to the determination of the target card missing contained in the card image, outputting first prompt information, wherein the first prompt information is used for prompting at least one of the target card missing contained in the card image, the missing position of the target card and the placement position of the target card.

In an alternative embodiment, before the key point detection is performed on the acquired card image, the method further includes:

and sending out second prompt information to prompt the user to place the target card in the appointed area.

In an alternative embodiment, the target card comprises one or any combination of an identification card, a bank card, a passport, a driver's license, a social security card.

A second aspect of the embodiments of the present application provides an image processing apparatus, including: the system comprises a prediction module, an integrity detection module and a storage module, wherein:

The prediction module is used for detecting key points of the acquired card image and obtaining predicted position information of a plurality of key points of a target card in the card image;

the integrity detection module is used for determining the integrity of the target card contained in the card image based on the predicted position information of a plurality of key points of the target card;

the storage module is configured to determine whether to store the card image based at least in part on an integrity of the target card contained in the card image.

Optionally, the plurality of key points of the target card include a plurality of vertices of the target card.

In an alternative embodiment, the integrity detection module is specifically configured to:

In an alternative embodiment, the integrity detection module is specifically further configured to:

In an alternative embodiment, the integrity detection module is further specifically configured to:

In an alternative embodiment, the device further comprises a turning module for:

before the integrity detection module determines the integrity of the target card contained in the card image based on the predicted position information of a plurality of key points of the target card, carrying out correction processing on the card image to obtain a card image after the correction processing;

The storage module is specifically used for determining whether to store the card image after the transfer processing.

In an alternative embodiment, the storage module is specifically configured to:

In an optional implementation manner, the system further comprises an identification module, a recognition module and a display module, wherein the identification module is used for carrying out text identification on the card image and determining a text identification result of the target card;

the storage module is further configured to determine whether to store the card image based at least in part on an integrity of the target card contained in the card image and a text recognition result of the target card.

In an optional implementation manner, the system further comprises an output module, configured to output first prompt information in response to determining that the target card contained in the card image is missing, where the first prompt information is used to prompt at least one of missing the target card, missing position of the target card, and adjusting placement position of the target card contained in the card image.

In an optional implementation manner, the system further comprises a prompt module, which is used for sending out second prompt information to prompt the user to place the target card in a designated area before the prediction module detects the key points of the acquired card image.

Optionally, the target card comprises one or any combination of an identity card, a bank card, a passport, a driving license and a social security card,

A third aspect of the embodiments provides an electronic device comprising a processor and a memory for storing a computer program configured to be executed by the processor for performing part or all of the steps as described in any of the methods of the first aspect of the embodiments of the present application.

A fourth aspect of the embodiments provides a computer-readable storage medium for storing a computer program, wherein the computer program causes a computer to perform some or all of the steps as described in any of the methods of the first aspect of the embodiments of the present application.

A fifth aspect of the present application provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of the first aspect and any one of its possible implementations.

In the embodiment of the application, the obtained card image is subjected to key point detection to obtain the predicted position information of a plurality of key points of the target card in the card image, and the integrity of the target card contained in the card image is determined based on the predicted position information of the plurality of key points of the target card, so that whether the card image is stored or not is determined based at least in part on the integrity of the target card contained in the card image, thereby being beneficial to improving the accuracy of card identification and improving user experience.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the technical aspects of the disclosure.

Fig. 1 is a schematic flow chart of an image processing method disclosed in an embodiment of the present application;

FIG. 2 is a schematic illustration of a card image as disclosed in an embodiment of the present application;

FIG. 3 is a flow chart of another image processing method disclosed in an embodiment of the present application;

FIG. 4 is a schematic diagram of the integrity of a target card disclosed in an embodiment of the present application;

fig. 5 is a schematic structural view of an image processing apparatus disclosed in an embodiment of the present application;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The term "and/or" in this application is merely an association relation describing an associated object, and means that three relations may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C. The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

The image processing apparatus according to the embodiments of the present application may be an apparatus capable of processing an image, and may be an electronic device, where the electronic device includes a terminal device, and in a specific implementation, the terminal device includes, but is not limited to, other portable devices such as a mobile phone, a laptop computer, or a tablet computer that has a touch sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).

The embodiments of the present application are described in detail below.

Referring to fig. 1, fig. 1 is a flowchart of an image processing method according to an embodiment of the disclosure, and as shown in fig. 1, the image processing method includes the following steps.

101. And detecting key points of the acquired card image to obtain the predicted position information of a plurality of key points of the target card in the card image.

The execution subject of the image processing method may be the above-described image processing apparatus, and for example, the image processing method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the image processing method may be implemented by way of a processor invoking computer readable instructions stored in a memory.

In this embodiment of the present application, the card image may be an image acquired by a camera, for example, a photograph taken by the camera of the terminal device, or a screenshot in a video taken by the camera of the terminal device, which is not limited in this embodiment of the present application.

The card image may include the target card, alternatively the target card may be rectangular, such as a rectangle, or any other shape. The shape of the target card is not limited in the embodiment of the application.

Optionally, the target card includes one or any combination of an identity card, a bank card, a passport, a driver's license, a driving license, and a social security card, and may also be a document such as a paper contract or a protocol, which is not limited in the embodiment of the present application.

And detecting the key points of the acquired card image to obtain the predicted position information of a plurality of key points of the target card in the card image. In one possible implementation, the key point detection may be performed on the card image using a convolutional neural network (Convolutional Neural Networks, CNN) to obtain predicted location information of a plurality of key points of the target card, where the predicted location information of the key points may include coordinates of the key points.

Specifically, the position information of the key point of the target card in the card image may be output, for example, two-dimensional coordinates of the plurality of key points may be output.

In image processing, a keypoint is essentially a feature that is an abstract description of a fixed region or spatial physical relationship, describing a combination or context within a certain neighborhood. It is not just a point information, or represents a location, but rather a combination of context and surrounding neighborhood.

For example, in the face key point detection task, there are 28 key points, or 64 or 128 key points which are popular nowadays, wherein each point represents a class of characteristics in different faces, and has a certain universality, and the class of characteristics not only comprises some characteristics of pixels, such as characteristic points of lips, but also comprises the positional relationship between lips and faces.

The neural network for implementing the keypoint detection may be an existing keypoint detection network, for example, one branch may be used for regression of the pixel points, and the other branch may be used for regression of the relationship of the pixel points. The definition method of the point labels and the relation labels lightens the difficulty of regression, the multi-stage calculation mode used in the network is equivalent to that each stage provides a certain prediction result, the next stage carries out multi-stage precision experiments on the result, and the similar method can be applied to various fields.

The key points of the target card may include a plurality of vertices of the target card, or may be points on a boundary of the target card, for example, refer to fig. 2, where the card image shown in fig. 2 includes the target card, and the key points include four vertices l, m, o, and n of the target card, and specifically, the predicted coordinates of the four key points may be obtained. Because the possibility that the position of the vertex (angle) exceeds the shooting range is high when the general certificate is not shot fully, the key point is selected as the vertex position of the target card, so that the accuracy of image integrity detection is higher, and the judgment is faster.

The image boundary in the embodiment of the application can also be called an image edge which is one of the most classical and fundamental problems in image processing and computer vision when the most basic characteristic edge of the image is detected, and the method has important application in the aspects of object identification, three-dimensional reconstruction, image matching, retrieval and the like.

The edges of such documents are often one of the most important information when locating such target cards (e.g., identification cards, bank cards, passports, driver's licenses, travel or social security cards, etc.) in a complex context. Optionally, the boundary of the target card in the card image, that is, the frame position of the certificate, can be determined by detecting the edge of the card image, so as to better determine the key points. Specifically, edge point position information in the card image, such as a roberts operator, a sobel operator, a prewitt operator and the like, can be obtained by using an edge detection algorithm, and edge contour information, such as a canny operator, can be obtained by using a subsequent tracking algorithm.

In an alternative embodiment, the boundary of the target card in the card image may be determined, and then the predicted position information of multiple key points of the target card in the card image may be obtained.

Specifically, a convex hull of the target card can be obtained through a key point detection algorithm, and then the boundary of the convex hull is used as the boundary of the target card.

The key point detection algorithm may be any method of inputting a plurality of points on a plane and outputting convex hulls of the points, such as a rotation karst method, a Graham scanning method, a Jarvis stepping method algorithm, and the like, and may also include a related algorithm in OpenCV. OpenCV is a cross-platform computer vision library based on BSD license (open source) release that can run on Linux, windows, android and Mac OS operating systems. The system is lightweight and efficient, is composed of a series of C functions and a small number of C++ classes, provides interfaces of Python, ruby, MATLAB and other languages, and realizes a plurality of general algorithms in the aspects of image processing and computer vision.

In a real vector space V, for a given set X, the intersection S of all convex sets containing X is referred to as the convex hull of X. The convex hull of X may be constructed with convex combinations of all points (X1,..xn) within X.

In general terms, given a set of points on a two-dimensional plane, a convex hull is understood as a convex closed figure formed by connecting the points of the outermost layer, which can contain all the points of the set of points, which can be represented in the card image as the boundary of the target card, on the other hand, at least one of the above-mentioned key points can be determined in these points.

After obtaining the predicted location information for the plurality of keypoints of the target card in the card image described above, step 102 may be performed.

102. And determining the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card.

The identification of the target card has wide application, and it is very important to detect whether the target card is missing or not, and in practical application, the target card is often required to be complete so as to keep the image of the target card for evidence storage, verification and the like. The integrity of the target card contained in the card image may be determined based on predicted position information of a plurality of key points of the target card. The integrity of the target card in the card image can be determined primarily by determining whether a plurality of key points of the target card are within the card image.

Specifically, whether the plurality of key points are located in the card image may be determined based on the predicted position information of the plurality of key points of the target card and the boundary position information of the card image;

and determining the integrity of the target card contained in the card image based on whether the plurality of key points are located in the card image.

The boundary position information of the card image can be obtained when the card image is processed, wherein the boundary position information of the card image can comprise boundary point coordinates of the card image, can be determined based on edge detection of pixel points, and can also be obtained through image information identification. Optionally, information such as resolution, image format, etc. of the card image may also be obtained. The boundary of the card image is formed by a plurality of straight lines, and can be generally rectangular formed by connecting four straight lines.

The positions of the key points and the boundaries of the card image can be analyzed and judged to determine whether the target card is missing in the card image. The above-mentioned deficiency is understood to mean that the target card does not show its integrity in the card image, for example, the target card shows incompleteness due to a photographing angle problem, and there is a partial deficiency, typically that the edge of the target card exceeds the image area.

In one possible embodiment, the determining the integrity of the target card included in the card image based on whether the plurality of key points are located in the card image may include:

and determining that the target card contained in the card image is missing in response to at least one of the plurality of keypoints being absent from the card image.

Based on the predicted position information of the key points and the boundary position information of the card image, it is determined whether the plurality of key points are located in the card image.

By determining whether the key points are in the card image, whether the target card is missing in the card image can be judged. When all the key points are in the card image, the target card can be determined that no deletion exists in the card image. And at least one of the plurality of key points is not in the card image, which means that the key point is outside the card image, that is, the target card is not completely shot (a part of the target card is beyond the card image during shooting), and the target card missing contained in the card image can be determined.

The selection and prediction of the key points can be proper, and judgment of too many key points is not needed, so that the data processing amount is reduced.

In some embodiments, the above-described keypoint detection may be implemented by a neural network, such as a convolutional neural network or a recurrent neural network, or the like. For example, the collected original card image or the preprocessed card image is input into a neural network for key point detection, so as to obtain key point information of a target card in the card image, wherein the convolutional neural network can comprise a plurality of convolutional layers, an activation function and a plurality of pooling layers. For example, the activation function may include a linear rectification function (Rectified Linear Unit, reLU) that may provide more efficient gradient descent and counter-propagation, avoiding gradient explosion and gradient extinction problems. The predicted position information of the key points can be obtained through the convolutional neural network, and the optimization can be performed based on the output confidence level during training.

In some embodiments, the neural network may output location information of a plurality of key points of the target card, or further output detection confidence information of the plurality of key points, such as scores, etc., which are not limited by the embodiments of the present disclosure.

According to the image processing method, key point detection is performed through the convolutional neural network, information extraction can be performed on the card image well without a recurrent neural network, the calculated amount is reduced, and the card identification accuracy is improved.

Optionally, the method further comprises: and outputting first prompt information for prompting one of the target card missing contained in the card image, the missing position of the target card, and the placement position of the target card or any combination thereof in response to determining that the target card contained in the card image is missing. The prompt information may be text information or voice information, where the text information may be displayed in any area in the card image or outside the card image, or pop-up prompt boxes, etc., which are not limited in this embodiment of the present application.

Through the first prompt information, the user can be prompted that the target card contained in the card image is missing, the image of the complete target card needs to be uploaded again, the verification or the evidence storage flow is completed successfully, the missing position of the target card can be indicated in detail in the first prompt information, for example, the missing position is marked in the card image in a mode of displaying the image, or the missing position is indicated by words, for example, "the upper left corner of the identity card in the uploaded image is missing". The user may be prompted to adjust the placement position of the target card through the first prompting information, specifically, in the process of verifying the certificate, the user may place the certificate at a designated position, the image processing device collects an image of the certificate (i.e. the card image), when the image processing device determines that the certificate contained in the image is missing, the first prompting information may be output to prompt the user to adjust the placement position of the target card, so as to obtain a complete certificate image, for example, "adjust the placement position of the certificate" or "please move the certificate to the upper right", and so on.

In some embodiments, the process may be performed in real time, and in the event that it is determined that the target card contained in the currently acquired card image is incomplete, the card image may be continuously acquired and the detection process repeated until a card image satisfying the condition is detected.

103. Determining whether to store the card image based at least in part on the integrity of the target card contained in the card image.

After the integrity check is performed on the target card in the card image in step 102 described above to obtain a result, it may be determined whether to store the card image.

In an alternative embodiment, the card image may be stored in response to determining that the target card contained in the card image is complete.

The target card without the deficiency meets the requirement, and the card image can be stored as a certificate of a user and also can be used as a basis for next step identity verification or auditing.

In an alternative embodiment, the method further comprises: performing text recognition on the card image, and determining a text recognition result of the target card;

the step 103 may specifically include:

The target card in the card image may contain text, text recognition can be performed on the card image to obtain the text recognition result, whether the target card is of a specified card type can be judged according to the text recognition result, and if the target card is complete and of the specified card type, the card image can be stored. If the target card is complete but not of the specified card type, the card image may not be stored.

In some embodiments, the above-described text recognition may be performed upon determining that the target card contained in the card image is complete (i.e., there is no missing). Therefore, text recognition is not performed under the condition that the target card in the card image is determined to be missing, the time for text recognition can be saved, and the processing efficiency is improved.

For example, when using a document image to perform online identity authentication, for example, some application programs need to upload an identity card, a bank card, etc. to perform identity authentication, text content can be extracted from the document image through text recognition, for example, optical character recognition (Optical Character Recognition, OCR), whether the document type to be acquired is a type of document to be acquired, for example, whether the document type is a designated bank, and information such as a document number, a card number, etc. in the document image can be extracted, so that identity authentication or information storage is facilitated. For another example, some illegal pictures may be uploaded on the social network, and if the illegal text content is contained in the pictures, the illegal text content can be automatically identified for subsequent processing.

It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.

In this embodiment of the present application, key point detection may be performed on an obtained card image, so as to obtain predicted position information of multiple key points of a target card in the card image, then, based on the predicted position information of multiple key points of the target card, the integrity of the target card included in the card image is determined, and then, based at least in part on the integrity of the target card included in the card image, whether to store the card image is determined, so that the integrity of the image may be detected, and the accuracy of image identification and information extraction may be improved.

Referring to fig. 3, fig. 3 is a schematic flow chart of another image processing method disclosed in an embodiment of the present application, and fig. 3 is further optimized based on fig. 1. The main body performing the steps of the embodiments of the present application may be an image processing apparatus as described above. As shown in fig. 3, the image processing method includes the steps of:

201. and sending out second prompt information to prompt the user to place the target card in the designated area.

The execution subject in the embodiments of the present application may be the image processing apparatus described above, and for example, the image processing method may be executed by a terminal device or a server or other processing devices, where the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the image processing method may be implemented by way of a processor invoking computer readable instructions stored in a memory.

The target card may be rectangular, such as a rectangle. The shape of the target card is not limited in the embodiment of the application. Optionally, the target card includes one or any combination of an identity card, a bank card, a passport, a driver's license, a driving license, and a social security card, and may also be a document such as a paper contract or a protocol, which is not limited in the embodiment of the present application.

Specifically, the image processing apparatus may send the second prompt information to prompt the user to place the target card in the specified area, and then may collect the card image of the target card through the camera, so that step 202 may be executed. The first prompt information may be text information or voice information, which is not limited in the embodiment of the present application.

202. And detecting key points of the acquired card image to obtain predicted position information of a plurality of key points of the target card in the card image.

The above step 202 may refer to the specific description of step 101 in the embodiment shown in fig. 1, which is not repeated here.

Optionally, before step 202, the method further includes:

and carrying out correction processing on the card image to obtain the card image after the correction processing.

The image processing device may perform preliminary image recognition to determine whether the target card in the card image is displayed in the forward direction before performing the key point recognition on the card image, and if the target card in the card image is not displayed in the forward direction, may perform the above-mentioned correction process on the card image to obtain the card image after the correction process, and then execute step 202.

Through the forward conversion processing, the card image of the target card which is displayed in the forward direction can not be converted into the forward direction for display, and the processing accuracy and the display effect of image recognition are improved.

203. And determining whether the plurality of key points are positioned in the card image based on the predicted position information of the plurality of key points of the target card and the boundary position information of the card image.

The step 102 in the embodiment shown in fig. 1 may be specifically described for obtaining the boundary of the card image, which is not described herein.

If it is determined that the plurality of keypoints are all located within the card image, step 206 may be performed.

Alternatively, if at least one first key point of the key points is not in the card image, step 204 may be performed.

204. And obtaining a minimum distance from at least one first key point in the plurality of key points to the boundary of the target card in response to the at least one first key point not being in the card image.

The predicted position information of the key point of the target card may include two-dimensional coordinates of the key point, that is, the position of the key point and the boundary position of the target card may be accurately determined in the card image according to the established coordinate system, so as to determine whether the key point is in the card image. If at least one of the key points is not in the card image, for convenience of description, the key point which is not in the card image is marked as a first key point, and the minimum distance from the first key point to the boundary of the target card can be further obtained for judgment.

The boundary of the card image is a straight line, and when the position of the key point and the boundary of the card image are known, the minimum distance from the at least one first key point to the boundary of the target card may be calculated according to a point-to-straight line distance formula.

Further, the determining may be performed on all the at least one first key point, to determine whether a minimum distance between the at least one first key point and a boundary of the target card is greater than a preset threshold.

If the minimum distances from the at least one first key point to the boundary of the target card are all less than or equal to the preset threshold, step 205 may be executed; if there is a key point with a minimum distance from the boundary of the target card greater than the preset threshold value among the at least one first key point, step 207 may be executed.

205. And determining that the target card contained in the card image is complete in response to the minimum distances from the at least one first key point to the boundary of the target card being less than or equal to the preset threshold value.

If the minimum distance from the at least one first key point to the boundary of the target card is less than or equal to the preset threshold, it may be understood that the distance from the key point not in the card image to the card image is not large, the integrity of the target card is not greatly affected, and it may be determined that the target card included in the card image is complete. The preset threshold value may be set and adjusted according to the requirement, which is not limited herein.

In the event that it is determined that the target card contained in the card image is complete, step 206 may be performed.

206. And storing the card image.

If the target card contained in the card image is determined to be complete, namely the target card is not missing, namely the card image meets the requirements, the card image can be stored and used as an information storage card of a user and also can be used as a basis for next step identity verification or auditing. Further alternatively, text recognition may be performed on the card image.

207. And determining that the target card contained in the card image is absent in response to the at least one first key point having a minimum distance from the boundary of the target card greater than the preset threshold.

If the minimum distance from at least one of the first key points to the boundary of the target card is greater than the preset threshold, it can be understood that the key point with the minimum distance from the boundary of the target card greater than the preset threshold is far beyond the card image, so as to affect the integrity and information identification of the target card, and it can be determined that the target card contained in the card image is missing.

If it is determined that the target card contained in the card image is missing, step 208 may be performed.

208. And outputting first prompt information, wherein the first prompt information is used for prompting and adjusting the placement position of the target card.

When the target card included in the card image is missing, the user may be prompted to adjust the placement position of the target card by outputting the first prompting information.

Specifically, in the process of verifying the certificate, the user may place the certificate at the designated position, the image processing device collects the image of the certificate (i.e. the card image), when the image processing device determines that the certificate contained in the image is missing, the image processing device may output the first prompt information to prompt that the target card contained in the card image is missing, and prompt the user to adjust the placement position of the target card, so as to obtain the complete certificate image again, for example, output "please adjust the placement position of the certificate". Or specifically "please move the document to the right and upward, and place it in the designated area. The first prompting information may be text information or voice information, which is not limited in the embodiment of the present application.

Referring to fig. 4, fig. 4 is a schematic diagram of card image processing disclosed in the embodiment of the present application, as shown in fig. 4, a target card a in an image a, where key points of the target card a are all in a region within a boundary of the image a, such as a point q, may determine that the target card a is complete in the image a. The target card B in the image B has a point p of one of the key points in the area outside the boundary of the image B, so that the target card B can be judged to have a defect in the card image B; alternatively, whether to store the image B may also be determined by determining whether the minimum distance h from the point p to the boundary of the target card B is greater than a preset threshold. The dashed line part in the figure is the complement part of the target card B beyond the image B.

The preset threshold can be set and modified according to card image processing tasks of different target cards, and the preset threshold is not limited in the embodiment of the application. For example, some core contents of the target card are concentrated in the middle of the certificate and far away from the edge of the certificate, and the preset threshold value set at the moment can be larger, so that even if the corners of the target card are missing in the card image, the information extraction and verification of the core contents are not influenced, and the detection standard can be flexibly adjusted according to the needs, so that the flexibility and the adaptability of the image processing method are improved.

In the embodiment of the application, the user is prompted to place the target card in the designated area by sending the second prompt information, the acquired card image is subjected to key point detection to obtain the predicted position information of a plurality of key points of the target card in the card image, whether the plurality of key points are located in the card image is determined based on the predicted position information of the plurality of key points of the target card and the boundary position information of the card image, then, in response to the existence of at least one first key point in the plurality of key points which is not located in the card image, the minimum distance from the at least one first key point to the boundary of the target card can be obtained, and determining that the target card contained in the card image is missing in response to the at least one first key point having the minimum distance to the boundary of the target card greater than the preset threshold, and outputting first prompt information for prompting and adjusting the placement position of the target card, detecting the integrity of the image, improving the accuracy of image recognition and information extraction, flexibly adjusting detection standards as required to complete different image processing tasks.

The foregoing description of the embodiments of the present application has been presented primarily in terms of a method-side implementation. It will be appreciated that the image processing apparatus, in order to implement the above-described functions, comprises corresponding hardware structures and/or software modules that perform the respective functions. Those of skill in the art will readily appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is implemented as hardware or computer software driven hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

The embodiment of the present application may divide the functional units of the image processing apparatus according to the above-described method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated in one processing unit. The integrated units may be implemented in hardware or in software functional units. It should be noted that, in the embodiment of the present application, the division of the units is schematic, which is merely a logic function division, and other division manners may be implemented in actual practice.

Referring to fig. 5, fig. 5 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present disclosure. As shown in fig. 5, the image processing apparatus 300 includes: a prediction module 310, an integrity detection module 320, and a storage module 330, wherein:

the prediction module 310 is configured to perform key point detection on the obtained card image, and obtain predicted position information of a plurality of key points of the target card in the card image;

the integrity detection module 320 is configured to determine the integrity of the target card included in the card image based on the predicted position information of the plurality of key points of the target card;

the storage module 330 is configured to determine whether to store the card image based at least in part on the integrity of the target card included in the card image.

Optionally, the integrity detection module 320 is specifically configured to:

determining whether the plurality of key points are located in the card image based on predicted position information of the plurality of key points of the target card and boundary position information of the card image;

Optionally, the integrity detection module 320 is specifically further configured to:

Optionally, the integrity detection module 320 is further specifically configured to:

and determining that the target card contained in the card image is absent in response to the at least one first key point having a minimum distance from the boundary of the target card greater than the preset threshold.

and determining that the target card contained in the card image is complete in response to the minimum distances from the at least one first key point to the boundary of the target card being less than or equal to the preset threshold value.

Optionally, the image processing apparatus 300 further includes a storage module 330 configured to store the card image if the target card has no deletion in the card image.

Optionally, the image processing apparatus 300 further includes a correction module 340, configured to:

the integrity detection module performs correction processing on the card image before determining the integrity of the target card contained in the card image based on the predicted position information of a plurality of key points of the target card, so as to obtain the card image after the correction processing;

the storage module 330 is specifically configured to determine whether to store the card image after the transfer processing.

Optionally, the storage module 330 is specifically configured to:

Optionally, the image processing apparatus 300 further includes an identification module 350, configured to perform text identification on the card image, and determine a text identification result of the target card;

the storage module 330 is further configured to determine whether to store the card image based at least in part on the integrity of the target card included in the card image and the text recognition result of the target card.

Optionally, the image processing apparatus 300 further includes an output module 360 configured to output, in response to determining that the target card contained in the card image is missing, first prompting information, where the first prompting information is used to prompt at least one of missing the target card contained in the card image, missing position of the target card, and adjusting a placement position of the target card.

Optionally, the image processing apparatus 300 further includes a prompting module 370, configured to send a second prompting message to prompt the user to place the target card in the specified area before the prediction module detects the key point of the acquired card image.

Optionally, the target card includes one or any combination of an identity card, a bank card, a passport, a driver's license, and a social security card.

The image processing method in the embodiments of fig. 1 and 3 described above can be implemented using the image processing apparatus 300 in the embodiment of the present application.

By implementing the image processing apparatus 300 shown in fig. 5, the image processing apparatus 300 may perform keypoint detection on an acquired card image to obtain predicted position information of a plurality of keypoints of a target card in the card image, determine the integrity of the target card included in the card image based on the predicted position information of the plurality of keypoints of the target card, and then determine whether to store the card image based at least in part on the integrity of the target card included in the card image, so as to detect the integrity of the image, and improve the accuracy of image recognition and information extraction.

Referring to fig. 6, fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 6, the electronic device 400 includes a processor 401 and a memory 402, wherein the electronic device 400 may further include a bus 403, the processor 401 and the memory 402 may be connected to each other through the bus 403, and the bus 403 may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus 403 may be classified into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 6, but not only one bus or one type of bus. The electronic device 400 may further include an input/output device 404, where the input/output device 404 may include a display screen, such as a liquid crystal display screen. The memory 402 is used for storing a computer program; the processor 401 is arranged to invoke a computer program stored in the memory 402 to perform some or all of the method steps mentioned in the embodiments of fig. 1 and 3 above.

By implementing the electronic device 400 shown in fig. 6, the electronic device 400 may perform keypoint detection on an acquired card image to obtain predicted position information of a plurality of keypoints of a target card in the card image, determine the integrity of the target card included in the card image based on the predicted position information of the plurality of keypoints of the target card, and then determine whether to store the card image based at least in part on the integrity of the target card included in the card image, so as to detect the integrity of the image, and improve the accuracy of image recognition and information extraction.

The present application also provides a computer storage medium storing a computer program that causes a computer to execute part or all of the steps of any one of the image processing methods described in the above method embodiments.

The present application also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform part or all of the steps of any one of the image processing methods as described in the method embodiments above.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, such as the division of the units, merely a logical function division, and there may be additional manners of dividing the actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, or may be in electrical or other forms.

The units (modules) described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a memory, including several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. And the aforementioned memory includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Those of ordinary skill in the art will appreciate that all or a portion of the steps in the various methods of the above embodiments may be implemented by a program that instructs associated hardware, and the program may be stored in a computer readable memory, which may include: flash disk, read-only memory, random access memory, magnetic or optical disk, etc.

The foregoing has outlined rather broadly the more detailed description of embodiments of the present application, wherein specific examples are provided herein to illustrate the principles and embodiments of the present application, the above examples being provided solely to assist in the understanding of the methods of the present application and the core ideas thereof; meanwhile, as those skilled in the art will have modifications in the specific embodiments and application scope in accordance with the ideas of the present application, the present description should not be construed as limiting the present application in view of the above.

Claims

1. An image processing method, the method comprising:

determining the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card; the determining the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card comprises: determining whether a plurality of key points are positioned in the card image based on predicted position information of the key points of the target card and boundary position information of the card image; determining the integrity of the target card contained in the card image based on whether the plurality of keypoints are located within the card image;

The determining the integrity of the target card contained in the card image based on whether the plurality of keypoints are located within the card image comprises: obtaining a minimum distance from at least one first keypoint of the plurality of keypoints to a boundary of the target card in response to the at least one first keypoint not being within the card image; determining that the target card contained in the card image is missing in response to the at least one first key point having a minimum distance from the boundary of the target card greater than a preset threshold; determining that the target card contained in the card image is complete in response to the minimum distances from the at least one first key point to the boundary of the target card being less than or equal to the preset threshold;

2. The image processing method of claim 1, wherein the plurality of key points of the target card comprise a plurality of vertices of the target card.

3. The image processing method according to claim 1 or 2, characterized by further comprising, before the determining of the integrity of the target card contained in the card image based on the predicted position information of the plurality of key points of the target card:

the determining whether to store the card image includes:

and determining whether to store the card image after the transfer processing.

4. The image processing method of claim 1 or 2, wherein the determining whether to store the card image based at least in part on the integrity of the target card contained in the card image comprises:

5. The image processing method according to claim 1 or 2, characterized in that the method further comprises:

6. The image processing method according to claim 1 or 2, characterized in that the method further comprises:

7. The image processing method according to claim 1 or 2, characterized by further comprising, before the performing of the key point detection on the acquired card image:

8. The image processing method according to claim 1 or 2, wherein the target card comprises one or any combination of an identification card, a bank card, a passport, a driver's license, a travel license, and a social security card.

9. An image processing apparatus, comprising: the system comprises a prediction module, an integrity detection module and a storage module, wherein:

the prediction module is used for detecting key points of the acquired card image and obtaining predicted position information of a plurality of key points of a target card in the card image; the integrity detection module is specifically configured to: determining whether a plurality of key points are positioned in the card image based on predicted position information of the key points of the target card and boundary position information of the card image; determining the integrity of the target card contained in the card image based on whether the plurality of keypoints are located within the card image;

The integrity detection module is also specifically configured to: obtaining a minimum distance from at least one first keypoint of the plurality of keypoints to a boundary of the target card in response to the at least one first keypoint not being within the card image; determining that the target card contained in the card image is missing in response to the at least one first key point having a minimum distance from the boundary of the target card greater than a preset threshold; determining that the target card contained in the card image is complete in response to the minimum distances from the at least one first key point to the boundary of the target card being less than or equal to the preset threshold;

10. The image processing device of claim 9, wherein the plurality of keypoints of the target card comprise a plurality of vertices of the target card.

11. The image processing apparatus according to claim 9 or 10, further comprising a turning module for:

12. The image processing apparatus according to claim 9 or 10, wherein the storage module is specifically configured to:

13. The image processing apparatus of claim 12, further comprising an identification module configured to perform text recognition on the card image, and determine a text recognition result of the target card;

14. The image processing apparatus according to claim 9 or 10, further comprising an output module configured to output, in response to determining that the target card contained in the card image is missing, first hint information for use in hint at least one of missing the target card contained in the card image, missing position of the target card, and adjustment of placement position of the target card.

15. The image processing apparatus according to claim 9 or 10, further comprising a prompt module configured to issue a second prompt message to prompt a user to place the target card in a specified area before the prediction module performs the keypoint detection on the acquired card image.

16. The image processing apparatus according to claim 9 or 10, wherein the target card comprises one or any combination of an identification card, a bank card, a passport, a driver's license, a travel license, a social security card.

17. An electronic device comprising a processor and a memory for storing a computer program configured to be executed by the processor for performing the method of any of claims 1-8.

18. A computer readable storage medium for storing a computer program, wherein the computer program causes a computer to perform the method of any one of claims 1-8.