KR102319492B1 - AI Deep learning based senstive information management method and system from images - Google Patents

Abstract

The present invention provides a personal information processing system using AI deep learning and a personal information processing method using the same, wherein the method comprises: a preprocessing step in which a preprocessing unit extracts an image file from a target file, detects an area of interest that is determined to contain personal information in the extracted image file, and performs processing for improving image recognition rate for the region of interest; a personal information extraction step in which a personal information area extraction unit extracts processing target information including personal information from the area of interest, detected by the preprocessing unit, using a CNN neural network; a pattern analysis step in which a pattern analysis unit analytically compares a pattern of the personal information, extracted by the personal information area extraction unit, with a preset personal information pattern to extract personal information of the processing target information; and a de-identification processing step in which a de-identification processing unit de-identifies the personal information extracted by the pattern analysis unit. As described above, it is possible to effectively detect and de-identify the personal information, retained by an individual or an institution, by analyzing text and image information included in the image file using the artificial intelligence neural network, then detecting the personal information (personal identification number information or fingerprint information, etc.) included therein and extracting the same, thereby effectively protecting the personal information.

Description

Personal information processing system using AI deep learning and personal information processing method using the same {AI Deep learning based senstive information management method and system from images }

The present invention relates to a personal information processing system using deep learning, and more specifically, by analyzing text and image information included in an image file using an AI artificial intelligence neural network, and then detecting and extracting the personal information included therein. It relates to a personal information processing system using deep learning that can effectively detect and de-identify personal information held by individuals or institutions.

In general, due to the development of computers, tasks performed within individuals or companies are being computerized, and the resulting data is shared or disclosed through a network. On the other hand, in the event that such data is leaked by hacking or due to a management mistake of the company (organization), files containing a large amount of personal information are disclosed at home and abroad, and the company or institution has a serious impact on external credibility. There is a problem with giving, and it is necessary for individuals, companies, and institutions to take countermeasures to prevent personal information from being exposed.

Accordingly, as an example of a technology for preventing the above-mentioned personal information exposure, Korean Patent Registration No. 10-1734442 is a method of protecting personal information in transacting goods using a posting on an Internet bulletin board. A step of requesting upload to a web server of a bulletin board, wherein the post includes a post header and a post body; extracting, by the web server, personal information included in the body of the post; storing the extracted personal information in a personal information database; and storing the post body in which the extracted personal information is replaced with a link in a body database.

On the other hand, recently, due to the popularization of mobile phones and the generalization of photography, many files in the form of images or videos are generated rather than text files. is the trend

However, as described above, in spite of the increasing trend of image files, the conventional technique for preventing personal information exposure is a method of extracting text, so it is difficult to detect personal information included in the image file on the server. However, it was not effective in de-identifying it, and there was a problem in protecting the personal information included in the image file.

Korean Patent Registration No. 10-1734442

The present invention uses an artificial intelligence neural network to effectively detect and de-identify personal information possessed by individuals or institutions by analyzing text and image information included in an image file and then detecting and extracting the personal information included therein. The purpose of this is to provide a personal information processing system using deep learning that can effectively protect personal information and a personal information processing method using the same.

According to one aspect of the present invention, the present invention extracts an image file from a target file, detects a region of interest that is determined to contain personal information from the extracted image file, and performs processing for improving the image recognition rate for the region of interest. a pre-processing unit to perform; a personal information region extraction unit for extracting processing target information including personal information from the region of interest detected by the pre-processing unit; a pattern analysis unit for extracting personal information of the processing target information by comparing and analyzing the pattern of the processing target information extracted by the personal information area extraction unit with a preset personal information pattern; It is possible to provide a personal information processing system using deep learning, characterized in that it comprises a de-identification processing unit for de-identification processing of the personal information extracted by the pattern analysis unit.

Here, the pre-processing unit includes: a template matching comparison unit that compares the image file with set templates and classifies the image according to the possibility of including personal information; a region detection unit for detecting the region of interest through a region detection algorithm in which a personal information inclusion format is learned by an artificial neural network from the image file selected by the template matching and comparing unit; and an image correction unit for geometrically correcting the image of the region of interest.

The template matching and comparison unit is configured to classify the image file into a formal format and an atypical format according to a template matching index of the image file and the templates, respectively, and the matching comparison algorithm according to the format matching index is a form It can be learned through an artificial neural network (CNN, Convolution Neural Network) for concordance index learning.

The set templates may be composed of templates of regular forms including personal information including driver's license, identification card, visa or foreigner's certificate.

The region detection algorithm detects a region of interest based on a feature point for determining a vertex, a corner, or a line included in the image file, but using a Rectangular Corner Localization (RCL) neural network. It may be configured to find an optimal ratio of the judgment weights for the feature points.

The region detection algorithm may be configured to convolutionally multiply a pixel value matrix of the image file with a gradient derivative matrix to detect a box shape.

The region detection algorithm may be configured to predict a vertex of a rectangle per entity, and detect the region of interest by determining whether a point of view is matched, rotated, or missing a vertex with respect to the region of interest image.

The singi region detection unit may be configured to detect a box shape by connecting the features composed of two corners, three line segments, or four vertices.

The region detector may be configured to extract a contour of the image file by using an image morphological filter in the image file so that the image characteristic of the region of interest is emphasized.

The image compensator includes a viewpoint correction module that performs perspective transformation so that a viewpoint of the region of interest image is positioned in a vertical direction so that the image of the region of interest forms a plane, and an inclined angle of the region of interest image. It may be configured to include a rotation correction module for rotation correction (rotation transform).

The pre-processing unit includes: an image-array unit for assigning a connection identification number to each of the image files extracted from the target file; It may be configured to further include; an image improvement unit for removing the quality impairing factors of the image file.

The connection identification number may include a file name, a page, and an image name for each of the image files so that the location information and conversion information of the image file can be confirmed.

The image improvement unit may be configured to remove quality-deteriorating factors including noise, blur, or occlusion of the image file through an image improvement algorithm learned through a CNN neural network.

The personal information region extraction unit may be configured to extract the processing target information from the region of interest through an extraction algorithm learned through a CNN neural network.

The processing target information may be composed of personal information composed of text.

The personal information area extraction unit is configured to detect a text unit including text in the area of interest through an extraction algorithm learned through a CNN neural network, and extract a text string from the text unit, a predetermined length in the area of interest An image having a height of and a region in which a space is continuous may be detected by the text unit.

The personal information area extraction unit may be configured to store the corresponding position and coordinates of the writing unit, and to store and manage the connection identification number data of the corresponding image file including the area of interest.

The processing target information may be composed of personal information composed of a fingerprint image or a face image.

The personal information region extraction unit is configured to detect an image pattern in the region of interest through an extraction algorithm learned through a CNN neural network, and extract a fingerprint image or a facial image from the image pattern, connected to a feature point in the region of interest It may be configured to detect a pattern region having a fingerprint pattern or a face pattern to be used as the image pattern.

The pattern analysis unit, a personal information pattern analysis unit that compares and analyzes the text with a preset personal information pattern including a resident number, a driver's license number, a passport number and a foreigner number and a fingerprint, and a resident registration card, a driver's license, a passport , and a document form analysis unit that compares and analyzes a preset document format including a residence certificate and an alien certificate, wherein the preset personal information pattern may be generated by a personal information generation logic.

The personal information pattern analysis unit and the document form analysis unit may be configured to compare and analyze the processing target information through a pattern analysis algorithm learned through a CNN neural network.

The document form analysis unit, if the personal information text extracted from the region of interest does not show complete correspondence with the personal information pattern, but has a high degree of similarity, rechecks whether the form matches the pattern through pattern matching.

The de-identification processing unit may be configured to process the text in any one or more manners of mosaic processing, hiding processing, encryption processing, pseudonymization processing, total processing, data deletion, data categorization, and masking.

Meanwhile, when it is determined that the target file has a detection region of a fingerprint image or a facial image, the region detection unit may apply a correction value to the detection algorithm so that the detection frequency of the region of interest increases.

According to another aspect of the present invention, a preprocessor extracts an image file from a target file, detects a region of interest that is determined to contain personal information from the extracted image file, and improves the image recognition rate for the region of interest. a pre-processing step of performing processing; a personal information extraction step of extracting, by a personal information region extraction unit, processing target information including personal information from the region of interest detected from the pre-processing unit using a CNN neural network; a pattern analysis step in which a pattern analysis unit compares and analyzes the pattern of the personal information extracted by the personal information area extraction unit with a preset personal information pattern to extract the personal information of the processing target information; It is possible to provide a personal information processing method using deep learning, characterized in that the de-identification processing unit includes a de-identification processing step of de-identifying the personal information extracted by the pattern analysis unit.

The pre-processing step includes: an image array step in which an image-array unit assigns a connection identification number to each of the image files extracted from the target file; an image improvement step in which an image improvement unit removes a quality degrading factor of the image file; The template matching and comparison unit compares the image file with set templates composed of template groups of regular forms including personal information including driver's license, identification card or foreigner's certificate, and compares the image according to the possibility of including personal information. step; detecting, by a region detection unit, the region of interest from the image file selected by the template matching and comparing unit, through a region detection algorithm in which a personal information inclusion format is learned by an artificial neural network; The image correction unit may include a correction step of correcting the viewpoint and rotation of the image of the region of interest.

The region detection unit, when the image file is classified into a fixed format by the template matching and comparing unit, uses an image morphological filter to convert the image file into the image file so that the image characteristic of the region of interest is highlighted. may be configured to extract the contour of

The region detection algorithm detects a region of interest based on a feature point including a vertex, a corner, or a line included in the image file, but using a Rectangular Corner Localization (RCL) neural network. It may be configured to find an optimal ratio of judgment weights for the feature points.

The image improvement unit removes quality-deteriorating factors including noise, blur or occlusion of the image file through an image improvement algorithm learned through a CNN neural network, but the quality-degrading pixel ( pixel) can be trained to find the removal value.

The template matching and comparison unit divides the image file into a formal format and an atypical format according to a template matching index of the image file and the templates, respectively, and the matching comparison algorithm according to the format matching index is a format matching index It can be learned through an artificial neural network (CNN, Convolution Neural Network) for learning.

The processing target information is personal information composed of text, and the personal information area extraction unit detects a text unit including text in the area of interest through an extraction algorithm learned through a CNN neural network, and the text from the text unit It may be configured to extract a character string, and it may be configured to detect a region in which an image having a predetermined length and height and a space are continuous in the region of interest as the writing unit.

The processing target information is personal information composed of a fingerprint image or a facial image, and the personal information region extraction unit detects an image pattern in the region of interest through an extraction algorithm learned through a CNN neural network, and a fingerprint from the image pattern. Doedoe configured to extract an image or facial image, it may be configured to detect a pattern region having a fingerprint pattern or facial pattern connected to a feature point in the region of interest as the image pattern.

The pattern analysis unit, through a pattern analysis algorithm learned through a CNN neural network, analyzes the personal information pattern to compare and analyze the text with a preset personal information pattern including resident number, driver's license number, passport number and foreign number and fingerprint and a document form analysis unit that compares and analyzes the text with a preset document form including a resident registration card, a driver's license, a passport, a residence certificate, and an alien certificate.

The de-identification processing unit may be configured to process the text in any one or more manners of mosaic processing, hiding processing, encryption processing, pseudonymization processing, total processing, data deletion, data categorization, and masking.

Meanwhile, when it is determined that the target file has a detection region of a fingerprint image or a facial image, the region detection unit may apply a correction value to the detection algorithm so that the detection frequency of the region of interest increases.

The personal information processing system using deep learning and the personal information processing method using the same according to the present invention can provide the following effects.

First, after analyzing the text and image information included in the image file using an artificial intelligence neural network, it detects and extracts the personal information (personal identification number information or fingerprint information, etc.) included in the image file. By effectively detecting and de-identifying personal information, personal information can be effectively protected.

Second, by using an artificial intelligence neural network, the image file is classified into atypical form and formal form by comparing the form according to the form index standard of the setting template, and processing accordingly can shorten the detection time of personal information contained in the image file. have.

Third, the detection performance can be improved by using the artificial intelligence neural network to determine the target position (ROI), which is determined to have a region of interest including text, in the image file.

Fourth, the location of the text box is identified by extracting the geometric singularity of the image file, and the identified area of interest can be accurately de-identified using a transformation matrix according to the setting criteria. However, it can effectively detect and de-identify personal information held by institutions.

1 is a diagram showing an environment configuration of a personal information processing system using deep learning according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a personal information processing system using deep learning according to the present invention.
3 is a view showing a terminal screen when the template matching comparison unit constituting the personal information processing system according to the present invention determines the form similarity as an index score.
4 is a diagram showing image correction by the image correction unit constituting the personal information processing system according to the present invention and a matrix therefor.
5 is a diagram showing an embodiment of a pre-processing unit processing method constituting a personal information processing system according to the present invention.
6 is a diagram showing a region detection method of the region detection unit constituting the personal information processing system according to the present invention.
7 is an exemplary view showing an example of the viewpoint correction and rotation correction processing in the pre-processing unit constituting the personal information processing system according to the present invention.
8 is a diagram showing a text extraction process by the personal information area extraction unit constituting the personal information processing system according to the present invention.
9 is a view showing a personal information pattern extraction process by the pattern analysis unit constituting the personal information processing system according to the present invention.
10 is a diagram illustrating a de-identification process after extracting personal information using the personal information processing system according to the present invention.
11 is a diagram illustrating an example of de-identification processing for a fingerprint by a de-identification processing unit in the personal information processing method according to the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, the personal information processing system (hereinafter referred to as 'personal information processing system') using deep learning according to an embodiment of the present invention examines image files in personal PCs or file servers owned by institutions or individuals to obtain personal information The purpose is to block the leakage (upload/download). Meanwhile, at this time, the types of files used by individuals are various, such as PDF, image files, and document editor (word/Hwp) files. In the present invention, various types of image files including OLE image files are targeted.

First, FIG. 1 shows the configuration of a network/server environment for image file detection according to the present invention. Referring to the drawing, the scan server finds the internal PCs while the scan server runs the scan server with personal information detection logic. A number of PCs are used in institutions such as government offices, and the internal network is connected to the external Internet. In the case of a structure in which packets are sent out through the gateway device, it can be divided into upload direction detection and download direction detection according to the type of query to the web server. In general, each corresponds to a file upload/download method divided into a get/post method.

2 is a block diagram showing the configuration of the personal information processing system of the present invention. Referring to the drawings, the personal information processing system may include a pre-processing unit 100 , a personal information area extraction unit 200 , a pattern analysis unit 300 , and a de-identification processing unit 400 . have.

First, the pre-processing unit 100 is a page unit or an image unit (Region of Interest (ROI) that is expected to contain personal information, hereinafter referred to as a Text Box or Text Box area) It plays a role in removing noise, blur, occlusion, etc., and the preprocessing process represents the process before extracting personal information text from the text box.

That is, the pre-processing unit 100 extracts an image file from the target file, detects a region of interest that is determined to contain personal information from the extracted image file, and performs processing for improving the image recognition rate for the region of interest. plays a role

Specifically, the pre-processing unit 100 includes an image-array unit 110 , an image improvement unit 120 , a template matching index comparison unit, an area detection unit 140 , and a geometric It may be configured to include a conversion unit 150 .

The image array unit 110 serves to assign a linking address to each of the image files extracted from the target file. Specifically, the image array unit 110 performs an image separation operation for each page of an image file consisting of a plurality of pages, separates the OLE object, and connects it in a file unit/page unit/image unit (Box region) Assign an identification number.

The connection identification number may be configured to include a file name, a page, and an image name for each of the image files so that the location information and conversion information of the image file can be checked, and the text extraction of the personal information area extraction unit 200 Additional information obtained in the process (Text Box location information and transformation angle information) is added and completed, and it is composed of image location information and conversion information.

The image improvement unit (pre-processing operation unit, 120) may be configured to remove quality-deteriorating factors through an image improvement algorithm learned through a CNN neural network. That is, the image improvement unit 120 removes noise of the image file through a noise removal algorithm, removes blur through a blur processing algorithm, and occlusions through an occlusion processing algorithm. can be processed

The image improvement unit 120 removes the quality-deteriorating factor through an image improvement algorithm learned through the CNN neural network in order to remove the page-by-page noise according to a threshold value, and the CNN neural network removes the quality-degrading element pixel It can be learned to find the removal value of (pixel).

The template matching and comparing unit 130 compares the image file with set templates and classifies the image according to the possibility of including personal information.

The template matching comparison unit 130 may apply the matching comparison algorithm learned through the CNN neural network in order to index the similarity by comparing it with an existing registration template through deep learning learning.

The template matching and comparing unit 130 processes these image files in a standard form so as to reduce the detection speed of image files including the existing registration form. The template matching and comparison unit 130 applies the matching comparison algorithm learned through the CNN neural network in order to convert the binarization processing unit into a black-and-white image in the case of a stereotypical image, and then transmits it to the viewpoint correction unit and the rotation correction unit. On the other hand, the template matching comparison unit 130 transmits the atypical image file to the area detection unit 140 which is the next process.

The template matching and comparing unit 130 is configured to classify the image file into a formal format and an atypical format according to a template matching index of the image file and the templates, respectively, and matching according to the format matching index The comparison algorithm can be learned through a convolutional neural network (CNN) for template matching index learning.

The set templates may be composed of templates of regular forms including personal information including driver's license, identification card, visa or foreigner's certificate.

Hereinafter, the template matching and comparing unit 130 will be described in detail. First, assuming that the template matching comparison unit knows in advance that the target image file to be inspected is, for example, an A4-sized image from which a driver's license is copied, the preparation time for the pre-processing operation can be drastically reduced. In other words, when an image file of the corresponding format (driver's license) is accepted as an input file by an artificial neural network completed through deep learning for a specific form (driver's license), the region of interest to be extracted more quickly (Text Box area) can be determined, and text detection can be performed.

FIG. 3 is a diagram illustrating an embodiment of the template matching and comparing unit, and shows a terminal screen when the form similarity of the template matching and comparing unit 130 is determined by the form matching index score. Referring to the drawing, the artificial neural network that has learned the four forms can determine the similarity of which form the input image file corresponds to as an exponential score (3543.6980/972/1917), and in this case, template_2 with the highest similarity score The format of the input file can be inferred as the format (corresponding to the index score of 6980). At this time, the neural network has learning data (class) for 4 types of forms, so if the corresponding form is detected in the image file, it is judged as “standard form” to be judged as

As described above, the template matching and comparing unit 130 classifies a template that has been previously through neural network learning into a formal form, and has this template information as a class of neural network learning information in advance. Compared to atypical forms that do not have, there is an advantage in that the detection speed can be processed quickly.

1) The region detection unit 140 detects the region of interest through an ROI extraction algorithm in which the personal information inclusion format is learned by an artificial neural network from the image file selected by the template matching and comparing unit 130 . do

The region of interest extraction algorithm detects a region of interest based on a feature point for determining a vertex, a corner, or a line included in the image file, but using a Rectangular Corner Localization (RCL) neural network. It may be configured to find an optimal ratio of judgment weights for the feature points.

The region detection algorithm may be configured to convolutionally multiply a pixel value matrix of the image file with a gradient derivative matrix to detect a box shape.

The region detection algorithm may be configured to predict a vertex of a rectangle per entity, and detect the region of interest by determining whether a point of view is matched, rotated, or missing a vertex with respect to the region of interest image.

2) The region detection unit 140 may be configured to detect a box shape by connecting the features composed of two corners, three line segments, or four vertices.

3) The region detection unit 140 may be configured to extract a contour of the image file by using an image morphological filter in the image file so that the image characteristic of the region of interest is highlighted. .

4) When it is determined that there is a detection region of a fingerprint image or a face image in the target file, the region detection unit 140 may apply a correction value to the region of interest detection algorithm to increase the detection frequency of the region of interest. This is because if there is a fingerprint image or facial image in the target file, if personal information related thereto is leaked together, the damage caused by the personal information leakage will be serious. am. Accordingly, when it is determined that the target file has a detection region for a fingerprint image or a face image, the region detection unit 140 may increase the region of interest detection frequency to detect more regions of interest.

As described above, the region detection unit 140 needs to determine a text box, which is a region of interest (ROI) region, in order to extract personal information, and a complete box-shaped contour is required. ), a weight is given to the feature points (features, points, corners, line segments of a certain thickness) of the image, and if the score is higher than a certain point, it is determined as a region of interest.

5) Meanwhile, a basic and effective method for extracting a region of interest by the region detection unit 140 is to find a corner. A corner is a characteristic point that plays an important role in distinguishing geometric shapes, and it can be said that it is the basis for shape recognition. In this case, since the definition of a corner is a part in which an image value (pixel value) changes rapidly in two or more directions, the localization of the corresponding corner can provide an important clue in shape recognition.

6) The region detection unit 140 receives the image file determined as an atypical format from the template matching and comparison unit 130, and attempts to detect a plurality of regions of interest (Text Box) in a page unit, improving the ease of detecting regions of interest. To do this, we use the null detection algorithm learned through the CNN neural network, and detect the feature points using the CNN neural network.

7) The region detection unit 140 transmits location information on the detected region of interest to the image array unit 110 to complete the linking address information.

Next, looking at the correction (conversion) process of the geometric transformation unit,

1) The geometric transformation unit 150 geometrically corrects the image to secure a parallel/vertical point of view of personal information detection, and transfers transformation information to the image array unit 110 for each extracted ROI image. so that it can be converted back to the original location (linking address update operation).

2) The geometric transformation unit 150 includes a viewpoint correction module that performs perspective transformation such that a viewpoint of the region of interest image is positioned in a vertical direction so that the image of the region of interest forms a plane; It may be configured to include a rotation correction module for rotation-correcting the tilted angle of the region image (rotation transform).

3) Here, the geometric transformation unit 150 calculates the rotation correction amount centered on the vertical axis and the horizontal axis on the horizontal plane of the viewpoint correction module and the rotation correction amount centered on the vertical axis on the vertical plane through a correction algorithm. can be calculated.

4 shows image correction by the geometric transformation unit constituting the personal information processing system according to the present invention and a matrix therefor. ) shows the method of viewpoint correction and rotation correction by changing the geometric composition to increase the accuracy of the image.

First, (a) shows viewpoint correction, and viewpoint correction corrects if the target file is tilted according to whether the viewpoint is aligned with the vertical image based on the viewpoint to form a 90 degree angle with the object. A floor plan is obtained, and for this purpose, the current model is transformed into a matrix.

(b) shows rotation correction, and when the image is placed at a constant angle with the horizontal plane due to rotation correction copy, etc., the angle is shifted by x degrees to change to a parallel state without the original angle.

(c) is described as a matrix transformation matrix in which the point of (1.0) is changed to the value of (1, tan@) for this correction. It can be expressed in the same way as a transformation matrix between The result of such a matrix operation is stored for each text box and, if necessary, can be moved back to the original location.

Looking at the processing process by the personal information area extraction unit below,

1) The personal information region extraction unit 200 serves to extract processing target information including personal information from the region of interest detected by the pre-processing unit 100 .

2) The personal information area extraction unit extracts text from the image of the area of interest (Text Box) recommended by the preprocessor 100 .

3) The personal information region extraction unit 200 may be configured to extract the processing target information from the region of interest through an extraction algorithm learned through a CNN neural network.

The processing target information may be composed of personal information composed of text.

4) The personal information region extraction unit 200 is configured to detect a text unit including text in the region of interest through an extraction algorithm learned through a CNN neural network, and extract a text string from the text unit, In the region of interest, an image having a predetermined length and height and a region in which blanks are continuous may be detected by the writing unit.

5) The personal information area extraction unit 200 may be configured to store the corresponding position and coordinates of the writing unit and to store and manage the connection identification number data of the corresponding image file including the area of interest.

The processing target information may be composed of personal information composed of a fingerprint image or a face image.

6) The personal information region extraction unit 200 is configured to detect an image pattern in the region of interest through an extraction algorithm learned through a CNN neural network, and extract a fingerprint image or a facial image from the image pattern, the It may be configured to detect a pattern region having a fingerprint pattern or a facial pattern connected to a feature point in the region of interest as the image pattern.

The pattern analysis unit 300 compares whether the processing target information extracted from the personal information area extraction unit 200 matches the personal information pattern, and compares the processing target information extracted by the personal information area extraction unit 200. It serves to extract the personal information of the processing target information by comparing and analyzing the pattern with a preset personal information pattern.

In this case, the preset personal information pattern may be generated by a personal information generation logic.

The pattern analysis unit 300 includes a personal information pattern analysis unit that compares and analyzes the text with a preset personal information pattern including a resident number, a driver's license number, a passport number, and a foreign number and a fingerprint, It may be configured to include a document form analysis unit that compares and analyzes a preset document form including a driver's license, passport, residence certificate and alien certificate.

The personal information pattern analysis unit and the document form analysis unit may be configured to compare and analyze the processing target information through a pattern analysis algorithm (form index algorithm) learned through a CNN neural network.

The document form analysis unit, if the personal information text extracted from the region of interest does not show complete correspondence with the personal information pattern, but has a high degree of similarity, rechecks whether the form matches the pattern through pattern matching.

The de-identification processing unit 400 serves to de-identify the personal information extracted by the pattern analysis unit 300 .

The de-identification processing unit 400 may be configured to process the text in any one or more manners of mosaic processing, hiding processing, encryption processing, pseudonymization processing, total processing, data deletion, data categorization, and masking.

Hereinafter, a personal information processing method (hereinafter referred to as 'personal information processing method') using deep learning according to an embodiment of the present invention will be described.

First, since the configuration of the personal information processing system used in the personal information processing method of the present invention is as described above, a detailed configuration description thereof will be omitted, and only the personal information processing method of the present invention will be described in detail below. do it with

First, in the personal information processing method, the pre-processing unit 100 extracts an image file from a target file, and detects a region of interest that is determined to contain personal information from the extracted image file, thereby improving the image recognition rate for the region of interest. Perform a pre-processing step to perform processing for

The pre-processing step includes an image array step in which an image-array unit 110 assigns a connection identification number to each of the image files extracted from the target file, and an image improvement unit 120 performs the image The image improvement step of removing the quality impeding factors of the file, and the template matching and comparing unit 130 are set consisting of template groups of regular forms containing personal information including driver's license, identification card or foreigner's certificate for the image file. A template comparison step of classifying images according to the possibility of including personal information compared with templates, and a personal information inclusion format by an artificial neural network in the image file selected by the template matching and comparing unit 130 by the area detection unit 140 It may include detecting the region of interest through the learned region of interest extraction algorithm, and a correction step in which the geometric transformation unit 150 corrects the viewpoint and rotation of the region of interest image.

First, the pre-processing step is a pre-work for the text extraction (personal information extraction process) process, and the process of finding an image file (Box Region) of the region of interest and handing it over to the personal information extraction process (step 2) is shown in FIG. Let's take a look and see.

5, the pre-processing process (step 1) includes an image array unit (image-array work unit), an image improvement unit (Pre-processing work unit), a template matching index comparison unit (template matching index comparison unit), It consists of a region detection unit (Box detection unit) and a geometric transformation unit (Perspective Transform/Rotation Transform processing unit).

First, as an input-type file for personal information detection, image files can be generally divided into two types. An image file is 1) a pure image file, or 2) an OLE (Object Linking Embedded) type in a specific document. It may include a document file included as

It is assumed that the image file presented as a sample in the drawing consists of 3 pages, each page has one image, image_1 and image_2 are images included in the informal format, and image_3 is the image included in the formatted format.

First, when the image file (File_name) is input to the image array unit 110, for identification of all image units to be processed later, the linking address is subdivided into file units/page units/image units (Box Region). is given At this time, the image array unit 110, through the connection identification number, the location information for confirming the location of the image, the conversion information for checking the conversion information of the image, respectively.

Image separation in OLE documents is also carried out through this operation. In the drawing, File_name indicates a case in which a file containing an image is separated by page unit and a link identification number is assigned when an OLE object is separated. However, at the current stage, since the image unit included in the page is not yet identified, the subsequent connection identification number information includes File_name/page_1 / image #??, File_name/page_2 / image #??, File_name/page_3 / image #? Like ?, only page-level information is included.

Then, the image improvement unit (Pre-Processing processing unit) performs pre-processing on a page-by-page basis. By applying the image enhancement algorithm learned through the CNN neural network, using a threshold value for each page, blur carry out processing This is to remove noise and to facilitate the subsequent extraction (detection) of personal information.

Afterwards, the template matching index comparison unit performs a form comparison operation using the matching comparison algorithm learned through the CNN neural network (Template Recognition CNN) in order to shorten the inspection time. In this work, a template form with high similarity (with the highest index score) is recommended to determine whether the page being inspected is a form learned through the neural network. If the similarity scores for all of them are remarkably low, the formal/atypical form is judged based on the threshold. At this time, if it is determined that the format is a regular format, it is transmitted to the binarization processing unit to shorten the inspection time. In the drawing, page_3 is the standard form.

Then, in the area detection unit (Box detection unit), using the area detection algorithm learned through the CNN neural network, three features of a rectangular edge, a line of constant thickness, and a point are detected. extract In order to determine the region of interest (Text Box) region, the region detection unit 140 may assign weights for each item based on feature points even when the region of interest is not a complete rectangular region of interest made of line segments and points, and obtains a certain score or more In one case, it can be recommended as a text box. This process is applied to the process for extracting figures of a model consisting of four sides, such as a parallelogram and a trapezoid, in addition to a rectangle. This is a method that can be used in various IDs, forms, and certificates, which are considered to be known as areas of interest (text areas) to some extent rather than a process of finding a general contour. That is, it is determined by using a score in which weights are assigned to a point (vertex) where two lines meet, a constant thickness of an outline, a discontinuous boundary inside and outside a region, and the like. If possible, it is desirable to extract four points per box of interest, but even if it consists of three points and an edge line, it can be assumed as one box of interest.

The area detection unit 140 extracts three feature points of a corner of a rectangle, an edge line of a certain thickness, and a point and assigns weights to them, but weights for the corners among the three elements A rectangle can be identified by making it the largest. As an example of this, by applying the region detection algorithm learned through the CNN neural network to find the region of interest (Text Box), two edges or three line segments ( line) or a rectangle composed of 4 points can be recommended, and for the detection of rectangles including parallelograms and trapezoids, dx/dy (gradient derivative matrix) in the pixel value matrix of the original image file By convolution of , parallelograms and rhombic rectangles can also be recommended/detected in the form of a text box.

The region detection unit 140 may finally perform binary processing on the recommended image (Box region) of the region of interest, and facilitate the text detection operation performed in the second stage operation.

As shown in Table 1 below, when the sum of the weights is equal to or greater than a certain value, an embodiment is shown in which the region of interest (Box) is determined.

item detection number weight vertex 4 5 Corner 2 10 Edge Line 2 2

Referring to FIG. 6 , if the rectangle of the region of interest to be detected is in a state in which the rectangular shape in the image is stably maintained without relatively large deformation (distortion or rotation), the three cases shown in FIG. In the case of falling, three line segments, or four vertices, it is considered as a condition that satisfies the minimum geometric rectangle, and is recommended as a text box. In addition, the shape and position of corners and edges of the image can be identified through convolution on the original image using the gradient derivative matrix as shown below. corner) part and edge can be processed by applying kernel filters in the dx and dy directions to the pixel value of image extracted from the image file, respectively.

Here, dx and dy are gradient derivatives matrices that calculate the rate of change of pixel values in the x and y directions, respectively. A kernel filter is a basic filter that processes a convolution operation, and a geometric configuration that satisfies the three conditions as shown in FIG. 6 can be found by using the convolution result for the dx and dy components. That is, if only the dx result or the dy result is satisfied, it means that a line segment in the x-axis or y-axis direction exists in a certain section. .

When the image of the original file is displayed as Matrix "M", when the Matrix shape of the image converted by applying the Kernel filter (gradient derivatives matrix) above is displayed as M`, the size (Magnitude) and angle 'alpha (theta)' )' can be expressed as follows.

When applied as above, there is an advantage in that the theta value can be used to detect not only rectangles, but also parallelograms and rhombic rectangles.

In the drawing, if there is one image in each of page_1 and page_2, the atypical form processed by the area detection unit 140 is finally a connection identification number ( linking address). And this information is directly transferred to the image array unit 110 to update the image array (Image-array) information.

When the feature points are extracted by the region detection unit 140, a region of interest (Text Box) can be recommended (defined) based on these feature points, and binarization processing is performed on a page-by-page basis including a plurality of recommended regions of interest (Text Box) in the page. do the work When the binarization process is completed, the image area within the region of interest is transferred to the next task for extracting personal information text. In this way, when extracting a text box, a corresponding image file can be simplified by changing the color of a pixel to a gray scale for parts other than the text box. In this way, a contour is completed from the detected feature points to determine a text box, and other areas are completely excluded from the detection target area.

The geometrical transformation unit 150 performs geometric correction on the image unit (Box region) of the region of interest extracted by the region detection algorithm in the viewpoint correction module and the rotation correction module.

7 is an exemplary view showing an example of the viewpoint correction and rotation correction processing in the preprocessor constituting the personal information processing system according to the present invention, and shows the execution process of the preprocessor 100 according to the shape of the image. Referring to the drawings, the image is divided into A-type (slanted), B-type (trapezoidal, caused by not maintaining the correct angle of incidence 90 degrees during photography or scanning), or normal C-type (normal 90-degree quadrilateral) can be In the case of type A, after detection, compensation is made as much as the inclination angle through the rotation compensation module, and after moving the angle to the original parallel position, it moves to the second step process. In the case of type B, after detection, the viewpoint correction module converts the actual tilted viewpoint to gaze vertically, and then moves to the second step process. In the case of type C, the normally detected image of the region of interest does not undergo additional transformation and proceeds directly to a two-step process. As in the previous process, this process is subdivided into file unit → page unit → image unit and has a linking address. save the information The connection identification number is managed so that it can be used during the box masking of the de-identification processing unit 400 later.

The following shows an embodiment of managing the connection identification number. The coordinates of four corner points are taken for the position of the image unit (Box region) of the region of interest, and when the region of interest (Text Box) is rotated, the page angle is If you record the value in radians as a reference, File_name/page_1/image_#11/(x11,y11),(x12,y12),(x13,y13),(x14,y14)/(z1 rad) and File_name It can be expressed as /page_2/image_#21/(x21, y21), (x22, y22), (x23, y23), (x24, y24)/(z2 rad).

Referring to FIG. 7 , when there is no template matching index comparison unit of FIG. 5 , all of them are regarded as atypical forms, and image pre-processing and personal information extraction are performed. In response to A-type, B-type, and C-type geometric images, the perspective correction module and rotation correction module designate and manage the matrix transformation unit in which the area of interest (Text Box) is converted based on the page, and when masking, After converting to the original position using the transformation matrix, the operation is performed.

After going through the pre-processing process, the personal information region extraction unit 200 uses the extraction algorithm learned through the CNN neural network to process target information including personal information in the region of interest detected from the pre-processing unit 100. It goes through the personal information extraction step.

The personal information extraction step may be configured including a text unit detection unit, a text box pre-processing unit, and an OCR processing unit, and the extraction algorithm learned through the CNN neural network may be applied to each process. Here, the extraction algorithm learned through the CNN neural network can use the OCR engine in case of text region detection, and in the case of fingerprint data, it can be learned by creating a lot of data by its own data generation technique.

Referring to FIG. 8 , after receiving the text box information (connection identification number, image), which is the result of the first step of the text unit detection unit 400, the internal area of all areas of interest (Text Box), When an image with a certain length and height and a space are continuous, it is recognized as a text image bulk and a CNN neural network that determines that a text unit is included in the corresponding image unit (Box Region) is applied.

The box preprocessing unit performs preprocessing by applying a CNN neural network for ease of text detection before the OCR processing unit step after a text unit is detected in the text box, and the text unit is detected in the area of interest (Text). Box), blur processing is performed by applying a threshold value, and quality-degrading pixels are removed.

The OCR processing unit detects a character string by using an extraction algorithm for text detection with respect to the preprocessed region of interest (Text Box).

The text unit detection network neural network 400 is a CNN neural network responsible for the process of first checking the existence of a string in the recommended box before extracting the text inside the box among several recommended areas of interest (Text Box). use The CNN neural network performs the task of finding word units of a certain height and spaces between text units. Currently, it is called a text unit, not a character detection, because only geometric lines and a certain height of the image are checked. That is, as shown in FIG. 8 , an image of a region of interest (Text Box) expected to contain text that has undergone a geometric correction step through the first step is recommended and transmitted to the text unit detection neural network 400 . The corresponding region of interest (Text Box) may include text or may not include text. When the text box information (connection identification number, image) is provided and there are consecutive images and spaces with a certain length and height in the text box, it is recognized as an image containing text and the corresponding interest It is judged that the text unit is included in the area. In the figure, a text unit was detected in only one Text Box in the text unit detection neural network 400 among the three areas of interest (Text Box) recommended in step 1, and it was determined that the text box contained only the text box (here, string " AB E ").

Thereafter, in order to accurately detect a character string in the corresponding region of interest (Text Box), a preprocessing operation is performed only on the corresponding region of interest (Text Box). The box preprocessor uses an image enhancement algorithm learned through a CNN neural network that can remove the blur part of the image or unnecessary pixels by using a threshold value. Afterwards, the pre-processed Box image (image including the "AB E" string) is input to the OCR neural network to extract the correct text for the corresponding string in the Box. In the OCR neural network, 1. the correct character string is extracted from the text unit (“AB E”), 2. the corresponding position (coordinate) of the detected character string is stored in the linking address, and masking processing is performed. I manage data for original image verification work.

Then, a pattern analysis step in which the pattern analysis unit 300 compares and analyzes the pattern of the personal information extracted by the personal information area extraction unit 200 with a preset personal information pattern to extract the personal information of the processing target information go through

Referring to FIG. 9 , the character string (the character string "AB E" in the drawing) detected through the previous OCR neural network goes through 1. a personal information pattern detection process and 2. a document form detection process, respectively. The reason for the document form detection process separate from the personal information pattern detection step is as follows. For example, if there are 12 digits detected in the personal information pattern detection process (the original resident number is 6+7 digits = 13 digits), due to one number that was not detected due to a problem of resolution, the pattern was detected failure may occur. However, if it can be determined that the document form is a resident registration card even if the pattern detection fails in this way, it can be concluded that the 12 detected numbers in the box area were not detected because one number that should be included was not detected. Accordingly, the personal information pattern detection unit 310 and the document form detection unit 330 perform complementary roles as described above.

In the present invention, through the pattern analysis algorithm learned through the CNN neural network (Template Recognition Convolution Neural Network) to enable such document form extraction, the matching comparison learned through the CNN neural network of the template matching and comparing unit 130 used in step 1 algorithms can be used.

The personal information pattern analysis process (three-step operation) by the pattern analysis unit 300 may include a personal information pattern detection unit 310 and a template matching index registration form detection unit. The personal information pattern detection unit 310 applies the personal information pattern form to the character string extracted from the OCR processing unit of the text extraction process to determine whether resident registration number, driver's license number, foreigner number, and passport number are included. Registration form detection unit, using a pattern analysis algorithm learned through CNN (Template Recognition CNN) that can determine a specific document / certificate form of the pre-processing step (step 1), in the personal information pattern detection unit 310 of a specific pattern When personal information extraction fails, it can be inferred whether or not personal information is included in comparison with the result of the personal information pattern detection unit 310 complementary to each other.

After the above process, the de-identification processing unit 400 undergoes a de-identification processing step of de-identifying the personal information extracted by the pattern analysis unit 300 .

In this regard, regarding personal information found after extraction of personal information, if the personal information is used without permission or is stored even after the period of use has elapsed, measures to de-identify the personal information so that a third party cannot verify it is needed The de-identification processing can be divided into pseudonymization processing/total processing/data deletion/data categorization/masking, etc. In the present invention, the process of masking when general personal information is detected will be described.

Step 3 When personal information is found while completing the personal information pattern extraction process, the process of de-identifying (masking) by checking the de-identification option for the personal information string in the included area of interest (Text Box) An example will be described with reference to FIG. 10 .

Referring to the drawings, first 1) when personal information is detected in step 3, related information (image data / location information) is transmitted to the masking processing unit 800 . Then, 2) Masking processing unit masks the image information of the detected personal information string in the text box according to the processing option. Thereafter, 3) the image array unit 110 is queried for the location information (Linking address) of the image to be de-identified to check whether there is conversion information. At this time, 4) If there is conversion information, it is converted to the original location using conversion matrix information, and the image file is saved, and 5) Linking address information is updated.

On the other hand, since the fingerprint is also included in the personal information that can identify an individual, the processing target information may be personal information composed of a fingerprint image or a face image. 11 shows a case of mosaic processing/hidden processing when processing target information is a fingerprint image.

According to the above, the personal information processing method using deep learning of the present invention, when an image comes in (various formats: png, jpg, tif, multipage tif, pdf, ole (such as images in docx, etc.), these images converted into a processable image format.

With respect to the image extracted above, it is first checked whether there is a 'form'. Here, 'form' means a rectangular shape. Most ID cards, passbook scans, passport scans, and certain forms have a rectangular shape, so in order to increase the accuracy of processing, a square is found and extracted separately.

The extraction of the 'form' is made through a CNN neural network (Rectangular Corner Localization neural network), and this rectangular corner localizer is configured to be extracted even if various viewpoints are entered into the rectangle, rotated, or one vertex is not visible. On the other hand, if a rectangular area is found, perspective transform is performed and the rectangular area is extracted as a separate flat image. At this time, the homography matrix used for perspective transform can be used again when making things like masking or preview using the inverse matrix when doing post-processing later.

Thereafter, the degree of rotation of the image extracted from the rectangular region and the entire image is mitigated through a rotation correction neural network. In addition, in detecting the text area, the rotation correction neural network can be trained so that rotation is completely invariant, but the rotation correction neural network can be trained to be standardized to allow some rotation and prevent excessive rotation. can

On the other hand, considering the probability that the image from which the rectangular region is extracted may be incorrectly rotated by 90 degrees, 180, 270, etc., a neural network that rotates in the forward direction is easy.

Such a neural network is a neural network that simply receives an image input and predicts how many degrees it is rotated counterclockwise from the horizontal. Then, the 'text area' is detected in the image extracted from the rotation-relaxed rectangular area and the original whole image. Specifically, text area detection can be done in two ways, and those that can be run on the server are provided in the form of an API as a heavy neural network, and the method of running the neural network by calling the API (PV, server scan, etc.) For local solutions, it is a method of running a neural network directly embedded in a solution with a neural network that uses a light backbone neural network.

Text region detection is performed using a neural network that understands the concept of spelling units, and it detects spellings and words by localizing the letters and estimating the probability that both letters are one word. In addition to this, it can be used by drawing additional lines by line, but for things such as preview and post-processing, it is easy to detect the text area by word unit. The coordinates from which the text area is extracted are used again for preview and masking during post-processing.

Then, the extracted text area is considered as an image following each, and 'text' is extracted through the OCR (Text Recognition) neural network. Specifically, text recognition is performed on a text region based on a neural network. Before that, preprocessing is performed to reduce noise and separate the text from the background to make it stand out. In this case, it is possible to use an image morphological filter or the like and perform image binarization.

Thereafter, the personal information pattern is detected through pattern matching using the document filter in the common filter, and the nature of the rectangle or image is identified through natural language processing with all 'texts' from any rectangle or image.

For example, if the spelling of 'resident registration' is detected, and one of the boxes below detects 12 digits and is not detected in the resident number pattern "6-7 digits" + checksum, these 12 digits of the OCR There is a high probability that all 13 digits were not selected due to false positives, and the text of resident registration and the location of the number indicate that these 12 digits are highly likely to be resident numbers. With this entity recognition, it is possible to detect more patterns that are not caught by the document filter by OCR false positives. In the case of other IDs, such as 'name' and 'address' can be identified, and various applications are possible.

Thereafter, various post-processing may be performed on the extracted personal information patterns. For example, you can localize the location of the personal information pattern in the original image by using personal information masking or homography of point correction and coordinates of text area detection, and masking can be done by coloring this area invisibly. In addition, it is possible to highlight and display a rectangular area in which a pattern is confined and easy to see by the user in order to preview the personal information and the detection result. In addition, specific keyword detection can determine which specific keywords are in the image, and certain combinations of texts through specific form detection and entity recognition express a resident registration card. By learning concepts such as representing bank documents and representing passports, you can check whether an image has a specific form.

Thereafter, the location of the fingerprint is additionally determined. Since the fingerprint can also be used as personal information, a neural network detecting the fingerprint is used to determine the location of the fingerprint.

On the other hand, by dividing the image into three compartments, foreground, background, and traps, a new image can be created by allowing foregrounds and trap images to have various rotation angles, positions, sizes, color depths, and brightness in the background, especially in real life. Since most of the fingerprint data that can be found is the fingerprint on the back of the resident registration card, it automatically creates various rolled-fingerprints on the fingerprint part of the resident registration card, making it the back side of the resident registration card with various noises and adding it to the background. can

Although the present invention has been described with reference to the embodiments shown in the drawings, which are merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: pre-processing unit 110: image array unit
120: image improvement unit 130: template matching comparison unit
140: area detection unit 150: geometric transformation unit
200: personal information area extraction unit 300: pattern analysis unit
400: de-identification processing unit

Claims (10)

a pre-processing step of extracting an image file from the target file by a pre-processing unit, detecting a region of interest that is determined to contain personal information from the extracted image file, and performing processing to improve the image recognition rate for the region of interest;
a personal information extraction step of extracting, by a personal information region extraction unit, processing target information including personal information from the region of interest detected from the pre-processing unit using a CNN neural network;
a pattern analysis step in which a pattern analysis unit compares and analyzes the pattern of the personal information extracted by the personal information area extraction unit with a preset personal information pattern to extract the personal information of the processing target information;
The de-identification processing unit is performed including a de-identification processing step of de-identifying the personal information extracted by the pattern analysis unit:
The pre-processing step is
an image array step in which an image-array unit assigns a connection identification number to each of the image files extracted from the target file;
an image improvement step in which an image improvement unit removes a quality degrading factor of the image file;
The template matching and comparison unit compares the image file with set templates consisting of template groups of regular forms including personal information including driver's license, identification card or foreigner's certificate, and compares the image according to the possibility of including personal information. step;
detecting, by a region detection unit, the region of interest from the image file selected by the template matching and comparing unit, through a region detection algorithm in which a personal information inclusion format is learned by an artificial neural network;
A method of processing personal information using AI deep learning, characterized in that it includes; a geometrical transformation unit correcting the viewpoint and rotation of the region of interest image.
delete The method of claim 1,
The area detection unit,
When the image file is classified into a stereotyped format by the template matching and comparison unit, the contour of the image file is calculated using an image morphological filter to highlight the image characteristics of the region of interest. extract,
The geometric transformation unit,
Personal information processing method using AI deep learning, characterized in that correcting the image file from which the outline is extracted.
The method of claim 1,
The region detection algorithm is
A region of interest is detected based on feature points including a vertex, a corner, or a line included in the image file, but using a Rectangular Corner Localization (RCL) neural network to determine the weight of the decision on the feature point. Personal information processing method using AI deep learning, characterized in that it is configured to find the optimal ratio.
The method of claim 1,
The image improvement unit,
Personal information processing method using AI deep learning, characterized in that it removes quality-impairing factors including noise, blur, or occlusion of the image file through an image improvement algorithm learned through a CNN neural network .
The method of claim 1,
The template matching comparison unit,
According to the template matching index of the image file and the templates, the image file is divided into a formal format and an atypical format, respectively, and a matching comparison algorithm according to the format matching index is an artificial neural network ( A personal information processing method using AI deep learning, characterized in that it is learned through CNN, Convolution Neural Network).
The method of claim 1,
The processing target information is
Personal information consisting of text,
The personal information area extraction unit,
It is configured to detect a text unit containing text in the region of interest through an extraction algorithm learned through a CNN neural network, and extract a text string from the text unit, wherein an image and a space having a certain length and height in the region of interest are Personal information processing method using AI deep learning, characterized in that the continuous area is detected by the writing unit.
The method of claim 1,
The processing target information is
Personal information consisting of a fingerprint image or facial image,
The personal information area extraction unit,
It is configured to detect an image pattern in the region of interest through an extraction algorithm learned through a CNN neural network, and extract a fingerprint image or face image from the image pattern, having a fingerprint pattern or facial pattern connected to a feature point in the region of interest. Personal information processing method using AI deep learning, characterized in that the pattern area is detected as the image pattern.
8. The method of claim 7,
The pattern analysis unit,
A personal information pattern analysis unit that compares and analyzes the text with a preset personal information pattern including a resident number, driver's license number, passport number and foreign number and fingerprint through a pattern analysis algorithm learned through CNN neural network;
Personal information processing method using AI deep learning, characterized in that it comprises a document form analysis unit that compares and analyzes the text with a preset document form including a resident registration card, driver's license, passport, residence certificate and alien certificate.
8. The method of claim 7,
The de-identification processing unit,
Personal information processing method using AI deep learning, characterized in that it is configured to process the text in any one or more ways of mosaic processing, hiding processing, encryption processing, pseudonymization processing, total processing, data deletion, data categorization, and masking.

Images