KR102548770B1 - System and method of calculating face similarity using matrix calculation - Google Patents

Abstract

The present invention discloses a system and method for calculating facial similarity using a matrix operation. The facial similarity calculation method includes the steps of collecting a plurality of video call data including the customer's face, deriving a facial image from the collected video call data and assigning it to each customer's account, among a plurality of customers, Determining the test target for facial similarity determination, calculating the vector value of the facial image belonging to the account of the customer included in the test target, the number of video calls of the customer for each customer, and each video call Generating a matrix using the vector value of the face image corresponding to , converting the size of the matrix to be the same for each customer, and calculating the facial similarity using an operation between the transformed matrices. include

Description

System and method of calculating face similarity using matrix calculation {System and method of calculating face similarity using matrix calculation}

The present invention relates to a system and method for calculating facial similarity using a matrix operation. Specifically, the present invention relates to a system and method for quickly calculating facial similarity of a video call by using a matrix including vector values of face images extracted from a video call.

The contents described in this part merely provide background information on the present embodiment and do not constitute prior art.

Recently, due to the development of smart devices and networks, and the development of various network services, various tasks, including banking, which were previously performed face-to-face, have been converted to online/wireless non-face-to-face processing.

For non-face-to-face business processing, financial companies conduct video calls with customers if necessary, and calculate facial similarity to determine whether or not the customer conducting the video call is the person in question.

Specifically, financial companies calculate facial similarity with other video calls or reference images for video calls with customers, whether the customer and the blacklist are similar, and whether the faces of a specific customer's multiple video calls match each other. , It is possible to determine whether a face of a specific customer for a video call matches a face for a video call of another customer.

Through this, financial companies can periodically determine whether a customer's identity has been stolen, and can prevent financial accidents and secure the stability of financial services.

However, in periodically performing the method of calculating the facial similarity described above, as the amount of video calls accumulated in the server increases, the system resources used for determining the facial similarity increase, and the time required to calculate the facial similarity also increases rapidly. there was a problem with

Therefore, there was a need for a facial similarity calculation method capable of performing the same operation with fewer resources in a short time.

An object of the present invention is to provide a facial similarity calculation system and method capable of quickly processing facial similarity between facial images extracted from a large amount of video calls.

In addition, an object of the present invention is to calculate the facial similarity of a plurality of facial images by calculating vector values for facial images using a deep learning module and using a matrix including vector values corresponding to the maximum number of video calls. It is to provide a system and method that can be calculated quickly.

In addition, an object of the present invention is to compare facial similarity with a reference value to determine whether or not there is an abnormality, and to relearn the deep learning module using the case in which the abnormality is confirmed, thereby increasing the accuracy of the facial similarity judgment. It is to provide a calculation system and method.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention not mentioned above can be understood by the following description and will be more clearly understood by the examples of the present invention. It will also be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations indicated in the claims.

A facial similarity calculation method according to an embodiment of the present invention includes the steps of collecting a plurality of video call data including the customer's face, deriving a facial image from the collected video call data, and assigning it to each customer's account. Step, among a plurality of customers, determining a test target for determining facial similarity, calculating a vector value of a facial image belonging to an account of a customer included in the test target, and performing a video call of the corresponding customer for each customer Generating a matrix using the number and the vector value of the facial image corresponding to each video call, converting the size of the matrix to be the same for each customer, and using the operation between the converted matrices, and calculating facial similarity.

In addition, the matrix is expressed as a product of a first matrix related to the number of video calls and a second matrix related to vector values of facial images corresponding to each video call, and the first matrix is the image of the corresponding customer. It includes one or more entries or elements corresponding to the number of currencies, and the entries or elements may be positive integers.

In addition, the step of converting the size of the matrix to be the same may include setting the number of video calls of the customer with the most call history as the 'maximum number of video calls' based on the number of video calls of the customers belonging to the test target; , converting all the first matrix constituting the matrix of each customer to the same size based on the maximum number of video calls.

In addition, the calculating of the vector value may include detecting a landmark in the facial image, arranging the facial image using the detected landmark, and deep learning for the aligned facial image. Calculating the vector value using a module may be included.

In addition, the step of determining whether or not there is an abnormality by comparing the calculated facial similarity with a preset reference value, and if the abnormality is found, reconfirming whether or not the same person is the same person based on the result of the visual inspection for the case; If there is an error in determining whether or not there is an anomaly, a step of re-learning the deep learning module using the image for the corresponding case may be further included.

In addition, in the step of calculating the vector value, the vector value is calculated using a pre-learned deep learning module so that the same vector value is output for the image of the same person, but the deep learning module calculates the face It includes an input layer using an image as an input node, an output layer using the facial similarity as an output node, and one or more hidden layers disposed between the input layer and the output layer, and between the input node and the output node. The weights of nodes and edges may be updated by the learning process of the deep learning module.

In addition, in the step of calculating the vector value, the vector value is calculated using a pretrained deep learning module so that the same vector value is output for the image of the same person, but the deep learning module is different from each other. A plurality of neural network modules to which images are input, a similarity determination module to calculate a similarity between values output from the plurality of neural network modules, and a weight module to adjust a weight for each calculated similarity; A feedback module may be included to transmit feedback about an error of the result value output from the weight module to the plurality of neural network modules.

In addition, the step of determining the inspection target includes receiving a selection for any one of a plurality of inspection methods, and determining the inspection target based on the selected inspection method, wherein the inspection method includes a customer; The first inspection method for determining the facial similarity between reference images registered in the blacklist, the second inspection method for determining the facial similarity between the facial images of a plurality of video calls belonging to a specific customer, and the video calls of different customers A third inspection method for determining the degree of facial similarity between facial images may be included.

In addition, the second inspection method and the third inspection method may determine facial similarity using the matrix converted to the same size for each customer.

In addition, whether to perform the first inspection method may be determined based on whether the blacklist is updated or whether the customer is a new customer.

In addition, a facial similarity calculation method according to another embodiment of the present invention includes (a) collecting a plurality of video call data including a customer's face, (b) deriving a facial image from the collected video call data Step (c) calculating the vector value of the derived facial image and assigning it to the account of the corresponding customer, (d) repeating steps (a) to (c) for a plurality of customers, (e) each Generating a matrix using the number of video calls of the corresponding customer for each customer and the vector value of the face image corresponding to each video call, (f) converting the size of the matrix to be the same for each customer, and (g) calculating facial similarity using the transformed inter-matrix operation.

On the other hand, a facial similarity calculation system according to some embodiments of the present invention includes a data collection unit that collects a plurality of video call data including a customer's face, and a test that determines a test target for determining facial similarity among a plurality of customers. A target determination unit, a facial image derivation unit for deriving a facial image from the plurality of video call data of the examination target, and a calculation unit for calculating a facial similarity with respect to the facial image, wherein the calculation unit is A feature calculation unit that calculates the vector value of the face image belonging to the account of the included customer, and the number of video calls of the customer for each customer and the vector value of the face image corresponding to each video call. It includes a matrix generator that generates a matrix, a matrix converter that converts the size of the matrix to be the same for each customer, and a facial similarity calculator that calculates the facial similarity using an operation between the transformed matrices.

In addition, the matrix is expressed as a product of a first matrix related to the number of video calls and a second matrix related to vector values of facial images corresponding to each video call, and the first matrix is the image of the corresponding customer. It includes one or more entries or elements corresponding to the number of calls, and the matrix conversion unit determines the number of video calls of customers with the most call history based on the number of video calls of customers belonging to the test target. is set to the 'maximum number of video calls', and based on the maximum number of video calls, the size of the first matrix constituting the matrix of each customer may be converted to the same size.

In addition, the feature calculation unit calculates the vector value using a pre-learned deep learning module so that the same vector value is output for images of the same person, and the deep learning module uses the face image as an input node. It includes an input layer, an output layer having the face similarity as an output node, and one or more hidden layers disposed between the input layer and the output layer, and nodes and edges between the input node and the output node. Weights may be updated by the learning process of the deep learning module.

In addition, the inspection target determination unit receives a selection of any one of a plurality of inspection methods, and determines the inspection subject based on the selected inspection method, wherein the inspection method is performed by a customer and a reference registered in a blacklist. A first inspection method for determining facial similarity between images, a second inspection method for determining facial similarity between facial images for a plurality of video calls belonging to a specific customer, and facial images for video calls of different customers A third inspection method for determining similarity may be included.

The facial similarity calculation system and method of the present invention calculates a vector value for a facial image extracted from a video call using a deep learning module, and uses a matrix including vector values corresponding to the maximum number of video calls. It is possible to quickly calculate the facial similarity between facial images extracted from video calls. Through this, the present invention can shorten the time required to calculate the similarity between a large number of video calls, thereby increasing the speed of determining whether an abnormal customer exists.

In addition, the facial similarity calculation system and method of the present invention compares the facial similarity with a preset reference value to determine whether there is an abnormality, and uses a case in which the abnormality is confirmed to output a vector value necessary for calculating the facial similarity. By retraining the module, the accuracy of facial similarity judgment can be increased. Through this, the present invention can increase the accuracy of determining the degree of similarity for a plurality of video calls, prevent financial accidents, and improve the stability of providing financial services.

In addition to the above description, specific effects of the present invention will be described together while explaining specific details for carrying out the present invention.

1 is a conceptual diagram for explaining a system for performing a facial similarity calculation method according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the configuration of the financial institution server of Figure 1;
3 is a diagram for explaining a facial similarity calculation method according to some embodiments of the present invention.
4 is a flowchart for explaining a facial similarity calculation method according to an embodiment of the present invention.
FIG. 5 is a diagram for explaining details of a step of deriving the facial image of FIG. 3 and calculating a vector value for it.
FIG. 6 is a diagram for explaining an example of calculating facial similarity in a plurality of video calls in FIG. 5 .
FIG. 7 is a diagram for explaining an example of calculating a facial similarity between a video call and a reference image in FIG. 5 .
8 is a block diagram for explaining the deep learning module in step S140 of FIG. 3 .
9 is a block diagram for explaining an example of the deep learning module of FIG. 8 .
10 is a block diagram for explaining a learning step of the deep learning module of FIG. 8 .
FIG. 11 is a diagram for explaining a process of determining a test target for determining facial similarity in S120 of FIG. 3 .
12 is a flowchart for explaining a process of re-learning a deep learning module according to an inspection method in some embodiments of the present invention.
13 is a diagram for explaining hardware implementation of a system that performs a facial similarity calculation method according to some embodiments of the present invention.

Terms or words used in this specification and claims should not be construed as being limited to a general or dictionary meaning. According to the principle that an inventor may define a term or a concept of a word in order to best describe his/her invention, it should be interpreted as meaning and concept consistent with the technical spirit of the present invention. In addition, the embodiments described in this specification and the configurations shown in the drawings are only one embodiment in which the present invention is realized, and do not represent all of the technical spirit of the present invention, so they can be replaced at the time of the present application. It should be understood that there may be many equivalents and variations and applicable examples.

Terms such as first, second, A, and B used in this specification and claims may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term 'and/or' includes a combination of a plurality of related recited items or any one of a plurality of related recited items.

Terms used in this specification and claims are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. It should be understood that terms such as "include" or "having" in this application do not exclude in advance the possibility of existence or addition of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification. .

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs.

Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In addition, each configuration, process, process or method included in each embodiment of the present invention may be shared within a range that does not contradict each other technically.

Hereinafter, a method for calculating facial similarity according to an embodiment of the present invention and a system for performing the same will be described in detail with reference to FIGS. 1 to 13 .

1 is a conceptual diagram for explaining a system for performing a facial similarity calculation method according to an embodiment of the present invention.

Referring to FIG. 1 , a system according to an embodiment of the present invention includes a financial company server 100 , a customer terminal 200 and a counselor terminal 300 .

The financial company server 100 (hereinafter, the server) may provide various non-face-to-face financial services to the customer terminal 200 . In addition, the server 100 may perform identity verification or identity authentication to determine whether the customer is the customer prior to providing financial services to the customer.

The server 100 may use a video call as a means for customer identification or authentication. To this end, the server 100 may mediate a video call between the customer terminal 200 and the counselor terminal 300, and perform identity verification or identity authentication of the customer based on the collected video call data.

At this time, the server 100 may extract the customer's facial image from the video call data using the facial similarity calculation method, and perform identity verification or identity authentication of the customer using the extracted facial image. Subsequently, when identification or identity verification is completed, the server 100 may provide the customer terminal 200 with a financial service requested by the customer (or a customer-customized financial service).

However, it is obvious that the facial similarity calculation method performed in the server 100 is not limited to the above operation, and can be applied and performed in various embodiments. Hereinafter, for convenience of description, an example of performing authentication of a customer using video call data derived during a video call with a customer will be described.

The server 100 may operate as a performer of a facial similarity calculation method. The server 100 may receive video call data from the customer terminal 200 and perform a facial similarity calculation method based on the received video call data.

Specifically, the facial similarity calculation method performed in the server 100 may operate by setting the test target differently according to the test method.

For example, the server 100 includes a first inspection method for determining the degree of similarity between a customer and a pre-registered blacklist, a second inspection method for determining whether or not faces of a plurality of video calls of a specific customer match each other, and , It can be used in a third inspection method for determining whether a face for a video call of a specific customer matches a face for a video call of another customer.

However, these are only a few examples of inspection methods performed in the server 100, and the present invention is not limited thereto. Hereinafter, for convenience of description, only the first to third inspection methods will be described as examples.

The server 100 may select and perform the second or third inspection method for each preset period. In addition, the server 100 may select and perform any one of the first to third inspection methods when there is a new customer or when the blacklist is updated.

In performing the first to third inspection methods, the server 100 needs to extract facial images from a large number of video call data and determine facial similarities for various combinations of the extracted facial images. Therefore, as the number of video calls stored in the server 100 increases, the resources required to calculate the facial similarity increase, and the time required to calculate the facial similarity may also increase rapidly. To solve this problem, the server 100 may generate a matrix for each customer and use it for calculation of facial similarity.

Here, the matrix may include a video call history of the corresponding customer and a vector value of a face image corresponding to each video call. For example, the matrix may be expressed as a product of a first matrix related to the number of video calls belonging to the customer and a second matrix related to vector values of facial images corresponding to each video call belonging to the customer. can In this case, the first matrix and the second matrix may include one or more entries or elements corresponding to the number of video calls. In this case, each component or element may be composed of a 'positive integer'.

Additionally, the vector value of the facial image included in the matrix may be generated by a pre-trained deep learning module. A description of the deep learning module of the present invention will be described in detail below with reference to FIGS. 8 to 10 .

However, in order to calculate the facial similarity using matrix operation, the matrix for each customer must be composed of matrices of the same size, but since the number of video calls performed for each customer is different, the size of the matrix constituting the matrix for each customer may be different from each other.

Accordingly, the server 100 may set the number of video calls of the customer with the most call history among the customers to be examined as the 'maximum number of video calls' so that a uniform matrix operation is possible. Subsequently, the server 100 may convert the first matrix constituting the matrix of each customer to the same size based on the 'maximum number of video calls'.

For example, when the number of video calls of a specific customer is 3, the customer matrix may include a first matrix having a size of 1x3 and a second matrix having a size of 3x1. At this time, if the number of video calls of the customer with the largest number of video calls among the customers to be inspected (ie, the maximum number of video calls) is 5, the server 100 sets the size of the first matrix of the customer to 1x5 and the size of the second matrix to 5x1. In this process, a scalar value or vector value of size 0 can be added to each matrix. However, this is only one example, and the present invention is not limited thereto.

Through this, the matrix of each customer is converted into the same size (or the same dimension), and the server 100 can quickly determine the facial similarity between a plurality of video calls by using the matrix operation. A detailed description of this will be described below.

Meanwhile, in the present invention, the server 100 and the customer terminal 200 may be implemented as a server-client system.

Specifically, the server 100 may constitute an electronic document management system (EDMS) that stores and manages various information and data on customers. At this time, the server 100 may operate each customer's account in the form of a database.

Through this, the server 100 can classify, store and manage video call data and previously input facial images (eg, ID images or facial images detected in the past) for each customer account, and provide financial information. and various services related to video calls, etc. may be provided through a terminal application installed in the customer terminal 200 .

In this case, the terminal application may be a dedicated application for receiving video call data or providing financial services, or a web browsing application. Here, the dedicated application may be an application embedded in the customer terminal 200 or an application downloaded from an application distribution server and installed in the customer terminal 200 .

The customer terminal 200 refers to a communication terminal capable of operating an application in a wired/wireless communication environment. In FIG. 1, the customer terminal 200 is shown as a smart phone, which is a type of portable terminal, but the present invention is not limited thereto, and as described above, it can be applied to a device capable of operating a financial application without limitation. there is. For example, the customer terminal 200 may include various types of electronic devices such as a personal computer (PC), a laptop computer, a tablet computer, a mobile phone, a smart phone, and a wearable device (eg, a watch type terminal).

In addition, although only one customer terminal 200 is shown in the drawing, the present invention is not limited thereto, and the server 100 may operate in conjunction with a plurality of customer terminals 200 .

Additionally, the customer terminal 200 includes an input unit for receiving a customer's input, a display unit for displaying visual information, a communication unit for sending and receiving signals to and from the outside, a camera unit for photographing the customer's face, and converting the customer's voice into digital data. It may include a microphone unit that converts, and a controller that processes data, controls each unit inside the customer terminal 200, and controls data transmission/reception between units. Hereinafter, commands executed by the controller in the customer terminal 200 according to the customer's command are collectively referred to as those performed by the customer terminal 200 .

Meanwhile, the counselor terminal 300 operates in association with the server 100 and may be a counterpart performing a video call with the customer terminal 200 . Although not clearly shown in the drawing, the server 100 operates in association with a plurality of counselor terminals 300, and when a video call request is received from the customer terminal 200, one of the plurality of counselor terminals 300 can be matched with the customer terminal 200 that requested the video call by selecting .

The server 100 serves as a relay so that a mutual video call can be performed between the matched customer terminal 200 and the counselor terminal 300 . In this case, the server 100 may store and manage details of a video call between the customer terminal 200 and the counselor terminal 300 .

Meanwhile, the communication network 400 serves to connect the server 100 , the customer terminal 200 and the counselor terminal 300 . That is, the communication network 400 refers to a communication network that provides an access path so that the customer terminal 200 or the counselor terminal 300 can transmit and receive data after accessing the server 100 . The communication network 400 may be, for example, a wired network such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), ISDNs (Integrated Service Digital Networks), wireless LANs, CDMA, Bluetooth, satellite communication, etc. However, the scope of the present invention is not limited thereto.

Hereinafter, a detailed configuration of the server 100 of the present invention will be described.

Figure 2 is a block diagram for explaining the configuration of the financial institution server of Figure 1;

Referring to FIG. 2 , the server 100 includes a data collection unit 110, an examination target determination unit 120, a facial image derivation unit 130, and a calculation unit 140.

First, the data collection unit 110 receives video call data between the customer terminal 200 and the counselor terminal 300 and stores it in a database (not shown). A database (not shown) may classify, store, and manage data for each customer account. In addition, the data collection unit 110 may also receive information on whether or not the blacklist is updated and whether new customers are introduced.

The test subject determination unit 120 determines a test subject for determining facial similarity among a plurality of customers stored in the database. The inspection target determining unit 120 may receive a selection of one of a plurality of inspection methods and determine the inspection subject based on the selected inspection method.

For example, the inspection target determination unit 120 may receive selection of the first to third inspection methods described above, and in this case, the inspection target of the first inspection method may be a specific customer and a blacklist. The inspection target of the 2nd inspection method can be a plurality of video calls belonging to a specific customer's account, and the inspection subject of the third inspection method can be a video call belonging to a plurality of customer's accounts. That is, the inspection target may be defined differently according to the inspection method.

In addition, whether or not to perform the inspection method may be automatically determined according to whether a preset condition is met, whether a preset time arrives, and when video calls exceeding a preset reference value are accumulated.

The face image derivation unit 130 derives a face image for a plurality of video call data of a subject to be examined. A detailed description of the method of deriving the facial image will be described later with reference to FIGS. 5 to 7 .

The calculation unit 140 calculates the facial similarity with respect to the derived facial image. Here, the calculation unit 140 may include a feature calculation unit 141, a matrix generation unit 143, a matrix conversion unit 145, and a facial similarity calculation unit 147.

At this time, the feature calculation unit 141 calculates a vector value of a facial image belonging to an account of a customer included in the inspection target. At this time, the vector value of the facial image may be calculated using a pre-learned deep learning module.

Subsequently, the matrix generating unit 143 performs an operation of generating a matrix by using the number of video calls of the corresponding customer for each customer and the vector value of the face image corresponding to each video call.

Subsequently, the matrix conversion unit 145 performs an operation of equally converting the size of the matrix for each customer.

Subsequently, the facial similarity calculation unit 147 performs an operation of calculating the facial similarity using the converted inter-matrix operation.

A detailed description of the operation of each module of the calculation unit 140 will be described in detail while explaining the facial similarity calculation method. Hereinafter, the facial similarity calculation method performed in the server 100 of the present invention will be described in detail.

3 is a diagram for explaining a facial similarity calculation method according to some embodiments of the present invention. 4 is a flowchart for explaining a facial similarity calculation method according to an embodiment of the present invention.

First, referring to FIG. 3 , in the facial similarity calculation method according to some embodiments of the present invention, the server 100 first collects video call data for each customer through a video call with the customer terminal 200 (S110). . The collected video call data may belong to the customer's account, and data related to a plurality of video calls may be stored in each customer's account.

Subsequently, the server 100 determines an examination target for facial similarity determination (S120). The inspection target may be defined differently according to the inspection method, and whether to perform the inspection method may be determined depending on whether a preset condition is met, whether a preset time arrives, and whether video calls exceeding a preset threshold are accumulated.

For example, when there is a new subscribed customer or when a blacklist is updated, the above-described first inspection method may be performed. In addition, when a preset time arrives or when video calls over a preset reference value are accumulated, the second or third test method may be performed.

However, these are just a few examples, and the conditions under which the first to third inspection methods are performed can be variously modified and implemented. In addition, the first to third inspection methods may be performed by a system administrator's command.

At this time, in the first inspection method, the inspection target can be a specific customer and a blacklist, in the second inspection method, the inspection subject can be a plurality of video calls belonging to the account of a specific customer, and in the third inspection method, The object may be a video call belonging to a plurality of customer accounts.

Subsequently, when the examination target is specified, the server 100 derives a facial image from the video call data of the examination target (S130). For example, the server 100 may extract some frames through sampling of video call data, and may extract a facial image corresponding to a customer's face from some of the extracted frames. A specific method of extracting a facial image will be described later in detail with reference to FIG. 5 .

Subsequently, the server 100 calculates a vector value (or feature) for the extracted facial image using the deep learning module (S140).

Here, the deep learning module outputs one vector value for one image. Also, the deep learning module may be trained to output the same (or similar) vector values for images of the same person. Details of the structure and training method of the deep learning module will be described later in detail with reference to FIGS. 8 to 10 .

Next, the server 100 creates a matrix for the customer by using the 'number of video calls' of the specific customer and the vector value of the face image corresponding to each video call (S150). Server 100 may create a unique matrix for each customer.

In this case, the matrix may be expressed as a product of a first matrix related to the number of video calls belonging to the customer and a second matrix related to vector values of face images corresponding to each video call belonging to the customer. In this case, the first matrix and the second matrix may include one or more entries or elements corresponding to the number of video calls.

For example, when three pieces of video call data are attributed to a specific customer's account, the customer's matrix may consist of a first matrix with a size of 1x3 and a second matrix with a size of 3x1. In this case, each component of the first matrix may have a scalar value, and each component of the second matrix may have a vector value. In other words, the first matrix may be composed of reference numbers or valid values representing a specific video call, and the second matrix may be composed of vector values of face images related to a specific video call.

However, this is only one example, and it goes without saying that the combination of matrices constituting the matrix and the size and components of each matrix may be variously modified.

Subsequently, the server 100 converts the size of the matrix for each customer into the same (S160). This means an operation of equalizing the size of a matrix so that a uniform matrix operation can be performed in the server 100 .

Specifically, referring to FIG. 4 , the server 100 sets the number of video calls of the customer with the most call history among the customers subject to examination as the 'maximum number of video calls' (S161).

Next, the server 100 converts all the matrices constituting the matrix of each customer into the same size based on the 'maximum number of video calls' (S163).

For example, when the number of video calls of a specific customer is 3, the customer matrix may include a first matrix having a size of 1x3 and a second matrix having a size of 3x1. At this time, if the number of video calls of the customer with the largest number of video calls among the customers to be inspected (ie, the maximum number of video calls) is 5, the server 100 sets the size of the first matrix of the customer to 1x5 and the size of the second matrix to 5x1. size can be converted. Also, in this process, the server 100 may add a scalar value having a size of 0 to the first matrix and a vector value having a size of 0 to the second matrix. Through this, the size of the matrix of each customer included in the inspection target can be converted to be the same.

However, this is only one embodiment of converting the size of the matrix of the present invention to be the same, and the method of converting the size of the matrix of the present invention can be variously modified and implemented by those skilled in the art.

Subsequently, referring to FIG. 3 again, the server 100 calculates the facial similarity for the examination target through inter-matrix calculation using the transformed matrix (S170). Here, the server 100 may calculate the similarity between the respective images through a method for calculating the similarity between vectors (eg, cosine distance). However, the present invention is not limited thereto, and the server 100 may use various methods for calculating facial similarity, and since detailed descriptions thereof have already been disclosed, they will be omitted here.

Through this, the present invention can calculate the facial similarity between facial images extracted from a large amount of video calls in a short time, shorten the time required to calculate the similarity between video calls, and increase the speed of judgment on the existence of abnormal customers. can

Additionally, although not clearly shown in the drawings, the server 100 may compare the calculated facial similarity with a preset standard value (or standard range) to determine whether the examination target is abnormal.

Next, when abnormalities are found, the server 100 may request a visual inspection for abnormalities by forwarding the corresponding case to the counselor terminal 300 or manager terminal (not shown). Through this, the server 100 may reconfirm whether or not the person is the same person.

Subsequently, the server 100 may determine whether there is an error in determining whether or not there is an abnormality based on the visual inspection result received from the counselor terminal 300 .

If there is an error in determining whether or not there is an anomaly, the server 100 may relearn the deep learning module using the image for the corresponding case to update the system so that the same error does not occur.

Through this, the present invention can increase the accuracy of facial similarity judgment by retraining the deep learning module using the case in which abnormality is confirmed. Accordingly, the present invention can increase the accuracy of determining the degree of similarity for multiple video calls, prevent financial accidents, and improve the stability of financial services.

FIG. 5 is a diagram for explaining details of a step of deriving the facial image of FIG. 3 and calculating a vector value for it. FIG. 6 is a diagram for explaining an example of calculating facial similarity in a plurality of video calls in FIG. 5 . FIG. 7 is a diagram for explaining an example of calculating a facial similarity between a video call and a reference image in FIG. 5 . Hereinafter, overlapping descriptions with the above descriptions will be omitted, and descriptions will focus on differences.

Referring to FIG. 5, after the test subject for facial similarity determination is determined (S120 in FIG. 3) and a facial image is derived from the video call data of the customer to be tested (S130 in FIG. 3), the server 100 detects the customer's face (ie, facial image) from the sampling image of the video call data (S210).

Here, face detection means detecting a part recognized as a human face in a specific image as a bounding box. For example, referring to step S11 of FIG. 6 , the customer's facial part in the sampled image may be displayed as a rectangular bounding box.

In this case, the server 100 may detect the customer's face in the sampling image using a pre-learned deep learning model (eg, MTCNN, Retinaface, or Blazeface). Of course, the deep learning model used in the server 100 may be variously modified and used.

Subsequently, the server 100 detects a facial landmark from the facial image within the detected bounding box (S220). Here, facial landmarks refer to parts constituting facial features such as eyes, nose, mouth, jawline, and bridge of the nose. The number of facial landmarks may be variously set from a minimum of 5 points (eg, 2 eyes, 1 nose, and 2 mouth) to 68 points, 96 points, and 109 points.

In this case, the server 100 may detect facial landmarks within the detected bounding box using a pre-learned deep learning model (eg, MTCNN, Retinaface, or Blazeface). Similarly, the deep learning model used in the server 100 may be variously modified and used.

Next, the server 100 performs facial alignment based on the detected facial landmarks (S230). Here, the face alignment means, if the face detected in the image faces an angle other than the front, the face image is rotated and aligned so as to face the front center direction (ie, step S12 of FIG. 6 ).

For example, when using a 5-point landmark, the server 100 may use a method of forming a straight line between the eyes, measuring an angle between the straight line and a horizontal horizontal line, and rotating the facial image by an opposite angle. can However, it goes without saying that the face alignment method may be variously modified and implemented.

Subsequently, the server 100 extracts features (ie, vector values) from the aligned facial images using a deep learning module (S240). The server 100 may derive a vector value for a facial image using a pre-learned deep learning module.

Here, the deep learning module outputs one real vector value for one image. The deep learning module can substitute real-valued vectors for facial features, and the distance between each vector represents a degree of similarity. For example, the deep learning module may output a '512-dimensional vector value' or a '512-dimensional floating 32 vector'. However, this is only an example of a vector value, and the present invention is not limited thereto.

In this case, the deep learning module may be trained to output the same (or similar) vector values for images of the same person. Details of the structure and training method of the deep learning module will be described later in detail with reference to FIGS. 8 to 10 .

Subsequently, the server 100 may generate a matrix for each customer based on the extracted vector value, and calculate facial similarity between matrices using matrix operation (S250). Through this, the server 100 can quickly calculate facial similarities for a large number of video calls.

Subsequently, the server 100 may compare the calculated facial similarity with a preset reference value (or reference range) to determine whether the customer has an abnormality (S260). In addition, in the case of samples in which the facial similarity was judged incorrectly, the deep learning module can be used for re-learning. This will be described later with reference to FIG. 12 .

Meanwhile, FIG. 6 schematically illustrates a process of determining similarities of facial images with respect to a plurality of video call data.

Specifically, referring to FIG. 6 , the server 100 receives different first video call data VD1 and second video call data VD2. In this case, each of the video call data VD1 and VD2 may be related to the same customer (eg, the second inspection method) or different customers (eg, the third inspection method).

Subsequently, the server 100 performs a sampling process of extracting a specific frame for each of the video call data VD1 and VD2 (S11 and S21).

For example, the server 100 may sample frames of the video call data VD1 and VD2 at regular time intervals or derive and sample video frames having an optical flow smaller than a reference value.

As another example, the server 100 may detect a section in which a specific voice pattern is detected and sample a specific frame within the section.

As another example, the server 100 may operate a pose detection algorithm on the extracted image data VD. When a predetermined pose is detected by the pose detection algorithm, the server 100 may terminate the pose detection algorithm and extract an image frame related to the detected pose.

However, these are only some examples of deriving an image frame, and the present invention is not limited thereto.

Next, the server 100 detects a face image from the sampled frame through face detection (S12, S22).

Next, the server 100 performs face alignment so that the detected face image faces the center of the front (S13, S23). At this time, the server 100 may perform facial alignment through detection of the aforementioned facial landmarks.

Subsequently, the server 100 extracts vector values (ie, features) for the face image using the pre-learned deep learning module (S14 and S24).

Next, the server 100 determines the similarity of the extracted vector values and calculates the facial similarity for each facial image (S41). At this time, the server 100 may generate a matrix including the extracted vector values for each customer, and quickly calculate the facial similarity of the plurality of images through matrix operation.

In addition, FIG. 7 schematically illustrates a process of determining the similarity of a facial image between specific video call data and a reference image.

Specifically, referring to FIG. 7 , the server 100 receives video call data (VD) and reference image data (ID) of a specific customer. Here, the reference image data may be a customer's identification card image or a pre-registered blacklist image (eg, the first inspection method). At this time, the server 100 may receive the customer's ID image from the customer terminal 200 or receive and use a blacklist or customer's ID image previously registered in the database.

Subsequently, the server 100 performs a sampling process of extracting a specific frame for each video call data (VD) (S11). Since a detailed example of the sampling process has been described above, redundant information will be omitted here.

Subsequently, the server 100 detects each face image through face detection from the sampled frame and the reference image data (ID) (S12, S32).

Next, the server 100 performs face alignment so that the detected face image faces the center of the front (S13, S33). At this time, the server 100 may perform facial alignment through detection of the aforementioned facial landmarks.

Next, the server 100 extracts vector values (ie, features) for each facial image using the pre-learned deep learning module (S14, S34).

Next, the server 100 determines the similarity of the extracted vector values and calculates the facial similarity for each facial image (S42). At this time, the server 100 may generate a matrix including the extracted vector values for each customer, and quickly calculate the facial similarity of the plurality of images through matrix operation.

Hereinafter, a deep learning module for extracting a vector value of a facial image will be described in detail.

8 is a block diagram for explaining the deep learning module in step S140 of FIG. 3 .

Referring to FIG. 8 , the deep learning module (DM) may receive image data about a customer and output a vector value of the image as an output thereof. At this time, the image data may be pre-extracted and aligned face images, and the output vector value may be a 512-dimensional floating 32 vector. However, it goes without saying that the format of the output vector value may be variously modified.

The deep learning module (DM) may derive a unique vector value for a facial image using an artificial neural network trained on the basis of big data. Accordingly, the deep learning module (DM) may output the same vector value for the same image.

The deep learning module (DM) may perform artificial neural network learning using mapping data for separate parameters derived based on input data. The deep learning module (DM) may perform machine learning on parameters input as learning factors. In this case, the memory of the server 100 may store data used for machine learning and result data.

To explain in more detail, deep learning technology, a type of machine learning, learns by going down to a deep level in multiple stages based on data.

Deep learning represents a set of machine learning algorithms that extract core data from a plurality of data while stepping up.

The deep learning module (DM) may use various known deep learning structures. For example, the deep learning module (DM) may use a structure such as a convolutional neural network (CNN), a recurrent neural network (RNN), a deep belief network (DBN), or a graph neural network (GNN).

Specifically, CNN (Convolutional Neural Network) extracts the basic features of an object when a person recognizes an object, and then performs complex calculations in the brain to recognize the object based on the result. It is a simulated model.

RNN (Recurrent Neural Network) is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes over time.

DBN (Deep Belief Network) is a deep learning structure composed of multiple layers of RBM (Restricted Boltzman Machine), a deep learning technique. When a certain number of layers is obtained by repeating RBM (Restricted Boltzman Machine) learning, a DBN (Deep Belief Network) having a corresponding number of layers may be configured.

GNN (Graphic Neural Network, hereinafter, GNN) represents an artificial neural network structure implemented in a way to derive similarities and feature points between modeling data using modeling data modeled on the basis of data mapped between specific parameters. .

Meanwhile, learning of the artificial neural network of the deep learning module (DM) can be performed by adjusting the weight of the connection line between nodes (and adjusting the bias value if necessary) so that a desired output is produced for a given input. In addition, the artificial neural network may continuously update a weight value by learning. In addition, a method such as back propagation may be used to learn the artificial neural network.

Meanwhile, the memory of the server 100 may be equipped with an artificial neural network pre-learned through machine learning.

The deep learning module (DM) may perform a machine learning-based improvement process recommendation operation using modeling data for the derived parameters as input data. In this case, both semi-supervised learning and supervised learning may be used as machine learning methods of the artificial neural network. In addition, the deep learning module (DM) can be controlled to automatically update the artificial neural network structure for outputting an accurate vector value for sample data in which abnormalities are found according to settings.

Additionally, although not clearly shown in the drawings, in another embodiment of the present invention, the operation of the deep learning module (DM) may be implemented in the server 100 or a separate cloud server (not shown).

Hereinafter, the configuration of the deep learning module (DM) according to the above-described embodiment of the present invention will be described.

9 is a block diagram for explaining an example of the deep learning module of FIG. 8 .

Referring to FIG. 9, the deep learning module (DM) has an input layer (input) having extracted and sorted facial images as input nodes, and an output layer (Output) having vector values for feature points of the corresponding facial images as output nodes. and M hidden layers disposed between the input layer and the output layer.

Here, a weight may be set to an edge connecting nodes of each layer. The presence or absence of these weights or edges can be added, removed, or updated in the learning process. Therefore, through the learning process, weights of nodes and edges disposed between k input nodes and i output nodes may be updated.

All nodes and edges may be set to initial values before the deep learning module (DM) performs learning. However, when information is accumulated and input, the weights of nodes and edges are changed, and in this process, the parameters input as learning factors (i.e., voice data and voice patterns for each section) and the values assigned to output nodes (i.e., Voice similarity for each section) may be matched.

Additionally, when using a cloud server (not shown), the deep learning module (DM) may receive and process a large number of parameters. Therefore, the deep learning module (DM) can perform learning based on massive data.

The weights of nodes and edges between an input node and an output node constituting the deep learning module (DM) may be updated by the learning process of the deep learning module (DM). In addition, it goes without saying that parameters output from the deep learning module (DM) can be additionally expanded with various data in addition to vector values for images.

For example, the deep learning module (DM) may be implemented by receiving a plurality of different images and outputting a similarity of the plurality of input images. However, this is only an example of one extended embodiment and the present invention is not limited thereto.

Hereinafter, in an embodiment of the present invention, a process in which the deep learning module (DM) of the present invention is learned using a triplet loss structure will be described. However, this is only one example of the learning process of the deep learning module, and the present invention is not limited thereto.

10 is a block diagram for explaining a learning step of the deep learning module of FIG. 8 .

Referring to FIG. 10, the deep learning module (DM) includes a plurality of neural network modules 11 into which different images are input, and a similarity determination module 13 (distance calculator) that calculates the similarity of values output from each neural network module. ), a weight module 15 (weight calculator) for adjusting the weight for each calculated similarity, and a feedback module 17 (feedback module) for providing feedback on the error of the result value.

The deep learning module (DM) basically uses a machine learning algorithm of triplet loss. Accordingly, the neural network module 11 includes three different neural network submodules 11a, 211b, and 211c, and different images are input to each of the submodules 11a, 11b, and 11c.

For example, a reference image I1 (anchor image) to be judged is input to the first submodule 11a, and a positive object including the same object as the reference image I1 is input to the second submodule 11b. A positive image (I2) is input, and a negative image (I3) similar to I1 and I2 may be input to the third sub-module 11c.

In this case, the weight of the neural network may be shared among the submodules 11a, 11b, and 11c.

The output values Av, Pv, and Nv output from each of the submodules 11a, 11b, and 11c may have vector values, and each vector value may have the same format as the vector value described above.

Subsequently, the output values Av, Pv, and Nv output from each of the submodules 11a, 11b, and 11c are input to the similarity determination module 13. Also, a ground truth (GT) for the input reference image I1 is input to the similarity determination module 13 .

The similarity determination module 13 calculates the similarity between the respective values using the input output values Av, Pv, and Nv and the reference value GT. For example, the similarity determination module 13 may calculate the similarity of input values using a cosine distance function.

At this time, the similarity determination module 13 determines the first similarity between the first resultant value (Av) for the reference image and the second resultant value (Pv) for the positive image, and the first resultant value (Av) for the negative image. 3 The second similarity between result values (Nv), the third similarity between the second result value (Pv) and the third result value (Nv), the fourth similarity between the first result value (Av) and the reference value (GT), the second A fifth degree of similarity between the result value Pv and the reference value GT may be derived and transmitted to the weight module 15 . Also, although not shown in the drawing, the similarity determination module 13 may additionally derive a sixth similarity between the third result value Nv and the reference value GT and transmit it to the weight module 15 .

Next, the weight module 15 may apply a preset weight to the received similarity and output a result value T(t) at the first time point. For example, the weight module 15 applies a first weight to the first to third similarities, and applies a second weight different from the first weight to the fourth and fifth similarities, thereby applying a resultant value at the first point in time (T (t)) can be derived.

Subsequently, the result value T(t) output from the weight module 15 may be provided to the feedback module 17, and the feedback module 17 may receive the first value received from the weight module 15 at a first time point. A difference value between the result value T(t) and the second result value T(t-1) received at the second time point is derived, and the derived value is provided to the neural network module 11 as a feedback value. can do.

The neural network module 11 may adjust the weight of each neural network submodule 11a, 11b, and 11c using the received feedback value.

In addition, the deep learning module (DM) can operate by dividing into a learning mode and a performance mode.

In the learning mode, the deep learning module (DM) may be trained to increase the accuracy of determining the similarity of each image through a preset training dataset. The deep learning module (DM) may output vector values having the same or high similarity for the reference image and the positive image including the same object through sufficient learning using the dataset.

In the execution mode, only one image may be input to the deep learning module (DM), and accordingly, the deep learning module (DM) may output a vector value for the input image using the trained neural network.

Vector values output from the deep learning module (DM) can be stored in the customer's account to correspond to each input image, and can be used to generate the customer's matrix as described above.

FIG. 11 is a diagram for explaining a process of determining a test target for determining facial similarity in S120 of FIG. 3 .

Referring to FIG. 11 , the server 100 may receive a selection of one of a plurality of preset examination methods and determine a test target for facial similarity calculation based on the selected examination method.

In addition, the server 100 may automatically perform a specific inspection method depending on whether a preset condition is met, whether a preset time has arrived, and video calls exceeding a preset reference value are accumulated, and determine the test subject accordingly. .

Specifically, when the server 100 receives a selection for any one of a plurality of inspection methods from the administrator, the server 100 determines the inspection target based on the inspection method (S310).

For example, when the server 100 receives a selection for the first inspection method, the server 100 may perform an operation of determining facial similarity between a specific customer and a reference image registered in a preset blacklist. (S341). In this case, the inspection target may be video call data for a specific customer and a reference image registered in a preset blacklist.

As another example, when the server 100 receives a selection for the second inspection method, the server 100 may perform an operation of determining facial similarity between facial images of a plurality of video calls belonging to a specific customer. (S343). In this case, the inspection target may be video call data belonging to a specific customer's account.

As another example, when the server 100 receives a selection for the third inspection method, the server 100 may perform an operation of determining facial similarity between facial images for video calls belonging to different customers. It can (S345). In this case, the inspection target may be video call data belonging to a specific customer's account and video call data belonging to another customer's account.

In this case, the second inspection method and the third inspection method may calculate facial similarity using a unique matrix converted to the same size for each customer as described above.

In addition, the first inspection method matches the size of the matrix corresponding to the number of reference images of the blacklist registered in advance with the size of the matrix corresponding to the number of video calls of the customer to be inspected, and then calculates the facial similarity through matrix operation. can be calculated. If there are a plurality of customers to be inspected in the first inspection method, the size of the matrix to be converted may be determined based on the largest value among the number of reference images and the number of video calls to the plurality of customers.

However, the inspection method according to the embodiment of the present invention is not limited to the first to third inspection methods, and additional inspection methods or modified inspection methods may be used.

Meanwhile, when the server 100 does not receive the inspection method, the server 100 may determine whether to implement the inspection method according to a preset condition.

For example, when a new customer newly subscribes to the server 100 occurs, the server 100 may perform the first inspection method described above (S320).

In addition, even if there is no new customer newly subscribed to the server 100, if there is an update in the blacklist managed by the server 100, the server 100 may perform the first inspection method described above (S330).

In addition, although not clearly shown in the drawings, when a preset period arrives, the server 100 may automatically perform the above-described second inspection method or third inspection method.

However, it is obvious that the conditions for determining whether or not to implement this specific inspection method can be variously modified and implemented.

Additionally, in the first inspection method, the server 100 may calculate facial similarity between all customers and the blacklist reference image.

For example, the server 100 may extract a 512-dimensional vector value for each of the customer's face image for each video call and the blacklist reference image. In this case, a matrix of 'number of video calls' x 'vector value of 512 dimensions' is created for each customer, and a matrix of 'number of reference images' x 'vector value of 512 dimensions' is created for the blacklist. do. next. The server 100 calculates facial similarity for '512-dimensional vector values' with respect to the two matrices of the customer and the blacklist. As a result of this, a matrix of result values related to 'number of video calls' x 'number of reference images' can be calculated. Subsequently, the server 100 may determine whether each value included in the result value matrix exceeds a predetermined reference value, and determine whether the corresponding video call is similar to the blacklist.

Meanwhile, in the second inspection method, the server 100 may calculate facial similarities for a plurality of video calls for each customer with respect to all customers.

For example, if the number of video calls is n for a specific customer, the number of comparison combinations of facial images for a plurality of video calls is “n combinations 2”. That is, the server 100 must calculate facial similarity as many times as the number of cases of each combination. If this is extended to all customers, the facial similarity calculation should be repeated as many times as the number of combinations of video calls for each customer. However, since the number of video calls for each customer is different, it is necessary to sequentially calculate facial similarity as many as the number of combinations for each customer. In this case, a lot of load may occur in the system. In order to solve this problem, the present invention sets the number of video calls of the customer with the most call history as the 'maximum number of video calls', and sets the '512-dimensional 0 vector' to the 'maximum number of video calls' for all customers. By adding to each customer's matrix as a basis, a matrix is created so that all customers have the same number of '512-dimensional vector values'. Through this, each customer can have a matrix of the same size (or dimension), and since all matrices have the same size, the server 100 can speed up the facial similarity calculation by using the matrix operation.

On the other hand, in the third inspection method, the server 100 may calculate facial similarity for each of a plurality of video calls for each different customer with respect to all customers. The third inspection method may operate substantially the same as the second inspection method, and may differ only in a method for determining whether or not there is an abnormality by comparing the calculated facial similarity with a reference value.

Hereinafter, in performing the above-described second and third inspection methods, the process of determining whether or not the facial image is abnormal using the facial similarity calculation result and re-learning the deep learning module based on this will be described in detail. let it do

12 is a flowchart for explaining a process of re-learning a deep learning module according to an inspection method in some embodiments of the present invention. Hereinafter, for convenience of description, an example in which the counselor terminal 300 determines whether or not there is an error will be described. However, it goes without saying that in the present invention, the subject of determination of abnormality may be changed and performed.

Referring to FIG. 12 , following step S170 of calculating the facial similarity of FIG. 3 , the server 100 determines whether the calculated facial similarity is smaller than a preset reference value (S410).

Subsequently, when the facial similarity is smaller than the reference value, the server 100 determines whether the performed test method is the second test method (S421).

Subsequently, the facial similarity is calculated by the second inspection method, and when the facial similarity is smaller than a preset reference value, the server 100 transfers the case to the counselor terminal 300 to perform a visual inspection for abnormalities. request (S423). That is, since the server 100 has performed the second test method for determining whether the test subject is the same person, the counselor terminal 300 determines again whether the test target is the same person (S425).

If, as a result of the visual inspection of the counselor terminal 300, if the case is determined to be the same person, the server 100 relearns the deep learning module using the image of the case (S427). That is, since the corresponding case corresponds to a case in which images of the same person are mistakenly judged to have low similarity, the deep learning module relearns the case so that an accurate similarity can be derived in the future.

On the other hand, if the facial similarity is greater than the reference value, the server 100 determines whether the performed examination method is the third examination method (S431).

Subsequently, the facial similarity is calculated by the third inspection method, and when the facial similarity is greater than a preset reference value, the server 100 transfers the case to the counselor terminal 300 to perform a visual inspection for abnormalities. request (S433). That is, since the server 100 has performed the third test method for determining whether the test target is a different person, the counselor terminal 300 determines again whether the test target is the same person (S435).

If, as a result of the visual inspection of the counselor terminal 300, it is determined that the corresponding case is not the same person, the server 100 relearns the deep learning module using the image of the corresponding case (S437). That is, since the corresponding case corresponds to a case in which images of different persons are mistakenly judged to have a high degree of similarity, the deep learning module similarly relearns the case so that an accurate degree of similarity can be derived in the future.

The re-learning process of the deep learning module may be performed whenever an abnormality in judgment is found, or may be performed at once when a predetermined number or more cases are accumulated.

Through this relearning process of the deep learning module, the present invention can increase the accuracy of facial similarity determination. Through this, the present invention can increase the accuracy of determining the similarity of a plurality of video calls, prevent the occurrence of financial accidents, and improve the stability of financial services.

13 is a diagram for explaining hardware implementation of a system that performs a facial similarity calculation method according to some embodiments of the present invention.

Referring to FIG. 13 , the server 100 performing the facial similarity calculation method according to some embodiments of the present invention may be implemented as an electronic device 1000. The electronic device 1000 may include a controller 1010, an input/output device 1020 (I/O), a memory device 1030, an interface 1040, and a bus 1050. . The controller 1010 , the input/output device 1020 , the memory device 1030 and/or the interface 1040 may be coupled to each other through a bus 1050 . At this time, the bus 1050 corresponds to a path through which data is moved.

Specifically, the controller 1010 includes a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphic processing unit (GPU), a microprocessor, a digital signal processor, a microcontroller, and an application processor (AP). , application processor), and logic elements capable of performing functions similar thereto.

The input/output device 1020 may include at least one of a keypad, a keyboard, a touch screen, and a display device.

The memory device 1030 may store data and/or programs.

The interface 1040 may perform a function of transmitting data to a communication network or receiving data from the communication network. Interface 1040 may be wired or wireless. For example, the interface 1040 may include an antenna or a wired/wireless transceiver. Although not shown, the memory device 1030 is an operating memory for improving the operation of the controller 1010 and may further include a high-speed DRAM and/or SRAM. The memory device 1030 may store programs or applications therein.

The server 100 and the customer terminal 200 according to embodiments of the present invention may be systems formed by connecting a plurality of electronic devices 1000 to each other through a network. In this case, each module or combinations of modules may be implemented as the electronic device 1000 . However, this embodiment is not limited thereto.

Additionally, the server 100 may include a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, and a network attached storage (NAS) system. It may be implemented as at least one of a system, a redundant array of inexpensive disks (RAID) system, and an electronic document management (EDMS) system, but the present embodiment is not limited thereto.

In addition, the server 100 may transmit data through a network using the customer terminal 200 . The network may include a network based on wired Internet technology, wireless Internet technology, and short-range communication technology. Wired Internet technology may include, for example, at least one of a local area network (LAN) and a wide area network (WAN).

Wireless Internet technologies include, for example, Wireless LAN (WLAN), DMNA (Digital Living Network Alliance), Wireless Broadband (Wibro), WiMAX (World Interoperability for Microwave Access: Wimax), HSDPA (High Speed Downlink Packet Access), High Speed Uplink Packet Access (HSUPA), IEEE 802.16, Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), Wireless Mobile Broadband Service (WMBS) And it may include at least one of 5G New Radio (NR) technology. However, this embodiment is not limited thereto.

Short-range communication technologies include, for example, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra-Wideband (UWB), ZigBee, Near Field Communication: At least one of NFC), Ultra Sound Communication (USC), Visible Light Communication (VLC), Wi-Fi, Wi-Fi Direct, and 5G NR (New Radio) can include However, this embodiment is not limited thereto.

The server 100 communicating through the network may comply with technical standards and standard communication methods for mobile communication. For example, standard communication methods include GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only) At least one of Wideband CDMA (WCDMA), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTEA), and 5G New Radio (NR) can include However, this embodiment is not limited thereto.

The above description is merely an example of the technical idea of the present embodiment, and various modifications and variations can be made to those skilled in the art without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but to explain, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of this embodiment should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights of this embodiment.

Claims (15)

In the facial similarity calculation method performed in the server associated with the customer terminal,
Collecting a plurality of video call data including a customer's face;
deriving facial images from the collected video call data and assigning them to each customer's account;
Determining a test target for determining facial similarity among a plurality of customers;
Calculating a vector value of a facial image belonging to an account of a customer included in the inspection target;
generating a matrix for each customer by using the number of video calls of the corresponding customer and the vector value of the face image corresponding to each video call;
converting the size of the matrix to be the same for each customer based on the number of video calls of the customer with the most call history among the customers belonging to the test target; and
Comprising the step of calculating facial similarity using the transformed inter-matrix operation
Facial similarity calculation method.
According to claim 1,
The matrix is
It is expressed as a product of a first matrix related to the number of video calls and a second matrix related to vector values of face images corresponding to each video call,
The first matrix includes one or more entries or elements corresponding to the number of video calls of the corresponding customer,
The component or the element is a positive integer
Facial similarity calculation method.
According to claim 2,
The step of converting the size of the matrix to be the same,
Setting the number of video calls of the customer with the most call history as the 'maximum number of video calls';
Based on the maximum number of video calls, converting all the sizes of the first matrix constituting the matrix of each customer to be the same
Facial similarity calculation method.
According to claim 1,
The step of calculating the vector value is,
Detecting a landmark in the face image;
Aligning the face image using the detected landmark;
For the aligned facial image, calculating the vector value using a deep learning module
Facial similarity calculation method.
According to claim 4,
Comparing the calculated facial similarity with a preset reference value to determine whether there is an abnormality;
If the abnormality is found, reconfirming whether the person is the same person based on the result of the visual inspection for the case;
Further comprising the step of re-learning the deep learning module using the image for the corresponding case if there is an error in the determination of the abnormality
Facial similarity calculation method.
According to claim 1,
The step of calculating the vector value is,
Calculate the vector value using a pre-trained deep learning module so that the same vector value is output for the image of the same person,
The deep learning module,
An input layer having the facial image as an input node;
An output layer having the facial similarity as an output node;
Including one or more hidden layers disposed between the input layer and the output layer;
The weights of nodes and edges between the input node and the output node are updated by the learning process of the deep learning module.
Facial similarity calculation method.
According to claim 1,
The step of calculating the vector value is,
Calculate the vector value using a pre-trained deep learning module so that the same vector value is output for the image of the same person,
The deep learning module,
A plurality of neural network modules into which different images are input;
a similarity determination module for calculating a similarity between values output from the plurality of neural network modules;
A weight module for adjusting weights for each of the calculated similarities;
And a feedback module for transmitting feedback about the error of the result value output from the weight module to the plurality of neural network modules.
Facial similarity calculation method.
According to claim 1,
The step of determining the inspection target includes receiving a selection for any one of a plurality of inspection methods and determining the inspection target based on the selected inspection method;
The inspection method,
A first inspection method for determining facial similarity between a customer and a reference image registered in a blacklist;
A second inspection method for determining facial similarity between facial images of a plurality of video calls belonging to a specific customer;
Including a third inspection method for determining the degree of facial similarity between facial images of different customers' video calls
Facial similarity calculation method.
According to claim 8,
The second inspection method and the third inspection method determine facial similarity using the matrix converted to the same size for each customer.
Facial similarity calculation method.
According to claim 8,
In the first inspection method, whether to perform is determined based on whether the blacklist is updated or whether the customer is a new customer
Facial similarity calculation method.
In the facial similarity calculation method performed in the server associated with the customer terminal,
(a) collecting a plurality of video call data including a customer's face;
(b) deriving a facial image from the collected video call data;
(c) calculating a vector value of the derived facial image and assigning it to the customer's account;
(d) repeating steps (a) to (c) for a plurality of customers;
(e) generating a matrix for each customer by using the number of video calls of the corresponding customer and the vector value of the face image corresponding to each video call;
(f) converting the size of the matrix to be the same for each customer based on the number of video calls of the customer with the most call history among the customers belonging to the test target; and
(g) calculating facial similarity using the transformed inter-matrix operation
Facial similarity calculation method.
a data collection unit that collects a plurality of video call data including a customer's face;
Among a plurality of customers, a test target determining unit for determining a test target for facial similarity determination;
a facial image derivation unit deriving a facial image for the plurality of video call data of the test subject; and
Including a calculation unit for calculating the facial similarity with respect to the facial image,
The calculation unit,
A feature calculation unit for calculating a vector value of a facial image belonging to an account of a customer included in the inspection target;
For each customer, a matrix generator for generating a matrix using the number of video calls of the corresponding customer and the vector value of the face image corresponding to each video call;
a matrix converter for converting the size of the matrix to be the same for each customer based on the number of video calls of the customer with the most call history among the customers belonging to the test target;
Comprising a facial similarity calculation unit for calculating facial similarity using the transformed inter-matrix operation
Facial similarity calculation system.
According to claim 12,
The matrix is
It is expressed as a product of a first matrix related to the number of video calls and a second matrix related to vector values of face images corresponding to each video call,
The first matrix includes one or more entries or elements corresponding to the number of video calls of the corresponding customer,
The matrix conversion unit,
The number of video calls of the customer with the most call history is set as the 'maximum number of video calls',
Converting the size of the first matrix constituting the matrix of each customer to be the same based on the maximum number of video calls
Facial similarity calculation system.
According to claim 12,
The feature calculation unit,
Calculate the vector value using a pre-trained deep learning module so that the same vector value is output for the image of the same person,
The deep learning module,
An input layer having the facial image as an input node;
An output layer having the facial similarity as an output node;
Including one or more hidden layers disposed between the input layer and the output layer;
The weights of nodes and edges between the input node and the output node are updated by the learning process of the deep learning module.
Facial similarity calculation system.
According to claim 12,
The inspection target determining unit,
Receiving a selection for any one of a plurality of inspection methods, and determining the inspection target based on the selected inspection method;
The inspection method,
A first inspection method for determining facial similarity between a customer and a reference image registered in a blacklist;
A second inspection method for determining facial similarity between facial images of a plurality of video calls belonging to a specific customer;
Including a third inspection method for determining the degree of facial similarity between facial images of different customers' video calls
Facial similarity calculation system.

Images