KR20210004195A - Integrated image generation apparatus of representative face using face feature embedding - Google Patents

Abstract

The present invention relates to an integrated image output device of a representative face using face feature embedding. To this end, the integrated image output device comprises: an image segment generating module for receiving original image data, which is an original image desired to be reprocessed for each person, and dividing the original image data into a plurality of pieces through scene change detection to generate a plurality of image segments; a face clustering module for receiving the image segments from the image segment generating module, and generating representative face data which is a face image of a representative person included in the original image data by clustering a face of a person in each of the image segments; and an image integration module for receiving selected face data, which is information on an image of the person desired to be reprocessed by a user, and integrally merging the image segments including the selected face data to generate integrated image data.

Description

An integrated image generation apparatus of representative face using face feature embedding

The present invention relates to an integrated image output apparatus of a representative face using facial feature embedding.

Recently, there is a growing demand to share and distribute various video contents by reprocessing them, rather than directly consuming them. As a specific example, K-POP idol fans reprocess existing broadcast images for each member to produce short video clips, or the broadcaster itself provides services such as summarizing the images for each person or producing a separate highlight image.

However, until now, most of the works mentioned above employ a method in which individuals manually find and edit a frame in which a specific person appears, so the work is very inefficient. There are technologies that can be used in the process of reprocessing people-centered content such as face recognition/identification, but in most cases, a large amount of data on the person is required to classify a specific person. There are many difficulties.

(Patent Document 1) Republic of Korea Patent Publication 10-2019-0021130, Face image-based similar image detection method and device, Samsung Electronics Co., Ltd.

Accordingly, an object of the present invention is a face that automatically edits and provides only images in which a desired person appears according to a user's selection based on a face clustering technique based on a machine learning algorithm that trains a face dataset composed of small people. It is to provide a high-speed image extraction apparatus using a clustering technique.

Hereinafter, specific means for achieving the object of the present invention will be described.

It is an object of the present invention to generate an image segment to generate a plurality of image segments by receiving original image data, which is an original image desired to be reprocessed for each person, and dividing the original image data into a plurality through scene change detection. module; A face clustering module configured to generate representative face data, which is a face image of a representative person included in the original image data, by receiving a plurality of the image segments in the image segment generation module and clustering the faces of a person in each of the image segments; And an image integration module configured to receive selected face data, which is information on an image of a person desired to be reprocessed by the user, from a user client, and to generate integrated image data by merging the image segments including the selected face data. Including, based on a face clustering technique based on a machine learning algorithm in which a face dataset composed of small people is trained, only the image in which the desired person appears according to the user's selection is automatically edited and output. This can be achieved by providing a high-speed image extraction apparatus using a face clustering technique.

As described above, the present invention has the following effects.

First, according to an embodiment of the present invention, a process in which a person manually identifies and edits a person appearing in every frame is provided as a semi-automatic service, thereby improving the efficiency of various person-centered image reprocessing work.

Second, according to an embodiment of the present invention, it is possible to accurately identify a large-scale person and provide it to a user by using a network in which a small-scale person data set is learned.

Third, according to an embodiment of the present invention, it is easy for K-POP idol fans to reprocess existing broadcasts, videos taken directly, etc., and share them on SNS mainly by individual members, and highlight broadcast programs provided by major broadcasters. It becomes easier to distribute reprocessed images mainly for people in summary images and the like. In addition, it becomes easier to extract each player's activity image from the entire game video in a sports event and provide it to consumers individually. Also, people who appear in a long video such as CCTV are individually classified and the desired person (ex. It can be used to find the point at which the public offender) appeared.

The following drawings attached to the present specification illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention together with the detailed description of the present invention, so the present invention is limited to the matters described in such drawings. And should not be interpreted.
1 is a schematic diagram showing a high-speed image extraction apparatus using a face clustering technique according to an embodiment of the present invention;
2 is a schematic diagram showing a configuration of a face clustering module 20 according to an embodiment of the present invention;
3 is a schematic diagram showing the configuration of the clustering module 24 according to an embodiment of the present invention.

Hereinafter, exemplary embodiments in which the present invention can be easily implemented by those of ordinary skill in the art will be described in detail with reference to the accompanying drawings. However, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

In addition, the same reference numerals are used for parts having similar functions and functions throughout the drawings. Throughout the specification, when a specific part is said to be connected to another part, this includes not only the case that it is directly connected, but also the case that it is indirectly connected with another element interposed therebetween. In addition, the inclusion of a specific component does not exclude other components unless specifically stated to the contrary, but means that other components may be further included.

High-speed image extraction device using face clustering technique

1 is a schematic diagram showing a high-speed image extraction apparatus using a face clustering technique according to an embodiment of the present invention. As shown in FIG. 1, a high-speed image extraction apparatus 1 using a face clustering technique according to an embodiment of the present invention includes an image segment generation module 10, a face clustering module 20, and an image integration module 30. ) Can be included. The high-speed image extraction device 1 using the face clustering technique according to an embodiment of the present invention is processed by a processing module of a computing device such as a specific web server, a virtual server such as a cloud server, a smartphone, a tablet PC, and a desktop PC. And may be configured to be stored in a memory module of each device.

The image segment generation module 10 receives the original image data 100, which is the original image desired to be reprocessed for each person, and divides the original image data 100 into a plurality of pieces through scene change detection. This module creates video segments.

The face clustering module 20 receives an image segment from the image segment generation module 10, and clusters the faces of a person in each image segment to represent a representative person's face image included in the entire original image data 100. It is a module that generates face data 200. The generated representative face data 200 may be transmitted to user clients such as smartphones, tablets, desktops, and laptops through a web or an app. The face clustering module 20 according to an embodiment of the present invention may be performed by face image clustering based on a machine learning algorithm. Various faces detected in the entire image are clustered for each person A representative picture is provided to the user.

The image integration module 30 receives selected face data 300, which is information on an image of a person that the user wants to reprocess, from a user client, and integrates the image segment including the selected face data 300 (Video merging). This is a module that generates the integrated image data 310. The generated integrated image data 310 may be transmitted (including streaming) to user clients such as smartphones, tablets, desktops, and laptops through a web or an app.

Regarding the specific configuration of the face clustering module 20, FIG. 2 is a schematic diagram showing the configuration of the face clustering module 20 according to an embodiment of the present invention. As shown in Fig. 2, the face clustering module 20 according to an embodiment of the present invention includes a face detection module 21, a landmark detection module 22, a standardization module 23, and a clustering module 24. Can include.

The face detection module 21 is a module that detects a face part from the received image segment 110 and generates face detection data (eg, bounding box). The face detection module 21 according to an embodiment of the present invention may use a face detection algorithm fine-tuning YOLO, RCNN, Faster RCNN, and the like. Alternatively, a face detection algorithm that fine-tuned a network such as AlexNet previously learned with ImageNet can be used. Furthermore, it is possible to use existing computer vision algorithms such as Boosting the Haar-like Feature of Viola-jones.

The landmark detection module 22 is a module that generates landmark data by detecting a landmark of a corresponding face based on the face detection data generated by the face detection module 21. The landmark detection module 22 according to an embodiment of the present invention may be configured with a Cascade CNN-based architecture or an architecture including an auto encoder.

The standardization module 23 is a module for generating standardized face data by standardizing the input face detection data based on the landmark data generated by the landmark detection module 22.

The clustering module 24 generates and outputs representative face data 200, which is a representative picture of each person, by clustering various faces detected in the entire image for each person based on the standardized face data generated by the standardization module 23. It is a module.

Regarding the specific configuration of the clustering module 24, the clustering module 24 according to an embodiment of the present invention can be configured in the form of Supervised Learning, and includes a linear/logistic regression analysis and a support vector machine. Vector Machine), Multi-layer perceptron, Naive-Bayesian Classification, Random Forest Classification, Neural Network, etc.It can be composed of various machine learning algorithms. For convenience of explanation, the following describes an example in which the clustering module 24 is configured as an artificial neural network according to an embodiment of the present invention. Hereinafter, the convolution layer may be mixed with “CONV layer” and “Conv. layer” for convenience of description, and the convolutional neural network is “ConvNet”, “CNN”, etc. Can be used interchangeably.

3 is a schematic diagram showing the configuration of the clustering module 24 according to an embodiment of the present invention. As shown in FIG. 3, the clustering module 24 according to an embodiment of the present invention is composed of an artificial neural network module including a convolution layer 241, a pooling layer 242, and a face clustering layer 243. I can.

According to an embodiment of the present invention, the standardized face data 230, which is an INPUT input image to the clustering module 24, may have 47 horizontal, 55 vertical, and RGB channels, and at this time, the input standardized face data 230 The size is [47x55x3]. The convolution filter (Conv.Filter) is connected to a part of the standardized face data 230, which is an input image, and calculates the dot product of the connected region and its weight, and the kernel size is [4x4x3 It can be composed of ]. The resulting volume, Conv. layer, 241 has the same size as [44x52x20]. The RELU layer is an activation function applied to each element, such as max(0,x). The RELU layer does not change the volume size ([44x52x20]). As a result, we create an activation map. The pooling layer 242 performs downsampling on the "horizontal, vertical" dimension to output a reduced volume (Activation map) such as [22x26x20]. After that, the second convolution layer, the second pooling layer, the third convolution layer, the third pooling layer, and the fourth convolution layer are connected, and n nodes are immediately connected from the fourth convolution layer. The face clustering layer 243, which is an excitation output layer, may be configured to be connected.

In the clustering module 24 according to an embodiment of the present invention, unlike a general ConvNet structure, the face clustering layer 243, which is an output layer, is directly connected to the last convolution layer. That is, the clustering module 24 according to an embodiment of the present invention uses only the face feature embedding step, not the final identification result, in the face identification network learned from the face dataset for n people. Accordingly, even if a face that is not included in the learning data appears in the target image, it is not possible to know what kind of person the person is, but it is possible to classify the objects into individual persons A and B, and thus can be applied to the present invention system.

In the conventional general ConvNet, n fully-connected layers (FCs) are connected after the last convolution layer or the last pooling layer. The fully-connected (FC) layer 105 calculates class scores and outputs an output layer 106 having a size of [1x1x10]. The FC layer is connected to all elements of the previous volume and is responsible for the final identification.

As described above, the ConvNet of the clustering module 24 according to an embodiment of the present invention passes the original image (normalized face data, 230) composed of pixel values through each layer, and embeds the corresponding standardized face data as a specific vector. Data can be created. Some layers have parameters, while others do not. In particular, CONV layers are activation functions that include not only the input volume but also the weight and bias. On the other hand, RELU/POOL layers are fixed functions. The parameters of the CONV layer are learned with gradient descent so that the class score for each image is the same as the label of the image.

The parameters of the CONV layer consist of a series of learnable filters. Each filter is small in the horizontal/vertical dimension but covers the entire depth in the depth dimension. In the forward pass, each filter is slid (convolved precisely) to the horizontal/vertical dimensions of the input volume, and a two-dimensional activation map is created. When sliding the filter over the input, a dot product is performed between the filter and the input volume. Through this process, ConvNet learns a filter that activates for a specific pattern at a specific location of the input data. The stacking of these activation maps in the depth dimension becomes the output volume. Therefore, each element of the output volume only handles a small area of the input, and neurons in the same activation map share the same parameters as the result of applying the same filter.

According to an embodiment of the present invention, a vanishing gradient problem in which an error is diluted in a layer at the front end occurs while applying a chain rule in back propagation, so that ReLU may be used instead of a sigmoid function. When using the sigmoid function, all values must be calculated, but the ReLU function can reduce the amount of computation in a significant part, thereby improving the computing speed. Regularization can be improved by the ReLU function.

In addition, in the learning of the clustering module 24 according to an embodiment of the present invention, the standardized face data 230 input in a training session is randomly cropped to diversify the input data into a plurality of patches. , It is possible to improve the accuracy in the inference session.

In addition, in the learning of the clustering module 24 according to an embodiment of the present invention, the SoftMax Loss is assumed to be Identification Loss, and the Loss using Euclidean Distance is assumed to be Verification Loss, and accuracy using a multi-task learning session Can improve.

As described above, those skilled in the art to which the present invention pertains will appreciate that the present invention can be implemented in other specific forms without changing the technical spirit or essential features thereof. Therefore, the above-described embodiments are illustrative in all respects and should be understood as non-limiting. The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present invention.

The features and advantages described herein are not all inclusive, and in particular many additional features and advantages will become apparent to those skilled in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used herein has been selected primarily for readability and for teaching purposes, and may not be chosen to describe or limit the subject matter of the invention.

The above description of embodiments of the present invention has been presented for purposes of illustration. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Those skilled in the art will understand that many modifications and variations are possible in light of the above disclosure.

Therefore, the scope of the invention is not limited by the detailed description, but by any claims in the application on which it is based. Accordingly, the disclosure of the embodiments of the present invention is illustrative and does not limit the scope of the present invention described in the following claims.

1: Fast image extraction device using face clustering technique
10: video segment module
20: Face clustering module
21: face detection module
22: landmark detection module
23: standardization module
24: clustering module
30: video integration module
100: original image data
110: video segment
200: representative face data
230: standardized face data
241: convolution layer
242: pooling layer
243: Face clustering layer
300: Select face data
310: Integrated video data

Claims (1)

An image segment generation module for receiving original image data, which is an original image desired to be reprocessed for each person, and generating a plurality of image segments by dividing the original image data into a plurality of pieces through scene change detection;
A face clustering module configured to generate representative face data, which is a face image of a representative person included in the original image data, by receiving a plurality of the image segments in the image segment generation module and clustering the faces of a person in each of the image segments; And
An image integration module configured to receive selected face data, which is information on an image of a person desired to be reprocessed by a user, from a user client, and to generate integrated image data by merging the image segments including the selected face data;
Including,
The face clustering module includes an artificial neural network including a convolution layer, and a face clustering layer, which is an output layer of the artificial neural network, is directly connected to the last convolution layer, and faces for a plurality of people The representative face data is generated using only the face feature embedding step, not the final identification result in the face identification network learned with the dataset,
Characterized in that, based on a face clustering technique based on a machine learning algorithm in which a face dataset consisting of small-scale people is trained, only the image in which the desired person appears according to the user's selection is automatically edited and output.
An integrated image output device of representative faces using facial feature embedding.

Images