KR20230141371A - Face tracking device and face tracking method using image generation network - Google Patents

Abstract

The present invention relates to a face tracking device using an image generation network, and to track an object in an image input from a user, at least one additional image containing different characteristics of at least one object from the initial image of the image. an image generation network unit that generates; an information extraction unit that extracts feature information present in the initial image and at least one additional image; an object tracking model generator that generates an object tracking model for tracking a face based on the feature information; and an object tracking unit that tracks the face in the image input from the user using the object tracking model. Through this, various appearances of the object that change depending on the movement of the camera and the environment are created through the image generation network, making it possible to track the object.

Description

FACE TRACKING DEVICE AND FACE TRACKING METHOD USING IMAGE GENERATION NETWORK}

The present invention relates to a face tracking device and a face tracking method using an image generation network. More specifically, the present invention relates to a face tracking device and a face tracking method using an image generation network to determine the characteristics of an object that changes depending on the camera movement and the environment and perform object tracking. This relates to a face tracking device and face tracking method using an image generation network.

Conventional object tracking technology plays an important role as a basic cutting-edge technology in various fields such as autonomous vehicles, road traffic monitoring, and medical systems.

Object tracking technology is a technology that recognizes specific objects in successive frames, that is, images, and performs the process of continuously displaying the objects. It includes a single object tracking algorithm that tracks a single object through a learned machine learning model, and a multi-object tracking algorithm. It is classified as a multi-object tracking algorithm that tracks objects.

The single object tracking algorithm is an algorithm that tracks only a specific object regardless of the number of objects in the frame and the class of the object, while the multi-object tracking algorithm is an algorithm that sequentially tracks all classes in the frame, such as the object to be tracked.

These single or multiple object tracking algorithms require a separate initialization step because they use a deep learning model that has already been trained to recognize objects, and use a separate object recognition algorithm in the first frame or directly place the object in the bounding box. A problem arises that needs to be flagged.

In addition, the algorithm that tracks objects using deep learning tracks objects through the features included in the objects, but the features of the objects can easily change depending on various factors such as light, shadows, occlusion by other objects, and camera shake, so the initial Since the feature information of the object obtained cannot be used, a problem occurs in which face tracking accuracy is low.

Therefore, when a single or multiple object tracking algorithm tracks an object in an image, research and development on technology to track the object with higher accuracy is needed.

(Republic of Korea) Registered Patent Publication No. 10-2167760

The present invention was created to solve the above problems, and the purpose of the present invention is to extract feature information from an image generated through an image generation network and generate an image that tracks the face of an object based on the extracted feature information. The aim is to provide a face tracking device and a face tracking method using a network.

A face tracking device using an image generation network according to an embodiment of the present invention to achieve the above object is a face tracking device that uses an image generation network to track an object in an image input from a user, and includes an initial part of the image. an image generation network unit generating at least one additional image including different features of at least one object from the image; an information extraction unit that extracts feature information present in the initial image and at least one additional image; an object tracking model generator that generates an object tracking model for tracking a face based on the feature information; and an object tracking unit that tracks the face in the image input from the user using the object tracking model.

At this time, the additional image may be generated including at least one face area of the object present in the initial image transformed at different angles.

In this regard, the image generation network unit may include an area setting unit that sets a face area of the object present in the initial image; a feature vector generator that extracts features from the face area and inputs the features into a preset neural network model to generate different feature vectors; and an additional image generator that generates the additional image in which the face area of the object is transformed at different angles based on the different feature vectors.

Accordingly, the object tracking model generator analyzes feature information extracted from each of the initial image and the at least one additional image to classify the object as the foreground when the face area is a specific object, and the face area is a specific object. If it is not an object, a classification information filter generator that generates a classification information filter that classifies the object as a background and calculates classification information; And generating an object frame information filter that displays the face area of the object included in each of the initial image and the at least one additional image in the form of a frame and calculates frame information of the face area of the object displayed in the frame form. It may include an object frame information filter creation unit.

In addition, the object tracking model generator may further include a model generator configured to generate the object tracking model by performing a convolution operation on the classification information filter and the object frame information filter.

Accordingly, the object tracking unit receives real-time feature information extracted from the image input from the user from the information extraction unit, and applies the real-time feature information to the object tracking model to track and display the specific object, Classify whether the face area of the real-time feature information is foreground or background using a classification information filter, and display the face area of the real-time feature information in the form of a frame using the object frame information filter to select the face area from the image. Objects can be tracked and marked.

Meanwhile, a face tracking method according to an embodiment of the present invention for achieving the above object is a method performed by a face tracking device using an image generation network, and includes an input step of receiving an image from a user; An image generation step of generating at least one additional image including different features of at least one object from the initial image of the video; An information extraction step of extracting feature information present in the initial image and at least one additional image; An object tracking model generation step of generating an object tracking model for tracking a face based on the feature information; and an object tracking step of tracking the face in an image input from the user using the object tracking model.

At this time, the additional image may be generated including at least one face area of the object present in the initial image transformed at different angles.

In this regard, the image generating step includes: an area setting step of setting a face area of the object present in the initial image; A feature vector generation step of extracting features from the face area and inputting the features into a preset neural network model to generate different feature vectors; and an additional image generating step of generating the additional image in which the face area of the object has a deformed shape at different angles based on the different feature vectors.

Accordingly, the object tracking model generation step analyzes feature information extracted from each of the initial image and the at least one additional image to classify the object as the foreground when the face area is a specific object, and the face area is a specific object. A classification information filter creation step of generating a classification information filter that classifies the object as a background if it is not a specific object and calculates classification information; And generating an object frame information filter that displays the face area of the object included in each of the initial image and the at least one additional image in the form of a frame and calculates frame information of the face area of the object displayed in the frame form. It may include an object frame information filter creation step.

In addition, the object tracking model generating step may further include a model generating step of generating the object tracking model by performing a convolution operation on the classification information filter and the object frame information filter.

Therefore, the object tracking step receives real-time feature information extracted from the image input from the user in the information extraction step, and applies the real-time feature information to the object tracking model to track and display the specific object, The classification information filter is used to classify whether the face area of the real-time feature information is foreground or background, and the object frame information filter is used to display the face area of the real-time feature information in the form of a frame to select the face from the image. You can track and mark specific objects.

According to one aspect of the present invention described above, by providing a face tracking device and a face tracking method using an image generation network, various appearances of the object that change depending on the movement of the camera and the environment are generated through the image generation network to track the object. It can be used as training data for deep learning models.

As a result, a face tracking device and a face tracking method using an image generation network with high tracking accuracy can be provided to users.

Figure 1 is a block diagram of a face tracking device using an image generation network according to an embodiment of the present invention.
FIG. 2 is a block diagram of the image generation network unit of FIG. 1.
FIG. 3 is a block diagram of the object tracking model creation unit of FIG. 1.
Figures 4 and 5 are example diagrams of a face tracking device using the image generation network of Figure 1.
6 to 8 are flow diagrams of a face tracking method according to an embodiment of the present invention.

The detailed description of the present invention described below refers to the accompanying drawings, which show by way of example specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different from one another but are not necessarily mutually exclusive. For example, specific shapes, structures and characteristics described herein may be implemented in one embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description that follows is not intended to be taken in a limiting sense, and the scope of the invention is limited only by the appended claims, together with all equivalents to what those claims assert, if properly described. Similar reference numbers in the drawings refer to identical or similar functions across various aspects.

In addition, the features and advantages of the present invention will become clearer with the detailed description based on the accompanying drawings, and the terms and words used in the present specification and claims should not be interpreted in the usual, dictionary meaning, and the inventor should not In order to explain one's invention in the best way, it should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that the concept of the term can be appropriately defined.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a block diagram of a face tracking device using an image generation network according to an embodiment of the present invention, FIG. 2 is a block diagram of the image generation network unit of FIG. 1, and FIG. 3 is a block diagram of the object tracking model creation unit of FIG. 1. These are diagrams, and FIGS. 4 and 5 are exemplary diagrams of a face tracking device using the image generation network of FIG. 1.

Referring to FIG. 1, a face tracking device (hereinafter referred to as a face tracking device) 10 using an image generation network is an image generation network that generates different images from an initial image in order to track an object in an image input from a user. It is designed to create an object tracking model using and to track and display specific objects in the image using the object tracking model.

Specifically, object tracking technology is a technology used in various fields such as autonomous driving, road traffic surveillance, and medical systems to track specific objects within the currently used video.

This object tracking technology is comprised of a deep learning model to track objects within an input image. In order to track an object, the characteristics of the tracked object according to various environmental changes are learned in advance using a lot of learning data. Uses a deep learning model.

However, object tracking technology that includes a deep learning model has a great advantage in accurately tracking objects in an image in real time over offline object tracking methods due to its fast calculation speed, but the performance of the deep learning model There is a problem that is greatly affected by .

Then, object tracking technology performed including these deep learning models is affected in real time by various factors such as light, occlusion, and unstable camera movement within the input image, so higher learning data is required, and for this purpose, , an object tracking device that uses an image generation network to generate separate images must be developed.

Accordingly, the face tracking device 10 of the present invention can be prepared to generate an additional image corresponding to the initial image in the video using an image generation network that is a deep learning model.

In addition, the face tracking device 10 of the present invention generates an object tracking model based on the feature information and a Siamese network that extracts feature information from the initial image and additional images, and uses the object tracking model to track a specific object. A proposal network (Region Proposal Network, RPN) may be established.

At this time, even if it is not the image generation network, Siamese network, or local proposal network proposed in the present invention, a deep learning model that generates an image through a known deep learning model and extracts features from the generated image to determine the object can be used. there is.

To this end, as shown in FIG. 1, the face tracking device 10 according to an embodiment of the present invention includes an image generation network unit 11, an information extraction unit 13, an object tracking model creation unit 15, and It may include an object tracking unit 17.

In addition, the face tracking device 10 may be installed and executed with software (application) for performing a face tracking method, and includes an image generation network unit 11, an information extraction unit 13, and an object tracking model creation unit. (15) and the object tracking unit 17 may be controlled by software for performing a face tracking method executed in the face tracking device 10.

In addition, although not shown in the drawing, the face tracking device 10 includes an input unit that receives an image from the user, a storage unit that stores at least one deep learning model used in the face tracking device 10, and various information from the user's external device. It may further include components such as a communication unit and an output unit for transmitting and receiving video and image information, and these components may also be controlled by software for performing a face tracking method executed in the face tracking device 10. .

Additionally, the face tracking device 10 may be a separate terminal or a partial module of the terminal. Additionally, the image generation network unit 11, the information extraction unit 13, the object tracking model creation unit 15, and the object tracking unit 17 may be formed as an integrated module or may be comprised of one or more modules. However, on the contrary, each configuration may be comprised of a separate module.

Additionally, the face tracking device 10 can be mobile or fixed. The face tracking device 10 may be in the form of a server or engine, and may be a device, apparatus, terminal, user equipment (UE), mobile station (MS), or wireless device. ), handheld device, etc.

Additionally, the face tracking device 10 can run or produce various software based on an operating system (OS), that is, a system. The operating system is a system program that allows software to use the hardware of the device, and includes mobile computer operating systems such as Android OS, iOS, Windows Mobile OS, Bada OS, Symbian OS, Blackberry OS, Windows series, Linux series, Unix series, etc. It can include all computer operating systems such as MAC, AIX, and HP-UX.

First, the image generation network unit 11 generates at least one additional image including different characteristics of at least one object from the initial image of the image input from the user.

Referring to FIG. 2, the image generation network unit 11 extracts features from the area setting unit 111 and the face area for setting the face area of the object present in the initial image of the image, and inputs the features into a preset neural network model. A feature vector generator 113 that generates different feature vectors by inputting them, and an additional image generator 115 that generates additional images in which the face area of the object is transformed at different angles based on the different feature vectors. may include.

In this regard, the region setting unit 111, the feature vector generating unit 113, and the additional image generating unit 115 are provided in a configuration that is performed through one deep learning model, as shown in FIG. 4, or each The configuration may be prepared as a configuration performed through a separate deep learning module.

More specifically, Figure 4 is an example of generating at least one additional image including different features from an image input through the image generation network unit 11. Referring to Figure 4, the image generation network unit 11 Based on the first image of the image input from the user, that is, the initial image, an image in which the angle of the object present in the initial image is modified can be additionally created.

In this regard, when an image is input to the image generation network unit 11, the area setting unit 111 may set the face area of the object by assigning a preset area to the object existing in the initial image of the image.

Additionally, the area setting unit 111 may project the face area of the object present in the initial image into the latent space. Here, the area setting unit 111 projects the initial image into the latent space in order to confirm the distribution of object features existing in the initial image in a high-dimensional form.

To this end, the area setting unit 111 uses the COCO data set as learning data and can project the face area of the object present in the initial image into the latent space using the Inception-V2 network that learned the COCO data.

Meanwhile, the feature vector generator 113 may extract features existing in the face area of the object and generate different feature vectors based on the features.

More specifically, the feature vector generator 113 extracts only the features of the face area of the object from the initial image projected by the area setting unit 111, and inputs the features of the extracted face area into a preset neural network model to create different A feature vector can be created.

In this regard, the feature vector generator 113 uses a style GAN (Generative Adversarial Network) neural network, which consists of a mapping network that stores the features of the input image and an image generation network that generates an image using the features stored in the mapping network. It can be used as a model.

The feature vector generator 113 can align the feature distribution of the face area using a mapping network designed in style GAN, and extract feature vectors that need to be adjusted from the feature distribution sorted through AdaIN.

Here, the mapping network is a fully connected network composed of 8 FC layers and classifies pre-learned data according to layer depth. When the feature distribution consisting of the data distribution is input, the feature distribution is stored, and in the mapping process By removing the resulting distortion as much as possible, the correlation between feature distributions can be reduced.

Additionally, the feature vector generator 113 can adjust features that change according to learning data learned in advance by the mapping network.

For example, if the fifth feature vector present in the feature distribution sorted according to pre-learned data is a feature of the overall facial shape, the feature vector generator 113 selects the hair or face color of a specific object included in the initial image. The feature vector can be adjusted so that the back is transformed into different shapes.

Through this, the feature vector generator 113 uses a neural network model to extract features from the feature distribution of the face area converted by the region setting unit 111, and adjusts the feature vectors of the extracted features to have different values. A feature vector can be created.

Meanwhile, the additional image generator 115 may generate an additional image based on the initial image using the neural network model used by the feature vector generator 113, that is, style GAN.

Here, the feature vector generator 113 and the additional image generator 115 are described as separate components for easy description of the present invention, but as shown in FIG. 4, one deep learning model is performed on the same deep learning model. It can be prepared as a configuration, and it can be prepared as a plurality of configurations performed through different deep learning models.

More specifically, the additional image generator 115 uses the image generation network of style GAN to apply feature vectors with different values generated by the feature vector generator 113 to the initial image, thereby creating a new image corresponding to the initial image. Additional images can be created. Here, the additional image generator 115 may use PGAN (Progressive GAN) as an image generation network.

By using PGAN as an image generation network, the additional image generator 115 can solve the resolution problem of the existing GAN by gradually increasing or decreasing the size of input/output data.

Additionally, the additional image generator 115 may adjust the sizes of the initial image and the additional image in order to input the initial image and the additional image into the information extraction unit 13, which will be described in detail below.

More specifically, the additional image generator 115 may convert the initial image and the additional image generated with a resolution of 512x512 to a size of 127x127x3.

Here, if the size converted by the additional image generator 115 is less than 127x127, a problem of poor performance occurs when extracting features. To solve this problem, the additional image generator 115 can convert the image to a size of 127x127x3 by filling the image with the RGB average values of all pixels.

Therefore, it is most desirable for the additional image generator 115 to convert the input size to 127x127x3, considering the time it takes to generate the image and the accuracy of the generated image.

Meanwhile, the image generation network unit 11 may further include a size adjustment unit 117 that adjusts the size of the initial image before generating an additional image from the initial image.

The size adjustment unit 117 can convert the size of the initial image to 512x1 to be used in the style GAN used by the feature vector generation unit 113 and the additional image generation unit 115.

In this way, the image generation network unit 11 of the present invention can generate additional images from the initial image in the image input from the user.

Accordingly, at least one additional image may be generated including the face area of the object present in the initial image transformed at different angles.

The information extraction unit () extracts feature information present in the initial image and at least one additional image generated by the image generation network unit ().

More specifically, referring to Figure 5, the information extraction unit () may use a Siamese network using two identical CNN networks based on VGC-16.

Here, the Siamese network is a network that has a structure in which input terminals exist in pairs in order to simultaneously receive a plurality of images, and extracts feature information of the simultaneously input images.

The information extraction unit may input the size-converted initial image and at least one additional image to the Siamese network as a pair to an input terminal to extract feature information present in each image.

Meanwhile, the object tracking model generator 155 generates an object tracking model for tracking a face based on feature information extracted from each image.

To this end, referring to FIG. 3, the object tracking model creation unit 155 may be provided including a classification information filter creation unit 151, an object frame information filter creation unit 153, and a model creation unit 155. .

More specifically, the classification information filter generator 151 analyzes feature information extracted from each of the initial image and at least one additional image, and when the face area is a specific object, classifies the object as the foreground, and the face area is a specific object. If it is not an object, a classification information filter can be created that classifies the object as the background and calculates classification information.

Here, the classification information filter generator sets the classification information to include a true value when the classification information filter determines the object is the foreground, and sets the classification information to include a false value when the classification information filter determines the object is the background. You can.

Therefore, the classification information filter generator 151 classifies the object as a foreground or background depending on whether the object is a specific object, and calculates classification information including a true value or a false value according to the classification result. You can create filters.

Meanwhile, the object frame information filter generator 153 displays the face area of the object included in each of the initial image and at least one additional image in the form of a frame, and calculates frame information of the face area of the object displayed in the form of a frame. You can create a frame information filter.

More specifically, the object frame information filter generator 153 sets the object frame information filter to display the face area of the object included in each image in the form of a frame, and includes an x-axis value according to the face area displayed in the frame form, The y-axis value, length value, and height value can be set in advance to be calculated as frame information for each image.

Therefore, the object frame information filter generator 153 generates an object frame information filter that calculates frame information including the x-axis value, y-axis value, length value, and height value according to the face area of the object appearing in each image. can do.

In this regard, the model generator 155 may generate an object tracking model by performing a convolution operation on the classification information filter and the object frame information filter.

More specifically, the model generator 155 may use a Region Proposal Network (RPN) to perform a convolution operation on the classification information filter and the object frame information filter. Here, RPN is a deep learning model that predicts the score for the location of the object and the boundary of the object, and is a model that performs calculations on features using the feature map extracted from CNN.

First, the model generator 155 can align the dimensions of the classification information filter and the object frame information filter using the stack layer.

Thereafter, the model generator 155 may perform a convolution operation in which the dimensionally aligned classification information filter and object frame information filter are converted to a size of 6x6x (2k x 512)x3. Here, k is the number of anchors, which are predefined bounding boxes of various sizes.

In addition, the model generator 155 can further perform a convolution operation to convert the size of the classification information filter and object frame information filter to 1x1 size, thereby adjusting the number of channels while maintaining the existing filter as much as possible. Here, the model generator 155 further performs the convolution operation to convert the size to 1x1 to include the same channel dimension as the image in order to track the object in the image converted to feature information.

Accordingly, the object tracking model generator 155 tracks and displays the object from the image converted into feature information through the classification information filter generator 151, the object frame information filter generator 153, and the model generator 155. You can create an object tracking model that does this.

Meanwhile, the object tracking unit () tracks the face in the image input from the user using an object tracking model.

More specifically, referring back to FIG. 5, the object tracking unit () may receive feature information extracted from an image input by the information extraction unit ().

In addition, the object tracking unit () applies the real-time feature information delivered from the information extraction unit () to the object tracking model to track and display a specific object, and uses a classification information filter to ensure that the face area of the real-time feature information is in the foreground or It is possible to track and display a specific object that is a face from an image by classifying whether it is a background and displaying the face area of real-time feature information in the form of a frame using an object frame information filter.

In this regard, the object tracking unit () uses a classification information filter to input real-time feature information into the classification information filter, predict an anchor corresponding to the face area of the object included in the real-time feature information, and predict the object within the predicted anchor. Filtering may be performed to classify whether is a specific object or background.

In addition, the object tracking unit () uses the object frame information filter to input real-time feature information into the object frame information filter, set the coordinate value of the face area included in the real-time feature information, and set the real-time feature information based on the set coordinate value. Filtering may be performed to display the facial area according to the information in the form of a frame.

Accordingly, the object tracking unit () tracks and tracks an object in an image from which feature information is extracted in real time using an object tracking model generated from feature information extracted from the initial image and additional images generated based on the initial image. By displaying this completed specific object in the form of a frame, a specific object, that is, a face, can be tracked and displayed in the video in real time.

Meanwhile, FIGS. 6 to 8 are flow diagrams of a face tracking method according to an embodiment of the present invention. The face tracking method according to an embodiment of the present invention is the same as the face tracking device 10 shown in FIGS. 1 to 5. Since this is done in terms of configuration, the same reference numerals as the face tracking device 10 of FIGS. 1 to 5 will be assigned, and repeated descriptions will be omitted.

Referring to FIG. 6, the face tracking method according to an embodiment of the present invention is a face tracking method performed by the face tracking device 10 using an image generation network, and includes an input step 610, an image generation step 630, It includes an information extraction step (650), an object tracking model creation step (670), and an object tracking step (690).

First, the face tracking device 10 receives an image from the user in an input step 610.

Thereafter, in the image generation step 630, the face tracking device 10 generates at least one additional image including different features of at least one object from the initial image of the image.

More specifically, referring to FIG. 7, the image generation step 630 includes the region setting step 631 of setting the face region of the object present in the initial image, extracting features from the face region, and entering the features into a preset neural network model. A feature vector generation step 633 of generating different feature vectors by inputting them, and an additional image generation step 635 of generating additional images in which the face area of the object is transformed at different angles based on the different feature vectors. may include.

In this regard, the face tracking device 10 extracts feature information present in the initial image and at least one additional image in the information extraction step 650.

Next, the face tracking device 10 generates an object tracking model for tracking the face based on each feature information in the object tracking model creation step 670.

To this end, the object tracking model creation step may be prepared as a classification information filter creation step (), an object frame information filter creation step (), and a model creation step ().

More specifically, the face tracking device 10 analyzes feature information extracted from each of the initial image and at least one additional image in the classification information filter creation step () to classify the object as the foreground if the face area is a specific object, If the face area is not a specific object, a classification information filter can be created that classifies the object as the background and calculates classification information.

In addition, the face tracking device () displays the face area of the object included in each of the initial image and at least one additional image in the form of a frame in the object frame information filter creation step (), and displays a frame of the face area of the object displayed in the form of a frame. You can create an object frame information filter that produces information.

Through this, the face tracking device () can generate an object tracking model by performing a convolution operation on the classification information filter and the object frame information filter in the model creation step ().

Next, the face tracking device 10 tracks the face in the image input from the user using an object tracking model in the object tracking step 690.

More specifically, when the object tracking step 690 receives real-time feature information extracted from the image input from the user in the information extraction step 650, the real-time feature information is applied to the object tracking model to track and display a specific object. You can.

At this time, the object tracking step 690 uses a classification information filter to classify whether the face area of the real-time feature information is foreground or background, and uses an object frame information filter to display the face area of the real-time feature information in the form of a frame from the image. It can track and display specific objects, such as faces.

Accordingly, the face tracking device 10 using an image generation network can perform a face tracking method to track and display a specific object in an image input from the user.

This face tracking method can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc., singly or in combination.

The program instructions recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or may be known and usable by those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and perform program instructions, such as ROM, RAM, flash memory, etc.

Examples of program instructions include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the invention and vice versa.

Although various embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and may be used in the technical field to which the invention pertains without departing from the gist of the invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

10: Face tracking device using image generation network
11: Image generation network unit
111: Area setting unit
113: Feature vector generation unit
115: Additional image creation unit
117: size control unit
13: Information extraction unit
131: Classification information extraction unit
133: Feature information extraction unit
15: Object tracking model creation unit
17: Object tracking unit

Claims (12)

In a face tracking device using an image generation network that tracks a face in an image input from a user,
an image generation network unit generating at least one additional image including different features of at least one object from the initial image of the image;
an information extraction unit that extracts feature information present in the initial image and at least one additional image;
an object tracking model generator that generates an object tracking model for tracking a face based on the feature information; and
A face tracking device using an image generation network, comprising: an object tracking unit that tracks the face in an image input from the user using the object tracking model.
According to paragraph 1,
The additional image above is,
A face tracking device using an image generation network, wherein at least one facial area of the object present in the initial image is generated including a deformed shape at different angles.
According to paragraph 2,
The image generation network unit,
an area setting unit that sets a face area of the object present in the initial image;
a feature vector generator that extracts features from the face area and inputs the features into a preset neural network model to generate different feature vectors; and
A face tracking device using an image generation network, comprising: an additional image generator that generates the additional image in which the face area of the object has a deformed shape at different angles based on the different feature vectors.
According to paragraph 3,
The object tracking model creation unit,
By analyzing the feature information extracted from each of the initial image and the at least one additional image, if the face area is a specific object, the object is classified as the foreground, and if the face area is not a specific object, the object is classified as the background a classification information filter generator that generates a classification information filter that calculates classification information; and
An object that displays a face area of the object included in each of the initial image and the at least one additional image in the form of a frame, and generates an object frame information filter for calculating frame information of the face area of the object displayed in the frame form. A face tracking device using an image generation network, including a frame information filter generator.
According to paragraph 4,
The object tracking model creation unit,
A face tracking device using an image generation network, further comprising a model generator configured to generate the object tracking model by performing a convolution operation on the classification information filter and the object frame information filter.
According to clause 5,
The object tracking unit,
The information extraction unit receives real-time feature information extracted from the image input from the user, applies the real-time feature information to the object tracking model to track and display the specific object, and uses the classification information filter to track and display the specific object. Classifying whether the face area of the real-time feature information is foreground or background, and displaying the face area of the real-time feature information in the form of a frame using the object frame information filter to track and display the specific object that is the face from the image, Face tracking device using image generation network.
In a face tracking method performed by a face tracking device using an image generation network,
An input step of receiving an image from a user;
An image generation step of generating at least one additional image including different features of at least one object from the initial image of the video;
An information extraction step of extracting feature information present in the initial image and at least one additional image;
An object tracking model generation step of generating an object tracking model for tracking a face based on the feature information; and
A face tracking method including; an object tracking step of tracking the face in an image input from the user using the object tracking model.
In clause 7,
The additional image above is,
A face tracking method in which at least one face area of the object present in the initial image is generated including a deformed shape at different angles.
According to clause 8,
The image creation step is,
An area setting step of setting a face area of the object present in the initial image;
A feature vector generation step of extracting features from the face area and inputting the features into a preset neural network model to generate different feature vectors; and
A face tracking method comprising: generating the additional image in which the face area of the object has a deformed shape at different angles based on the different feature vectors.
According to clause 9,
The object tracking model creation step is,
By analyzing the feature information extracted from each of the initial image and the at least one additional image, if the face area is a specific object, the object is classified as the foreground, and if the face area is not a specific object, the object is classified as the background A classification information filter creation step of generating a classification information filter that calculates classification information; and
An object that displays a face area of the object included in each of the initial image and the at least one additional image in the form of a frame, and generates an object frame information filter for calculating frame information of the face area of the object displayed in the frame form. A face tracking method comprising: creating a frame information filter.
According to clause 10,
The object tracking model creation step is,
A model generation step of generating the object tracking model by performing a convolution operation on the classification information filter and the object frame information filter.
According to clause 11,
The object tracking step is,
In the information extraction step, real-time feature information extracted from the image input from the user is received, the real-time feature information is applied to the object tracking model to track and display the specific object, and the classification information filter is used to track and display the specific object. Classifying whether the face area of the real-time feature information is foreground or background, and displaying the face area of the real-time feature information in the form of a frame using the object frame information filter to track and display the specific object that is the face from the image, Face tracking method.