KR102569572B1 - A system for providing virtual wearing image - Google Patents

Abstract

In the present invention, a system for providing a virtual wearing image is disclosed. The system for providing a virtual worn image includes a database storing a set of first captured images generated from a camera module capturing a subject at various different imaging angles, and a subject and a background captured on the first captured image. An image segmentation unit for dividing into different regions, an object detection unit for extracting an object from a second captured image different from the first captured image, and a posture detection unit for detecting information on a capturing angle of the object extracted from the object detecting unit , The subject image of the first captured image is obtained by querying the database with reference to the imaging angle of the object output from the posture detection unit and taking the first captured image captured at the closest imaging angle most similar to the imaging angle of the second captured image. It may include; an image matching unit that synthesizes the object of the second captured image.
According to the present invention, a virtual wearing image providing system capable of providing a natural virtual wearing image between a subject and an object through posture and scale matching between a subject and an object captured in different postures and scales can be provided. there is.

Description

Virtual wearing image providing system {A system for providing virtual wearing image}

The present invention relates to a system for providing a virtual wearing image.

Small fashion items, such as accessories or ornaments, have high decorative value on their appearance due to the luster of precious metals or high reflectivity, but due to their relatively small volume or three-dimensional design rather than a flat design like clothing, There is an aspect that it is not easy to make a purchase decision only with the image of the product itself presented in the description, and there is a tendency to make a purchase decision considering the image of the product itself and the wearing image of the product together, and the seller of accessories or accessories From the standpoint of the product, by presenting the image of the product itself and the wearing image of the product together in the product description, it is possible to reduce future A/S such as return or exchange of the product due to consumer's error regarding the size or shape of the product. It is preferred to present a wearing image together.

One embodiment of the present invention provides a virtual wearing image providing system capable of providing a natural virtual wearing image between a subject and an object through posture and scale matching between a subject and an object captured in different postures and scales. include

In order to solve the above and other problems, the system for providing a virtual wearing image of the present invention,

a database storing a set of a group of first captured images generated from a camera module that captures a subject at various different imaging angles;

an image divider for dividing a subject and a background captured on the first captured image into different regions;

an object detection unit for extracting an object from a second captured image different from the first captured image;

a posture detection unit for detecting information on an imaging angle of the object extracted from the object detection unit; and

The database is queried with reference to the imaging angle of the object output from the posture detection unit, and the first captured image captured at the closest imaging angle most similar to the imaging angle of the second captured image is taken, and the subject image of the first captured image is obtained. It may include; an image matching unit that synthesizes the object of the second captured image.

For example, the imaging angle of the subject is,

Rotation angles indicating different angular positions along the trajectory of the camera module traveling around the subject in parallel with the support surface on which the subject is placed; and

The trajectory of the camera module may include a tilting angle indicating an angular position inclined from the support surface.

For example, the space where the subject is placed has a Y-axis height on a plane of the X-axis-Z-axis forming the support surface, and the X-axis corresponds to a direction in which the camera module directs the subject,

The rotation angle and the tilt angle correspond to rotation directions centered on the Y-axis and the X-axis, respectively,

The imaging angle may not include a yawing angle with respect to a rotational direction about the Z-axis that changes the depth of field of the camera module.

For example, information on different rotation angles and tilting angles may be linked and stored in the database as information on the imaging angle of each of the first captured images.

For example, the image divider may divide the subject captured on the first captured image, the fluorescently colored light emitting indicator, and the background into different regions.

For example, in the image information of the subject captured on the first captured image, a light emitting indicator captured on the first captured image together with the subject and a target position set in relation to the wearing position on the second captured image are aligned with each other. If possible, it can be copied to a target position on the second captured image.

For example, the first captured image includes images of three channels of R, G, and B;

The image segmentation unit selects the subject, the light emitting indicator, and the background based on different first and second threshold values calculated from the distribution of pixel values of at least one channel among the three R, G, and B images. It can be divided into 3 different areas.

For example, the first captured image is an image capturing accessories as a subject,

The second captured image may be an image capturing a person's face as an object.

For example, the object detection unit,

a feature extraction unit for extracting features from the second captured image; and

A classifier for binary classification on whether or not a face is an object by taking the feature detected from the feature extraction unit as an input.

For example, the object detection unit,

A feature map obtained by extracting features of the second captured image from the convolution between the second captured image and the learned kernel is calculated by taking the second captured image as an input, and an object is obtained from a linear combination between the calculated feature map and the learned weight. It may include a convolution neural network (CNN neural network) for outputting a predicted probability regarding whether or not a face as .

For example, the posture detection unit takes the face image extracted from the object detection unit as an input, extracts eye and lip regions from the face image, and connects both eyes and lip regions to each other. A part based model can create

For example, the part model includes a first line component connecting both eye regions, a second line component connecting one eye region and a lip region, and connecting the other eye region and lip region to each other. It includes a third line segment component that

The posture detection unit may include a linear regression neural network for outputting a predicted value about an imaging angle of an object by receiving angles and lengths of each of the first to third line segment components as inputs.

For example, the imaging angle of the object may include a rotation angle and a tilt angle of a face as an object.

For example, the linear regression neural network may take the angle and length of each of the first to third line segment components as inputs, and output a predicted value related to the scale of the object together with the imaging angle of the object.

For example, scale information is calculated directly from the part model, or

The posture detector outputs scale information from the linear regression neural network, and the scale information output from the posture detector may correspond to a length of a major axis of the face region as the object.

For example, the image matching unit may obtain a square error sum between a rotation angle and a tilt angle as the imaging angle detected from the second captured image and a rotation angle and tilt angle as the imaging angle of the first captured image stored in the database. An imaging angle of the first captured image at which (sum of squared errors, SSE) is minimized may be recognized as closest to an imaging angle detected from the second captured image.

For example, the image matching unit may copy subject image information divided from the background of the first captured image to a target location of an object image extracted from the second captured image.

For example, the image matching unit refers to the size of the subject image divided from the background in the first captured image and the scale information output from the posture detection unit to correspond to a preset size ratio between the subject and the object. The geometrically transformed subject image obtained by enlarging or reducing the subject image may be copied to a target location of an object image extracted from the second captured image.

For example, the size ratio between the subject and the object may be stored in association with an entire set of a group of first captured images in which the subject is captured at different imaging angles.

For example, the image matching unit may use image information of a subject captured on the first captured image such that a light emitting indicator captured on the first captured image and a target position set in relation to the wearing position on the second captured image are aligned with each other. may be copied to a target location on the second captured image.

The system for providing a virtual wearing image according to an embodiment of the present invention can provide matched virtual wearing images for each other through posture and scale matching between a subject and an object, each captured in different postures and scales. A natural virtual wearing image between the subject and the object may be provided through posture and scale matching between the subject and the object.

1 is a diagram showing the overall configuration of a system for providing a virtual wearing image according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining an imaging angle of a camera module for generating a first captured image capturing a subject.
3A to 3E exemplarily illustrate different views of a first captured image generated from a camera module.
4A to 4E are diagrams showing different second captured images in which various objects are captured by way of example.
FIG. 5 is a diagram for explaining different first and second types of object detectors applied in an embodiment of the present invention.
FIG. 6 is a diagram for explaining an operation for detecting features of a facial image by applying a Harr like feature (operator) in the feature extraction unit shown in FIG. 5 .
FIG. 7 is a diagram for explaining the classifier shown in FIG. 5, and for convenience of understanding, a diagram showing a simplified model of a multilayer neural network in which a plurality of unit neurons are connected is shown.
FIG. 8 is for explaining the CNN neural network shown in FIG. 5, and is a diagram showing a simplified model of the CNN neural network for convenience of understanding.
9 and 10 are drawings for explaining a part based model generated from the posture detection unit shown in FIG. 1 .
FIG. 11 is a diagram for explaining the linear regression neural network shown in FIG. 1, and for convenience of understanding, a diagram showing a simplified model of a multilayer neural network connected to a plurality of unit neurons is shown.
12A to 12E illustrate composite images synthesized so that the image of the subject captured in FIGS. 3A to 3E and the image of the object captured in FIGS. 4A to 4E are matched to each other through posture and scale matching. Drawings showing the enemy are shown.

Hereinafter, a system for providing a virtual wearing image according to a preferred embodiment of the present invention will be described with reference to the accompanying drawings.

1 is a diagram showing the overall configuration of a system for providing a virtual wearing image according to an embodiment of the present invention.

A virtual wearing image providing system according to an embodiment of the present invention is a system for providing matched virtual wearing images for each other through posture and scale matching between a subject O and an object F. In one embodiment of the invention, a natural virtual wearing image between the subject O and the object F is provided through posture and scale matching between the subject O and the object F captured in different postures and scales. can do. More specifically, a system for providing a virtual wearing image according to an embodiment of the present invention includes a group of first captured images including the same subject O captured along different imaging directions, and each A camera module (C) generating a set of first captured images including a group of first captured images to which a different capturing direction is assigned to each captured image, and a subject (O) captured on the first captured image; An image division unit for dividing a background (B) into different regions, an object detection unit for extracting an object (F) from a second captured image different from a first captured image, and an object (F) extracted from the object detection unit A posture detecting unit for detecting information on the capturing angle of the image, and a database storing a set of first captured images are queried by referring to the capturing angle of the object F outputted from the posture detecting unit, and a second captured image is captured. An image matching unit may be configured to take a first captured image captured at a closest imaging angle most similar to the angle and synthesize the image of the subject O of the first captured image onto the object F of the second captured image.

FIG. 2 is a diagram for explaining an imaging angle of the camera module C for generating a first captured image capturing a subject O. Referring to FIG.

In one embodiment of the present invention, the camera module (C) is the central subject at an angular position equally divided along the 360 angle while aiming at the central subject (O) and going around the subject (O) at 360 degrees. By capturing (O), it is possible to create a set of first captured images including a group of first captured images capturing the same subject O at different imaging angles.

Through this specification, the imaging angle may refer to the optical axis direction of the camera module C generating the first captured image, and the optical axis direction of a group of optical lenses and imaging devices provided in the camera module C are aligned with each other This may mean an angle directed toward the subject O, and in one embodiment of the present invention, the imaging angle is along a circumferential trajectory of the camera module C that travels around the subject O at an angle of 360 degrees. , the rotation angle (θ) representing the angular position measured from the reference angle (0 degree) among the angles of 360 degrees, and the angular position of the trajectory of the camera module (C) itself inclined from the support surface on which the subject (O) is placed It may include a tilting angle (α). As will be described later, in one embodiment of the present invention, each of the first captured images capturing the same subject O may be given a predetermined imaging angle, wherein the imaging angle is the rotation angle θ as described above and It may include a tilt angle α, and for example, in one embodiment of the present invention, the information about the imaging angle may include information about a pair of a rotation angle θ and a tilt angle α. there is. In one embodiment of the present invention, the posture of the subject O and the posture of the object F may be matched with each other based on the information about the imaging angle, and the posture of the subject O and the posture of the object F can be recognized as being matched with respect to each other under the same or closest imaging angles (rotation angle θ and tilt angle α) with respect to each other.

In the system for providing a virtual wearing image according to an embodiment of the present invention, the subject (O) may mean an image of an accessory captured on a first captured image, and the object (F) is the first captured image may refer to a face image of a person captured on a second captured image different from the image of a person's face captured on a second captured image, and in an embodiment of the present invention, an accessory type image captured on the first captured image and a person captured on the second captured image It is possible to provide virtual wearing images synthesized so that the images are matched to each other through posture and scale matching.

In one embodiment of the present invention, the camera module (C) can generate a group of captured images captured at various imaging angles for the same subject (O), and the group of captured images are rotated at different rotational angles ( θ) and a group of captured images capturing the subject O along the tilting angle α. In one embodiment of the present invention, in order to generate a group of captured images, a 360 degree angle along the circumference of the camera module C along the circumferential trajectory of the camera module C as shown in FIG. 2 is set at a constant rotation angle θ For each rotation angle θ divided by ) unit, for example, for each position of any one rotation angle θ, the circumferential trajectory of the camera module C is a constant inclination from the support surface of the subject O By tilting in units of angles, it is possible to obtain captured images for positions of various tilting angles α for each rotational angle θ. ) can be created.

In one embodiment of the present invention, the rotation angle θ and the tilt angle α may be defined as rotations centered on three different coordinate axes set in a 3D space where the subject O is disposed. . X-Y-Z coordinate axes set in the 3D space where the object O is disposed may form a coordinate system that has the location of the object O as an origin and rotates with a rotation angle θ. For example, the space in which the object O is disposed may be defined as a three-dimensional space having a Y-axis as a height on a plane of X-Z axes in which the object O is disposed, and a rotation angle θ around the Y-axis A tilting angle α centered on the X-axis may be defined on the plane of the X-Z axes rotating together with the Y-axis. In one embodiment of the present invention, the camera module (C) for capturing the subject (O) can capture the subject (O) while directing the subject (O) along various rotation angles (θ) and tilting angles (α). In this case, the X axis defining the tilting angle α may correspond to a direction facing the camera module C from the subject O or a direction of the camera module C, and the Z axis is the center A yawing angle to be may form a depth of field of the camera module (C) for the subject (O).

In one embodiment of the present invention, the subject (O) and the object (F) can align their postures with respect to each other through the information of the imaging angle, and the subject (O) and the object (F) correspond to each other By matching in the desired posture, it is possible to provide a natural virtual wearing image. At this time, in one embodiment of the present invention, the coordinate axis of the X-Y-Z axis set in the three-dimensional space in which the subject O or object F is placed (for example, the position of the subject O or object F is set as the origin, Considering only the position of the rotation angle (θ) around the Y-axis and the position of the tilt angle (α) around the X-axis among the X-Z coordinate axes that rotate with the rotation angle (θ) around the Y-axis), the subject Align the pose of (O) and the object F, and may not consider the position of the yaw angle centered on the Z axis. For example, the rotation angle (θ) between the subject O and the object F ) and the position of the tilting angle α, it is possible to recognize that the postures of the subject O and the object F are aligned relative to each other, and the subject O and the object F It is possible to provide virtual wearing images expressed in postures matched with each other.

In one embodiment of the present invention, the position and tilt angle of the rotation angle (θ) defined only by any two coordinate axes among three different coordinate axes set in a three-dimensional space in which the subject (O) or object (F) is placed Aligning the poses of the subject O and the object F using the position of (α) is a visual and The visual visibility according to the change in the yaw angle of the subject O or object F is relatively lower than the visual visibility, for example, the two-dimensional image set on the captured image of the subject O or object F Information corresponding to the depth of shooting relatively on a plane (difference in depth according to the yawing angle centered on the Z axis) can be considered to have relatively low visual visibility, and the three-dimensional space in which the subject (O) or object (F) is placed Generating a first captured image capturing the subject O according to different imaging angles for each position of the rotation angle θ centered on the X-Y-Z axis set on the image, the position of the tilt angle α, and the position of the yaw angle , Since a considerable degree of work (imaging work for each different imaging angle) is required for generating the first captured image, relatively visual visibility is relatively low without considering the yaw angle Posture information of the subject O or F may be matched using only the high rotation angle θ and tilt angle α.

In one embodiment of the present invention, a group of first captured images captured at various different imaging angles for the same subject O may be stored in a storage device in association with information on the imaging angles, and the information on the imaging angles A group of first captured images to which is assigned may form a database related to the first captured images, and each of the group of first captured images includes, as information on a capturing angle, information of a rotation angle θ and a tilt angle α. Information about the pair may be stored. In one embodiment of the present invention, a database related to the first captured image in which the subject O is captured is transmitted to a computing device that implements the second captured image in which the object F is captured, and the first captured image image A virtual wearing image may be provided through posture and scale matching between the subject O captured in the image and the object F captured on the second captured image.

3A to 3E are different views showing the first captured image generated from the camera module C by way of example.

A system for providing a virtual wearing image according to an embodiment of the present invention may include an image dividing unit for dividing two different types of areas of a subject O and a background B captured on a first captured image. The image segmentation unit extracts only image information of the subject O from the first captured image in a method of segmenting the subject O and the background B captured on the first captured image (object/background segmentation). image matting can be performed. In one embodiment of the present invention, the first captured image may capture an image of the subject O under a light source for illuminating a periphery of the subject O, and a light source surrounding the subject O may be captured. By forming the black background (B) to absorb the illumination light, it is possible to form a sharp contrast between the subject (O) and the background (B) in the first captured image, and thus, according to an embodiment of the present invention The image division unit may calculate the frequency of occurrence of each pixel value between 0 and 255 with a histogram for all pixels of the first captured image, and analyze the histogram to obtain a threshold value at a valley point between the two peaks Based on , it is possible to divide an area above a threshold value and an area below a threshold value. For example, in one embodiment of the present invention, the image segmentation unit may divide the first captured image into two light and dark regions based on one threshold value, and more specifically, relatively having a pixel value equal to or greater than the threshold value. It can be divided into a bright subject (O) and a relatively dark background (B) having a pixel value less than a threshold value.

In various embodiments of the present invention, the image division unit analyzes the histogram of the first captured image, and two peaks appearing on the histogram, for example, a high frequency appearing in a set of pixels (relatively low pixel values) constituting the background (B) A valley having a relatively low frequency between the peak of and a peak of a high frequency appearing in a set of pixels (relatively high pixel value) constituting the subject O is taken and set as a threshold value, and the background (B ) and the subject O. For example, a group of pixels having similar low pixel values and connected to each other is divided into one class constituting the background B, and each other as another class. Another group of pixel sets having similar high pixel values and connected to each other may be divided into another class constituting the subject O.

In various embodiments of the present invention, in the image segmentation unit, a set of pixels belonging to the same class, such as each of the subject O and the background B, have similar characteristics, for example, similar pixel values, and are classified into the same class. It is determined that the distribution of pixel values within a set of pixels to which they belong is uniform is excellent in segmentation performance, and while changing the threshold value, a threshold value that minimizes variance within a set of pixels belonging to the same class is selected, or pixels belonging to the same class A threshold value that minimizes an objective function or a cost function is selected so that the variance within the set is small, and the first captured image is divided into two classes (background B and subject O) based on the selected threshold value can be divided In addition, in various embodiments of the present invention, it is determined that segmentation performance is excellent when the variance of pixel values is large among pixel sets of different classes, such as different backgrounds (B) and subjects (O), and the threshold value The first captured image may be divided into two classes (background B and subject O) based on the selected threshold value by selecting a threshold value maximizing variance between pixel sets belonging to different classes while changing . In one embodiment of the present invention, the image segmentation unit divides the first captured image into two different light and dark regions based on one threshold, and the image segmentation unit divides the first and second thresholds into different regions. Based on the value, the first captured image may be divided into three different light and dark areas.

In one embodiment of the present invention, on the first captured image, together with the subject O, a light emitting directing unit I for indicating a wearing position of the subject O may be captured. For example, the light emitting directing unit (I) may include fluorescent coloring to appear as a high pixel value under a light source that illuminates the subject O. For example, in one embodiment of the present invention, the first captured image may include R, G, and B 3-channel images, and the light emitting directing unit (I) displays a specific channel (e.g., For example, it may have a relatively high pixel value on an image of R channel) or on images of specific channels. It is possible to divide three different light and dark regions from the image based on the first and second threshold values. Dividing the first captured image into three different light and dark areas, a subject (O) area consisting of a set of pixels less than the second threshold and greater than or equal to the second threshold, and a background (B) area consisting of a set of pixels less than the second threshold. You may.

In one embodiment of the present invention, the image segmentation unit divides the first captured image into an area equal to or greater than a threshold value and an area less than the threshold value based on a threshold value calculated based on a distribution of pixel values on the first captured image. In this case, the first captured image may include images of 3 channels R, G, and B, and at this time, the image division unit independently performs image division for each of the images of different R, G, and B channels. Image division can be performed, and the background (B) and the subject (O) can be divided on the first captured image in a way of integrating the image division on the images of each of the R, G, and B channels. For example, on the first captured image, together with the subject O, a light emitting directing unit I for indicating a wearing position of the subject O may be captured, and the light emitting directing unit I is a fluorescent color Depending on the color, it may have a relatively high pixel value on an image of a specific channel (eg, R channel) or on an image of specific channels. It is possible to divide three different light and dark regions based on the first and second threshold values from the image of at least one of the channels, and, for example, a light emitting indicator (I) composed of a set of pixels equal to or higher than the first threshold value A region of the subject (O) consisting of a set of pixels less than the first threshold and greater than or equal to the second threshold, and a region of the background (B) consisting of a set of pixels less than the second threshold, the first captured image is divided into each other. It can also be divided into three other light and dark regions. At this time, the image segmentation unit independently performs image segmentation on each of the three R, G, and B channel images constituting the first captured image, and synthesizes the image segmentation on the R, G, and B channel images. Thus, the background (B), the subject (O), and the light emitting directing unit (I) can be divided on the first captured image.

In various embodiments of the present invention, the image segmentation unit performs region segmentation using thresholding to divide the background (B) and the subject (O) based on a threshold value calculated from the distribution of pixel values of the first captured image. Alternatively, the image segmentation unit may perform region segmentation using clustering (eg, k-means clustering). In addition, in various embodiments of the present invention, in that areas such as the background (B) and the subject (O) are a set of pixels continuously connected to each other, together with the distribution of pixel values on the first captured image, the first captured image It can be divided into different classes by considering the adjacency of pixel positions along the y-axis and the x-axis corresponding to the matrix of .

In the comparative example contrasted with the present invention, the amount of change in pixel values (eg, first order differential value, second order differential value) between adjacent pixels on the first captured image is calculated, and according to the calculated amount of change in pixel values , For example, when the first derivative forms a local maximum and the second derivative corresponds to a zero crossing, the corresponding pixel is between the background (B) and the subject (O). It is determined that it corresponds to an edge pixel forming the boundary of , and a line segment dividing the background (B) and the subject (O) can be detected by connecting the edge pixels. For example, an edge or an edge connected to each other The background (B) and the subject (O) may be divided into different types of regions based on line segments.

In this comparative example contrasted with the present invention, since the boundary line dividing the background (B) and the subject (O) corresponds to a point where a change in contrast occurs, the boundary line should conceptually coincide with the edge. However, in order to form the same area as the background (B) and the subject (O), the edge or line segments connecting the edges should form a closed loop that surrounds the areas belonging to different classes in a closed form. Since an edge detected from an image or a line segment connecting the edge does not form a closed loop in most cases, in one embodiment of the present invention, the image segmentation unit detects an edge component on the first captured image. Instead of dividing the background B and the subject O on the first captured image in this manner, as described above, the background B and the subject O may be divided by region division.

In one embodiment of the present invention, image information of the subject O divided from the background B on the first captured image may be stored together with information about a capturing angle associated with the corresponding first captured image, , For example, the image information of the subject O may be stored in a manner in which pixel values of the background B excluding the subject O are replaced with 0 among pixel values of 0 to 255, and the image information of the subject O may be stored. Along with the image information, information about an imaging angle, eg, a rotation angle θ and a tilt angle α at which the subject O is captured may be stored in a storage device. In one embodiment of the present invention, the image segmentation unit may divide the subject O and the background B into different areas on the first captured image, and may divide the subject O from the background B The image information may be stored through image processing on the first captured image by replacing the pixel value of the background (B) with 0 in the first captured image generated by the camera module (C). The related image information may correspond to a first captured image processed to clearly distinguish the subject O and the background B on the first captured image generated by the camera module C, and in this sense, the present invention 1 captured image may include the first captured image generated by the camera module C and the first captured image image-processed so that the subject O and the background B divided by the image segmentation unit are clearly distinguished. . In this way, the first captured image in which the subject O and the background B are divided may be stored in a storage device together with information on the capturing angle to form a database related to the first captured image.

4A to 4E are diagrams showing different second captured images in which various objects F are captured by way of example.

In one embodiment of the present invention, the subject O and the object F are imaged through posture and scale matching between the subject O captured on the first captured image and the object F captured on the second captured image. It is possible to provide a virtual wearing image synthesized with images. In one embodiment of the present invention, the pose of the subject O captured on the first captured image can be grasped from information on the captured angle stored in association with the first captured image, and the object captured on the second captured image The posture of (F) may be calculated through a machine learning module described later. In one embodiment of the present invention, the poses of the subject O captured on the first captured image and the object F captured on the second captured image are different from each other on the first and second captured images. O) and the object F can be identified from the information of the captured imaging angle, and in this sense, the posture of the subject O and the object F may mean the information of the imaging angle, and the present invention In one embodiment, the matching of postures between the subject O and the object F may mean matching imaging angles of the subject O and the object F to each other.

In one embodiment of the present invention, the imaging angle of the subject O captured on the first captured image is a camera module C capturing the subject O at various rotation angles θ and tilting angles α that are different from each other. When the first captured image is generated from, the first captured image generated from the camera module (C) and the rotation angle (θ) and the tilting angle (α) are linked and stored through the database of the first captured image. Therefore, a separate calculation for calculating the imaging angle of the subject O captured on the first captured image is not required. However, the posture of the object F captured on the second captured image or the rotation angle θ′ and the tilt angle α′ of the object F may be calculated from the second captured image through a separate operation. , The system for providing a virtual wearing image according to an embodiment of the present invention may include a machine learning module for calculating information about the posture of the object F from the second captured image.

In the system for providing a virtual wearing image according to an embodiment of the present invention, the object F may mean a face image of a person captured on a second captured image, and the subject O may refer to a first captured image In one embodiment of the present invention, posture and scale matching are performed between the image of accessories captured on the first captured image and the image of a person captured on the second captured image. Through this, it is possible to provide virtual wearing images synthesized to match each other. In this sense, the object F captured on the second moving image may mean a person image. In one embodiment of the present invention, a person image as an object F is detected from the second captured image, but in particular, a face image is extracted as a wearing part of an accessory type from the person image, and posture and scale information are extracted from the extracted face image Detect and match (posture and scale matching) with the first captured image in which the accessories as the subject O divided from the background B are captured, and the accessories captured on the first captured image and the second captured image image In a method of synthesizing the face image extracted from the second captured image and the image of accessories captured on the first captured image in a form matched to each other through posture and scale matching of the face image detected from can provide a wearing image of

The machine learning module according to an embodiment of the present invention may include an object detection unit for detecting a face image as an object F and a posture detection unit for detecting posture and scale information from the detected face image.

FIG. 5 is a diagram for explaining different first and second types of object detectors applied in an embodiment of the present invention. FIG. 6 is a diagram for explaining an operation for detecting features of a facial image by applying a Harr like feature (operator) in the feature extraction unit shown in FIG. 5 . FIG. 7 is a diagram for explaining the classifier shown in FIG. 5, and for convenience of understanding, a diagram showing a simplified model of a multilayer neural network in which a plurality of unit neurons are connected is shown. FIG. 8 is for explaining the CNN neural network shown in FIG. 5, and is a diagram showing a simplified model of the CNN neural network for convenience of understanding.

A machine learning module according to an embodiment of the present invention may include a first type of object detector including a feature extractor and a classifier or a second type of object detector including a CNN neural network. The first type of object detection unit may perform binary classification on whether or not a face is present using a feature extraction unit extracting features from the second captured image and a feature detected by the feature extraction unit as inputs.

In one embodiment of the present invention, the feature extraction unit converts an input second captured image into a plurality of pyramid images including multi-scale images including various scales (or multi-resolution), and the multi-scale images Features can be extracted while moving a window or an operator on the image. In an embodiment of the present invention, a feature capable of distinguishing a face image from a non-face image can be detected while moving a Harr like feature (operator) over the multi-scale image. For example, FIG. 6 As shown in , the quasi-Haar operators (f1, f2) apply a weight of -1 to the sum of intensities in the black area and apply a weight of +1 to the sum of intensities in the white area (f1, f2). ), for example, from the first operator (f1) applied to the eyes of the face and the glabella between the eyes, that is, from the first operator (f1) applied to the relatively dark eyes and the relatively bright glabella. The computed features may have a certain tendency. In addition, the feature calculated from the second operator f2 located at the boundary between the forehead and the hair, that is, the second operator f2 applied to relatively dark hair and relatively light forehead, may also have a certain tendency. In this way, the feature detected from the feature extractor may be input to a classifier, and as shown in FIG. 7, the classifier derives a value from a linear combination of each element constituting the detected feature and the learned weight (W). 2 Classify a face image and a non-face image on a captured image, and perform binary classification on whether or not a face exists. In one embodiment of the present invention, the classifier is an activation function, for example, a binary combination of an input vector in which a plurality of elements are arranged in an array form as a feature extracted from the feature extraction unit and a learned weight (W) vector. By applying a sigmoid function suitable for classification, it is possible to calculate a predicted probability of whether a face exists, and the weight (W) vector is a training data set prepared in advance, and a class label related to whether a face is a target A cost function (or loss) to minimize the error between the estimated class label of the classifier (estimation of whether or not a face) and the given target label, with the given feature (for example, the feature extracted from the feature extractor) as an input vector. function) can be learned in such a way as to update the weight vector so that it converges toward the minimum value.

A machine learning module according to another second type of the present invention may include a second type of object detection unit including a convolution neural network (CNN). Referring to FIG. 8 , the second type of object detection unit takes the second captured image itself as an input and performs synthesis with a learned weight matrix (kernel or filter) in order to calculate a feature map obtained by extracting features from the second captured image. The feature map calculated by performing multiplication (convolution) is flattened into a one-dimensional feature vector, and a linear combination or linear combination with learned weights applied to each element forming the feature vector is performed. By applying an activation function, it is possible to calculate a predicted probability of whether a face exists. For example, in one embodiment of the present invention, the CNN neural network implemented in the object detection unit includes a plurality of convolutional layers to extract high-level features from low-level features, and from the extracted features (feature maps). It may include a dense layer responsible for classifying whether or not a face exists. In addition, the convolution layer may include a plurality of kernels (or filters, weight matrices), and a plurality of kernels (each element of the kernel) on the second captured image (pixel values of each pixel forming the second captured image). As , the weight W) may be multiplied to generate a feature map in which the depth dimension increases while the 2D size of the matrix decreases.

The CNN neural network may be applied with max pooling or average pooling in order to reduce the size of the feature map generated from the convolutional layer (down sampling), and the dense layer is a two-dimensional array of matrices. A three-dimensional feature map having depth and depth dimensions is flattened into a feature vector of one-dimensional features, and an activation function is applied to a weighted linear combination or linear combination of a plurality of elements forming the one-dimensional feature vector. By applying, as a binary classification of whether or not a face is present, an estimated class label (a probability of whether a face image exists in the second captured image) may be output.

The weight matrix (kernel or filter) of the CNN neural network implemented in the object detector is a training data set including a plurality of training data and a target class label given for each training data (whether or not a face image exists in the input training data). Backpropagation algorithm that calculates the gradient of the cost function to be minimized so that the error between the clay label predicted from the CNN neural network and the given target class label is reduced. can be applied to update the weight vector (kernel or filter).

In one embodiment of the present invention, an image pyramid including a plurality of images of multiplexed resolution or multiplexed scale is generated for the second captured image, and an operator or kernel ( or filter), extracts features or feature maps, and performs binary classification on whether a face is detected from the feature or feature map, that is, whether a face is detected in a given image, and as an output of a learned classifier or a learned CNN neural network. , we can calculate the predicted probability for binary classification.

9 and 10 are drawings for explaining a part based model generated from the posture detection unit shown in FIG. 1 .

In one embodiment of the present invention, the posture detection unit may detect the posture and scale information of the object F by using the face image extracted from the object detection unit as an input. For example, the posture detector may detect eye and lip regions from a face image and generate a triangular part-based model that connects both eye and lip regions. In an embodiment, the posture detection unit may detect both eye and lip regions from a face image by applying the following algorithm. For example, the posture detection unit may calculate an area in which a change in contrast is prominent in all directions and detect it as an eye area, and may detect an eye area by applying an edge detection algorithm that detects an edge from a change in contrast. . More specifically, the posture detection unit calculates the amount of change and direction of pixel values for a set of pixels around the pixel of interest from the face image, and calculates the amount of change of each pixel value for the direction binarized in 8 steps in all directions (8). direction), an area in which a change in pixel value is maximum in the area may be detected and determined as an eye area.

For example, the posture detection unit applies a window (filter or mask) of a certain size or an operator (a first-order differential operator, for example, a Sobel operator) to a set of pixels arranged adjacent to each other on a face image (region operation) area operation) to calculate the amount of change in the pixel value of the pixel of interest in the window, and in various embodiments of the present invention, the posture detection unit calculates 2 values such as the Laplacian operator for a set of pixels arranged next to each other on the face image. An eye region is detected from a face image by applying a differential differential operator and detecting zero crossing by applying a secondary differential operator to a candidate group of eye regions calculated by applying the primary differential operator. You can do it. For example, the posture detection unit according to an embodiment of the present invention calculates a first derivative and a second derivative for a set of pixels arranged adjacent to each other, and according to the amount of change in the calculated pixel values, for example, all An area in which a first derivative value along a direction (eight directions) forms a local maximum and a second derivative value is zero-crossing may be determined as an eye area.

The posture detector may determine a region having a relatively high pixel value in the face image of the R channel from the face images of the three channels R, G, and B as the lip region. Based on the threshold value calculated from the distribution of pixel values of the face image, region segmentation can be performed using thresholding that divides the lip region and other regions other than the lips, and region using clustering such as k-means clustering. Through segmentation, the lip region may be detected. For example, in one embodiment of the present invention, the posture detection unit analyzes the histogram of the face image of the R channel and two peaks appearing on the histogram, for example, a set of pixels (relatively high pixels) forming a lip region. value) and a valley with a relatively low frequency between the peaks with a relatively low frequency appearing in a set of pixels (relatively low pixel values) forming an area other than the lips, and setting it as a threshold value. Based on , the lip area may be detected from the face image by dividing the lip area and other areas other than the lips from the face image.

Referring to FIGS. 9 and 10 , the posture detection unit may generate a part model that connects both eye and lip regions detected from the face image in a triangular shape, and uses characteristics of the part model as input to obtain posture information. It may include a linear regression neural network that detects. For example, the triangular component model includes a first line segment component (L1) connecting both eye regions, a second line component (L2) connecting one eye region and a lip region, and the other eye region. and a third line segment component (L3) connecting the lip area to each other, and the posture detector may include an angle of the first to third line segment components (L1 to L3), and the first to third line segment components (L1 to L3). The length of L3) is calculated, and the linear regression neural network can calculate a predicted value for a posture from each face image by using the calculated feature as an input.

FIG. 11 is a diagram for explaining the linear regression neural network shown in FIG. 1, and for convenience of understanding, a diagram showing a simplified model of a multilayer neural network connected to a plurality of unit neurons is shown.

The linear regression neural network is a feature of the generated part model, the angle of the first to third line segment components (L1 to L3) of the triangular part model, and the length (feature) of the first to third line segment components (L1 to L3). It is possible to detect the posture information of each face image from , and the linear regression neural network applies an activation function to a linear combination or a prior combination of each element constituting the feature of the part model and the learned weight, so that each face A predicted value of a posture for an image may be calculated. In one embodiment of the present invention, the linear regression neural network applies an activation function to a linear combination of an input vector in which a plurality of elements are arranged in an array form as a feature extracted from a part model and a learned weight (W) vector to determine the posture. The weight (W) vector is a pre-prepared learning data set, and the target values for the posture (rotation angle θ` of the object F, tilting angle α`, and target value of the scale) are Given the features (features of the part model) as input vectors, the error between the estimated values of the predicted posture of the linear regression neural network (estimated values of rotation angle θ`, tilt angle α`, and scale of object F) and the given target value is It can be learned in such a way as to update the weight (W) vector so that the cost function to be minimized to be reduced converges toward the minimum value.

Referring to FIG. 10, the linear regression neural network calculates the tilt angle (α) as the angle of the first line segment component (L1) connecting the eye regions on both sides of the face image and the pose of the object (F) through learning as described above. `) can be analyzed, and the correlation between the relative lengths of the first to third line segment components (L1 to L3) and the rotation angle (θ`) can be analyzed, and the first to third line segments can be analyzed. The correlation between the length and scale of the components (L1 to L3) can be analyzed. For example, in one embodiment of the present invention, the linear regression neural network changes the tilt angle α′ of the object F according to the angle or inclination of the first line segment component L1 connecting the eye regions to each other. Depending on the correlation, predicting the tilting angle α` of the object F from the angle or inclination of the first line segment component L1, or each angle or angle of the first to third line segment components L1 to L3 The tilting angle α′ of the object F may be predicted from the tilt. In addition, the linear regression neural network connects both eye regions to each other according to the relative lengths of the first to third line segment components L1 to L3, for example, according to the rotation angle θ′ of the object F. From the relative lengths of the first to third line segment components L1 to L3, according to the correlation that the change in the length of the first line segment component L1 is greater than the change in the lengths of the second and third line segment components L3. The rotation angle θ` of the object F may be predicted. In addition, according to the correlation that the size of the face area changes according to the lengths of the first to third line segment components L1 to L3, the size of the face area is determined from the lengths of the first to third line segment components L1 to L3. can be predicted In one embodiment of the present invention, the size of the face region is calculated directly from the lengths of the first to third line segment components (L1 to L3) of the face region, for example, from the glabella located between the eye regions on both sides. It can be defined as the distance (S) to the lip area, and in this case, prediction by a linear regression neural network is not required, and when the face area is viewed as a generally elliptical shape, the length of the major axis of the elliptical shape can be regarded as the size of the face area. In this case, the linear regression neural network may output a prediction value for the long axis length of the face region from learning data (lengths of first to third line segment components L1 to L3) given a target value for the long axis length of the face region, , The linear regression neural network including the learned weight predicts the long axis length of the face region when the lengths of the first to third line segment components (L1 to L3) are input as characteristics (characteristics of the part model) of the face region. value can be printed.

In one embodiment of the present invention, the image matching unit refers to the scale information output from the posture detection unit and the size of the subject image divided from the background in the first captured image, and sets the subject (O) and the object (F) in advance. ), the geometrically transformed subject O image obtained by enlarging or reducing the subject image may be copied to a target position of the object image F extracted from the second captured image. At this time, the size ratio between the subject O and the object F may be designated in association with the entire set of first captured images of the subject O captured at different imaging angles, and the present invention In an embodiment of the above, the set of first captured images of the same subject O captured at different imaging angles is the same subject captured at different rotational angles θ and tilting angles α, and the same Since the size ratio between the subject O and the object F may be designated as the same for the subject O, a set of first captured images of the same subject O captured at different imaging angles is obtained. A size ratio between the same subject O and object F with respect to the set of first captured images of the group may be linked and stored in the stored database.

A system for providing a virtual wearing image according to an embodiment of the present invention queries a group of databases related to a first captured image with reference to posture information detected from a second captured image, and detects information about a second captured image. A first captured image captured in the closest posture most similar to the posture information is taken, and an image of the subject O divided from the background B from the corresponding first captured image is displayed on the object F of the second captured image. An image matching unit for synthesizing may be included.

12A to 12E illustrate composite images synthesized so that the image of the subject captured in FIGS. 3A to 3E and the image of the object captured in FIGS. 4A to 4E are matched to each other through posture and scale matching. Drawings showing the enemy are shown.

The image matching unit refers to a rotation angle (θ′) and a tilt angle (α′) as attitude information detected from the second captured image, searches a database for the first captured images, and obtains a plurality of first captured images and As the attitude information stored in association, a first captured image captured in a posture most similar to the posture detected from the second captured image among the rotation angle θ and the tilt angle α is taken, and the background ( The image of the subject O, which is region-divided from B), may be synthesized on the face region detected from the second captured image.

In one embodiment of the present invention, the image matching unit determines a rotation angle (θ′) and a tilting angle (α′) as the imaging angle detected from the second captured image, and the imaging of the first captured image stored in the database. The imaging angle of the first captured image at which the sum of squared errors (SSE) between the rotation angle θ and the tilting angle α as an angle is minimized is determined by the latest It can be recognized as having been touched. For example, in one embodiment of the present invention, the image matching unit determines the difference between the rotation angle θ′ detected from the second captured image and the rotation angle θ of each first captured image stored in the database. Rotation angle that minimizes the sum of the square errors obtained by adding the square and the square of the difference between the tilt angle α` detected from the second captured image and the tilt angle α of each first captured image stored in the database ( θ) and a tilting angle α, the first captured image may be taken and recognized as a pair matching the posture with the second captured image.

The image matching unit may copy a pixel value of each pixel forming an area of the subject (O) divided from the background (B) on the first captured image to a target position (wearing position, composite position) on the second captured image, For example, the target position (wearing position, composite position) on the second captured image may be indicated according to a user's input, for example, through drag and drop of a mouse as a user's input. The target position (wearing position) indicated according to the user's input and the position of the light emitting directing unit I divided on the first captured image (the position of the light emitting directing unit I divided into areas on the first captured image) are mutually related. The image of the subject O, in which the image of the light emitting directing unit I is combined, may be synthesized into a face area on the second captured image so that the position is aligned. Information (pixel value of each pixel forming the subject O) is copied to the target position indicated according to the user's input, with the position of the light emitting indicator (I) included in the image information of the subject O as the reference position. , The image information of the subject O may be copied to the face image of the second captured image so that the position of the light emission directing unit I is aligned with the target position.

Although the present invention has been described with reference to the embodiments shown in the accompanying drawings, this is merely exemplary, and those skilled in the art can make various modifications and equivalent other embodiments therefrom. you will understand the point.

O: subject F: object
θ, θ`: rotation angle α, α`: tilting angle
C: Camera module B: Background
I: light emitting indicator f1, f2: first and second operators
L1 to L3: first to third line segment components

Claims (20)

a database storing a set of a group of first captured images generated from a camera module that captures a subject at various different imaging angles;
an image divider for dividing a subject and a background captured on the first captured image into different regions;
an object detection unit for extracting an object from a second captured image different from the first captured image;
a posture detection unit for detecting information on an imaging angle of the object extracted from the object detection unit; and
The database is queried with reference to the imaging angle of the object output from the posture detection unit, and the first captured image captured at the closest imaging angle most similar to the imaging angle of the second captured image is taken, and the subject image of the first captured image is obtained. An image matching unit for synthesizing a second captured image onto an object;
The imaging angle of the subject is,
Rotation angles indicating different angular positions along the trajectory of the camera module traveling around the subject in parallel with the support surface on which the subject is placed; and
The trajectory of the camera module includes a tilting angle indicating an angular position inclined from the support surface,
The space where the subject is placed has a Y-axis height on a plane of the X-axis-Z-axis forming the support surface, and the X-axis corresponds to a direction in which the camera module directs the subject,
The rotation angle and the tilt angle correspond to rotation directions centered on the Y-axis and the X-axis, respectively,
The imaging angle does not include a yawing angle with respect to a rotational direction around the Z axis that changes the depth of field of the camera module,
The posture detection unit,
With the face image extracted from the object detection unit as an input, a first line segment component connecting both eye regions to each other, a second line segment component connecting one eye region and lip region to each other, and the other eye region and lip Creating a part based model including a third line segment component connecting the regions to each other;
A linear regression neural network for outputting a predicted value for an imaging angle of an object by taking angles and lengths of each of the first to third line segment components as inputs,
The imaging angle of the object includes a rotation angle and a tilt angle of the face as an object, and does not include a yaw angle. delete delete According to claim 1,
The virtual wearing image providing system according to claim 1 , wherein information on different rotation angles and tilting angles is linked and stored in the database as information on the imaging angle of each of the first captured images. According to claim 1,
The image dividing unit divides the subject captured on the first captured image, the fluorescently colored light emitting indicator, and the background into different regions, respectively. According to claim 5,
The image information of the subject captured on the first captured image is such that the light emission indicator captured on the first captured image together with the subject and the target position set in relation to the wearing position on the second captured image are aligned with each other. 2 A virtual wearing image providing system characterized in that it is copied to a target position on the captured image. According to claim 5,
The first captured image includes images of three channels of R, G, and B;
The image segmentation unit selects the subject, the light emitting indicator, and the background based on different first and second threshold values calculated from the distribution of pixel values of at least one channel among the three R, G, and B images. Virtual wearing image providing system, characterized in that divided into three different areas. According to claim 1,
The first captured image is an image capturing accessories as a subject,
The second captured image is a virtual wearing image providing system, characterized in that the image captured a person's face as an object. According to claim 1,
The object detection unit,
a feature extraction unit for extracting features from the second captured image; and
A classifier for binary classification on whether or not a face is an object by taking the feature detected from the feature extraction unit as an input. According to claim 1,
The object detection unit,
A feature map obtained by extracting features of the second captured image from the convolution between the second captured image and the learned kernel is calculated by taking the second captured image as an input, and an object is obtained from a linear combination between the calculated feature map and the learned weight. A virtual worn image providing system comprising a convolution neural network (CNN neural network) for outputting a predicted probability of whether or not a face is a face. delete delete delete According to claim 1,
The linear regression neural network takes the angles and lengths of each of the first to third line segment components as inputs, and outputs a prediction value related to the scale of the object together with the imaging angle of the object Virtual wearing image, characterized in that provision system. According to claim 1,
Calculate scale information directly from the part model, or
The posture detection unit outputs scale information from the linear regression neural network, and the scale information output from the posture detection unit corresponds to the length of a major axis of the face region as the object. According to claim 1,
The image matching unit calculates a sum of squared error between a rotation angle and a tilt angle as the imaging angle detected from the second captured image and a rotation angle and a tilt angle as the imaging angle of the first captured image stored in the database. A system for providing a virtual wearing image, characterized in that recognizing a capturing angle of a first captured image at which errors (SSE) are minimized is closest to a capturing angle detected from a second captured image. According to claim 1,
The image matching unit copies the subject image information divided from the background on the first captured image to a target position of the object image extracted from the second captured image. According to claim 1,
The image matching unit enlarges the subject image to correspond to a preset size ratio between the subject and the object by referring to the scale information output from the posture detection unit and the size of the subject image divided from the background in the first captured image. Alternatively, the virtual wearing image providing system characterized in that copying the reduced geometrically transformed subject image to a target location of the object image extracted from the second captured image. According to claim 18,
The system for providing a virtual wearing image, characterized in that the size ratio between the subject and the object is stored in association with the entire set of first captured images of the subject captured at different imaging angles. According to claim 1,
The image matching unit may set the image information of the subject captured on the first captured image to the second captured image so that the light emitting indicator captured on the first captured image and a target position set in relation to the wearing position on the second captured image are aligned with each other. A system for providing a virtual wearing image, characterized in that for copying to a target position on a captured image.

Images