KR20230083347A - The Apparatus and method for generating the Back side image - Google Patents

Abstract

The present invention includes the steps of generating human pose data composed of M joint coordinates from first image information, generating a label map from the first image information, and the first joint coordinates and the label among the M joint coordinates. When the separation distance of the map is less than or equal to the preset separation distance, the step of generating the label map as the first body label corresponding to the first joint coordinates and the step of generating the first body label correspond to all the M joint coordinates. sequentially, generating M body labels and generating a body label map composed of the M body labels, wherein the separation distance between the M joint coordinates and the label map exceeds a predetermined separation distance. In one case, generating a non-human label map and generating second image information composed of the human label map and the non-human label map; dividing the first image information into N patches to generate N patches of first image information, and generating a first patch map composed of the N patches of first image information; dividing into N patches to generate N second image information patches, and generating a second patch map composed of the N second image information patches; The step of dividing the information patch into A number of sub-patches and generating a first sub-patch group composed of the A number of sub-patches corresponding to the first image information patch, and the step of generating the first sub-patch group are described above. It is sequentially performed on all N first image information patches, generating N sub-patch groups corresponding to each of the N first image information patches, and generating a sub-patch map composed of the N sub-patch groups. and calculating an average value by summing the color information values of the A sub-patches constituting the first sub-patch group, changing the average value to the color information values of the A sub-patches, and changing the color information values of the A number of sub-patches. The steps of generating a first average sub-patch group composed of A number of sub-patches to which values are reflected and the steps of generating the first average sub-patch group are sequentially performed on all N sub-patch groups, generating a sub-patch group and generating an average sub-patch map composed of the N average sub-patch groups; A plurality of first average sub-patch groups are set, and the plurality of first average sub-patch groups are set in proportion to a first weight of the first average sub-patch group and a separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups. A second weight of average sub-patch groups is set, a first color information value in which the first weight is reflected in the first average sub-patch group, and a second weight value in the color information of the plurality of adjacent average sub-patch groups The steps of generating a back color information value by reflecting the average value of the reflected second color information values, generating a rear sub patch group reflecting the back color information value, and generating the rear sub patch group are the average sub-patch sequentially performing on all N average subpatch groups constituting the map, generating N rear subpatch groups, and generating a rear subpatch map composed of the N rear subpatch groups; A patch and a sub-patch are composed of a plurality of pixels, and the pixel is an address value of the pixel, location information of the pixel, and color information of the pixel.

Description

Apparatus and method for generating the back side image information {The Apparatus and method for generating the Back side image}

The present invention relates to a method and apparatus for generating rear image information using a front image, and more particularly, to a method and apparatus for providing a stereoscopic image by extracting or generating a rear image using artificial neural network technology.

In the field of content development, attempts to develop and utilize content using 3D characters are increasing, starting with the development of 2D character content due to the development of overall performance of hardware and software. When using 3D characters, there are various advantages, such as being able to reuse existing 3D characters in various software, expressing three-dimensional animation and applying various control methods. However, there are technical limitations of 3D character production, and various researches and developments are being conducted to overcome them.

In particular, there is a method of generating a 3D mesh using an artificial neural network for a front image captured such as a person. However, when there is no rear image, it is difficult to generate a 3D mesh image.

In order to solve this problem, a method and apparatus for generating rear image information by extracting and generating a rear image based on a photographed front image of a person are required.

Republic of Korea Registration No. 10-2245220 (2021.04.27. Notice) Republic of Korea Patent Publication No. 10-1906431 (2018.10.11. Notice) Republic of Korea Patent Publication No. 10-1829733 (2018.02.09. Notice)

An object of the present invention is to provide a method and apparatus for generating rear image information by extracting and generating a rear image based on a photographed front image of a person.

A method and apparatus for generating rear image information to solve the above problems,

Generating human pose data composed of M joint coordinates from first image information, generating a label map from the first image information, and spacing between the first joint coordinates and the label map among the M joint coordinates When the distance is less than or equal to the preset separation distance, the step of generating the label map as a first body label corresponding to the first joint coordinates and the step of generating the first body label are sequentially performed for all the M joint coordinates. , generating M body labels and generating a body label map composed of the M body labels, and when the separation distance between the M joint coordinates and the label map exceeds a predetermined separation distance, generating a non-human label map, and generating second image information composed of the human body label map and the non-human label map; dividing the first image information into N patches to generate N patches of first image information, and generating a first patch map composed of the N patches of first image information; dividing into N patches to generate N second image information patches, and generating a second patch map composed of the N second image information patches; The step of dividing the information patch into A number of sub-patches and generating a first sub-patch group composed of the A number of sub-patches corresponding to the first image information patch, and the step of generating the first sub-patch group are described above. It is sequentially performed on all N first image information patches, generating N sub-patch groups corresponding to each of the N first image information patches, and generating a sub-patch map composed of the N sub-patch groups. and calculating an average value by summing the color information values of the A sub-patches constituting the first sub-patch group, changing the average value to the color information values of the A sub-patches, and changing the color information values of the A number of sub-patches. The steps of generating a first average sub-patch group composed of A number of sub-patches to which values are reflected and the steps of generating the first average sub-patch group are sequentially performed on all N sub-patch groups, generating a sub-patch group and generating an average sub-patch map composed of the N average sub-patch groups; A plurality of first average sub-patch groups are set, and the plurality of first average sub-patch groups are set in proportion to a first weight of the first average sub-patch group and a separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups. A second weight of average sub-patch groups is set, a first color information value in which the first weight is reflected in the first average sub-patch group, and a second weight value in the color information of the plurality of adjacent average sub-patch groups The steps of generating a back color information value by reflecting the average value of the reflected second color information values, generating a rear sub patch group reflecting the back color information value, and generating the rear sub patch group are the average sub-patch sequentially performing on all N average subpatch groups constituting the map, generating N rear subpatch groups, and generating a rear subpatch map composed of the N rear subpatch groups; A patch and a sub-patch are composed of a plurality of pixels, and each pixel stores an address value of the pixel, location information of the pixel, and color information of the pixel.

In addition, in the step of generating the rear sub-patch group to which the back color information value is reflected, the positional coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map determine the human body label map. Only when the positional coordinates are the same as one of the positional coordinates of the constructing patches, a first rear subpatch group in which the back color information value is reflected is generated, and N average sub-patches constituting the average sub-patch map If the positional coordinates of the first average sub-patch group among the groups are not the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map, the color information value of the first average sub-patch group is reflected. In the step of generating a second back subpatch group and a back subpatch map composed of the N back subpatch groups, the first average subpatch among the N average subpatch groups constituting the average subpatch map The step of generating a rear subpatch group in which the back color information value is reflected is performed only when the positional coordinates of the group are the same as one of the positional coordinates of the positional coordinates of the patches constituting the body label map. It is sequentially performed on all N average sub-patch groups constituting the patch map to generate S first rear sub-patch groups, and among the N average sub-patch groups constituting the average sub-patch map, the first average sub-patch If the positional coordinates of the group are not the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map, a second sub-patches group in which the color information value of the first average sub-patch group is reflected is obtained. The generating step is sequentially performed on all N average subpatch groups constituting the average subpatch map to generate T second rear subpatch groups, the S first rear subpatch groups, and the T It is characterized in that a rear sub-sub patch map consisting of two second sub-sub patch groups is generated.

In addition, in the plurality of first adjacent average sub-patch groups, the positional coordinates of the plurality of first adjacent averaged sub-patch groups are identical to the positional coordinates of one of the positional coordinates of the patches constituting the body label map. to be characterized

In addition, the first weight is set to a value greater than the second weight,

이고, The sum of the first weight and the second weights set to the plurality of first adjacent average subpatch groups is 1, and the first weight C is

ego,

이고,The second weight w is

ego,

W, which is a correlation between the first weight and the second weight, is

이고,

ego,

Z is the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups.

이고, 상기 제 2 가중치인 w는

이고, 상기 제 1가중치와 상기 제 2가중치들의 상관 관계식인 W은

이고, 상기 Z는 상기 제 1평균서브패치군과, 상기 복수개의 제 1 인접평균서브패치군의 패치군수의 합이고, 상기 패치, 서브패치는 복수개의 픽셀로 구성되고, 상기 픽셀은 상기 픽셀의 주소 값, 상기 픽셀의 위치정보 및 상기 픽셀의 색상정보가 저장되어 있는 것을 특징으로 한다.In addition, the apparatus for extracting rear image information includes a human pose generator for generating human pose data composed of M joint coordinates from the first image information, a label map generator for generating a label map from the first image information, and the M When the separation distance between the first joint coordinate of the number of joint coordinates and the label map is less than or equal to the preset separation distance, the label map is generated as a first body label corresponding to the first joint coordinates, and all the M joint coordinates are generated. M body labels are sequentially generated for , a body label map composed of the M body labels is generated, and when the separation distance between the M joint coordinates and the label map exceeds a preset separation distance, the non-human body A body label map generating unit that generates a label map and generates second image information composed of the human body label map and the non-body label map, and N first images by dividing the first image information into N patches An information patch is generated, a first patch map composed of the N first image information patches is generated, the second image information is divided into N patches to generate N second image information patches, and the N second image information patches are generated. A second patch map composed of second image information patches is generated, the first image information patch among the N first image information patches is divided into A sub-patches, and the first image information patch corresponding to the first image information patch is generated. generating a first sub-patch group composed of A number of sub-patches, and sequentially generating N sub-patch groups corresponding to each of the N first image information patches; A sub-patch map is created by generating a sub-patch map composed of the N sub-patch groups, an average value is calculated by summing the color information values of the A number of sub-patches constituting the first sub-patch group, and the average value is calculated. The color information values of the A sub-patches are changed, and a first average sub-patch group composed of A sub-patches reflecting the changed color information values is generated, and N average sub-patch groups are sequentially generated for all N sub-patch groups. and a patch map generation unit generating sub-patch groups and generating an average sub-patch map composed of the N average sub-patch groups, wherein the patch map generation unit comprises the N average sub-patch groups constituting the average sub-patch map. An average sub-patch including the N average sub-patch groups, provided that the positional coordinates of the first average sub-patch group are the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map. In the map, a plurality of first adjacent average sub-patch groups surrounding the first average sub-patch group are set, and the first weight value of the first average sub-patch group and the first average sub-patch group and the plurality of adjacent averages are determined. A second weight of the plurality of average sub-patch groups is set in proportion to the separation distance between the sub-patch groups, the first color information value in which the first weight is reflected in the first average sub-patch group, and the average of the plurality of adjacent sub-patch groups A back color information value is created by reflecting the average value of the second color information values in which the second weight is reflected in the color information of the sub-patch group, and a first rear sub-patch group to which the rear color information value is reflected is generated, and the average value of the second color information value is reflected. In all N average sub-patch groups constituting the sub-patch map, S number of first secondary sub-patch groups are sequentially generated, and among the N average sub-patch groups constituting the average sub-patch map, the first average sub-patch group is selected. If the positional coordinates are not the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map, a second sub-patches group in which the color information value of the first average sub-patch group is reflected is generated; , T second rear subpatch groups are sequentially generated from all N average subpatch groups constituting the average subpatch map, the S first rear subpatch groups and the T second rear subpatch groups A back side subpatch map consisting of groups is generated, and the plurality of first adjacent average subpatch groups have positional coordinates of the plurality of first adjacent averaged subpatch groups among positional coordinates of patches constituting the body label map. position coordinates identical to one position coordinate, the first weight is set to a value greater than the second weight, and the sum of the first weight and the second weights set to the plurality of first adjacent average subpatches is 1 , and the first weight C is

, and the second weight w is

And W, which is the correlation between the first weight and the second weight, is

where Z is the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups, the patches and sub-patches are composed of a plurality of pixels, and the pixels are It is characterized in that an address value, location information of the pixel, and color information of the pixel are stored.

The present invention has an advantage of providing a 3D stereoscopic image based on front image information by providing a method and apparatus for generating rear image information using input front image information.

In addition, the present invention has the advantage of reducing the amount of processing operation by applying a human body label map when generating rear image information using input front image information.

1 is a device and system diagram for generating rear image information according to the present invention.
2 is a flowchart illustrating a method of generating rear image information according to the present invention.
3 is a conceptual diagram illustrating a method of generating rear image information according to the present invention.
4 is a conceptual diagram of generating human pose data using input first image information according to an embodiment of the present invention.
5 is a diagram of joint coordinates that are human pose data using input first image information according to an embodiment of the present invention.
6 is a conceptual diagram of generating a label map by segmentation using input first image information according to an embodiment of the present invention.
7 is a detailed flowchart for generating patch information using first image information and second image information according to an embodiment of the present invention.
8 is an exemplary view showing a human body label map and a non-human body label map according to an embodiment of the present invention.
9 is a conceptual diagram illustrating a rear sub-patch map generated by dividing a first input image and a second input image into a plurality of patches according to an exemplary embodiment of the present invention.
10 is a conceptual diagram illustrating a sub-patch group in which a first image patch separated from a first input image is divided into sub-patches according to an example of the present invention.
11 is a conceptual diagram of generating a first average sub-patch group by performing an average operation and pixel information of sub-patches constituting a sub-patch group according to an example of the present invention.
12 is a conceptual diagram for assigning first and second weights to a first average sub-patch group and a first adjacent average sub-patch group according to an example of the present invention.
13 is a conceptual diagram of generating a rear subpatch map by applying first and second weights according to an exemplary embodiment of the present invention.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content will be thorough and complete and the spirit of the present invention will be sufficiently conveyed to those skilled in the art.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in this application, they should not be interpreted in ideal or excessively formal meanings. don't

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings.

1 is a device and system diagram for generating rear image information according to the present invention. Referring to FIG. 1 , an apparatus or system for generating rear image information according to the present invention may include a server 100 or a terminal 200, the server 100, and a terminal 200. In general, it is preferable to configure a device and system for generating rear image information with the server 100.

The server 100 includes an imaging unit 110, a human pose generator 120, a label map generator 130, a body label map generator 140, and a patch map generator 150, The communication unit 160 may be selectively applied and configured.

The imaging unit 110 may capture and generate the first image information and then store it, or store image information that has already been stored.

The human pose generation unit 120 generates human pose data composed of M joint coordinates from the input first image information.

The label map generator 130 creates a label map from the first image information.

The human body label map generator 140 converts the label map into a first human body corresponding to the first joint coordinates when a separation distance between a first joint coordinate of the M joint coordinates and the label map is equal to or less than a predetermined separation distance. labels, M body labels are sequentially generated for all the M joint coordinates, a body label map composed of the M body labels is created, and the separation distance between the M joint coordinates and the label map is When the predetermined separation distance is exceeded, a non-human label map is generated, and second image information composed of the human body label map and the non-human label map is generated.

The patch map generating unit 150 separates the first image information into N patches to generate N patches of first image information, and generates a first patch map composed of the N patches of first image information; The second image information is divided into N patches to generate N second image information patches, a second patch map composed of the N second image information patches is generated, and the N first image information patches are generated. The first image information patch is divided into A number of sub-patches, a first sub-patch group composed of the A number of sub-patches corresponding to the first image information patch is generated, For each of the N first image information patches, N sub-patch groups corresponding to each of the N first image information patches are sequentially generated, a sub-patch map is generated by generating a sub-patch map composed of the N sub-patch groups, and the first sub-patch map is generated. An average value is calculated by summing the color information values of the A sub-patches constituting the patch group, the average value is changed to the color information value of the A sub-patches, and the A sub-patches to which the changed color information values are reflected A first average sub-patch group is generated, N average sub-patch groups are sequentially generated for all N sub-patch groups, and an average sub-patch map composed of the N average sub-patch groups is generated.

In addition, the patch map generator 150 Only when the location coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map are the same as the location coordinates of one of the location coordinates of the patches constituting the body label map. In the average sub-patch map including the N average sub-patch groups, a plurality of first adjacent average sub-patch groups surrounding the first average sub-patch group are set, and a first weight of the first average sub-patch group and , A second weight value of the plurality of average sub-patch groups is set in proportion to the separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups, and the first weight value is applied to the first average sub-patch group. A back color information value is generated by reflecting the average value of the first color information value reflected and the second color information value in which the second weight is reflected in the color information of the plurality of adjacent average sub-patch groups, and the rear color information value A first rear sub-patch group reflected by this is generated, S number of first rear sub-patch groups are sequentially generated in all N average sub-patch groups constituting the average sub-patch map, and the average sub-patch map is constructed. If the position coordinates of the first average sub-patch group among the N average sub-patch groups are not the same as the position coordinates of one of the position coordinates of the patches constituting the body label map, the first average sub-patch group A second rear subpatch group in which the color information value is reflected is generated, T second rear subpatch groups are sequentially generated in all N average subpatch groups constituting the average subpatch map, and the S first subpatch groups are generated. A rear sub-sub patch map composed of a rear sub-sub patch group and the T second rear sub-sub patch groups is generated.

In addition, the patch map generating unit 150 is configured such that the positional coordinates of the plurality of first adjacent average sub-patch groups are the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map. A group of first adjacent average subpatches is set.

이고, 상기 제 2 가중치인 w는

이고, 상기 제 1가중치와 상기 제 2가중치들의 상관 관계식인 W은

이고, 상기 Z는 상기 제 1평균서브패치군과, 상기 복수개의 제 1 인접평균서브패치군의 패치군수의 합으로 설정한다. In addition, the patch map generator 150 sets the first weight to a value greater than the second weight, and the sum of the first weight and the second weights set in the plurality of first adjacent average subpatches is set to 1, and the first weight C is

, and the second weight w is

And W, which is the correlation between the first weight and the second weight, is

where Z is set as the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups.

here, The patch and sub-patch are composed of a plurality of pixels, and each pixel stores the address value of the pixel, location information of the pixel, and color information of the pixel.

The communication unit 160 performs a function of transmitting specific information, and may use wired or wireless communication, limited to a system in which a general communication technology is applied and configured between the server 200 and the terminal 100.

Here, the specific information means data such as first image information, second image information, first patch map, second patch map, and rear patch map.

Since the terminal 200 has the same configuration as the server 100, description thereof is omitted. A detailed description of the operation of the server 100 will be described later.

2 is a flowchart illustrating a method of generating rear image information according to the present invention. 3 is a conceptual diagram illustrating a method of generating rear image information according to the present invention. 2 and 3, generating human pose data using first image information (S210), generating label map data using the first image information (S220), and the human pose data Generating the body label map using (S230), generating second image information using the human body label map and non-body label map (S240), the first image information and the second image information It may be configured to include generating patch information using patch information (S250) and generating back image information using the patch information (S260).

In the step of generating human pose data using the first image information (S210), the image unit 110 or 210 in the terminal 200 or the server 100 converts the image acquired from the camera into the first image information or the terminal 200 or the first image information stored in the server 100 is used to generate human pose data.

The first image information generally refers to data or data files that store video or images in various formats such as a JPG file, which is a file format for storing data. In addition, the first image information may include an image image including human body related information in the first image information, and in particular, the human body related information may include a front image and may not include a rear image. Conversely, it may include a rear image and may not include a front image.

4 is a conceptual diagram of generating human pose data using input first image information according to an embodiment of the present invention. Referring to FIG. 4 , the human pose generators 120 and 220 generate human pose data using the first image information.

Using a known human pose generation algorithm, the known human pose generation algorithm is applied, and iteratively learns using a Convolutional Neural Network (CNN), a type of deep neural network, and finally a human pose vector map generate Here, the convolutional neural network (CNN) extracts human body-related information stored in the first image information, outputs the coordinates of each human body joint using the extracted human body information, and outputs the output Among the joint coordinates of the human body, the coordinates with high predictive accuracy are selected to set the 2D joint coordinates. Then, 3D joint coordinates are generated using the 2D joint coordinates. Using a 3D standard human body mesh data model (SMPL, Skinned Multi-Person Linear Model), the 2D joint coordinates are converted to the 3D standard human body. It is applied to the mesh data model to generate matched 3D joint coordinates, that is, a human pose vector map.

Here, the human pose generally means a designated number of joints such as nose, elbow, ankle, etc. for a person detected on an input original image and a 3D coordinate vector on each image coordinate system, and the coordinates are It is a known technique that is normalized to a value between 0 and 1. Here, the generated human pose vector map refers to FIG. 4 .

5 is a diagram of joint coordinates that are human pose data using input first image information according to an embodiment of the present invention. Referring to FIG. 5 , in one embodiment, the human pose data is composed of 23 joint coordinates, each joint coordinate is given a name and a unique number, and the joint coordinates are used when a human body label map is created thereafter. It can be. The joint coordinates may be arbitrarily set as needed. Also, if necessary, a human pose vector map, which is a 3D coordinate, can be used.

6 is a conceptual diagram of generating a label map by segmentation using input first image information according to an embodiment of the present invention. Referring to FIG. 6 , when the human pose generators 120 and 220 generate human pose data using the first image information, the label map generators 130 and 230 use the first image information to create labels. A step of generating map data (S220) is performed.

In general, the meaning of segmentation (Semantic Image Segmentation) is to detect human body information on image information, and set pixels with detected human body information as True, and pixels without detected human body information as False. It refers to a technique of constructing a label map with the same size as the original by specifying pixel values.

A known label map generation algorithm is used, the known label map generation algorithm is applied, and a convolutional neural network (CNN), which is a type of deep neural network, is used for repeated learning to finally generate a label map. .

Here, in the label map extraction method, the label map is set to the same image size as the image size of the image information, and a value of 0 to 255 is set using a black-and-white channel instead of a general RGB channel.

A pixel corresponding to human body information among the image information is set to 255, and a pixel not corresponding to human body information among the image information is set to 0 to create and store a label map.

By applying the label map extraction method to a convolutional neural network (CNN), it is repeatedly learned to extract a label map by isolating only pixel information corresponding to the human body region. The data size of the extracted label map is reduced compared to the transmitted image information.

To compensate for this data loss, a label map is finally generated by applying a known convolutional neural network (convolutional neural network with an encoder-decoder structure). Here, the segmentation result label map is referred to FIG. 6 .

After the label map generator (130, 230) generates label map data using the first image information, the human body label map generator (140, 240) generates the human body label map using the human pose data. (S230) is performed.

8 is an exemplary view showing a human body label map and a non-human body label map according to an embodiment of the present invention.

5 and 8, the human body label map generators 140 and 240 measure the distance between the joint coordinates of the human data and the label map, and if the measured distance is smaller than a preset distance, Set the label corresponding to the corresponding joint coordinates as the human body label.

Here, the body label map generators 140 and 240 may arbitrarily set a predetermined distance, and the number of body labels may be changed according to the set value. Here, it is preferable to set the predetermined distance so that the number of body labels corresponding to joint coordinates can be determined.

In addition, when the joint coordinates are composed of H pieces, the distance between each of the joint coordinates and the label map is measured for the H number of joint coordinates, and when the measured distance is smaller than the preset distance, the corresponding joint coordinates and Corresponding labels are set as human body labels, and H body labels are generated by sequentially and repeatedly performed, and a body label map 810 composed of the generated H body labels is generated.

Here, the distance means a general distance specific term, Euclidean distance.

In addition, when there are 23 joint coordinates, which are the human data described above, 23 human body labels corresponding thereto are generated. That is, when H number of joint coordinates are used, H number of body labels are generated, and a body label map 810 composed of the generated H number of body labels is generated.

After the body label map is generated, the body label map generators 140 and 240 generate second image information using the body label map and the non-body label map (S240).

Here, the non-body label map 820 means a region corresponding to a case where the joint coordinates and a predetermined distance are greater than each other.

Here, the preset distance may be arbitrarily set.

That is, the second image information is composed of a human body label map 810 corresponding to the human body region of the first image information and a non-body label map 820 not corresponding to the human body region of the first image information.

After the second image information is generated, the patch map generators 150 and 250 generate patch information using the first image information and the second image information (S250).

7 is a detailed flowchart for generating patch information using first image information and second image information according to an embodiment of the present invention. Referring to FIG. 7, generating a first patch map by dividing the first image information into specific units (S710), and generating a second patch map by dividing the second image information into specific units (S710). S720) and generating a rear patch map using the first patch map and the second patch map (S730).

First, the patch map generators 150 and 250 generate a first patch map by dividing the first image information into specific units (S710).

9 is a conceptual diagram illustrating a rear sub-patch map generated by dividing a first input image and a second input image into a plurality of patches according to an exemplary embodiment of the present invention. 10 is a conceptual diagram illustrating a sub-patch group in which a first image patch separated from a first input image is divided into sub-patches according to an example of the present invention.

Referring to FIGS. 9 and 10, a patch and a sub-patch, which are the separated specific units, will be described.

Referring to FIG. 9, the patch map generators 150 and 250 divide the first image information into N patches to generate N first image information patches, Create the first patch map.

Here, the patch map generators 150 and 250 may arbitrarily set P and Q, and may set P and Q identically or differently. In general, since a video image is expressed and stored in units of pixels, a patch separated into P*Q=N pieces is also stored in units of pixels. A known technique is applied to this and detailed description thereof will be omitted.

The patch map generator (150, 250) separates the first image information and generates P*Q patches expressed as (P, Q) from the first patch expressed as (1,1), A term is defined by defining the generated P*Q, that is, N, first image information patches as a first patch map.

In addition, the patch map generator 150, 250 separates the second image information into N patches to generate N second image information patches, and the second patch composed of the N second image information patches create a map

The patch map generator (150, 250) separates the second image information and generates a patch of P*Q represented by (P, Q) from the first patch represented by (1, 1), A term is defined by defining the generated P*Q, that is, N, second image information patches as a second patch map.

Here, the patch map generators 150 and 250 may arbitrarily set P and Q, and may set P and Q identically or differently. In general, a video image is expressed, stored, and structured in units of each pixel, and patches separated into N pieces are also stored in units of pixels. A known technique is applied to this, and detailed description thereof will be omitted.

Here, the pixel is a well-known technical term, and the information stored in the pixel information is described within the scope required for the present invention. The pixel stores certain information and expresses the information in terms of color and brightness. The schedule information may include an address value of the pixel, location information corresponding to the address value of the pixel, and color information. The location information includes location coordinate information of the pixel, and the color information is composed of RGB values, which is a color information display format that is a known technology, and each value of RGB may be set in the range of 0 to 255 and used.

Here, mapping the (1,1) first image information patch of the first patch map and the (1,1) second image information patch of the second patch map in a 1:1 manner is expressed as synchronization. The (1,2) first image information patch of the first patch map and the (1,2) first image information patch of the second patch map are mapped 1:1, and sequentially the first patch map ( The first image information patch of P, Q) and the second image information patch of (P, Q) of the second patch map are mapped 1:1, and the first patch map and the second patch map are respectively Synchronization is performed by mapping the configuration patch of

Here, for synchronization of N first image information patches constituting the first patch map and N second image information patches constituting the second patch map, position coordinates constituting each patch may be used, The synchronized first image information patch and the second image information patch are mapped with the same location coordinates and synchronized. Accordingly, the N first image information patches constituting the first patch map and the N second image information patches constituting the second patch map each store the same location information.

Here, the N first image information patches constituting the first patch map are composed of N sub-patch groups.

The sub-patch is composed of one or a plurality of pixels.

Here, the pixel is a well-known technical term, and the information stored in the pixel information is described within the scope required for the present invention. The pixel stores certain information and expresses the information in terms of color and brightness. The schedule information may include an address value of the pixel, location information corresponding to the address value of the pixel, and color information. The location information includes location coordinate information of the pixel, and the color information is composed of RGB values, which is a color information display format that is a known technology, and each value of RGB may be set in the range of 0 to 255 and used.

The sub-patch is composed of J pixels, and the J number may be arbitrarily set. Preferably, it can be set to J = 4, but is not limited thereto. Also, the patch may be composed of K sub-patches, and the K number may be arbitrarily set. Preferably, K = 4 may be set, but is not limited thereto. Here, the specific unit constituting the sub-patch means a pixel composed of J pieces, and the specific unit constituting the patch means a sub-patch composed of K pieces.

Again, referring to the step of generating a first patch map by dividing the first image information into specific units (S710), the patch map generator 150, 250 separates the first image information into N patches. Thus, N first image information patches are generated, and a first patch map composed of the N first image information patches is generated.

In addition, the first image information patch is divided into A number of sub-patches, and a sub-patch group composed of the A number of sub-patches corresponding to the first image information patch is generated.

In addition, all the N first image information patches are sequentially divided into A sub-patches, N sub-patch groups corresponding to each of the N first image information patches are generated, and the N sub-patches Create a sub-patch map composed of groups.

Here, the N sub-patch groups constituting the sub-patch map and the N first image information patches constituting the first patch map are composed of the plurality of pixels described above, and the addresses that are storage information of the pixels Value, location information and color information corresponding to the address value of the pixel are also stored.

After the sub-patch map is generated, an average value is calculated by summing the color information values of the A sub-patches constituting the first sub-patch group, and the average value is changed to the color information value of the A sub-patches. and a first average sub-patch group composed of A sub-patches in which the changed color information values are reflected.

In addition, the average value is calculated by summing the color information values of the A number of sub-patches, the average value is changed to the color information value of the A number of sub-patches, and the A number of sub-patches reflecting the changed color information values are configured. The process of generating the first average sub-patch group is sequentially performed on all the N sub-patch groups to generate N average sub-patch groups, and an average sub-patch map composed of the N average sub-patch groups is obtained. generate

11 is a conceptual diagram of generating a first average sub-patch group by performing an average operation and pixel information of sub-patches constituting a sub-patch group according to an example of the present invention.

Referring to FIG. 11, one patch is composed of 4 sub-patches, and the sub-patch is composed of 4 pixels.

Here, as described above, one pixel includes three types of color information in RGB form, and each expression value is expressed as a natural number from 0 to 255. Four pixels constituting sub-patch 1 have color information described therein, and color information is described for four pixels constituting sub-patches 2 to 4, respectively. A first average sub-patch group is generated by summing and averaging each color information, and the color information for which the sum and average operations have been performed is reflected in the generated first average sub-patch group.

As described above, the sub-patch can be set to any number of pixels, and the first patch can also be set to any number of sub-patches.

When the patch map generator (150, 250) generates a first patch map by dividing the first image information into specific units, the patch map generator (150, 250) divides the second image information into specific units. A step of separating and generating a second patch map (S720) is performed.

More specifically, the second image information is divided into N patches to generate N second image information patches, and a second patch map composed of the N second image information patches is generated.

In general, the N second image information patches of the second patch map composed of the N second image information patches are composed of the same number of patches as the N first image information patches constituting the first patch map. desirable.

As a result, N first image information patches constituting the first patch map, N average sub-patch groups constituting the average sub-patch map, and N second image information constituting the second patch map The patch is preferably composed of the same number of N patches.

After the second patch map is generated by dividing the second image information into specific units, the patch map generator (150, 250) generates a rear patch map using the first patch map and the second patch map. Step S730 is performed.

Among the examples, the first example will be described.

In the first embodiment, a plurality of first adjacent average sub-patch groups surrounding the first average sub-patch group in the average sub-patch map including the N average sub-patch groups are set, and the first average sub-patch group is set. A second weight of the plurality of average sub-patch groups is set in proportion to a first weight of the sub-patch group and a separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups, The rear color information value is obtained by reflecting the average value of the first color information value in which the first weight is reflected in the sub-patch group and the second color information value in which the second weight is reflected in the color information of the plurality of adjacent average sub-patch groups. The steps of generating, generating a rear subpatch group in which the rear surface color information value is reflected, and generating the rear subpatch group are sequentially performed on all N average subpatch groups constituting the average subpatch map, N back sub patch groups are generated, and a back sub patch map composed of the N back sub patch groups is created.

9 to 13, FIG. 9 shows a rear sub-patch map generated using a first patch map and a second patch map.

The patch map generation units 150 and 250 use an average sub-patch map composed of N average sub-patch groups generated from the N first image information patches of the first patch map, that is, by performing a predetermined operation to obtain N A posterior subpatch map composed of posterior subpatches is created.

Here, the patch map generation units 150 and 250 randomly select a first average sub-patch group, which is one of the N average sub-patch groups constituting the average sub-patch map, and select a plurality of average sub-patch groups surrounding the first average sub-patch group. A number of first adjacent average sub-patch groups are set.

Referring to FIG. 12, when the patch map generators 150 and 250 set the first average sub-patch group 1210 in the center, 8 adjacent average sub-patch groups are set, and the set 8 average sub-patch groups are This is the first adjacency average sub-patch group 1220.

Here, the patch map generators 150 and 250 may arbitrarily set a first weight to the first average sub-patch group 1210. For example, after setting 0.4, the first adjacent average sub-patch group 1220 The second weight may be arbitrarily set, and for example, 0.075 may be set.

It is preferable that the sum of the first weight of the first average sub-patch group and the second weight of the 8 average sub-patch groups constituting the first adjacent average sub-patch group 1220 is 1, and the first average sub-patch group It is preferable to set the first weight higher than the second weight of the first adjacent average sub-patch group 1220 . In addition, it is preferable to equally set the second weights of the 8 average sub-patch groups constituting the first adjacent average sub-patch group 1220 .

For example, the first weight and the second weight of the first adjacent average sub-patch group 1220 satisfy the following formula condition.

이고, 상기 제 2 가중치인 w는

이고, 상기 제 1가중치와 상기 제 2가중치들의 상관 관계식인 W은

이고, 상기 Z는 상기 제 1평균서브패치군과, 상기 복수개의 제 1 인접평균서브패치군의 패치군수의 합으로 설정한다Here, the first weight is set to a value greater than the second weight, the sum of the first weight and the second weights set to the plurality of first adjacent average subpatches is set to 1, and the first weight C is

, and the second weight w is

And W, which is the correlation between the first weight and the second weight, is

Z is set as the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups.

Referring back to FIG. 11, the steps of generating a rear-semi-sub patch group will be described. The first average sub-patch group 1210 stores RGB color information, and by applying the first weight, a sum value to which the first weight is applied to each RGB color information value is obtained.

In addition, each of the eight average sub-patch groups constituting the first adjacent average sub-patch group 1220 also stores RGB color information values, and a sum value to which the second weight is applied to each RGB color information value is obtained.

The sum value to which the first weight is applied stored in the first average sub-patch group 1210 and the RGB color information values of each of the eight average sub-patch groups constituting the first adjacent average sub-patch group 1220 Calculate the average value of the summed values with 2 weights applied.

The average value is replaced with each RGB color information value of the first average sub-patch group 1210 and stored, and the first average sub-patch group 1210 replaced by each RGB color information value is stored. 1 It is created as a rear sub patch group.

The method described above is sequentially performed on all N average subpatch groups constituting the average subpatch map to generate the rear subpatch groups, and N rear subpatch groups are generated. A posterior subpatch map composed of posterior subpatches is created.

After the rear sub-patch map is generated, rear image information is generated using the patch information (S260).

In order to remove noise between the N rear sub-patches groups, the noise applies a known edge blending algorithm, and performs a Gaussian Blur operation on the edge pixels of each rear patch to perform edge blending to remove noise. and create the final back surface texture map.

Among the examples, the second example will be described. In the second embodiment, in order to reduce the amount of data calculation, the weight-based operation of the first embodiment is performed only on the body label patches corresponding to the body label map among the second image information to generate the first rear sub-patches group, In the case of the non-human label map, a second sub-sub-patch group is generated by applying an average sub-patch group without performing a separate weight calculation, and the first sub-sub-patch group and the second sub-sub-patch group are obtained. applied to create a posterior subpatch map.

Specifically, the first image information patch of (1,1) of the first patch map and the second image information patch of (1,1) of the second patch map are mapped 1:1. is expressed as synchronization, and the (1,2) first image information patch of the first patch map and the (1,2) first image information patch of the second patch map are mapped 1:1, sequentially. By mapping the (P, Q) first image information patch of the first patch map and the (P, Q) second image information patch of the second patch map in a 1:1 ratio, the first patch map and the second image information patch Synchronization is performed by mapping each configuration patch of the two-patch map.

Here, for synchronization of N first image information patches constituting the first patch map and N second image information patches constituting the second patch map, position coordinates constituting each patch may be used, The synchronized first image information patch and the second image information patch are mapped with the same location coordinates and synchronized. Accordingly, the N first image information patches constituting the first patch map and the N second image information patches constituting the second patch map each store the same location information.

In addition, as described above, the N sub-patch groups and the N first average sub-patch groups are stored with only color information values changed, and the N sub-patch groups and the N first average sub-patch groups In the patch group, positional information of each of the N first image information patches is stored as it is. Accordingly, the N first average sub-patch groups are synchronized with the N second image information patches constituting the second patch map, and store the same location coordinates.

In addition, as described above, each of the N second image information patches constituting the second patch map has location information stored therein, and each of the location information includes a human body label map and a non-body label map, respectively. stored in the patch.

Accordingly, the N first average sub-patch groups may be synchronized based on location information of the body label map and the non-body label map.

When the position coordinates of the first average sub-patch group among the N first average sub-patch groups are the same as the position coordinates of one of the position coordinates of the patches constituting the body label map, weight calculation is performed to After 1st sub-patch group is created, the location coordinate of the first average sub-patch group among the N first average sub-patch groups is the same as the location coordinate of one of the location coordinates of the patches constituting the non-human label map. In the case of coordinates, weight calculation is not performed, and the first average sub-patch group is replaced with the second rear sub-patch group.

As a result, if the first rear sub-patches group generated by performing the weight calculation is S pieces and the second rear sub-patches group generated without performing the weight calculation is T pieces, then S+T=N is satisfied, and the rear sub-patches A map is created.

Describing this in detail, in the case of patches constituting the human body label map, S first rear patch groups are generated through the following process.

A plurality of first contiguous average sub-patch groups surrounding the first average sub-patch group in the average sub-patch map including the N average sub-patch groups are set, and a first weight of the first average sub-patch group is determined; A second weight of the plurality of average sub-patch groups is set in proportion to a separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups, and the first weight is applied to the first average sub-patch group. A back color information value is generated by reflecting the average value of the reflected first color information value and the second color information value in which the second weight is reflected in the color information of the plurality of adjacent average sub-patch groups, and the rear color information value is The reflected back subpatch group is generated, and the step of generating the back subpatch group is sequentially performed on all N average subpatch groups constituting the average subpatch map to generate S first back subpatch groups. do.

9 to 13, FIG. 9 shows a rear sub-patch map generated using a first patch map and a second patch map.

The patch map generator (150, 250) uses an average sub-patch map consisting of N average sub-patch groups generated from N first image information patches of the first patch map, that is, performs a predetermined operation to obtain S number of sub-patch maps. A first rear subpatch group is generated.

Here, the patch map generation units 150 and 250 randomly select a first average sub-patch group, which is one of the N average sub-patch groups constituting the average sub-patch map, and select a plurality of average sub-patch groups surrounding the first average sub-patch group. A number of first adjacent average sub-patch groups are set.

Referring to FIG. 12, when the patch map generators 150 and 250 set the first average sub-patch group 1210 in the center, 8 adjacent average sub-patch groups are set, and the set 8 average sub-patch groups are This is the first adjacency average sub-patch group 1220.

Here, the patch map generators 150 and 250 may arbitrarily set a first weight to the first average sub-patch group 1210. For example, after setting 0.4, the first adjacent average sub-patch group 1220 The second weight may be arbitrarily set, and for example, 0.075 may be set.

It is preferable that the sum of the first weight of the first average sub-patch group and the second weight of the 8 average sub-patch groups constituting the first adjacent average sub-patch group 1220 is 1, and the first average sub-patch group It is preferable to set the first weight higher than the second weight of the first adjacent average sub-patch group 1220 . In addition, it is preferable to equally set the second weights of the 8 average sub-patch groups constituting the first adjacent average sub-patch group 1220 .

Here, 8 average sub-patch groups among the first adjacent average sub-patch groups adjacent to the first average sub-patch group are areas not included in the location information values of patches constituting the human body label map, that is, the non-body label map In the case of a patch constituting , the second weight, which is the weight, is set to 0, and eight average sub-patch groups among the first adjacent average sub-patch groups adjacent to the first average sub-patch group constitute the human body label map. In the case of an area corresponding to the location information value of the patch, a second weight is applied.

이고, 상기 제 2 가중치인 w는

이고, 상기 제 1가중치와 상기 제 2가중치들의 상관 관계식인 W은

이고, 상기 Z는 상기 제 1평균서브패치군과, 상기 복수개의 제 1 인접평균서브패치군의 패치군수의 합으로 설정한다.For example, the first weight and the second weight of the first adjacent average sub-patch group 1220 satisfy the following formula condition. The first weight is set to a value greater than the second weight, the sum of the first weight and the second weights set to the plurality of first adjacent average subpatches is set to 1, and the first weight, C Is

, and the second weight w is

And W, which is the correlation between the first weight and the second weight, is

where Z is set as the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups.

Referring back to FIG. 11, the steps of generating a rear-semi-sub patch group will be described. The first average sub-patch group 1210 stores RGB color information, and by applying the first weight, a sum value to which the first weight is applied to each RGB color information value is obtained.

In addition, each of the eight average sub-patch groups constituting the first adjacent average sub-patch group 1220 also stores RGB color information values, and a sum value to which the second weight is applied to each RGB color information value is obtained.

The sum value to which the first weight is applied stored in the first average sub-patch group 1210 and the RGB color information values of each of the eight average sub-patch groups constituting the first adjacent average sub-patch group 1220 Calculate the average value of the summed values with 2 weights applied.

The average value is replaced with each RGB color information value of the first average sub-patch group 1210 and stored, and the first average sub-patch group 1210 replaced by each RGB color information value is stored. 1 It is created as a rear sub patch group.

The method described above is sequentially performed on all N average subpatch groups constituting the average subpatch map to generate S rear subpatch groups.

In the case of patches constituting the non-body label map, T second posterior patch groups are generated through the following process.

In other words, when the position coordinates of the first average sub-patch group among the N first average sub-patch groups are the same as the position coordinates of one of the position coordinates of the patches constituting the non-body label map, the weight Without performing the calculation, the first average sub-patch group is replaced with the second rear sub-patch group, thereby generating T second rear sub-patch groups.

The S first rear sub patch groups and the T second rear sub patch groups are applied to generate a final rear sub patch map composed of S+T=N rear sub patch groups.

After the rear sub-patch map is generated, rear image information is generated using the patch information (S260).

In order to remove noise between the N rear sub-patches groups, the noise applies a known edge blending algorithm, and performs a Gaussian Blur operation on the edge pixels of each rear patch to perform edge blending to remove noise. and create the final back surface texture map.

100: server 200: terminal
110: video unit 210: video unit
120: human pose generator 220: human pose generator
130: label map generator 230: label map generator
140: human body label map generator 240: human body label map generator
150: patch map generator 250: patch map generator
160: communication unit 260: communication unit
1000: image information 1010: patch
1020: sub-patch group 1210: first average sub-patch group
1220: first adjacent sub-patch group

Claims (5)

A rear image information generation method for generating rear image information of first image information including a front human body image using artificial neural network software,
generating human pose data composed of M joint coordinates from the first image information;
generating a label map from the first image information;
generating the label map as a first body label corresponding to the first joint coordinates when a separation distance between a first joint coordinate among the M joint coordinates and the label map is equal to or less than a predetermined separation distance;
generating M body labels by sequentially performing the generating of the first body labels for all the M joint coordinates, and generating a body label map composed of the M body labels;
When the separation distance between the M joint coordinates and the label map exceeds a predetermined separation distance, generating a non-human label map, and generating second image information composed of the body label map and the non-human label map step;
dividing the first image information into N patches to generate N first image information patches, and generating a first patch map composed of the N first image information patches;
generating N second image information patches by dividing the second image information into N patches, and generating a second patch map composed of the N second image information patches;
dividing the first image information patch among the N first image information patches into A number of sub-patches, and generating a first sub-patch group composed of the A number of sub-patches corresponding to the first image information patch;
The step of generating the first sub-patch group is sequentially performed for all of the N first image information patches to generate N sub-patch groups corresponding to each of the N first image information patches; generating a sub-patch map composed of N sub-patch groups;
An average value is calculated by summing the color information values of the A number of sub-patches constituting the first sub-patch group, the average value is changed to the color information value of the A number of sub-patches, and the color information values of the A number are reflected. generating a first average sub-patch group composed of N sub-patches;
The step of generating the first average sub-patch group is sequentially performed for all N sub-patch groups to generate N average sub-patch groups, and an average sub-patch map composed of the N average sub-patch groups is obtained. generating;
A plurality of first contiguous average sub-patch groups surrounding the first average sub-patch group in the average sub-patch map including the N average sub-patch groups are set, and a first weight of the first average sub-patch group is determined; A second weight of the plurality of average sub-patch groups is set in proportion to a separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups, and the first weight is applied to the first average sub-patch group. A back color information value is generated by reflecting the average value of the reflected first color information value and the second color information value in which the second weight is reflected in the color information of the plurality of adjacent average sub-patch groups, and the rear color information value is generating a post-reflected patch group; and
The step of generating the back subpatch group is sequentially performed on all N average subpatch groups constituting the average subpatch map to generate N back subpatch groups, which are composed of the N back subpatch groups. Including; generating a rear sub-patch map that is
The patch and the sub-patch are composed of a plurality of pixels, and the pixel stores an address value of the pixel, location information of the pixel, and color information of the pixel. According to claim 1,
In the step of generating the rear sub-patch group in which the back color information value is reflected, the positional coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map constitute the human body label map. Only when the positional coordinates are the same as one of the positional coordinates of the patches, a first rear sub-patch group in which the back color information value is reflected is generated, and among the N average sub-patch groups constituting the average sub-patch map, When the positional coordinates of the first average sub-patch group are not the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map, the second average sub-patch group reflects the color information value of the first average sub-patch group. Create a posterior sub-patch group;
The step of generating a rear sub patch map consisting of the N rear sub patch groups,
Only when the location coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map are the same as the location coordinates of one of the location coordinates of the patches constituting the body label map. , the step of generating the rear sub-patch group to which the back color information value is reflected is sequentially performed on all N average sub-patch groups constituting the average sub-patch map to generate S first rear sub-patch groups;
If the position coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map are not the same as the position coordinates of one of the position coordinates of the patches constituting the body label map, The step of generating the second rear sub-patch group in which the color information value of the first average sub-patch group is reflected is sequentially performed on all N average sub-patch groups constituting the average sub-patch map, so that T number of second rear-side sub-patch groups are obtained. By creating a sub-patch group,
and generating a rear surface image information generation method comprising the S number of first rear surface sub patch groups and the T number of second rear surface sub patch groups. According to claim 2,
The plurality of first adjacent average sub-patch groups,
The method of generating rear image information, characterized in that the positional coordinates of the plurality of first adjacent average sub-patch groups are the same as the positional coordinates of one of the positional coordinates of the patches constituting the body label map. According to claim 3,
The first weight is set to a value greater than the second weight,
the sum of the first weight and the second weights set to the plurality of first adjacent average sub-patch groups is 1;
The first weight C is

ego,
The second weight w is

ego,
W, which is a correlation between the first weight and the second weight, is

ego,
Z is the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups. In the rear image information extraction device for extracting the rear image information of the first image information including the front human body image using artificial neural network software,
a human pose generating unit generating human pose data composed of M joint coordinates from the first image information;
a label map generating unit generating a label map from the first image information;
When a separation distance between a first joint coordinate of the M joint coordinates and the label map is less than or equal to a predetermined separation distance, the label map is generated as a first body label corresponding to the first joint coordinates, and all of the M joint coordinates are generated. Sequentially generating M body labels for joint coordinates and generating a body label map composed of the M body labels;
When the separation distance between the M joint coordinates and the label map exceeds a predetermined separation distance, generating a non-human label map, and generating second image information composed of the body label map and the non-human label map a human body label map generating unit;
dividing the first image information into N patches to generate N patches of first image information, and generating a first patch map composed of the N patches of first image information;
dividing the second image information into N patches to generate N patches of second image information, and generating a second patch map composed of the N patches of second image information;
The first image information patch among the N first image information patches is divided into A number of sub-patches, and a first sub-patch group composed of the A number of sub-patches corresponding to the first image information patch is generated; For all first image information patches, N sub-patch groups corresponding to each of the N first image information patches are sequentially generated, and a sub-patch map composed of the N sub-patch groups is generated to generate a sub-patch map. create,
An average value is calculated by summing the color information values of the A number of sub-patches constituting the first sub-patch group, the average value is changed to the color information value of the A number of sub-patches, and the color information values of the A number are reflected. A first average sub-patch group composed of the N sub-patch groups is generated, N average sub-patch groups are sequentially generated for all N sub-patch groups, and an average sub-patch map consisting of the N average sub-patch groups is generated. Including a patch map generation unit to generate,
The patch map generator,
Only when the location coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map are the same as the location coordinates of one of the location coordinates of the patches constituting the body label map. In the average sub-patch map including the N average sub-patch groups, a plurality of first adjacent average sub-patch groups surrounding the first average sub-patch group are set, and a first weight of the first average sub-patch group and , A second weight value of the plurality of average sub-patch groups is set in proportion to the separation distance between the first average sub-patch group and the plurality of adjacent average sub-patch groups, and the first weight value is applied to the first average sub-patch group. A back color information value is generated by reflecting the average value of the first color information value reflected and the second color information value in which the second weight is reflected in the color information of the plurality of adjacent average sub-patch groups, and the rear color information value A first rear sub-patch group reflected by this is generated, and S first rear sub-patch groups are sequentially generated in all N average sub-patch groups constituting the average sub-patch map;
If the position coordinates of the first average sub-patch group among the N average sub-patch groups constituting the average sub-patch map are not the same as the position coordinates of one of the position coordinates of the patches constituting the body label map, A second subsequent sub-patch group in which the color information value of the first average sub-patch group is reflected is generated, and T second subsequent sub-patch groups are sequentially added to all N average sub-patch groups constituting the average sub-patch map. create,
generating a rear sub patch map composed of the S first rear sub patch groups and the T second rear sub patch groups;
The plurality of first adjacent average sub-patch groups,
The positional coordinates of the plurality of first adjacent average sub-patch groups are the same positional coordinates as the positional coordinates of one of the positional coordinates of the patches constituting the body label map;
The first weight is set to a value greater than the second weight,
the sum of the first weight and the second weights set to the plurality of first adjacent average sub-patch groups is 1;
The first weight C is

ego,
The second weight w is

ego,
W, which is a correlation between the first weight and the second weight, is

ego,
Z is the sum of the first average sub-patch group and the number of patch groups of the plurality of first adjacent average sub-patch groups;
The patch and the sub-patch are composed of a plurality of pixels, and the pixel stores an address value of the pixel, location information of the pixel, and color information of the pixel.

Images