WO2023214748A1

WO2023214748A1 - Method and device for measuring color in image, and computer-readable medium

Info

Publication number: WO2023214748A1
Application number: PCT/KR2023/005838
Authority: WO
Inventors: 김영민; 이준호; 장호준; 양용진; 박명삼; 김은정; 김정은
Original assignee: 서울대학교 산학협력단; 코스맥스 주식회사
Priority date: 2022-05-04
Filing date: 2023-04-28
Publication date: 2023-11-09
Also published as: KR20230155718A

Abstract

The present invention relates to a method and a device for measuring the color of skin in an image, and a computer-readable medium, and to a method and a device for measuring color in an image, and a computer-readable medium, the method and the device deriving shape information and light information from a captured image, subtracting the shape information and the light information from the image so as to derive reflectivity information, deriving a color parameter transform means on the basis of the difference between captured color information about a reference color range in patch reflectivity information about a patch and color information about the ground truth of a reference color, and applying the color parameter transform means to facial reflectivity information about the face so as to derive information about a colorimetry color that is an inherent color of the face, from which the influence of illumination is excluded.

Description

Colorimetric methods, apparatus, and computer-readable media in images

The present invention relates to a method, device, and computer-readable medium for skin color measurement in an image, which derives shape information and light information from a captured image and derives reflectance information by subtracting the shape information and light information from the image. And, based on the difference between the photographed color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color parameter conversion means is used. It relates to a colorimetric method, device, and computer-readable medium in an image that derives colorimetric color information, which is the unique color of the face, excluding the influence of lighting, by applying it to the reflectance information of the face.

As interest in beauty gradually increases, consumers' efforts to obtain accurate information about their skin condition continue. An example of this is the trend toward personal color. Personal color is a color that harmonizes with the body color of an individual to make the individual look lively and energetic. Makeup methods and clothing colors are different depending on the personal color. If you know your personal color, you can create an effective look according to the color. possible.

To this end, consumers visit skin medical facilities, skin care shops, personal color diagnosis companies, etc. and check their skin condition with the help of professional skin analysis devices. Accordingly, if accurate color can be measured with a device that can be easily used in daily life, its usability will be very high.

Meanwhile, user terminals equipped with cameras, such as smartphones, are becoming widely available, and the number of media that utilize them to convey color or visual information is increasing. Therefore, if the user's skin can be analyzed based on an image taken of the user's skin through the camera of the user terminal without expensive professional equipment, the user will be able to check the condition of their skin more easily. However, the colors captured by the camera are the result of additional lighting, camera sensor, and display corrections, so it is difficult to say that they delivered accurate colors.

Korean Patent Publication No. 10-2018-0082172 discloses a method and device for performing colorimetry on a user terminal. Prior Patent 1 noted that the conventional user terminal-based skin analysis technology has the limitation of not being able to accurately analyze the user's skin because it does not take into account the lighting characteristics in the space where the user's skin is photographed. Although technology has been disclosed to perform colorimetry or skin analysis by analyzing the characteristics of lighting, there is a problem in that it only considers the characteristics of the lighting and does not take into account the characteristics of the camera and the cognitive characteristics of human vision.

Accordingly, since the captured image information may change depending on the external environment such as camera settings, performance, and lighting, a color patch with the colorimetric reference color printed is used to minimize colorimetric errors due to the external environment when performing colorimetry. There is a need to develop a method of photographing the back with an object and performing colorimetry based on the color difference between the photographed object and the colorimetric reference color (ground truth, the exact color value for this is known).

In order to solve the above problems, an embodiment of the present invention is a colorimetry method performed in a computing system including one or more processors and one or more memories, and is a colorimetric method that calculates a facial image for a face area from an original image and a patch area for the patch area. An image derivation step of deriving a patch image; The face image is input to an inference model including one or more artificial neural networks to derive face shape information and light information including the intensity and direction of light in the face image, and the face shape information for the face image and a first subtraction application step of applying the subtraction to the light information to derive reflectance information of the face that minimizes the effect of color change due to the shape of the face to which the light is irradiated. A patch image containing one or more reference color areas is input into an inference model including one or more artificial neural networks, the shape information of the patch is derived, and the shape information of the patch and the light information are subtracted and applied to the patch image, A second subtraction application step of deriving reflectance information of the patch that minimizes the effect of color change due to the shape of the patch to which light is irradiated; And based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face A colorimetry method is provided, including a face color derivation step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means to the color parameter conversion means.

In one embodiment of the present invention, the shape information includes 3D information according to the actual shape derived from 2D information on the image about the object included in the image, and the light information includes information about the object included in the image. It includes the intensity and direction of the irradiated light, and the reflectance information is color information for each pixel that removes the influence of shadows that appear on the shape according to the intensity and direction of the light when light is irradiated to the shape of the object included in the image. It can be included.

In one embodiment of the present invention, the patch includes a plurality of reference color areas, and each reference color area has the same shape and area and may be arranged separately from each other.

In one embodiment of the present invention, the inference model includes a common model, a surface normal vector model, a light information inference model, and a reflectance model, and the common model includes one or more artificial neural networks, and determines the surface from the input image. Feature information that is commonly input to the normal vector model, the light information inference model, and the reflectance model is derived, and the surface normal vector model includes one or more artificial neural networks, and from the feature information in the image that is 2D information The shape information, which is 3D structure information containing the actual shape, is derived, the light information inference model includes one or more artificial neural networks, and the light information includes the intensity of light and the direction of light in the image from the feature information. Derived, the reflectance model includes one or more artificial neural networks, and reflectance information, which is color information for each pixel, can be derived from the feature information.

In one embodiment of the present invention, the first subtraction application step includes a facial feature information deriving step of deriving facial feature information by inputting a facial image into a common model including one or more artificial neural networks; A facial shape derivation step of deriving facial shape information by inputting the facial feature information into a surface normal vector model including one or more artificial neural networks; A light information derivation step of inputting the facial feature information into a light information inference model including one or more artificial neural networks to derive light information including the intensity and direction of light in the face image; And a facial reflectance derivation step of applying the shape information and light information of the face to the face image to derive reflectance information of the face reflecting only the color of the face. The second subtraction application step includes applying the patch image. A patch feature information deriving step of deriving patch feature information by inputting into a common model including one or more artificial neural networks; A patch shape derivation step of deriving shape information of the patch by inputting the patch feature information into a surface normal vector model including one or more artificial neural networks; It may include a patch reflectance derivation step of applying the shape information of the patch and the light information to the patch image to derive reflectance information of the patch reflecting only the color of the patch.

In one embodiment of the present invention, the colorimetric method further includes a model learning step, wherein the model learning step inputs a learning image into the common model and inputs learning feature information of the common model into the surface normal vector model. A learning shape information deriving step of deriving learning shape information; A learning light information deriving step of inputting the learning feature information into the light information inference model to derive learning light information; A learning reflectance information deriving step of deriving learning reflectance information by inputting the learning feature information into a reflectance model that derives reflectance information; And a predicted learning image derivation step of deriving a predicted learning image by combining the learning shape information, learning light information, and learning reflectance information, wherein the common Detailed parameter values or filter information of the model, the surface normal vector model, the light information inference model, and the reflectance model can be learned.

In one embodiment of the present invention, the face color derivation step includes a conversion means derivation step, and the conversion means derivation step includes the conversion of a preset color space for each of a plurality of reference color regions in the reflectance information of the patch. A color parameter extraction step of extracting each first color parameter information; Between a second color parameter derived by applying a color parameter conversion means including a plurality of correction values to a plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors A correction value determination step of determining the correction value so that the color difference value derived by the first color difference algorithm satisfies a preset standard; and a fourth color parameter derived by applying a color parameter conversion means including a plurality of correction values to the plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors. It may include a correction value changing step of changing some or all of the correction values in a direction that reduces the color difference value derived by the second color difference algorithm.

In one embodiment of the present invention, the color difference value by the first color difference algorithm is linear with respect to the color parameter, and the color difference value by the second color difference algorithm is non-linear with respect to the color parameter of the first color. The difference algorithm and the second color difference algorithm are different from each other, and the color parameter conversion means is implemented as a matrix including a plurality of correction values, and the number of columns or rows of the matrix is the number of elements of the first color parameter. Corresponds to a number, the second color parameter may be derived by matrix multiplying the matrix with respect to the first color parameter, and the fourth color parameter may be derived by matrix multiplying the matrix with respect to the first color parameter.

In order to solve the above problems, one embodiment of the present invention is a colorimetric device implemented in a computing system including one or more processors and one or more memories. The colorimetric device includes a facial image for the facial area from an original image and An image derivation step of deriving a patch image for the patch area; The face image is input to an inference model including one or more artificial neural networks to derive face shape information and light information including the intensity and direction of light in the face image, and the face shape information for the face image and a first subtraction application step of applying the subtraction to the light information to derive reflectance information of the face that minimizes the effect of color change due to the shape of the face to which the light is irradiated. A patch image containing one or more reference color areas is input into an inference model including one or more artificial neural networks, the shape information of the patch is derived, and the shape information of the patch and the light information are subtracted and applied to the patch image, A second subtraction application step of deriving reflectance information of the patch that minimizes the effect of color change due to the shape of the patch to which light is irradiated; And based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face A colorimetry device is provided that performs a facial color derivation step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means to the color parameter conversion means.

In order to solve the above problem, one embodiment of the present invention is a computer program stored in a computer-readable medium, including a plurality of instructions executed by one or more processors, wherein the computer program is configured to generate a face from an original image. An image derivation step of deriving a face image for the area and a patch image for the patch area; The face image is input to an inference model including one or more artificial neural networks to derive face shape information and light information including the intensity and direction of light in the face image, and the face shape information for the face image and a first subtraction application step of applying the subtraction to the light information to derive reflectance information of the face that minimizes the effect of color change due to the shape of the face to which the light is irradiated. A patch image containing one or more reference color areas is input into an inference model including one or more artificial neural networks, the shape information of the patch is derived, and the shape information of the patch and the light information are subtracted and applied to the patch image, A second subtraction application step of deriving reflectance information of the patch that minimizes the effect of color change due to the shape of the patch to which light is irradiated; And based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face A facial color derivation step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means to a computer program is provided.

According to one embodiment of the present invention, the colorimetry method can be easily performed even with a personal mobile terminal such as a smartphone.

According to one embodiment of the present invention, by performing colorimetry on an image taken together with the reference color included in the patch and the face, the effect of deriving the color information of the face is achieved by comparing the patch and the face in the same environment. It can be performed.

According to one embodiment of the present invention, since shape information is derived for each face and patch, it is possible to derive accurate reflectance information by compensating for differences in each characteristic and state in the image.

According to an embodiment of the present invention, by deriving shape information and light information from an acquired image, reflectance information that removes the influence of a shadow created by light irradiated on the shape can be derived from the image. there is.

According to one embodiment of the present invention, the reflectance information is derived by subtracting the shape information and light information that affect the reflectance information from the image, thereby achieving the effect of deriving more accurate reflectance information than when the reflectance information is directly derived from the image. It can be performed.

According to one embodiment of the present invention, since the patch includes a plurality of reference colors, it can exhibit the effect of responding to various facial colors.

According to one embodiment of the present invention, since the ground truth of each reference color of the patch is known, color parameters are determined based on the color information of each reference color (reference color in the reflectance information of the patch) in the image and the ground truth. It can be effective in deriving conversion means and deriving color difference information.

According to one embodiment of the present invention, a learning image is input in the model learning step, and the derived learning shape information, learning light information, and learning reflectance information are combined and compared with the learning image, thereby deriving a clear output value (learning image). This can have the effect of learning detailed parameter values or filter information of the artificial neural network.

According to embodiments of the present invention, colorimetry reflects color difference information in a non-linear color space that has characteristics that are robust to external environments such as camera settings, performance, and lighting, and can more accurately implement the color difference perceived by the human eye. Methods, devices, and computer-readable media may be provided.

According to one embodiment of the present invention, it is possible to perform colorimetry by reflecting non-linear color difference information that is closer to the actual perception of the human eye, such as CIEDE2000.

According to one embodiment of the present invention, it is possible to easily perform colorimetry using a personal mobile terminal such as a smartphone.

According to one embodiment of the present invention, non-linearly defined color differences can be approximated and optimized through repetitive calculations in the color space, thereby achieving the effect of simultaneously promoting computational efficiency and colorimetric accuracy. there is.

Figure 1 schematically shows a colorimetry method according to an embodiment of the present invention.

Figure 2 schematically shows the environment of a computing system in which a colorimetric method according to embodiments of the present invention is performed.

Figure 3 schematically shows shape information, light information, and reflectance information according to an embodiment of the present invention.

Figure 4 schematically shows a patch according to an embodiment of the present invention.

Figure 5 schematically shows a plurality of inference models according to an embodiment of the present invention.

Figure 6 schematically shows detailed steps of the first subtraction application step and the second subtraction application step according to an embodiment of the present invention.

Figure 7 schematically shows the model learning step according to an embodiment of the present invention.

Figure 8 schematically shows an example of a color difference region defined in the color space of CIE76.

Figure 9 schematically shows an example of a color difference region defined in the color spaces of CIE76, CIE94, and CIE2000.

Figure 10 schematically shows the process of deriving a color difference value that can be linearly defined.

Figure 11 schematically shows the detailed steps of the conversion means derivation step and the internal components of the colorimetric device according to embodiments of the present invention.

Figure 12 schematically shows the detailed process of the correction value determination step according to embodiments of the present invention.

Figure 13 schematically shows the overall processes of the conversion means derivation step according to embodiments of the present invention.

Figure 14 schematically shows the operation of the color parameter conversion means according to embodiments of the present invention.

Figure 15 schematically shows the detailed process in the correction value determination step according to embodiments of the present invention.

Figure 16 schematically shows the detailed process of the correction value changing step according to embodiments of the present invention.

Figure 17 schematically shows detailed steps of the correction value changing step according to embodiments of the present invention.

Figure 18 schematically shows the internal configuration of a computing device according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS Various embodiments and/or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to facilitate a general understanding of one or more aspects. However, it will also be appreciated by those skilled in the art that this aspect(s) may be practiced without these specific details. The following description and accompanying drawings set forth in detail certain example aspects of one or more aspects. However, these aspects are illustrative and some of the various methods in the principles of the various aspects may be utilized, and the written description is intended to encompass all such aspects and their equivalents.

Additionally, various aspects and features may be presented by a system that may include multiple devices, components and/or modules, etc. It is also understood that various systems may include additional devices, components and/or modules, etc. and/or may not include all of the devices, components, modules, etc. discussed in connection with the drawings. It must be understood and recognized.

As used herein, “embodiments,” “examples,” “aspects,” “examples,” etc. may not be construed to mean that any aspect or design described is better or advantageous over other aspects or designs. . The terms '~part', 'component', 'module', 'system', 'interface', etc. used below generally refer to computer-related entities, such as hardware, hardware, etc. A combination of and software, it can mean software.

Additionally, the terms “comprise” and/or “comprising” mean that the feature and/or element is present, but exclude the presence or addition of one or more other features, elements and/or groups thereof. It should be understood as not doing so.

Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by those skilled in the art to which the present invention pertains. It has the same meaning as Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the embodiments of the present invention, have an ideal or excessively formal meaning. It is not interpreted as

1.이미지에서의 측색 방법1. Colorimetric method in image

Objects have unique colors. However, when an image of a material is acquired through a camera, colors that are different from those seen in the actual object with the eyes may be derived from the image. The color that appears when a color is expressed in an image can change depending on the inherent reflectance, light information (direction, intensity and color of light, etc.) according to surrounding lighting, and the sensitivity of the camera sensor. Therefore, it is possible to use an expensive colorimeter to remove the effects of the lighting and sensors and find the exact color unique to the object. However, in embodiments of the present invention, a patch having a reference color that is relatively inexpensive and can be easily utilized is used. A method of finding the original color by photographing the object to be colorimetric is disclosed.

Figure 1 (A) corresponds to a drawing schematically showing the original image.

In the present invention, a patch, which is a specific physical object with a reference color, and an object to be colorimetric are photographed together with a camera. The camera may be the camera itself, or may correspond to a camera built into a smartphone, etc. An original image can be acquired with the camera, and the original image can include an object and a patch to be colorimetric. Preferably, the object and patch to be colorimetric can be acquired without overlapping. The patch may include one or more reference colors, and the object to be colorimetric may include various objects such as faces, hands, animals, and objects, but the description below will focus on faces.

The original image may include a face area including a face and a patch area including a patch, and the face area may include a measurement area for colorimetry. That is, the measurement area may correspond to part or the entire face area, and may be determined by the user.

Therefore, the face image derived from the original image may be an image including the face area, and the patch image derived from the original image may be an image containing the patch area.

Figure 1(B) corresponds to a diagram schematically showing the steps of the colorimetric method.

As shown in Figure 1 (B), it is a colorimetry method performed in a computing system including one or more processors and one or more memories, which derives a face image for the face area and a patch image for the patch area from the original image. Image derivation step (S3000); The face image is input to an inference model including one or more artificial neural networks to derive face shape information and light information including the intensity and direction of light in the face image, and the face shape information for the face image and a first subtraction application step (S3100) of subtracting the light information to derive reflectance information of the face that minimizes the effect of color change due to the shape of the face to which the light is irradiated; A patch image containing one or more reference color areas is input into an inference model including one or more artificial neural networks, the shape information of the patch is derived, and the shape information of the patch and the light information are subtracted and applied to the patch image, A second subtraction application step (S3200) of deriving reflectance information of the patch that minimizes the effect of color change due to the shape of the patch to which light is irradiated; And based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face It may include a face color derivation step (S3300) of deriving colorimetric color information of the measurement area by applying the color parameter conversion means.

The colorimetric method may include an image derivation step (S3000) of deriving a face image for the face area and a patch image for the patch area from the original image. Specifically, the original image in Figure 1 (A) may include a face area for a face and a patch area for a patch, and the face area is an area where only the face is extracted from the original image and may be included in the face image, The patch area is an area where only patches are extracted from the original image and may be included in the patch image.

The colorimetric method can input the face image into an inference model (2000) including one or more artificial neural networks to derive face shape information and light information from the face image, and add the face shape information and light information to the face image. It may include a first subtraction application step (S3100) in which light information is subtracted to derive reflectance information of the face.

Specifically, the inference model 2000 in the first subtraction application step (S3100) may include a common model 2100, a surface normal vector model 2200, and a light information inference model 2300, each of which It may contain one or more artificial neural networks. The common model 2100 can derive feature information from an image, and the feature information can be input to the surface normal vector model 2200 and the light information inference model 2300 to derive shape information and light information. . That is, facial feature information is derived by inputting the facial image into the common model 2100, and inputting the facial feature information into the surface normal vector model 2200 and the light information inference model 2300 to obtain facial shape information. and light information can be derived. The shape information of the face may be the shape of the face area, and the shape may mean 3D information including the actual shape, and the light information may be the direction and direction of light included in the original image or the face image. May include century. Details about the plurality of inference models, shape information, and light information described above will be described later.

Subsequently, reflectance information of the face can be derived from the face image based on the shape information of the face and the light information. The face image includes the shape information of the face, the light information, and the reflectance information of the face. By subtracting the shape information of the face and the light information from the face image, the reflectance information of the face can be derived. there is.

According to an embodiment of the present invention, the shape information and light information of the face are extracted from the face image and the reflectance information of the face is derived by subtracting the shape information and light information of the face from the face image, so that the face It can have the effect of deriving more accurate reflectance information than the reflectance information of the face derived directly from the image.

In addition, the colorimetric method can input the patch image into an inference model including one or more artificial neural networks to derive the shape information of the patch from the patch image, and add the shape information of the patch and the light information to the patch image. It may include a second subtraction application step (S3200) in which reflectance information of the patch is derived by applying subtraction.

Specifically, the inference model in the second subtraction application step (S3200) may include a common model 2100 and a surface normal vector model 2200, and each may include one or more artificial neural networks. The common model 2100 can derive feature information from an image, and the feature information can be input to the surface normal vector model 2200 to derive shape information. That is, the patch image can be input into the common model 2100 to derive patch feature information, and the patch feature information can be input into the surface normal vector model 2200 to derive patch shape information. The shape information of the patch may be the shape of the patch area, and the shape may mean 3D information including the actual shape.

Subsequently, reflectance information of the patch can be derived from the patch image based on the shape information of the patch and the light information (derived from the face image). The patch image includes the shape information of the patch, the light information, and the reflectance information of the patch. By subtracting the shape information of the patch and the light information from the patch image, the reflectance information of the patch can be derived. there is.

According to an embodiment of the present invention, the shape information of the patch and the light information from the face image are extracted from the patch image, and the shape information of the patch and the light information are subtracted from the patch image to derive the reflectance information of the patch. , it can have the effect of deriving more accurate reflectance information than the reflectance information of the patch derived directly from the patch image.

According to one embodiment of the present invention, the plurality of inference models (common model 2100 and surface normal vector model 2200) in the first subtraction application step (S3100) and the second subtraction application step (S3200) are each It may be the same inference model.

According to another embodiment of the present invention, the plurality of inference models (common model 2100 and surface normal vector model 2200) in the first subtraction application step (S3100) and the second subtraction application step (S3200) are each These may be different inference models.

Thereafter, a color parameter conversion means is derived based on the reflectance information of the patch, the reflectance information of the face, and the preset ground truth of the patch derived in the above-described step, and the color parameter conversion means is converted to the reflectance information of the face. By applying this information, colorimetric color information of the measured part of the face can be derived. Details will be described later in Figure 1 (C).

Figure 1(C) corresponds to a diagram schematically showing the face color derivation step (S3300).

As shown in (C) of Figure 1, it is a colorimetric method performed in a computing system including one or more processors and one or more memories, wherein the color information of the reference color area in the reflectance information of the patch and the ground truth of the reference color are used. Based on the difference between the color information, a color parameter conversion means is derived, and the color parameter conversion means is applied to the color information of the measurement area in the reflectance information of the face to derive colorimetric color information of the measurement area. It may include a face color derivation step (S3300).

Specifically, the patch includes one or more reference colors, and the reflectance information of the patch includes color information for each of the one or more reference color areas. In other words, the color information for each of one or more reference color areas of the reflectance information of the patch may be color information from the reflectance information of the patch derived from the patch area in the original image. For one or more reference colors included in the patch, color information of a preset ground truth is stored in the computing system, and the ground truth may correspond to a color parameter that is the true value of the color. A color parameter conversion means can be derived based on the difference between the color information of the reference color area and the color information of the ground truth of the reference color. The difference may be a color difference value derived by applying the corresponding algorithm to the color parameters of each of the color information of the reference color area and the color information of the ground truth of the reference color.

The color parameter conversion means can be applied to the color information of the measurement area of the face reflectance information to derive colorimetric color information of the measurement area. The color parameter conversion means includes a correction value of 1 or more, and applying the color parameter conversion means may mean applying the correction value. As an example of the correction value, color information in the reference color area and the reference color The difference value between the color information of the ground truth may correspond. The derivation and details of the color parameter conversion means will be described later.

The derived colorimetric color information of the measurement area is color information obtained by measuring the measurement area of the face area in the present invention, and is based on the true value (ground truth) or the true value (ground truth) of the actual facial area corresponding to the measurement area, not on the image. It is desirable that it corresponds to nearby color information.

The color information of the measurement part of the face reflectance information may mean the color information of the reflectance information on the face image for the measurement part (extracted for the corresponding measurement part from the reflectance information of the face).

In the face color derivation step (S3300), if the patch contains only one reference color, the color parameter conversion means will be the difference value itself between the color information of the reference color area and the color information of the ground truth. (i.e., the correction value of the color parameter conversion means may correspond to the difference value), and the difference value is applied to the color information of the measurement area in the reflectance information of the face to obtain colorimetric color information of the measurement area. can be derived.

Additionally, if the patch includes only one reference color, but the correction value of the color parameter conversion means does not include only the difference value, it may be implemented as a matrix including a plurality of correction values.

In the face color derivation step (S3300), if the patch includes a plurality of reference colors, the color parameter conversion means may be implemented as a matrix including a plurality of correction values.

According to an embodiment of the present invention, the face color derivation step (S3300) can be performed by deriving one reference color that has the smallest color difference from the color information of the measurement area in the reflectance information of the face among a plurality of reference colors. .

Therefore, if the patch includes a plurality of reference colors, in the face color derivation step (S3300), among the plurality of reference colors, one reference color has the smallest color difference from the color information of the measurement area in the reflectance information of the face. If derived, the correction value of the color parameter conversion means may include a difference value between the one reference color and the corresponding ground truth color information.

According to one embodiment of the present invention, by deriving the color parameter conversion means and correcting color information, it is possible to achieve the effect of performing color measurement by considering various factors.

In the embodiment shown in Figure 2 (A), the original image of the part and the patch that the user wants to colorize is simultaneously acquired by the camera module built into the user terminal, and the method described above and below is used inside the user terminal. This allows color information to be derived through accurate colorimetry.

In the embodiment shown in Figure 2 (B), an original image is obtained by simultaneously photographing the part and patch that the user wants to colorize using a camera module built into the user terminal, and the obtained original image is transmitted to the server system. Later, color information can be derived through accurate colorimetry within the server system by the method described above and later.

The colorimetric method according to an embodiment of the present invention may be implemented in various types of systems in addition to environments such as (A) and (B) of Figures 2, but in common, a specific physical object on which the colorimetric target and reference color are displayed (e.g. For example, it can be said that any method that analyzes the image for a patch) in a computing system is applicable.

The above-mentioned Figure 1 shows a form in which the object to be colorimetric (for example, human skin color) and a physical object to which a reference color is assigned are captured as one image, but the present invention is not limited to this, and the object to be colorimetric is It also includes an embodiment in which a physical object to which a reference color (for example, a person's skin color) and a reference color are assigned are acquired as a plurality of separate images, and a colorimetric method described later is performed. When the plurality of images are acquired, light information corresponding to each image can be derived.

As shown in FIG. 3, the shape information includes 3D information according to the actual shape derived from 2D information on the image about the object included in the image, and the light information is related to the object included in the image. It includes the intensity and direction of the irradiated light and the shadow, and the reflectance information is for each pixel that removes the influence of the shadow that occurs on the shape according to the intensity and direction of the light when light is irradiated to the shape of the object included in the image. Color information may be included.

Specifically, the image includes the shape of the object included in the image, the color of the object, and the intensity and direction of light irradiating the object.

The color of the object may change due to the performance of the camera that acquires the image, the intensity and direction of the light, the shadow caused by the light irradiated to the shape, etc. For example, the plaster image itself for sketching is white, but the color changes due to the shadow caused by the light irradiated on the plaster image. In the present invention, in order to exclude the influence of such color change and measure color, shape information and light information can be subtracted from the image.

The shape information is information excluding color and light from the image, may correspond to 3D information about the shape of an object included in the image, and may not include color information of the object.

The object that actually exists is 3D, but if the object is acquired as an image through a camera, the object may be included in the image as 2D information, and the shape of the object included in the image is 2D information.

Accordingly, by inputting the image into one or more inference models, shape information, which is 3D information that is the shape of the object that actually exists, can be derived from the 2D information shape of the object included in the image. For example, the shape information may be the plaster statue itself that is not affected by light, or it may be said to be 3D modeling that is not affected by light except for the color (reflectance information) displayed on the 3D modeling program.

In other words, the shape information may be information containing only the shape of the object in the image, excluding reflectance information and light information.

The light information may mean the intensity and direction of light irradiated to the object in the image, a shadow may be created on the shape of the object by the light information, and the angle of the shadow depending on the direction of the light, The shape, area, etc. may change, and the light and dark of the shadow may change depending on the intensity of the light. Accordingly, a shadow may be formed according to the light information and the color of the object may change.

The light information contained in the image can be derived by inputting the image into one or more inference models.

According to an embodiment of the present invention, since the face image and patch image are extracted from the original image, even if the light information is derived from the face image, the effect of deriving the light information from the original image can be achieved. Therefore, applying light information derived from the face image to the patch image can produce the same effect as applying light information derived from the patch image.

In an embodiment of the present invention, a face image and a patch image are derived from the original image, and the light information is derived from the face image,

The reflectance information may be color information that is not changed by the light information included in the image and the shadow created by the shape to which the light is irradiated. By subtracting the shape information and the light information from the image, the degree of reflectance can be derived.

That is, the image includes light information on the image, shape information about the shape, and reflectance information, and when the shape information and light information are combined, a shadow due to light is created on the shape of the object, and the shape information, When the light information and the reflectance information are combined, the image can be created with the color of the reflectance information changed by the shadow.

Meanwhile, the reflectance information derived from the image may include color changes due to camera performance. Therefore, according to an embodiment of the present invention, the colorimetric color information of the measured area is derived based on the color information of the ground truth of the preset patch and the reflectance information of the patch derived from the image, so that the reflectance information changed by the performance of the camera is derived. Through correction, it can be effective in deriving color information close to the true value.

As shown in FIG. 4, the patch includes a plurality of reference color areas, and each reference color area has the same shape and area and may be arranged separately from each other.

The ground truth of each reference color area may be preset and stored in the computing system, and the location or arrangement of each reference color may be stored in the computing system.

Each reference color area of the patch according to an embodiment of the present invention has the same shape and area and is arranged separately from each other, making it easy to derive the patch area from the inference model and obtain color information of each reference color area. It is possible to achieve a certain effect.

In Figure 4, reference colors #1 to reference colors are shown, but the number of reference colors a patch has is not limited to this.

As shown in Figure 5, the inference model includes a common model 2100, a surface normal vector model 2200, a light information inference model 2300, and a reflectance model 2400, and the common model 2100 is , includes one or more artificial neural networks, and derives feature information commonly input to the surface normal vector model (2200), the light information inference model (2300), and the reflectance model (2400) from the input image, and The surface normal vector model 2200 includes one or more artificial neural networks and derives shape information, which is 3D information, which is the actual shape, from the image, which is 2D information, from the feature information. The light information inference model 2300 has 1 It includes the above artificial neural network, and derives light information including the intensity of light and the direction of light in the image from the feature information, and the reflectance model 2400 includes one or more artificial neural networks, and derives light information including the intensity of light and the direction of light in the image from the feature information. Reflectance information, which is color information for each pixel, can be derived.

Specifically, the inference model in the present invention includes the common model 2100, the surface normal vector model 2200, the light information inference model 2300, and the reflectance model 2400, each of which is a convolutional neural network. It can be implemented in a form that includes one or more of artificial neural network models such as CNN, capsule network (CapsNet), and rule-based feature information extraction models, and preferably includes SFS NET and RI_render (3ddfa). You can.

The common model 2100 derives feature information from an input image, and the feature information may be an output value from which the shape information, light information, and reflectance information can be derived by a plurality of inference models. The image may include all images in the present invention, and preferably may include the original image, a face image and patch image derived from the original image, and a learning image. The common model 2100 can derive facial feature information when the input image is a face image, derive patch feature information when the input image is a patch image, and derive learning feature information when it is a learning image. The feature information may be commonly input to the surface normal vector model (2200), the light information inference model (2300), and the reflectance model (2400).

The surface normal vector model 2200 can receive the feature information from the common model 2100 and derive shape information. If the feature information is facial feature information, face shape information can be derived, if the feature information is patch feature information, patch shape information can be derived, and if the feature information is learning feature information, learning shape information can be derived.

The light information inference model 2300 can receive the feature information from the common model 2100 and derive light information. If the feature information is facial feature information, light information can be derived, and if the feature information is learning feature information, learning light information can be derived.

According to an embodiment of the present invention, the light information is derived by inputting the facial feature information into the light information inference model 2300, and is used in both the first subtraction application step (S3100) and the second subtraction application step. You can.

According to another embodiment of the present invention, facial light information is derived by inputting the facial feature information into the light information inference model 2300, and inputting the patch feature information into the light information inference model 2300 to obtain patch light information. can be derived and used in each appropriate step.

The reflectance model 2400 can receive the feature information from the common model 2100 and derive reflectance information. If the feature information is learning feature information, learning reflectance information can be derived.

In an embodiment of the present invention, the reflectance model 2400 is used in the model learning step to have the effect of learning the common model 2100, the surface normal vector model 2200, and the light information inference model 2300. You can.

In another embodiment of the present invention, the colorimetric method further includes a reflectance inference step, wherein the reflectance inference step includes inputting a face image into a reflectance model 2400 to derive reflectance information of the face; Inputting the patch image into the reflectance model 2400 to derive reflectance information of the patch; And based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face It may include the step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means.

Meanwhile, the common model 2100, the surface normal vector model 2200, and the light information inference model 2300 may be used in the first subtraction application step, and the common model 2100 and the surface normal vector model (2200) can be used in the second subtraction application step, and the common model (2100), the surface normal vector model (2200), the light information inference model (2300), and the reflectance model (2400) are models to be described later. It can be used in the learning phase.

The model used in each step may be the same model, or it may be a different model optimized for each step. For example, the common model 2100 derives facial feature information in the first subtraction application step, derives patch feature information in the second subtraction application step, and derives learning feature information in the model learning step. . The common model 2100 used in the first subtraction application step and the common model 2100 used in the second subtraction application step may be the same or different models. Therefore, in the case of different models, each model can be trained in the model learning step.

Figure 6 schematically shows the detailed steps of the first subtraction application step (S3100) and the second subtraction application step (S3200) according to an embodiment of the present invention.

Figure 6 (A) corresponds to a diagram showing detailed steps of the first subtraction application step (S3100).

As shown in (A) of FIG. 6, the first subtraction application step is a facial feature information derivation step ( S3110); A face shape derivation step (S3120) of deriving shape information of the face by inputting the facial feature information into a surface normal vector model (2200) including one or more artificial neural networks; A light information derivation step (S3130) of inputting the facial feature information into a light information inference model (2300) including one or more artificial neural networks to derive light information including the intensity and direction of light in the face image; and a facial reflectance derivation step (S3140) of deriving reflectance information of the face in which only the color of the face is reflected by applying the face shape information and light information to the face image.

Specifically, in the first subtraction application step (S3100), facial feature information is derived by inputting a resized face image by cutting out only the face area from the original image in (A) of FIG. 1 into the common model (2100) to derive facial feature information. It may include an extraction step (S3110). The facial feature information is in the form of an output value of the common model 2100, and can be derived from face shape information and light information.

The first subtraction application step (S3100) may include a face shape derivation step (S3120) of inputting the facial feature information into the surface normal vector model (2200) to derive the shape information of the face. The shape information of the face is shape information about the face, and may be information derived by excluding parts that do not include skin, such as hair, from the face image.

The first subtraction application step (S3100) may include a light information derivation step (S3130) in which light information is derived by inputting the facial feature information into the light information inference model (2300). The light information is not only used to derive reflectance information of the face from the patch image, but can also be used in the second subtraction application step (S3200) to derive reflectance information of the patch.

The first subtraction application step (S3100) applies the shape information and light information of the face derived from the face shape derivation step (S3120) and the light information derivation step (S3130) to the face image to obtain reflectance information of the face. It may include a facial reflectance derivation step (S3140). Specifically, by subtracting and applying the shape information of the face and the light information from the face image, reflectance information of the face that reflects only the color of the face can be derived.

Figure 6(B) corresponds to a diagram showing detailed steps of the second subtraction application step (S3200).

As shown in (B) of FIG. 6, the second subtraction application step (S3200) inputs the patch image into a common model 2100 including one or more artificial neural networks to derive patch feature information. Derivation step (S3210); A patch shape derivation step (S3220) of deriving shape information of the patch by inputting the patch feature information into a surface normal vector model (2200) including one or more artificial neural networks; It may include a patch reflectance derivation step (S3230) of deriving reflectance information of the patch in which only the color of the patch is reflected by applying the shape information of the patch and the light information to the patch image.

Specifically, in the second subtraction application step (S3200), patch feature information is derived by inputting a resized patch image by cutting out only the patch area from the original image in (A) of FIG. 1 into the common model (2100). An extraction step may be included. The patch characteristic information is in the form of an output value of the common model 2100 and can be derived as the shape information of the patch.

The second subtraction application step (S3200) may include a patch shape derivation step (S3220) of deriving the shape information of the patch by inputting the patch feature information into the surface normal vector model (2200). The shape information of the patch is shape information about the patch and may include shape information about a plurality of reference color areas.

The second subtraction application step (S3200) uses the shape information and light information of the patch derived in the light information derivation step (S3130) included in the patch shape derivation step (S3220) and the first subtraction application step (S3100). It may include a patch reflectance derivation step (S3230) of deriving reflectance information of the patch by applying it to the patch image. Specifically, the reflectance information of the patch reflecting only the color of the patch can be derived by subtracting the shape information and the light information of the patch from the patch image.

As shown in Figure 7, the colorimetric method further includes a model learning step,

In the model learning step, learning shape information is derived by inputting a learning image into the common model (2100) and inputting learning feature information of the common model (2100) into the surface normal vector model (2200) to derive learning shape information. step; A learning light information deriving step of inputting the learning feature information into the light information inference model (2300) to derive learning light information; A learning reflectance information deriving step of deriving learning reflectance information by inputting the learning feature information into a reflectance model (2400) that derives reflectance information; And a predicted learning image derivation step of deriving a predicted learning image by combining the learning shape information, learning light information, and learning reflectance information, wherein the common Detailed parameter values or filter information of the model 2100, the surface normal vector model 2200, the light information inference model 2300, and the reflectance model 2400 can be learned.

Specifically, the model learning step may aim to train a plurality of inference models included in the present invention to derive a predicted learning image derived based on the input learning image as identical as possible to the learning image, The plurality of inference models may include a common model (2100), a surface normal vector model (2200), a light information inference model (2300), and a reflectance model (2400). The plurality of inference models may be the same as the model included in the present invention, and specifically, the common model 2100 is an inference model for extracting facial feature information and patch feature information in the present invention, and the surface normal vector model. 2200 may be an inference model for extracting face shape information and patch shape information in the present invention, and the light information inference model 2300 may be an inference model for extracting light information in the present invention.

The common model 2100 may be an inference model that can derive feature information including shape information, light information, and reflectance information from an image. When the learning image is input to the common model 2100, learning feature information can be derived as an output value of the common model 2100. The learning feature information may include learning shape information, learning light information, and learning reflectance information of the learning image, and the form of the learning feature information includes the type of the common model 2100 and the information included in the common model 2100. It can be determined according to artificial neural networks, etc. The learning feature information may be input to the plurality of inference models. That is, the learning feature information can be input to the surface normal vector model (2200), the light information inference model (2300), and the reflectance model (2400).

The colorimetric method in the present invention may include a learning shape information derivation step of deriving learning shape information by inputting learning feature information derived by inputting a learning image into the common model 2100 into the surface normal vector model 2200. You can.

The surface normal vector model 2200 may be an inference model that can derive shape information from the feature information. The shape information is shape information about the object in the image input to the common model 2100, and may be 3D information that is the actual shape of the object. The shape information may be information about the actual shape that is not affected by light and therefore does not include a shadow or color (reflectance). For example, it may correspond to uncolored 3D modeling (shape) in which no shadow is formed even when viewed from all directions because information about light is not entered into the program.

The colorimetric method in the present invention may include a learning light information derivation step of inputting learning feature information derived by inputting a learning image into the common model 2100 to the light information inference model 2300 to derive learning light information. You can.

The light information inference model 2300 may be an inference model that can derive light information from the feature information. The light information is light information being irradiated to the shape included in the image input by the common model 2100, and may include light intensity and light direction. A shadow may be created on a shape in the image by light information in the image.

The colorimetric method in the present invention may include a learning reflectance information derivation step of deriving learning reflectance information by inputting the image into the reflectance model 2400.

The reflectance model 2400 may be an inference model that can derive reflectance information from the feature information. The reflectance information is color information about the corresponding shape in the image, and may be unchanged color information that is not affected by light information and/or the shadow.

The colorimetric method in the present invention may include a predicted learning image derivation step of deriving a predicted learning image by combining the learning shape information, the learning light information, and the learning reflectance information derived by each inference model.

When comparing the predicted learning image with the input learning image, the common model 2100, the surface normal vector model 2200, and the light information inference model 2300 are used so that the predicted learning image with a small difference can be derived. ), and detailed parameter values or filter information of the reflectance model 2400 can be learned. Therefore, according to an embodiment of the present invention, the model learning step repeats the above-described steps to reduce or minimize the difference between the prediction learning image and the learning image by using the common model 2100 and the surface normal vector model 2200. , the light information inference model 2300, and the reflectance model 2400 can have the effect of learning detailed parameter values or filter information.

2.색상파라미터변환수단을 도출하는 방법2. Method of deriving color parameter conversion means

CIE76 corresponds to a formula that determines color difference using the CIELAB coordinate set. The color difference by CIE76 can be expressed by the following equation.

(Here, the first color has color parameters of (L ^* ₁ , a ^* ₁ , b ^* ₁ ) in the Lab color system, and the second color has color parameters of (L ^* ₂ , a ^* ₂ , b ^* ₂ ). In this case, the color difference between the first color and the second color can be expressed as ΔE ^* _ab above, and the color difference has a scalar value)

As a result, the color difference defined in the color space of CIE76 has a linear relationship with detailed color parameters.

As shown in FIG. 8, when colors in the RGBY color system are displayed as circles and a certain range of color differences defined in the color space of CIE76 are displayed as circles, it can be seen that they have regularity, as shown in FIG. 3. You can.

In Figure 9, when the x-axis is set to the color parameter value of a and the y-axis is set to the color parameter value of b in the Lab color system, the color difference defined in each of the CIE76, CIE94, and CIE2000 color spaces is shown for the displayed color. The range is shown in a circle.

In the case of color difference defined by the color space of CIE76, which has a linear relationship with color parameters, it can be seen that it has a completely linear relationship (complete circle), but in the case of CIE94, the color difference range line appears in an oval shape, In the case of CIE2000, some parts follow CIE94, but for example, when the b value is -0.5 or less, it shows completely different behavior.

As will be explained later, because CIE2000 reflects human cognitive characteristics more accurately, it is difficult to express detailed color parameters in a single formula like in CIE76, and the concept of boundary exists, so the calculation of color difference varies for each area. This is because the formula itself also has non-linear elements.

However, in the case of color differences that humans actually perceive, it is difficult to recognize a completely linear relationship like CIE76, and the sensitivity of the difference may vary depending on the color. Color difference calculation according to the color space reflecting this point may be non-linear, such as CIE2000, and the colorimetric method according to embodiments of the present invention, which will be described later, performs colorimetry in consideration of such non-linear color difference calculation, thereby allowing a person's actual perception. More accurate colorimetry can be performed.

For example, the color difference value in CIE76 can be expressed as the following equation, as described above, and can be expressed as a distance in three-dimensional or multi-dimensional space.

Figure 10 corresponds to an example of the color difference for color parameters (corresponding to the X, Y, and Z coordinate systems in Figure 10) in the case where such color parameters and color difference values have a linear relationship . The meaning of “linear” in the present invention should be interpreted in a broad sense, including cases where the color difference itself or other indirect numerical information such as a transformation matrix related to the color difference can be solved by linear algebraic methods.

For example, if there are 3 colors, 3 color parameters are given, and there is a random color, and the color difference for the random color from each of the 3 colors is known, it is defined as the linear color difference above. In the case of CIE76, an arbitrary color can be known using an algebraic method, but in the case of CIE2000, an arbitrary color cannot be known using an algebraic method.

Alternatively, in the case of an algorithm defined as a linear color difference, the color difference consisting of the calculated value of (a-b) may be included.

The colorimetric method according to embodiments of the present invention is performed in a computing system including one or more processors and one or more memories, and each color in a preset color space for each of a plurality of reference color areas in the reflectance information of the patch is used. A color parameter extraction step (S100) of extracting first color parameter information; Between a second color parameter derived by applying a color parameter conversion means including a plurality of correction values to a plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors A correction value determination step (S200) of determining the correction value so that the color difference value derived by the first color difference algorithm satisfies a preset standard; and a fourth color parameter derived by applying a color parameter conversion means including a plurality of correction values to the plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors. A correction value changing step (S300) of changing some or all of the correction values in a direction that reduces the color difference value derived by the second color difference algorithm between; If the correction value changing step (S300) is performed one or more times and meets a preset standard, a color parameter conversion means determining step (S400) of finally determining a color parameter conversion means with the updated correction value; And a colorimetric step (S500) of applying the color parameter conversion means to the color information of the measurement area in the reflectance information of the face to derive colorimetric color information of the measurement area corresponding to the actual color.

Preferably, the color difference value obtained by the first color difference algorithm is linear with respect to the color parameter. This is to determine the initial correction value.

For example, the linear color difference value for the color parameter can be expressed by the following equations.

The color difference value by the second color difference algorithm is derived in a different form from the color difference value by the first color difference algorithm. Preferably, the color difference value obtained by the second color difference algorithm is non-linear with respect to the color parameter.

For example, the color difference value in CIEDE94 can be defined as follows.

Here, the first color has color parameters of (L ^* ₁ , a ^* ₁ , b ^* ₁ ) in the Lab color system, and the second color has color parameters of (L ^* ₂ , a ^* ₂ , b ^* ₂ ). In this case, the color difference between the first color and the second color can be expressed as ΔE ^* ₉₄ above, and the color difference has a scalar value.

Here, the calculation method of each variable follows the following equations.

k _L , K ₁ , and K ₂ correspond to constant values according to the application field of the corresponding color difference, and may have the following values depending on the field. These values may vary depending on the intended use of the color difference, and as an example, may follow Table 1 below.

Preferably, the color difference value by the first color difference algorithm is determined by an equation including at least one value (nth element of the first color parameter - nth element of the second color parameter). When the color difference value is determined by an equation including the calculated value of (nth element of the first color parameter - nth element of the second color parameter), the color difference algorithm can be said to be linear.

Preferably, the color difference value according to the second color difference algorithm includes a color difference value according to the CIEDE2000 standard. In another embodiment of the present invention, the color difference equation based on the second color difference algorithm may correspond to a color difference value that can be calculated non-linearly.

The color difference value defined by CIEDE2000 can be defined by the following equation.

Here, the variables referred to the same way are the same as the variables in CIEDE76 and CIEDE94 described above.

Here, the calculation method of each variable follows the formulas below:

Preferably, the correction value changing step (S300) includes a fourth color parameter derived by applying a color parameter conversion means including a plurality of correction values to the plurality of first color parameter information and the plurality of reference colors. The sum of the color difference values derived by the second color difference algorithm between the preset third color parameters corresponding to the ground truth for is repeatedly performed to converge according to the preset standard.

After the sum of the color difference values converges according to the preset standard, the color parameter conversion means is determined in step S400, and when colorimetry is performed, the color parameters ( By applying color parameter conversion means to the color information of the measurement area in the reflectance information of the face, accurate colorimetric results (color information of the measurement area) can be derived.

The color parameter extraction unit 1100, correction value determination unit 1200, correction value change unit 1300, color parameter conversion means determination unit 1400, and colorimetry unit 1500 of FIG. 11(B) are each described above. Perform steps S100, S200, S300, S400, and S500. Redundant explanations regarding this will be omitted.

The correction value determination step (S200) is based on the second color parameter derived by applying a color parameter conversion means including a plurality of correction values to the plurality of first color parameter information and the ground truth for the plurality of reference colors. The correction value is determined so that the color difference value derived by the first color difference algorithm between the set third color parameters satisfies a preset standard.

Specifically, a plurality of reference colors are captured in the patch image of FIG. 12. The reference color may correspond to the reference color area displayed on the patch described with reference to FIG. 4.

Afterwards, the first color parameter is extracted for each reference color captured in the patch image. In this case, it is desirable that the position or arrangement of each reference color is recorded in the computing system, and after performing a preprocessing process to perform this, for example, cutting out only the patch area and performing resizing, the position of each reference color is After extracting the pixel information, a process such as converting it to a first color parameter (or the pixel information itself may be the first color parameter) is performed.

A first color parameter is extracted for each of the plurality of reference colors. In Figure 12, for convenience, the description is based on an example defined by three detailed color parameters based on the Lab color coordinate system.

Thereafter, for the first color parameter, the second color parameter can be derived using a color parameter conversion means including a plurality of correction values. The color parameter conversion means corresponds to a means of correcting when the color is photographed differently due to external conditions such as lighting, camera settings, camera performance, and surrounding environment, and in embodiments of the present invention, it is implemented by a plurality of correction values. It may be, and if the reference color is one, it may be the color difference value itself between the color information of the ground truth and the second color parameter.

As the simplest example of a correction value, if there are L, a, and b, the addition or subtraction value for L, the addition or subtraction value for a, or the addition or subtraction value for b may correspond to this, or L, a, and b may be converted into a vector string. The internal elements of the matrix that are handled and perform matrix operations on them may correspond to correction values.

Meanwhile, for the reference color, the computing system stores color parameters corresponding to the groundtruth of the reference color. In this way, the color parameter conversion means can be primarily determined by using a patch containing a preset reference color and setting the color parameter conversion means in a direction to minimize the color difference between the photographed value of the reference color and the true value.

Preferably, in the correction value determination step, the correction value may be determined to minimize the sum of color difference values derived by the first color difference algorithm for a plurality of reference colors.

Preferably, the first color difference algorithm corresponds to an algorithm that can find a correction value that minimizes the sum of color difference values using a linear algebraic method, and typically the variables are (nth element of the first color parameter - CIEDE76, which exists only in a formula of the nth element of a two-color parameter, may correspond to this.

In embodiments of the present invention, a color parameter conversion means for color correction is determined using information about a reference color. When using the Lab color coordinate system, colors can be expressed in three-dimensional space, and the color difference corresponds to the distance between two points that refer to two colors in three-dimensional space. Below, we will explain how to derive the correction value of the color parameter conversion means based on the Lab color coordinate system.

In the equation below

corresponds to information about the color in the captured image, that is, the first color parameter, and is a 3x3 matrix to correct values changed by external factors such as lighting and camera environment.

Multiply by the original color (

) to find out. Original color (

) may correspond to the third color parameter, which corresponds to the color parameter corresponding to the ground truth for the reference color. T may correspond to a color parameter conversion means.

In this way, if the groundtruth color parameter value for the photographed reference color is known, the following optimization problem can be solved.

This method can be solved when the color space and distance are well defined, for example, linearly.

As an example of the first color difference algorithm, in color spaces such as sRGB and CIE76, colors can be expressed as three-dimensional vectors in Euclidean space, and in this case, the optimization problem of T can be solved using general linear algebra. In this case, additional processes such as tone mapping or gamut mapping may be added.

In step S1000, information on T corresponding to the color parameter conversion means, that is, a correction value, is determined in the first color difference algorithm.

Meanwhile, in the case of the CIEDE 2000 method, which is known to be cognitively superior, the distance function corresponding to the color difference value cannot be expressed as a point distance in space, and is a complex function representing the distance between two points.

It can be obtained with In cases like this, it is difficult to solve using existing linear algebra methods.

In embodiments of the present invention, after determining the T value by the first color difference algorithm as described above,

Change the T value so that the value decreases. In step S1100, this process is performed, preferably based on partial differential information at the T value determined in the correction value determination step.

Determine the direction of change in T that can reduce .

That is, in step S1000, the first and third color parameters are mapped to a color space that can linearly obtain T, such as Lab (based on CIE76), and the initial value T is generated by solving a linear equation.

Afterwards, in step S1100, as the value of T (each of the plurality of correction values constituting T) changes in the Lab space, an error (error) in CIEDE2000 is generated.

) finds the change amount information of T that changes or decreases, and updates T (T+ΔT).

More specifically, in S1100 and S1200, because it is necessary to find changes in relative values given as distance values rather than absolute values in space, the local shape of the function defined based on the distance value is expressed by projecting it into Lab space. Optimization will be carried out based on this, but for new colors, another

Since is defined, it is updated repeatedly.

The color parameter conversion means is implemented as a matrix including a plurality of correction values, and the number of columns or rows of the matrix corresponds to the number of elements of the first color parameter.

In FIG. 14, T ₁₁ to T ₃₃ may correspond to correction values, and a matrix composed of these may correspond to a color parameter conversion means.

Meanwhile, the second color parameter is derived by matrix multiplying the matrix with respect to the first color parameter, and the fourth color parameter is derived by matrix multiplying the matrix with respect to the first color parameter.

In embodiments of the present invention, the color parameter conversion means may be multiplied on the right side of the first color parameter as shown in FIG. 14, but may also be defined as multiplied on the left side of the first color parameter as described above.

As shown in FIG. 15, the first color parameter defined as L, a, and b can be converted to L', a', and b' by a color parameter conversion means (e.g., matrix T).

Meanwhile, the third color parameter corresponding to the groundtruth for the reference color may be expressed as L ^GT , a ^GT , and b ^GT , and may be expressed as a point in space as shown in FIG. 10 . In this case, the spatial distance between the points corresponding to L', a', and b' and L ^GT , a ^GT , and b ^GT may correspond to an example of a color difference value derived by the first color difference algorithm, , the T value is found so that the sum of the distances for a plurality of reference colors is minimized.

Basically, the correction value determination step can operate in the same manner as above.

Meanwhile, in the correction value change stage, a color difference algorithm similar to human perception, for example, CIEDE2000, is used, so the color difference between two points in the L, a, and b coordinate system does not appear completely linearly as a straight line. Since the space of the coordinate system changes, the T value is changed to find and reduce the change behavior of the T value so that the sum of the color difference between the photographed value converted by the T of the plurality of reference colors and the ground truth is minimized. Change T to find the optimal T.

In Figure 16, similar to Figure 12, the first color parameter is derived for the reference color area captured from the patch image.

Similarly, after applying the color parameter conversion means to the first color parameter to derive the fourth color parameter, the distance between the color defined by the fourth color parameter and the color defined by the third color parameter is calculated. The correction value of the color parameter conversion means is changed so that the sum of color difference values by the second color difference algorithm, which is closer to human perception than the first color difference algorithm, is reduced.

Specifically, the correction value changing step defines the sum of the color difference values derived by the second color difference algorithm between the fourth color parameter and the third color parameter for the plurality of reference colors as an error function, and Some or all of the correction values are changed based on change information including the first partial derivative of the error function with respect to the correction value.

For example, when the color parameter conversion means is defined as a matrix as shown in FIG. 14, the first partial differential value is (differential value for T ₁₁ of the error function, differential value for T ₁₂ of the error function, T of the error function Differential value for ₁₃ , Differential value for T ₂₁ of the error function, Differential value for T ₂₂ of the error function, Differential value for T ₂₃ of the error function, Differential value for T ₃₁ of the error function, T of the error function It may correspond to a vector consisting of 9 values (differential value for ₃₂ , differential value for T ₃₃ of the error function).

As the simplest example, if the vector of partial differential values at the current T is (-10, 0, 0, 0, 0, 0, 0, 0, 0), the error function can be reduced by increasing the T11 value. It can be seen that this increase in T11 may correspond to change information.

Preferably, the correction value changing step defines the sum of color difference values derived by a second color difference algorithm between the fourth color parameter and the third color parameter for the plurality of reference colors as an error function, and , some or all of the correction values are changed based on change information including the second partial differential value of the error function with respect to the correction value.

The second partial differential value may correspond to a Hessian matrix, and as described above, when the color parameter conversion means corresponds to a 3X3 vector, the first partial differential value may be expressed as a 1X9 vector, and the second The partial differential value can be expressed as a 9X9 matrix.

As described above, the correction value changing step defines the sum of the color difference values derived by the second color difference algorithm between the fourth color parameter and the third color parameter for the plurality of reference colors as an error function. It can be done.

In step S2000, a change vector for the change direction of the entire correction value is determined based on the first partial differential value of the error function with respect to the correction value. In other words, the first partial differential value is the direction of change of T, for example, if the color parameter conversion means is defined in the form shown in FIG. 9, the direction of change for the 9 correction values of T (e.g. (1, 1, 0) If ,0,0,0,0,0,0) corresponds to the first partial differential value, only T11 and T11 are reduced to the same size, and the rest are kept as is) is determined.

For example, a change vector can also be expressed as a matrix with the same dimension as T.

After changing the size while maintaining the direction of the change vector based on the second partial differential value of the error function with respect to the correction value, some or all of the correction value is changed by applying the change vector.

In other words, the second-order partial derivative determines the size of the change vector. For example, in the above matrix, the matrix T value can operate in 9 coordinate systems, and the error function can be calculated in each T. Since the first partial derivative value can find the direction that can reduce the error, and the second partial derivative value can derive curvature information of the error function, the optimal moving distance of the change vector can be derived.

In other words, if the change vector extracted with the first partial differential value is ΔT1, a scalar value a can be derived from the second partial derivative value, and a* ΔT1 corresponds to the final change amount information of T.

Afterwards, T+ a*ΔT1 can become a new color parameter conversion means, and in this state, the correction value change step (S300) can be repeated. As repetition progresses, the change in the error function may decrease or decrease below a certain value. At this time, S400 is performed and the color parameter conversion means can be determined.

Preferably, the first partial differential value and the second partial differential value can be calculated numerically by approximating the nth order Taylor series.

Preferably, the error function may be defined as f(x), x may correspond to a matrix or vector such as T described above, and f(x) may correspond to a function that outputs a differentiable scalar value. You can.

In this case, the initial x value, that is, x ₀ , can be found through the correction value determination step (S200) described above, and while updating x by deriving the first and second partial differential information from x0, f(x) We can find x that can minimize .

p _k at step k corresponding to the direction of change that changes x (corresponding to the above-mentioned change vector) can be expressed as follows by the solution of the Newton equation.

here

may correspond to an approximation of the Hessian matrix corresponding to the second partial derivative, which may be updated at each step,

May correspond to the first partial differential value (gradient value) at x _k .

Line search in the p _k direction can be used to find x _k+1 corresponding to the next x value, which is

It can be implemented by finding the value of r that minimizes .

As such, in embodiments of the present invention, the optimal T can be found by performing the process of deriving the change value of T several times in the direction of reducing the value of the error function at the current T. In addition to the above-mentioned method, various known numerical calculations, such as the Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS) algorithm, can be used.

As shown in FIG. 18, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, and an input/output subsystem ( It may include at least an I/O subsystem (11400), a power circuit (11500), and a communication circuit (11600). At this time, the computing device 11000 may correspond to the computing device 1000 shown in FIG. 1.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. . The memory 11200 may include software modules, instruction sets, or other various data necessary for the operation of the computing device 11000.

At this time, access to the memory 11200 from other components such as the processor 11100 or the peripheral device interface 11300 may be controlled by the processor 11100.

The peripheral interface 11300 may couple input and/or output peripherals of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute a software module or set of instructions stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem can couple various input/output peripherals to the peripheral interface 11300. For example, the input/output subsystem may include a controller for coupling peripheral devices such as a monitor, keyboard, mouse, printer, or, if necessary, a touch screen or sensor to the peripheral device interface 11300. According to another aspect, input/output peripherals may be coupled to the peripheral interface 11300 without going through the input/output subsystem.

Power circuit 11500 may supply power to all or some of the terminal's components. For example, power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may contain arbitrary other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may include an RF circuit to transmit and receive RF signals, also known as electromagnetic signals, to enable communication with other computing devices.

This embodiment of FIG. 18 is only an example of the computing device 11000, and the computing device 11000 omits some components shown in FIG. 18, further includes additional components not shown in FIG. 18, or 2. It may have a configuration or arrangement that combines more than one component. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 18, and may include various communication methods (WiFi, 3G, LTE) in the communication circuit 11600. , Bluetooth, NFC, Zigbee, etc.) may also include a circuit for RF communication. Components that can be included in the computing device 11000 may be implemented as hardware, software, or a combination of both hardware and software, including an integrated circuit specialized for one or more signal processing or applications.

Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to this embodiment may be composed of a PC-based program or a mobile terminal-specific application. The application to which the present invention is applied can be installed on the computing device 11000 through a file provided by a file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request from the computing device 11000.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), etc. , may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used by any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed over networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

A colorimetric method performed in a computing system including one or more processors and one or more memories,

An image derivation step of deriving a face image for the face area and a patch image for the patch area from the original image;

The face image is input to an inference model including one or more artificial neural networks to derive face shape information and light information including the intensity and direction of light in the face image, and the face shape information for the face image and a first subtraction application step of applying the subtraction to the light information to derive reflectance information of the face that minimizes the effect of color change due to the shape of the face to which the light is irradiated.

A patch image containing one or more reference color areas is input into an inference model including one or more artificial neural networks, the shape information of the patch is derived, and the shape information of the patch and the light information are subtracted and applied to the patch image, A second subtraction application step of deriving reflectance information of the patch that minimizes the effect of color change due to the shape of the patch to which light is irradiated; and

Based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face is derived. A colorimetric method comprising: applying the color parameter conversion means to derive colorimetric color information of the measurement area.

In claim 1,

The shape information is,

Contains 3D information according to the actual shape derived from 2D information on the image about the object contained in the image,

The light information is,

Includes the intensity and direction of light irradiated to the object contained in the image,

The reflectance information is,

A colorimetric method that includes color information for each pixel in which light is irradiated to the shape of an object included in an image and the influence of shadows that appear on the shape depending on the intensity and direction of the light is removed.

In claim 1,

The patch is,

A colorimetric method that includes a plurality of reference color areas, where each reference color area has the same shape and area and is arranged separately from each other.

In claim 1,

The inference model includes a common model, a surface normal vector model, a light information inference model, and a reflectance model,

The common model is,

It includes one or more artificial neural networks, and derives feature information commonly input to the surface normal vector model, the light information inference model, and the reflectance model from the input image,

The surface normal vector model is,

It includes one or more artificial neural networks, and derives the shape information, which is 3D structure information containing the actual shape, from the image, which is 2D information, from the feature information,

The light information inference model is,

Includes one or more artificial neural networks, and derives the light information including the intensity and direction of light in the image from the feature information,

The reflectance model is,

A colorimetric method comprising one or more artificial neural networks and deriving reflectance information, which is color information for each pixel, from the feature information.

In claim 1,

The first subtraction application step is,

A facial feature information deriving step of deriving facial feature information by inputting a facial image into a common model including one or more artificial neural networks;

A facial shape derivation step of deriving facial shape information by inputting the facial feature information into a surface normal vector model including one or more artificial neural networks;

A light information derivation step of inputting the facial feature information into a light information inference model including one or more artificial neural networks to derive light information including the intensity and direction of light in the face image; and

A facial reflectance derivation step of applying the shape information and light information of the face to the face image to derive reflectance information of the face in which only the color of the face is reflected,

The second subtraction application step is,

A patch feature information deriving step of deriving patch feature information by inputting the patch image into a common model including one or more artificial neural networks;

A patch shape derivation step of deriving shape information of the patch by inputting the patch feature information into a surface normal vector model including one or more artificial neural networks;

A patch reflectance derivation step of applying the shape information of the patch and the light information to the patch image to derive reflectance information of the patch reflecting only the color of the patch.

In claim 1,

The colorimetric method further includes a model learning step,

The model learning step is,

A learning shape information deriving step of inputting a learning image into the common model and inputting learning feature information of the common model into the surface normal vector model to derive learning shape information;

A learning light information deriving step of inputting the learning feature information into the light information inference model to derive learning light information;

A learning reflectance information deriving step of deriving learning reflectance information by inputting the learning feature information into a reflectance model that derives reflectance information; and

A predicted learning image derivation step of deriving a predicted learning image by combining the learning shape information, learning light information, and learning reflectance information, wherein the common model is used to reduce or minimize the difference between the predicted learning image and the learning image. , A colorimetric method for learning detailed parameter values or filter information of the surface normal vector model, the light information inference model, and the reflectance model.

In claim 1,

The face color derivation step includes a conversion means derivation step,

The conversion means derivation step is,

A color parameter extraction step of extracting first color parameter information in a preset color space for each of a plurality of reference color areas in the reflectance information of the patch;

Between a second color parameter derived by applying a color parameter conversion means including a plurality of correction values to a plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors A correction value determination step of determining the correction value so that the color difference value derived by the first color difference algorithm satisfies a preset standard; and

Between a fourth color parameter derived by applying a color parameter conversion means including a plurality of correction values to a plurality of first color parameter information and a preset third color parameter corresponding to the ground truth for the plurality of reference colors. A colorimetric method including; a correction value changing step of changing some or all of the correction values in a direction that reduces the color difference value derived by the second color difference algorithm of .

In claim 7,

The color difference value by the first color difference algorithm is linear with respect to the color parameter, and the color difference value by the second color difference algorithm is non-linear with respect to the color parameter. The algorithms are different from each other,

The color parameter conversion means is implemented as a matrix including a plurality of correction values, and the number of columns or rows of the matrix corresponds to the number of elements of the first color parameter,

The second color parameter is derived by matrix multiplying the matrix with respect to the first color parameter, and the fourth color parameter is derived by matrix multiplying the matrix with respect to the first color parameter.

A colorimetric device implemented in a computing system including one or more processors and one or more memories,

The colorimetric device is,

Based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face is derived. A colorimetric device that performs a facial color derivation step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means.

1. A computer program stored on a computer-readable medium comprising a plurality of instructions executed by one or more processors, comprising:

The computer program is,

Based on the difference between the color information of the reference color area in the reflectance information of the patch and the color information of the ground truth of the reference color, a color parameter conversion means is derived, and the color information of the measurement area in the reflectance information of the face is derived. A computer program comprising; a facial color derivation step of deriving colorimetric color information of the measurement area by applying the color parameter conversion means.