JP4682373B2 - Face image synthesis device, face image synthesis method, and face image synthesis program - Google Patents

Description

  The present invention relates to a face image composition device, a face image composition method, and a face image composition program.

  2. Description of the Related Art Conventionally, a face image synthesizing apparatus and method that can change the age impression (aging process) of a face image by manipulating blotches and wrinkles in the face image have been proposed (for example, Patent Document 1). reference).

  In such a face image synthesizing apparatus, for example, face images having different age impressions are synthesized by manipulating the shapes of face parts (face parts) such as eyes, nose, and mouth.

Conventionally, as a first method, a computer graphics (CG) stain and wrinkle are combined with a face image, and as a second method, a stain and wrinkle are extracted from a face image having spots and wrinkles, and the extracted blot and wrinkle are extracted. There is a way to operate.
Japanese Patent Laid-Open No. 2005-92349

  However, when the spots and wrinkles by conventional computer graphics (CG) are combined with the face image, unnaturalness remains in the combined face image.

  It is also very difficult to extract blots and wrinkles from young people who have few spots and wrinkles. As a result, the aging process for young people cannot be performed.

  An object of the present invention is to provide a face image composition device, a face image composition method, and a face image composition program capable of performing age manipulation on a face image of a young person and leaving no unnaturalness in the face image after age manipulation. It is to be.

A face image synthesizer according to a first aspect is a face image synthesizer for synthesizing a face image of a specified age or age group from a face image of a subject, and storing means for storing a plurality of face images in advance. Detecting means for detecting spots or wrinkles of a plurality of persons from the face images of the plurality of persons stored in the storage means, and an average face shape corresponding to the age or age group from the plurality of face images stored in the storage means A generation means for generating a certain age shape feature, a creation means for creating a blot model or a wrinkle model corresponding to the age or age group from the spots or wrinkles of a plurality of persons detected by the detection means , and a blot from the face image of the subject or an extraction means for extracting a no face shape wrinkles, extracted by the extraction means based on the corresponding age shape features on the age or age group that is specified among the generated by the generating means age shape feature And deforming means for deforming the subject's face shape that is, the stain model or wrinkle model corresponding to the age or age group that is specified among the stain model or wrinkle model corresponding to the age or age group that is created by the creation means A face image corresponding to a specified age or age group by applying the deformed blot model or wrinkle model to the face image of the subject after being deformed to match the face shape of the subject deformed by the deforming means. And applying means for synthesizing , and the creating means extracts the number or area of stains from a plurality of persons for each area divided into a plurality of face images, and calculates the number or area of stains for each extracted area for each area. When the images are arranged in ascending or descending order, the face image area having the number or area of the blot located at the approximate center is selected, and the spot pattern obtained by combining the selected area's blots is selected. The wrinkle component of the face image of multiple people is synthesized with the age shape feature generated by the generating means, the wrinkle component position is shifted so that the overlapping degree of the wrinkle component is the largest, and the wrinkle after the shift is The luminance of the component is integrated, the luminance of the integrated wrinkle component is divided by the number of people, and an area where the obtained luminance exceeds the reference value is set as a wrinkle of the wrinkle model .

In the face image synthesizer according to the present invention, the face images of a plurality of persons are stored in advance by the storage means, and the spots or wrinkles of the persons are detected from these face images by the detection means. Further , an age shape feature that is an average face shape corresponding to the age or age group is generated from the face images of a plurality of persons stored in the storage means by the generation means. Further, a blot model or a wrinkle model corresponding to the age or age group is created from the blots or wrinkles of a plurality of persons detected by the detection means.

On the other hand, a face shape having no blotches or wrinkles is extracted from the face image of the subject by the extracting means. Further, the face shape of the subject extracted by the extraction unit based on the age shape feature corresponding to the specified age or age group among the age shape features generated by the generation unit is deformed by the deformation unit. Thereby, the age of the face shape of the subject can be manipulated to a face shape corresponding to the specified age or age group. In addition, the stain model or the wrinkle model corresponding to the specified age or age group among the stain model or the wrinkle model created by the creation means has been deformed to match the face shape of the subject deformed by the deformation means. Thereafter, the deformed blot model or wrinkle model is applied to the face image of the subject by the applying means. Thereby, the face image of the subject corresponding to the designated age or age group is synthesized.

  In this way, the blot model or wrinkle model corresponding to the specified age or age group is applied to the face image of the subject, so that the conventional computer graphics (CG) blot and wrinkle model is converted into the face image. It is possible to prevent the occurrence of unnaturalness remaining in the face image after the synthesis as in the case of the synthesis. Thereby, the age of the face image of the subject can be manipulated to a natural face image corresponding to a desired age or age group.

In addition, by using a blot model or a wrinkle model created by the creating means, it is possible to easily perform an age operation on a face image of a young person who has almost no blots and wrinkles.
The location of the stain varies from person to person, but the number and area of the stain is common to people of the same age or age group. Therefore, for each area divided into a plurality of face images, a target area having a standard number or area of spots that is located approximately in the center of each area is selected, and the spots of the selected areas are combined. The reliability of the stain model can be improved by using the processed image as the stain model. Further, by using a common area among wrinkles of a plurality of people as a wrinkle model, a natural face image corresponding to a desired age or age group can be obtained.

  The detection unit may extract a small blob in each face image stored in the storage unit based on a difference between the luminance of each pixel and the luminance of the surrounding area, and detect a blot or a wrinkle from the extracted small blob. .

  In this case, in each face image stored in the storage unit, a small lump is detected by the detecting unit based on the difference between the luminance of each pixel and the luminance of the surrounding area, and a spot or a wrinkle is detected by the detecting unit. Detected. Here, the blob means an area where pixels having lower or higher luminance than the luminance of the peripheral area in the face image are connected.

  Since the stain is a portion having a different color on the skin surface, a small lump corresponding to the stain can be detected based on the difference between the luminance of each pixel and the luminance of the surrounding area. Further, since the wrinkle is a shadow formed by the unevenness of the skin surface, a small lump corresponding to the wrinkle can be detected based on the difference between the luminance of each pixel and the luminance of the surrounding area. By using the small blob thus detected, it is possible to easily detect spots and wrinkles.

A face image composition method according to a second aspect of the present invention is a face image composition method for compositing a face image of a specified age or age group from a face image of a subject, and storing a plurality of face images in advance. Detecting a blot or wrinkle of a plurality of persons from the stored face images of the plurality of persons, and generating an age shape feature that is an average face shape corresponding to the age or age group from the stored face images of the plurality of persons Creating a blot model or wrinkle model corresponding to the age or age group from the detected blots or wrinkles of a plurality of people, extracting a face shape having no blots or wrinkles from the face image of the subject, a step of deforming the generated subject's face shape extracted based on corresponding age shape features to the given age or age group of age-shaped features, age created or After deformed to match the face shape of the subject has been modified spots model or wrinkle model corresponding to the specified age or age of the stain model or wrinkle model corresponding to the age group, stain model after deformation Alternatively, the step of synthesizing the face image corresponding to the specified age or age group by applying the wrinkle model to the face image of the target person and the step of creating a plurality of people for each area divided into the plurality of face images When the number or area of spots is extracted, and the number or area of blots for each extracted area is arranged in ascending or descending order for each area, the area of the face image that has the number or area of blots located approximately in the center Is created by combining the selected region's blots as a blot model, and the wrinkle components of the face images of multiple people are combined with the generated age shape features, and the wrinkle component overlaps. The position of the wrinkle component is shifted to maximize the degree of wrinkle, the luminance of the wrinkle component after the shift is integrated, the luminance of the integrated wrinkle component is divided by the number of people, and the area where the obtained luminance exceeds the reference value It includes setting as a wrinkle of a wrinkle model .

In the face image composition method according to the present invention, face images of a plurality of people are stored in advance, and spots or wrinkles of the plurality of people are detected from these face images. Also, an age shape feature that is an average face shape corresponding to the age or age group is generated from the stored face images of a plurality of persons. Further, a blot model or wrinkle model corresponding to the age or age group is created from the detected blots or wrinkles of a plurality of persons.

On the other hand, a face shape having no spots or wrinkles is extracted from the face image of the subject. Further, the face shape of the subject extracted based on the age shape feature corresponding to the specified age or age group among the generated age shape features is transformed. Thereby, the age of the face shape of the subject can be manipulated to a face shape corresponding to the specified age or age group. In addition, after the blot model or wrinkle model corresponding to the specified age or age group of the created blot model or wrinkle model is deformed to match the deformed subject's face shape, the deformed blot model The model or wrinkle model is applied to the face image of the subject. Thereby, the face image of the subject corresponding to the designated age or age group is synthesized.

  In this way, the blot model or wrinkle model corresponding to the specified age or age group is applied to the face image of the subject, so that the conventional computer graphics (CG) blot and wrinkle model is converted into the face image. It is possible to prevent the occurrence of unnaturalness remaining in the face image after the synthesis as in the case of the synthesis. Thereby, the age of the face image of the subject can be manipulated to a natural face image corresponding to a desired age or age group.

In addition, by using the created blot model or wrinkle model, it is possible to easily perform an age operation on a face image of a young person who has few spots and wrinkles.
The location of the stain varies from person to person, but the number and area of the stain is common to people of the same age or age group. Therefore, for each area divided into a plurality of face images, a target area having a standard number or area of spots that is located approximately in the center of each area is selected, and the spots of the selected areas are combined. The reliability of the stain model can be improved by using the processed image as the stain model. Further, by using a common area among wrinkles of a plurality of people as a wrinkle model, a natural face image corresponding to a desired age or age group can be obtained.

A face image composition program according to a third aspect of the present invention is a face image composition program that can be executed by a computer and synthesizes a face image of a specified age or age group from a face image of a target person. Process for storing images, process for detecting spots or wrinkles of multiple persons from stored face images of multiple persons, and age that is an average face shape corresponding to age or age group from stored multiple face images Processing to generate shape features, processing to create a blot model or wrinkle model corresponding to the age or age group from the detected blots or wrinkles of multiple people, and facial shape that does not have blots or wrinkles from the face image of the subject a process for extracting, and processing for deforming the subject's face shape extracted based on corresponding age shape features to the given age or age of the generated age shape feature After deformed to match the face shape of the subject has been modified spots model or wrinkle model corresponding to the specified age or age of the stain model or wrinkle model corresponding to the age or age group has been created, modified The process of creating a face image corresponding to the specified age or age group by applying a later spot model or wrinkle model to the face image of the subject is executed by the computer. When the number or area of blots is extracted from multiple people for each divided area, and the number or area of the spots for each extracted area is arranged in ascending or descending order for each area, the blot is located at the approximate center. A face image area having the number or area of the selected face image is selected, and a combination of the selected area's blots is created as a blot model to generate wrinkle components for multiple face images. Synthesize the age shape feature, shift the position of the wrinkle component so that the overlapping degree of the wrinkle component is the largest, add the luminance of the wrinkle component after the shift, divide the luminance of the integrated wrinkle component by the number of people Including setting a region in which the luminance obtained exceeds a reference value as a wrinkle of the wrinkle model .

In the face image composition program according to the present invention, face images of a plurality of people are stored in advance, and spots or wrinkles of the plurality of people are detected from these face images. Also, an age shape feature that is an average face shape corresponding to the age or age group is generated from the stored face images of a plurality of persons. Further, a blot model or wrinkle model corresponding to the age or age group is created from the detected blots or wrinkles of a plurality of persons.

On the other hand, a face shape having no spots or wrinkles is extracted from the face image of the subject. Further, the face shape of the subject extracted based on the age shape feature corresponding to the specified age or age group among the generated age shape features is transformed. Thereby, the age of the face shape of the subject can be manipulated to a face shape corresponding to the specified age or age group. In addition, after the blot model or wrinkle model corresponding to the specified age or age group of the created blot model or wrinkle model is deformed to match the deformed subject's face shape, the deformed blot model The model or wrinkle model is applied to the face image of the subject. Thereby, the face image of the subject corresponding to the designated age or age group is synthesized.

  In this way, the blot model or wrinkle model corresponding to the specified age or age group is applied to the face image of the subject, so that the conventional computer graphics (CG) blot and wrinkle model is converted into the face image. It is possible to prevent the occurrence of unnaturalness remaining in the face image after the synthesis as in the case of the synthesis. Thereby, the age of the face image of the subject can be manipulated to a natural face image corresponding to a desired age or age group.

In addition, by using the created blot model or wrinkle model, it is possible to easily perform an age operation on a face image of a young person who has few spots and wrinkles.
The location of the stain varies from person to person, but the number and area of the stain is common to people of the same age or age group. Therefore, for each area divided into a plurality of face images, a target area having a standard number or area of spots that is located approximately in the center of each area is selected, and the spots of the selected areas are combined. The reliability of the stain model can be improved by using the processed image as the stain model. Further, by using a common area among wrinkles of a plurality of people as a wrinkle model, a natural face image corresponding to a desired age or age group can be obtained.

  ADVANTAGE OF THE INVENTION According to this invention, generation | occurrence | production of the unnaturalness which remains in the face image after a synthesis | combination like the case where the model of a blot and a wrinkle by computer graphics (CG) is synthesized with a face image can be prevented. Thereby, the age of the face image of the subject can be manipulated to a natural face image corresponding to a desired age or age group. In addition, it is possible to easily perform an age operation on a face image of a young person who has few spots and wrinkles.

(1) Basic idea First, the basic idea of the face image composition device and the face image composition method according to the present invention will be described.

  Blots and wrinkles are accumulated skin that changes due to various effects from the outside world in the process of human life. Stain is a change in tissue within the skin and pigmentation. On the other hand, wrinkles are grooves on the skin that are fixed by repeatedly bending the skin.

  For this reason, the blot is not a shadow formed by the unevenness of the skin, but exists as a portion having a different color on the skin surface on the image. Since wrinkles are grooves on the skin, they are shadows formed by irregularities on the skin surface on the image.

(2) Configuration of Face Image Synthesis Device Next, the configuration of the face image synthesis device for executing the face image synthesis method of the present embodiment will be described using the block diagram of FIG.

  The image processing apparatus 50 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, an input device 504, a display device 505, an external storage device 506, a recording medium driving device 507, and the like. A printing device 508.

  The input device 504 includes a keyboard, a mouse, a scanner, a digital camera, and the like, and is used for inputting various commands, data, and images. The ROM 502 stores a system program. The recording medium driving device 507 includes a CD (compact disk) drive, a DVD (digital versatile disk) drive, an FD (flexible disk) drive, and the like, and reads / writes data from / to a recording medium 509 such as a CD, DVD, or FD. .

  In the recording medium 509, a face image synthesis program is recorded. This face image synthesis program includes various image processing programs.

  The external storage device 506 includes a hard disk device or the like, and stores a face image synthesis program and various data read from the recording medium 509 via the recording medium driving device 507. The CPU 501 executes the face image synthesis program stored in the external storage device 506 on the RAM 503.

  The display device 505 includes a liquid crystal display panel, a CRT (cathode ray tube), and the like, and displays various images. The printing device 508 prints various images.

  As the recording medium 509 for recording the face image synthesis program, various recording media such as a semiconductor memory such as a ROM and a hard disk can be used. Further, the face image synthesis program may be downloaded to the external storage device 506 via a communication medium such as a communication line and executed on the RAM 503.

(3) Blemish / wrinkle model creation process The blot / wrinkle model creation process performed by the face image synthesizing apparatus of FIG. 1 will be described below with reference to the flowchart of FIG.

  Here, a process of creating a blot / wrinkle model for each age or each age group based on the face images of a plurality of subjects in each age or each age group will be described. In this case, it is assumed that the face image (hereinafter referred to as an original image) of the subject obtained by a digital camera or the like is stored in the external storage device 506 as image data.

  The image data includes red image data, green image data, and blue image data. Spots and wrinkles, which will be described later, are detected using the green image data. In this case, since the red image data is close to the skin color, it is possible to accurately detect the spots and wrinkles by using the green image data different from the skin color.

  Further, it is assumed that position information indicating the position of face parts (eyes, nose, mouth, etc.) in the original image is acquired in advance and stored in the external storage device 506. The position information of the face part may be acquired by a human input operation or may be automatically acquired from the original image.

  First, the CPU 501 acquires the number of subjects M and the age N (step S1). Note that the CPU 501 may acquire the age group of the target person instead of the age N. For example, when the acquired age group is in the 20s, the age of the subject is limited to 20 to 29 years.

  Next, the CPU 501 acquires an original image from the external storage device 506 (step S2). The acquired original image is stored in the first buffer area of the RAM 503.

  Subsequently, the CPU 501 detects a bright area from the acquired original image, and creates a binarized bright area image (step S3). Here, the bright region refers to a region having a luminance higher than the average luminance in the original image. Details of the bright area image creation processing will be described later. The bright area image is stored as image data in the second buffer area of the RAM 503.

  Further, the CPU 501 detects a dark area from the original image and creates a binarized dark area image (step S4). Here, the dark region refers to a region having luminance lower than average luminance in the original image. Details of the dark region image creation processing will be described later. The dark area image is stored as image data in the third buffer area of the RAM 503.

  Next, the CPU 501 performs blob (small blob) extraction and labeling in each of the bright area image and the dark area image (step S5). Note that a blob is a small block area in which pixels having a value of “1” are connected in a bright area image or a dark area image. Here, when there is a pixel having a value of “1” around the target pixel having a value of “1” (eight neighbors), the peripheral pixel having the value of “1” and the target pixel are regarded as a connected region. . A label is attached to each of the plurality of blobs (labeling). Details of blob extraction and labeling will be described later. The labeled bright area image and dark area image are stored as image data in the fourth and fifth buffer areas of the RAM 503, respectively.

  Next, a stain and a wrinkle component are detected, and a stain component and a wrinkle component are generated (step S6). In this case, blots and wrinkles can be distinguished from other regions (for example, face parts) from the shape and size of the blob. For example, the blot has a shape that is small compared to the size of the face and close to a circle or slightly distorted. Wrinkles have a linear or elongated shape but do not become significantly longer compared to the size of the face. A blot component is generated based on the blob detected as a blot, and a wrinkle component is generated based on the blob detected as the wrinkle. Details of the detection of blots and wrinkles and the generation process of blot components and wrinkle components will be described later. The spot component and the wrinkle component are stored as image data in the sixth and seventh buffer areas of the RAM 503, respectively.

  Subsequently, the CPU 501 extracts the face shape of the subject (step S7). Then, the CPU 501 subtracts 1 from the number M of subjects (step S8), and then determines whether the number M has become 0 (step S9). When the number of people M is not 0, the CPU 501 returns to the process of step S2 and repeats the processes of steps S2 to S8.

  In the process of step S9, when the number of persons M becomes 0, that is, when the processes of steps S2 to S8 are completed for all the target persons, the CPU 501 determines the age shape characteristics (average face of these target persons). Shape) is created (step S10).

  Next, the CPU 501 performs a wrinkle model creation process (step S11) and a spot model creation process (step S12). When the blot model creation process ends, the spot / wrinkle model creation process ends.

(4) Wrinkle Model Creation Process FIG. 3 is a flowchart showing the wrinkle model creation process. The wrinkle component used in the wrinkle model creation process is a multivalued image having a plurality of gradations.

  As shown in FIG. 3, first, the CPU 501 synthesizes the wrinkle component of each subject with the age shape feature created in the process of step S10 (step S21).

  Next, the CPU 501 detects the maximum value of the matching coefficient P (step S22). Here, the matching coefficient P will be described. FIG. 4 is an explanatory diagram for explaining the matching coefficient P. In FIGS. 4A and 4B and FIG. 4C described later, the age shape feature is not shown.

  The wrinkle component of one subject shown in FIG. 4A is synthesized with the age shape feature (step S21 described above). Let A be the area of the wrinkle component of this one subject.

  Similarly, the wrinkle component of the other subject shown in FIG. 4B is synthesized with the age shape feature in which the wrinkle component of one subject is synthesized (step S21 described above). Let B be the area of the wrinkle component of the other subject.

  Here, assuming that the wrinkle components of the one and the other subjects are at substantially the same position of the age shape feature, the wrinkle component of the other subject is combined with the age shape feature obtained by combining the wrinkle components of one subject. As shown in FIG. 4C, the wrinkle component of one subject having area A and the wrinkle component of the other subject having area B overlap. The area of this overlapping wrinkle component (hereinafter referred to as overlapping wrinkle component: black region in FIG. 4C) is C.

  Using the area C of the overlapping wrinkle component calculated in this way, the CPU 501 calculates the matching coefficient P based on the following equation.

P = C / A + C / B (1)
Next, the CPU 501 detects the position where the degree of overlap is greatest by shifting the position of the wrinkle component of one subject or the wrinkle component of the other subject, and uses the matching coefficient P at that position as the matching coefficient P. The maximum value of the coefficient P is detected (step S22 described above). Note that the process of shifting the position of the wrinkle component is performed, for example, on an area of 9 pixels × 9 pixels.

  Next, the CPU 501 determines whether or not the maximum value of the matching coefficient P is a predetermined value (for example, 0.5) or more (step S23 in FIG. 3). When the maximum value of the matching coefficient P is equal to or greater than the predetermined value, the CPU 501 integrates the luminance of the wrinkle component after shifting (step S24).

  Next, after the processing of step S24 and when the maximum value of the matching coefficient P is less than a predetermined value in step S23, the CPU 501 determines whether or not the processing has been completed for all wrinkle components (step). S25). When the process has been completed for all the wrinkle components, the CPU 501 determines whether or not the processes for all the subjects have been completed (step S26).

  When the processing of all the subjects is completed in the processing of step S26, the CPU 501 divides the integrated luminance of the wrinkle component by the number of subjects (step S27), and then a predetermined reference value (for example, the luminance is 3) The following wrinkle component area is erased (step S28). Thereby, a region where the luminance exceeds the reference value is set as a wrinkle component of the wrinkle model.

  On the other hand, when the process has not been completed for all the wrinkle components in the process of step S25, the CPU 501 returns to the process of step S22 and repeats the processes of steps S22 to S25.

  If the processes of all the subjects have not been completed in the process of step S26, the CPU 501 returns to the process of step S22 and repeats the processes of steps S22 to S26. Thereby, the brightness | luminance of a wrinkle component of a some subject is integrated | accumulated in the state which makes the matching coefficient P the maximum.

  Next, after the process of step S28, the CPU 501 stores the wrinkle model created as described above in the external storage device 506 (step S29). In the wrinkle model creation process, a region that does not exceed the reference value among regions other than the overlapping wrinkle component region may be set as a part of the wrinkle component of the wrinkle model.

(5) Blot Model Creation Processing Hereinafter, the blot model creation processing (step S12) will be described with reference to the drawings.

  FIG. 5 is a schematic diagram in which the original image of the subject person is simplified.

  The original image of the subject in FIG. 5 is divided into a plurality of regions. For example, a range from the tip of the head to the center of the eye is a region F1, a range from the center of the eye to the center of the mouth is a region F2, and a range from the center of the mouth to the tip of the chin is a region F3.

  For example, when there are the subject person A, the subject person B, the subject person C, the subject person D, and the subject person E, the number of spots of the subject person A in the area F1 is 12, and the number of spots of the subject person B is 20 The number of spots on subject C is 5, the number of spots on subject D is 10, and the number of spots on subject E is 8. In the area F1, when the number of blots of the subject persons A to E is arranged in the order of increasing (or decreasing order), the target person D having 10 spots is located in the middle.

  Based on such a result, the stain of the subject D is set as the stain component of the stain model in the region F1. Similarly, in the region F2 and the region F3, when the number of blots of each subject is arranged in order of increasing (or decreasing order), the blot of the subject positioned in the middle is set as a blot component of the blot model, respectively. . In the above example, the stain model is set based on the number of stains. However, the present invention is not limited to this, and the stain model may be set based on the stain area (total area).

  The location of the stain varies from person to person, but the number and area of the stain is common to people of the same age or age group. Therefore, by using the method of the present embodiment, a blot model with improved reliability can be created.

  In the above example, the number of subjects is five (subjects A to E) for simplification of explanation, but in reality, it is preferable to have 100 or more subjects as subjects. Further, when the number of subjects is an even number, blots of a plurality of people (for example, two people) as the subject located in the middle may be set as the stain component of the stain model.

  The wrinkle model and the spot model created by the blot / wrinkle model creation process are shown in FIGS. 6 (a) and 6 (b), respectively.

(6) Model application process Next, a model application process for applying the spot model and the wrinkle model to an original image of a certain subject will be described.

  FIG. 7 is a flowchart showing the model application process.

  As shown in FIG. 7, the CPU 501 acquires the original image of the subject (step S31). In this example, the CPU 501 acquires an original image of a 20-year-old woman shown in FIG.

  Next, the CPU 501 acquires the age after the age operation (step S32). In this case, when the operator operates the input device 504 of the image processing apparatus 50, the age after the age operation is determined. For example, the worker can select 40s, 50s, 60s and 70s as the age after the age operation.

  Subsequently, the CPU 501 selects a corresponding age shape feature, a spot model, and a wrinkle model (step S33), and then deforms the face shape of the original image based on the age shape feature (step S34).

  Next, the CPU 501 transforms the blot model and the wrinkle model so as to match the deformed face shape (step S35).

  Thereafter, the CPU 501 applies the blot model and wrinkle model transformed into the original image (step S36).

  FIG. 8B shows the result of applying the blot model and wrinkle model in the 40s to the original image, and FIG. 8C shows the result of applying the blot model and wrinkle model in the 70s to the original image.

(7) Effects in the present embodiment As described above, in the present embodiment, a stain model and a wrinkle model are created by the stain / wrinkle model creation process, and these are applied to the original image of the subject, so that the computer It is possible to prevent the occurrence of unnaturalness remaining in the face image after synthesis, such as when a blot and wrinkle by graphics (CG) is synthesized with the face image. Thereby, the reliability in the result after age operation can be improved.

  Further, by using the blot model and the wrinkle model created by the blot / wrinkle model creation process, it is possible to easily perform the age operation even on the original image of a young person who has almost no blots and wrinkles.

(8) Creation of Bright Area Image and Dark Area Image Next, creation of a bright area image (step S3) and creation of a dark area image (step S4) in the blot / wrinkle model creation process of FIG. 2 will be described.

  FIG. 9 is a diagram illustrating an example of a target pixel and peripheral pixels. 10 and 11 are flowcharts showing a bright area image creation process. Further, FIG. 12 and FIG. 13 are flowcharts showing a dark region image creation process.

  As shown in FIG. 9, the horizontal direction of the original image 100 is defined as the X direction, and the vertical direction is defined as the Y direction. In the example of FIG. 9, the number of pixels in the X direction of the original image 100 is 512, and the number of pixels in the Y direction is 512. A region of 10 pixels × 10 pixels around the target pixel 300 is referred to as a peripheral region 200.

  In the bright area image creation process and the dark area image creation process, the value of the target pixel 300 is determined based on the average value of the luminance in the peripheral area 200 while shifting the target pixel 300 pixel by pixel.

  In the bright area image creation processing of FIGS. 10 and 11, first, the CPU 501 sets the target pixel 300 and the peripheral area 200 (step S41).

  Next, the average value of the luminance of the pixels in the peripheral area 200 is calculated (step S42). Then, the difference between the luminance value and the average value of the target pixel 300 is calculated (step S43).

  Next, it is determined whether or not the difference between the luminance value and the average value of the pixel of interest 300 is a positive predetermined value or more (step S44).

  If the difference between the luminance value and the average value of the target pixel 300 is a positive predetermined value or more, “1” is set as the pixel value of the target pixel 300 (step S45). If the difference between the luminance value and the average value of the target pixel 300 is lower than the positive predetermined value, “0” is set as the pixel value of the target pixel 300 (step S46).

  Thereafter, it is determined whether or not the target pixel 300 is the last pixel in the X direction (step S47 in FIG. 11). If the target pixel 300 is not the last pixel in the X direction, the target pixel 300 and the peripheral region 200 are shifted by one pixel in the X direction (step S48). And it returns to step S42 and repeats the process of steps S42-S47.

  If the target pixel 300 is the last pixel in the X direction in step S47, it is determined whether or not the target pixel 300 is the last pixel in the Y direction (step S49). If the target pixel 300 is not the last pixel in the Y direction, the target pixel 300 is returned to the first pixel in the X direction, and the target pixel 300 and the peripheral region 200 are shifted by one pixel in the Y direction (step S50). And it returns to step S42 and repeats the process of steps S42-S49.

  If the target pixel 300 is the last pixel in the Y direction in step S49, the bright area image creation process ends.

  In this way, a bright area image indicating a bright area having a luminance higher than the average luminance in the original image 100 is created.

  Next, in the dark region image creation processing of FIGS. 12 and 13, first, the CPU 501 sets the target pixel 300 and the peripheral region 200 (step S51).

  Next, the average value of the luminance of the pixels in the peripheral area 200 is calculated (step S52). Then, the difference between the luminance value and the average value of the target pixel 300 is calculated (step S53).

  Next, it is determined whether or not the difference between the luminance value and the average value of the target pixel 300 is equal to or less than a predetermined negative value (step S54).

  If the difference between the luminance value and the average value of the target pixel 300 is equal to or less than a predetermined negative value, “1” is set as the pixel value of the target pixel 300 (step S55). If the difference between the luminance value of the target pixel 300 and the average value is higher than the negative predetermined value, “0” is set as the pixel value of the target pixel 300 (step S56).

  Thereafter, it is determined whether or not the target pixel 300 is the last pixel in the X direction (step S57 in FIG. 13). If the target pixel 300 is not the last pixel in the X direction, the target pixel 300 and the peripheral region 200 are shifted by one pixel in the X direction (step S58). And it returns to step S52 and repeats the process of steps S52-S57.

  If the target pixel 300 is the last pixel in the X direction in step S57, it is determined whether or not the target pixel 300 is the last pixel in the Y direction (step S59). If the target pixel 300 is not the last pixel in the Y direction, the target pixel 300 is returned to the first pixel in the X direction, and the target pixel 300 and the peripheral region 200 are shifted by one pixel in the Y direction (step S60). And it returns to step S52 and repeats the process of steps S52-S59.

  If the target pixel 300 is the last pixel in the Y direction in step S59, the dark region image creation process ends.

  In this way, a dark area image indicating a dark area having a luminance lower than the average luminance in the original image 100 is created.

  FIG. 14 is a diagram illustrating an example of an original image, and FIG. 15 is a diagram illustrating an example of a bright area image. The bright area image 600 in FIG. 15 is created from the original image 100 in FIG. 14 by the above-described bright area image creation processing. In addition, each number in FIG. 14 shows a brightness | luminance.

(9) Blob Extraction and Labeling Next, blob extraction and labeling will be described. 16 and 17 are diagrams for explaining blob extraction and labeling.

  Here, blob extraction refers to extracting a connected area of pixels having a value of “1” in a bright area image or a dark area image as a blob. Labeling refers to labeling each blob in order to identify each blob.

  In the bright area image 600 shown in FIG. 15, is there a pixel with the value “1” on the left side, diagonally upper left, upper side, or diagonally right of the target pixel while shifting the target pixel by one pixel in the X direction? Check for no. When the value of the target pixel first becomes “1”, the target pixel is labeled “A”. Next, when the value of the pixel of interest is “1” and there is a pixel having a value of “1” on the left side, diagonally upper left, upper side or diagonally upper right, The same label “A” is attached. When the value of the pixel of interest is “1” and there is no pixel with the value “1” on the left side, diagonally upper left, upper side or diagonally upper right, another label “B” is attached to the pixel of interest.

  After shifting the target pixel to the last pixel in the X direction, the target pixel is returned to the first pixel in the X direction and shifted by one pixel in the Y direction, and the above processing is performed. The above processing is repeated until the target pixel becomes the last pixel in the X direction and the Y direction.

  By the above processing, as shown in FIG. 16, the labels “A” and “B” are attached to the blob 601 of the bright region image 600, and the label “C” is attached to the blob 602.

  Next, in the bright region image 600 of FIG. 16, the target pixel is returned to the first pixel in the X direction and the Y direction, and the target pixel is shifted by one pixel in the X direction. Alternatively, it is checked whether or not there is a pixel labeled on the upper right side. If the pixel of interest is labeled and there is a pixel with a different label on the left, upper left, upper or upper right, all pixels with the same label as the target pixel are left Change to the same label as the pixel on the upper left or upper left or upper right.

  After shifting the target pixel to the last pixel in the X direction, the target pixel is returned to the first pixel in the X direction and shifted by one pixel in the Y direction, and the above processing is performed. The above processing is repeated until the target pixel becomes the last pixel in the X direction and the Y direction.

  As a result, as shown in FIG. 17, the label “B” in the blob 601 is changed to the label “A”.

  In this way, connected regions of pixels having a value of “1” can be extracted as blobs, and individual blobs can be identified by labels.

  In the bright region image 600 of FIG. 17, the blob 601 is identified by the label “A”, and the blob 602 is identified by the label “C”.

  Blob extraction and labeling are performed in the same manner for the dark region image.

(10) Generation of Blemish Component and Wrinkle Component Next, processing for detecting a blot and wrinkle from the labeled blob and processing for generating a blot component and a wrinkle component will be described.

(A) Blot detection From general face knowledge, a blot does not occupy a large area with respect to the face and is sufficiently small. However, the extremely small blob is not a stain because the observer will not be able to perceive it. Further, the shape of the blot is a shape close to a circle or slightly distorted from the circle.

  Here, all the blobs not detected as blots are removed (the pixel value is set to 0).

(B) Wrinkle detection From general facial knowledge, wrinkles are not significantly longer than face size. The wrinkle shape is linear or curved. On the other hand, the outline of the face part may have a similar shape. However, even when a person observes the face, it is often difficult to distinguish wrinkles and the contours of the facial parts from only these local images. In such a case, it is considered that a person uses the knowledge of a face. Therefore, if a blob contains a part of a face part, the blob is regarded as a contour line of the face part.

  Here, all the blobs not detected as wrinkles are removed (the pixel value is set to 0).

  FIG. 18 is a diagram for explaining the reference length of the original image. FIG. 19 is a flowchart showing a spot and wrinkle detection process. FIG. 20 is a diagram showing an example of a blob.

  In the spot and wrinkle detection processing, as shown in FIG. 18, the horizontal distance L1 between the centers of both eyes of the original image and the vertical distance L2 between the midpoint between the eyes and the mouth are set. The value multiplied by the coefficient is compared, the longer distance among them is set as the reference length, and the size of the extracted blob is normalized.

  Here, it is assumed that spots and wrinkles are detected from the blobs B1 to B8 in FIGS. In FIG. 20, a rectangular area inscribed by each blob B1 to B8 is referred to as a processing area 700.

  The horizontal length w of each blob B1 to B8 is the difference between the maximum coordinate and the minimum coordinate in the X direction of the blob B1 to B8, and corresponds to the horizontal length of the processing region 700. The vertical length h of each blob B1 to B8 is the difference between the maximum coordinate and the minimum coordinate in the Y direction of the blob B1 to B8, and corresponds to the vertical length of the processing region 700.

  First, the CPU 501 determines whether or not the number of pixels in the blob is equal to or greater than a predetermined value P1 (step S71). If the number of pixels in the blob is smaller than the predetermined value P1, the blob is excluded (step S79). Thereby, blobs that are too small are eliminated.

  If the number of pixels in the blob is equal to or greater than the predetermined value P1, the CPU 501 determines whether or not the relationship value fa between the aspect ratio of the blob and the density is equal to or greater than the predetermined value K (step S72). The predetermined value K is set to a value larger than 0 and smaller than 1.

  Here, the relation value fa is obtained by the following equation.

fa = α [1- | (4 / π) {tan ⁻¹ (w / h) − (π / 4)} |] + (1-α) (βs / wh) (2)
In the above equation (2), w is the horizontal length of the blob, h is the vertical length of the blob, s is the number of pixels in the block, and α and β are predetermined coefficients.

In [1- | (4 / π) · {tan ⁻¹ (w / h) − (π / 4)} |] in the first term of the above formula (2), the value of tan ⁻¹ (w / h) Represents an angle formed by a diagonal line of the processing region 700 with respect to the X direction. When the aspect ratio w / h of the blob is 1, the diagonal angle is 45 degrees, and the above value of the first term is 1. When the blob aspect ratio w / h is close to 0 or ∞, the diagonal angle is close to 90 degrees or 0 degrees, and the above-mentioned value of the first term is about 0.

  For example, in the blobs B1, B2, and B3 in FIGS. 20A, 20B, and 20C, the value of the first term is 1, and in the blob B4 in FIG. The value of is 0.

  In the second term of the above equation (2), the value of s / wh represents the density of the blob in the processing area 700.

  For example, in the blobs B1 and B4 in FIGS. 20A and 20D, the above-mentioned value of the second term is close to 1, and the blobs B2, B7, and B in FIG. 20B, FIG. In B8, the value of the second term is close to zero.

  The coefficient α is used to adjust the aspect ratio and the density weight, and is set so that 0 <α <1. Therefore, when the shape is close to a square or a circle and the density is high, the relation value fa is close to 1.

  For example, in the blob B1 in FIG. 20A, the relation value fa is close to 1.

  If the relationship value fa between the aspect ratio of the blob and the density is equal to or greater than the predetermined value K, the CPU 501 determines whether or not the number of pixels in the blob is equal to or less than the predetermined value P2 (step S77). The predetermined value P2 is set to a value larger than the predetermined value P1.

  When the number of pixels of the blob is not less than the predetermined value P2, that is, when it is larger than the predetermined value P2, the CPU 501 excludes the blob. This eliminates blobs that are too large.

  If the number of pixels in the blob is equal to or smaller than the predetermined value P2 in step S77, the CPU 501 determines that the blob is a blot (step S78). In this case, the blob B1 in FIG.

  As a result, it is determined that a blob having a pixel number not less than the predetermined value P1 and not more than the predetermined value P2 and having a relationship value fa between the aspect ratio and the density not less than the predetermined value K is a blot. For example, blob B1 in FIG. 20A is determined to be a stain when the number of pixels is not less than a predetermined value P1 and not more than a predetermined value P2, and when the number of pixels is smaller than the predetermined value P1 or larger than the predetermined value P2. Eliminated.

  If the relation value fa between the aspect ratio and the density is smaller than the predetermined value K in step S72, the CPU 501 determines whether or not the relation value fb between the size and density of the blob is equal to or larger than the predetermined value L ( Step S73). The predetermined value L is set to a value larger than 0 and smaller than 1.

  Here, the relation value fb is obtained by the following equation.

fb = max (w, h) / s (3)
In the above formula (3), max (w, h) indicates the longer value of the horizontal length w and the vertical length h of the blob. S is the number of pixels in the blob.

  In a thin line, the relation value fb is close to 1, and in a thick line, the relation value fb is close to 0.

If the relation value fb between the size and density of the blob is not equal to or greater than the predetermined value L, that is, smaller than the predetermined value L, the CPU 501 excludes the blob (step S76).

  For example, in the blob B3 in FIG. 20C, the relation value fb is close to zero. Therefore, blob B3 is eliminated.

  If the relation value fb between the size and density of the blob is greater than or equal to the predetermined value L in step S73, the CPU 501 determines whether or not the degree of blob variation fc is greater than or equal to the predetermined value M (step S74). The predetermined value M is set to a value larger than 0 and smaller than 1. For example, the predetermined value M is 0.12.

  In this case, the degree of variation fc of the blob is defined as the ratio of pixels in the blob that have a distance from the center of gravity of the blob that is longer than twice the standard deviation σ. The degree of variation fc is obtained by the following equation.

fc = u / s (4)
In the above equation (4), s is the number of pixels in the blob. U is the number of pixels outside the circle C centered on the blob's center of gravity W0.

  If the blob variation degree fc is smaller than the predetermined value M, the CPU 501 excludes the blob.

  For example, in the blobs B5, B6, and B8 in FIGS. 20E, 20F, and 20H, the value of the variation degree fc is small. Therefore, blobs B5, B6, B8 are eliminated.

  If the degree of blob variation fc is greater than or equal to the predetermined value M in step S74, the CPU 501 determines that the blob is wrinkled (step S75).

  Thereby, the number of pixels is equal to or greater than the predetermined value P1, the relationship value fa between the aspect ratio and the density is smaller than the predetermined value K, the relationship value fb between the size and the density is equal to or greater than the predetermined value L, and the degree of variation A blob in which fc is equal to or greater than a predetermined value M is determined as a wrinkle.

  For example, in the blobs B2, B4, and B7 in FIGS. 20B, 20D, and 20G, the degree of variation fc increases. Therefore, the blobs B2, B4, B7 are determined to be wrinkles.

  For the blob detected as a stain or wrinkle as described above, a stain component or a wrinkle component is generated based on the original image.

  First, a blob detected as a stain and a blob detected as a wrinkle are sorted.

  In the generation of a stain component, the value of a pixel outside the blob detected as a stain is set to “0”. For each pixel in the blob, calculate the median luminance of the pixels in the peripheral area of 10 pixels × 10 pixels including the pixel as the center, and calculate the absolute value of the difference between the luminance value of the original image and the median value. The absolute value of the difference is the pixel value of the blot component. In the generation of the stain component, a stain component corresponding to the dark region image is generated.

  Similarly, in the generation of the wrinkle component, the value of the pixel outside the blob detected as the wrinkle is set to “0”. For each pixel in the blob, calculate the median luminance of the pixels in the peripheral area of 10 pixels × 10 pixels including the pixel as the center, and calculate the absolute value of the difference between the luminance value of the original image and the median value. The absolute value of the difference is the value of the pixel of the wrinkle component. In the generation of the wrinkle component, a wrinkle component corresponding to the dark region image and a wrinkle component corresponding to the bright region image are generated.

  FIG. 21 is a diagram illustrating an example of a stain component or a wrinkle component. The value of the pixel outside the blob 601A, 602A is set to “0”. The value of each pixel in the blob 601A, 602A is calculated by calculating the median luminance value of the pixels in the peripheral area of 10 pixels × 10 pixels including the pixel in the original image. It is obtained by calculating the absolute value of the difference.

(11) Manipulation of face image shape Next, a method of synthesizing face images of different ages with the face image of the original image of the subject will be described. By manipulating the shape of the face image using this method, it is possible to obtain an impression of “feeling like this if a person of a certain face image is seen young” and “feeling like this if a person is seen old”. The configuration of the image processing apparatus for performing the above method is the same as that of the image processing apparatus 50 in FIG.

  In the following, the image is an individual face image, and a plurality of morphological features are two-dimensional coordinates of L feature points (x, y) constituting the shape features of face parts (face parts) such as eyes and mouths. There will be described a case where the attribute is age. Here, L is an integer of 2 or more, x is an x coordinate in the face image, and y is a y coordinate in the face image.

  In the present embodiment, first, as a feature vector (shape feature vector) representing an individual face image, not only the two-dimensional coordinates of L feature points (x, y) but also the image is not directly related. A weighted value of a known attribute value (age attribute value) that strongly influences the age characteristics is added as a variable a, and dimension compression by principal component analysis (PCA) is performed as shown below.

Specifically, the following procedure is performed. The shape feature vector of an individual face image is represented as a 2L + 1 dimensional vector F _i (i = 1, 2,..., M) as in the following equation. M is the number of individual face images used for principal component analysis.

F _i = [x ₁ , y ₁ , x ₂ , y ₂ , x ₃ , y ₃ ,..., X _L , y _L , a] ^T (5)
In the above formula (5), T represents transposition. The average _FAV is subtracted from each element (variable) so that the average value of the shape feature vectors of all the face images is arranged at the center of the shape feature vector space.

ΔF _i = F _i −F _AV (6)
Next, a new facial feature space G is defined by the following equation (7) using the above equation (6).

G = [ΔF ₁ , ΔF ₂ ..., ΔF _M ] (7)
Then, the covariance matrix S of the face feature space G is obtained by the following equation (8).

S = GG ^T (8)
By performing singular value decomposition on the covariance matrix S obtained from the above equation (8) by the following equation (9), each eigenvector can be obtained as the main component of the face feature space G.

S = UDU ^T (9)
U is a unitary matrix, and each column is an eigenvector. That is, the principal component scores of the first principal component in the first to Mth face images are arranged in order in the first column of the unitary matrix U, and the first to Mth face images in the second column. The principal component scores of the second principal component are arranged in order, and the third component principal component scores in the first to Mth face images are arranged in order in the third column. Similarly, the principal component score of the (2L + 1) principal component in the first to Mth face images is arranged in the Mth column of the unitary matrix U. D is a diagonal matrix, and the diagonal component is the eigenvalue of each principal component. That is, the diagonal elements of the diagonal matrix D are the eigenvalues lambda ₁ to [lambda] _{2L + 1} of the first principal component, second (2L + 1) the main component.

  The first principal component obtained here is strongly influenced by the component (age) added as the attribute value, as will be described later, and the value of the principal component score of the first principal component strongly increases the age attribute value. It will be reflected.

Here, the standard deviation of the first principal component is σ _1, and 3 times the variation range of the shape feature. Since the eigenvalue of the principal component is the variance of the principal component, the standard deviation σ ₁ of the first principal component is expressed by the following equation (10) from the eigenvalue λ ₁ of the first principal component.

σ ₁ ² = λ ₁ (10)
A range that is three times the standard deviation σ ₁ of the first principal component, that is, a range from − (3/2) σ ₁ to + (3/2) σ ₁ is defined as a change range of the face shape (shape feature). More than 90% of all face images belong to this change range. A principal component feature vector having + (3/2) σ ₁ as the principal component score of the first principal component and 0 as the principal component score of the other principal component is defined as P _max . Also, let P ( _min) be a principal component feature vector in which-(3/2) σ ₁ is the principal component score of the first principal component and the principal component score of the other principal component is 0. The principal component feature vector P _max corresponds to the oldest (or the youngest) face image in the average face image obtained from the M face images, and the principal component feature vector P _min is the average obtained from the M face images. Corresponds to the youngest (or oldest) face image in the face image.

P _max = [+ (3/2) σ ₁ , 0, 0,..., 0] ^T (11)
P _min = [− (3/2) σ ₁ , 0, 0,..., 0] ^T (12)
The face shape vectors F _max and F _min are reconstructed from the principal component feature vectors P _max and P _{min of the} above equations (11) and (12). The face shape vector _Fmax represents the oldest (or youngest) average face image, and the face shape vector _Fmin represents the youngest (or oldest) average face image. Then, the difference between the face shape vector F _max and the face shape vector F _min is extracted as a feature vector (age feature vector) F representing an age feature as in the following equation.

F = F _max −F _min (13)
However, since an attribute value is not required for the age feature vector F, the age feature vector F is a 2L-dimensional vector from which the age attribute value is deleted.

F = [x ₁ , y ₁ , x ₂ , y ₂ ,..., X _2L , y _2L ] ^T (14)
The age feature vector F in the above equation (14) represents the amount of movement of each feature point between the oldest average face image and the youngest average face image. For example, the variable x ₁ represents the amount of movement of the first feature point in the x coordinate, and the variable y ₁ represents the amount of movement of the first feature point in the y coordinate.

  Using the age feature vector F of the above equation (14), a face image of a different age of the individual can be synthesized from the original face image of the specific individual as follows.

First, configuring the shape feature vector F ^I a particular individual of the original face image as follows.

F ^I = [x ₁ , y ₁ , x ₂ , y ₂ ,..., X _2L , y _2L ] ^T (15)
The variables x ₁ , y ₁ , x ₂ , y ₂ ,..., X _2L , y _2L in the above equation (15) are the x and y coordinates of L feature points of the original face image.

Next, each variable x ₁ , y ₁ , x ₂ , y ₂ ,..., X _2L , y _2L of the age feature vector F of the above equation (14) is multiplied by the composition ratio b, and a modified vector F ′ of the following equation: Is calculated.

F ′ = [bx ₁ , by ₁ , bx ₂ , by ₂ ,..., Bx _2L , by _2L ] ^T (16)
The correction vector F ′ of the above equation (16) is added to or subtracted from the shape feature vector F ^I of the original face image of the above equation (15) as in the following equation to create a shape feature vector F ^{S in} which the age feature is mapped. .

F ^S = F ^I ± F ′ (17)
It can be synthesized age different face images on the original facial image by reconstructing a face image from the shape feature vector F ^S in the above equation (17). In this case, it is possible to synthesize a face image of any age by arbitrarily setting the value of the composition ratio b in the above equation (16).

  22 and 23 are flowcharts showing the processing of the image processing program executed in the image processing apparatus of FIG. Here, an example of a method for synthesizing a face image of another age of the individual from the original face image of the individual of a certain age is shown.

  First, the CPU 501 stores a plurality of individual face images input by the input device 504 as image data in the external storage device 506 (step S91). In this case, image data of a plurality of individual face images stored in advance in the database may be used.

  Next, the CPU 501 extracts coordinate values of a plurality of feature points of each face image based on the image data stored in the external storage device 506 (step S92). In this embodiment, the x coordinate and y coordinate of each feature point are extracted.

  Next, the CPU 501 acquires the age attribute value of each face image (step S93). The age attribute value may be input by the input device 504 for each face image, or an age attribute value stored in the database in advance for each face image may be used. In the present embodiment, an apparent age obtained by an age perception experiment described later is used as an age attribute value.

Further, the CPU 501 constructs the shape feature vector F _i (see formula (5)) of each face image using the coordinate values and age attribute values of the plurality of feature points extracted for each face image (step S94). .

Next, the CPU 501 calculates the eigenvalue σ ₁ of the first principal component from the shape feature vectors F _i of the plurality of face images by principal component analysis according to equations (6) to (10) (step S95). Further, the CPU 501 constructs the principal component feature vectors P _max and P _min of the first principal component of Expression (11) using the eigenvalue σ ₁ of the first principal component (step S96 in FIG. 23).

Next, the CPU 501 uses the eigenvalue σ ₁ of the first principal component to reconstruct the face shape vectors F _max and F _min of the first principal component of Expression (11) (Step S97), and the age of Expression (12) An age feature vector F representing the feature is extracted (step S98).

  Next, the CPU 501 synthesizes a face image of a desired age from an original face image of a specific individual using the age feature vector F (step S99).

  In this way, by executing the image processing program in the image processing apparatus, it is possible to extract facial shape features closely related to age features as age feature vectors F using a plurality of individual face images, A facial image of a desired age can be synthesized from an original facial image of a specific individual using the extracted facial shape features.

  As described above, since unknown face images can be synthesized by the image processing apparatus, the image processing method, and the image processing program of the present embodiment, they can be used for criminal investigations and the like.

  In the above embodiment, coordinate values of a plurality of feature points representing facial shape features are used as the morphological features of the image. However, textures such as skin tone, blotches, wrinkles, etc. (Texture) can also be used. Further, the perception of age may consider not only the shape inside the face but also the hairstyle and the like.

  Furthermore, in the above embodiment, the apparent age is used as the attribute, but the actual age may be used as the attribute instead of the apparent age. Moreover, you may use other attributes, such as a facial expression, a race, sex, and a body shape, as an attribute.

  When the attribute is real age, by applying the image processing method of the above embodiment, it is possible to extract facial shape features closely related to real age features using a plurality of individual face images, Furthermore, face images having different real ages can be synthesized from the original face image of a specific individual using the extracted facial shape features.

  When the attribute is a facial expression, expressions such as emotions are expressed with different values, and by applying the image processing method of the above-described embodiment, it is closely related to facial expression features using a plurality of individual facial images. The feature of the face shape can be extracted, and a face image with a different expression can be synthesized from the original face image of a specific individual using the extracted feature of the face shape.

  When the attribute is race, different races are represented by different values, and by applying the image processing method of the above embodiment, it is closely related to race characteristics using face images of multiple individuals The feature of the face shape to be extracted can be extracted, and the face image of a different race can be synthesized from the original face image of a specific individual using the extracted feature of the face shape.

  When the attribute is gender, the gender of the man and woman is represented by different values, and by applying the image processing method of the above embodiment, the facial shape features closely related to the gender features using a plurality of individual face images Furthermore, it is possible to synthesize a face image with different gender from an original face image of a specific individual using the extracted feature of the face shape.

  When the attribute is a body shape, different body shapes are represented by different values, and by applying the image processing method of the above embodiment, facial shape features closely related to the body shape features using a plurality of individual face images Furthermore, it is possible to synthesize a face image having a different body shape from an original face image of a specific individual using the extracted facial shape characteristics.

  Moreover, although the case where the image is a personal face image has been described in the above embodiment, the present invention is not limited to this. For example, the image may be an animal image.

(12) Correspondence relationship between each constituent element of claims and each part of the embodiment In the present embodiment, the external storage device 506 corresponds to the storage means, and the CPU 501 is the detection means, creation means, extraction means, and application means. , Generating means, deforming means, and computer.

1. Preliminary experiment When considering the shape features related to the aging of an individual's face, the real age may be used as an attribute value given to the principal component analysis variable. However, it is also empirically clear that, with the aging of the face, spots, wrinkles, skin texture and the like change with the shape. In the present embodiment, the characteristics of the apparent age, that is, the face that looks young and the shape of the face that looks old, were examined, not the facial shape characteristics that change with time. Since it is not sufficient to give the actual age as the attribute value of the apparent age, an age perception experiment (age evaluation experiment) was performed in order to obtain the attribute value of the apparent age.

1-1. Procedure For experimental stimulation in age perception experiments, from the facial expression image database collected and created by the applicant of the present application, there are 142 facial expressions, front-facing faces, and real ages in the late teens to late 30s. Images were used. The face image was a color image of 512 × 512 pixels, and the face was large enough to fit in the frame. The test subjects were 25 male students and 22 female students (18 to 22 years old).

  The experiment was performed according to the following procedure. The subject was asked to estimate the age of the face image displayed on the monitor and select the most suitable category from the age rating categories shown in Table 1.

  As shown in Table 1, ages were classified into 8 categories, and rating values of 0 to 7 were given to these categories in ascending order of age.

1-2. Results The average and standard deviation of the rating values were determined for each face image. FIG. 24 is a diagram in which the averages of the rating positions for each face image are plotted in the descending order. The horizontal axis of FIG. 24 shows the face images of 142 persons, and the vertical axis shows the average of the rating values of each face image. Triangle marks and circle marks represent the average rating values of male and female face images, respectively.

  The average was obtained for each gender of the face image from the standard deviation obtained from each face image. The average standard deviation was 0.85 for male face images and 0.89 for female face images. That is, the age is estimated with an error of about ± 5 years for both male and female faces.

  In addition, since face images differ between men and women and there is no relation between face images, comparison between face images is meaningless.

2. Age Feature Extraction Experiment Next, a face age feature extraction experiment was performed using the apparent age attribute value obtained by the age perception experiment. Extracted when an apparent age attribute value is added to face information (hereinafter referred to as attribute value addition condition) and when an apparent age attribute value is not added to face information (hereinafter referred to as no attribute value condition) Comparison of the shape characteristics to be performed was performed.

2-1. Procedure The facial image shape features include the facial image synthesis system (FUTON) developed by the applicant (Miyuki Tsujiike, Shigeru Mukai, Saiko Yoshikawa, Takashi Kato, Masaomi Oda, Shigeru Akamatsu, “ Facial Image Synthesis System for FUTON System-, "Science Technique, HIP97-39, pp.73-80, Jan.1998 and Shigeru Mukaida, Miyuki Tsujiike, Shigeru Akamatsu," Manual in Facial Image Synthesis System (FUTON system) Sampling evaluation, the coordinate values of 83 default feature points of "Science Technique, HIP99-49, pp. 13-18, Nov. 1999" were used.

  FIG. 25 is a diagram showing feature points used as variables of face feature vectors. In FIG. 25, 83 feature points are indicated by black circles, and symbols are assigned to identify each feature point.

  As face images, 142 face images of both men and women used in the preliminary experiment were used. However, normalization was performed so that the face width (the length between the feature points Fr3 and Fl3) was 230 pixels, and the straight line connecting the eyes of both eyes was horizontal.

For each male and female face image, dimension compression by principal component analysis was performed for each of the no attribute value condition and the attribute value addition condition.
-No attribute value condition In the no attribute value condition of the conventional method, only the coordinate values of 83 feature points were used as face information (variables of shape feature vectors). It represents the 1 face images as 166-dimensional shape feature vector F _N.

F _N = (x ₁ , y ₁ , x ₂ , y ₂ , x ₃ , y ₃ ,..., X ₈₃ , y ₈₃ ) ^T
Attribute value addition condition In the attribute value addition condition, as face information (variable of shape feature vector), in addition to the coordinate values of the 83 feature points, the average value of the evaluation values of each face image obtained in the preliminary experiment is 100. The doubled value was used as the apparent age attribute value a. One face image is represented as a 167-dimensional shape feature vector F _A.

F _A = (x ₁ , y ₁ , x ₂ , y ₂ , x ₃ , y ₃ ,..., X ₈₃ , y ₈₃ , a) ^T
The amount of change of the principal component was determined from the obtained eigenvalues for each principal component, and after reconstructing the shape feature vector for each principal component, the difference was visually compared.

2-3. Results The principal component scores of each principal component obtained from the no attribute value condition and the attribute value addition condition were analyzed for both male and female face images. If the principal component is strongly involved in the age change, when the face images are arranged in order of age, the principal component scores of the face images on the principal component should be arranged in ascending or descending order. Therefore, the principal component scores from the first principal component to the third principal component among the principal components obtained under each condition in the male and female face images were plotted.

  FIG. 26A is a diagram showing the principal component scores of the upper principal components obtained in the male face image without the attribute value condition, and FIG. 26B is the upper face obtained in the female face image without the attribute value condition. It is a figure which shows the main component score of a main component. FIG. 27A is a diagram showing the principal component scores of the upper principal components obtained under the attribute value addition condition in the male face image, and FIG. 27B is obtained under the attribute value addition condition in the female face image. It is a figure which shows the principal component score of the higher-order principal component. The horizontal axis of FIG. 26 and FIG. 27 shows the face images arranged in order of the apparent age (rating value), and the vertical axis shows the principal component score (PC score). A square mark indicates the principal component score of the first principal component, a circle mark indicates the principal component score of the second principal component, and a triangle mark indicates the principal component score of the third principal component.

  As shown in FIGS. 26 (a) and 26 (b), in the no attribute value condition, the principal component scores of face images arranged in order of age are arranged in ascending or descending order for both male face images and female face images. I couldn't find any ingredients.

  On the other hand, as shown in FIGS. 27A and 27B, in the attribute value addition condition, the male principal image and the female facial image are shown in a form in which the principal component scores of the first principal component are arranged almost in sequence. It was.

  From these results, it was found that the first principal component obtained under the attribute value addition condition is closely related to the feature that affects the apparent age of the face image.

  Next, in order to visually confirm what shape change each principal component is involved in, the coordinates of the feature points of the facial parts are used, using the change range obtained from the eigenvalues of the principal component to be confirmed. Was reconfigured. As shown below, the influence of the principal component on the face shape under the attribute value addition condition was examined.

  28, 29, 30 and 31 are reconstructed using the first principal component, the second principal component, the third principal component and the fourth principal component obtained in the attribute value addition condition in the male face image, respectively. It is a figure which shows the performed face shape. 32, 33, 34, and 35 show the first principal component, the second principal component, the third principal component, and the fourth principal component obtained in the attribute value addition condition in the female face image, respectively. It is a figure which shows the reconfigure | reconstructed face shape. In FIGS. 28 to 35, (a) and (b) show face shapes whose apparent ages are both extremes.

  Two face shapes of each principal component were compared from the reconstructed face images. The subjective impression of how each principal component is related to changes in shape characteristics is as follows. In the first principal component obtained with the attribute value addition condition for the male face image, the inside of the eyebrows changes up and down, and the outside changes up and down in the direction opposite to the inside. The nose and mouth change up and down, the width of the mouth narrows and widens slightly, and the chin becomes slightly larger and smaller. In the second main component, the forehead scales, the outside of the eyebrows changes horizontally (the size of the eyebrows changes), the distance between the eyes is slightly wider or narrower, and the nose, mouth and The chin changes greatly up and down. In the third main component, the inside of the eyebrows changes vertically, and the outside changes slightly in the horizontal direction. The nose and mouth change greatly up and down, and the chin changes into a square, slightly stretched shape, or a slightly smaller, pointed shape. In the fourth main component, the size of the forehead changes abruptly, and the inside of the eyebrows changes greatly up and down. The left side of the nose and mouth changes slightly in the horizontal direction, and the left side of the chin changes very slightly.

  On the other hand, in the no attribute value condition, the first principal component showed almost the same change as the second principal component of the attribute value addition condition, and the second principal component showed almost the same change as the third principal component of the attribute value addition condition. . The same applies to the subsequent main components.

  In the first principal component obtained with the attribute value addition condition for the female face image, the mouth changes up and down, and the width becomes wider and narrower. And the chin gets a little bigger or smaller. The second principal component was substantially the same as the second principal component obtained under the attribute value addition condition for the male face image. In the third principal component, the forehead scales up, the inside of the eyebrows slightly changes up and down, and the nose and mouth change slightly on the left side in the horizontal direction, and the chin is slightly stretched or slightly smaller It becomes a pointed shape. In the fourth main component, the face outline, nose, and mouth slightly change on the right side. And the width of the face outline changes.

  On the other hand, in the no attribute value condition, like the male face image, the first principal component shows almost the same change as the second principal component of the attribute value addition condition, and the second principal component is the third main component of the attribute value addition condition. It showed almost the same change as the component. The same applies to the subsequent main components.

2-3. Considering from the distribution of the principal component scores of each principal component, the principal component scores are in ascending order when the face images are arranged in the order of apparent age under the condition where there is no attribute value and the second principal component of the attribute value addition condition. Or because they were not in descending order, these principal components are not closely related to features that affect the apparent age. On the other hand, in the first principal component of the attribute value addition condition, when face images are arranged in the order of apparent age, the principal component scores are arranged almost in order for both male and female face images, and further, an age perception experiment (age evaluation experiment) ) And the distribution of the first principal component data (principal component score) are very similar to each other, and therefore the first principal component obtained under the attribute value addition condition Is closely related to the shape features that affect the apparent age.

  Regarding the change of the feature point coordinates that are changed to the feature point coordinates of the face parts by changing the value of the principal component score of each principal component, the first principal component of the attribute value addition condition is the youth based on anatomical knowledge It seems to be almost consistent with the aging of the season.

  In the image processing method according to the present invention, the principal component analysis is performed by adding not only the coordinate value of the feature point that is the face shape information but also the attribute value of apparent age that is not directly related to the structure of the face image. Because the principal component score of the first principal component quantitatively shows the relationship with the age change, and the qualitatively reasonable deformation was observed in the face shape reconstructed from the first principal component, It shows that shape features closely related to apparent age can be obtained.

3-1. Mapping to individual face images Apparent age features obtained by age feature extraction experiments were mapped to individual face images.

  FIG. 36 is a diagram showing a result of synthesizing face images of different ages by mapping the age shape feature onto the face image of the subject.

  36 (a) shows an original image of a 20-year-old subject, FIG. 36 (b) shows a face image synthesized so that the age looks 40 years, and FIG. 36 (c) shows an age 70 years. A synthesized face image is shown. The composition ratio was set to ± 50% of the change amount of the first principal component in the attribute value addition condition of the age feature extraction experiment.

The synthesis procedure is as follows. From the target's face image, we obtain the feature points of the 83 points used in FUTON, using the coordinate values of the feature points of shape feature vector F ^I of the original face images, and 166-dimensional vector F ^I.
F ^I = (x ₁ , y ₁ , x ₂ , x ₃ ,..., X ₁₆₆ , y ₁₆₆ ) ^T (15)
The shape feature vector F ^I of the original face image and the correction vector F ′ obtained by multiplying each element of the apparent age feature vector F by the composite ratio 0.5 are added or subtracted, and the face shape vector F ^S mapping the age feature is obtained. Created.

F ′ = (0.5x ₁ , 0.5y ₁ , 0.5x ₂ , 0.5y ₂ ... 0.5x ₈₃ , 0.5y ₈₃ ) ^T (16)
F ^S = F ^I ± F ′ (17)
The face shape is performed by FUTON shape morphing (facial synthesis techniques) that changes from ^{F I} to ^{F S.}

  As described above, in the above-described embodiment, the attribute value of apparent age that is not directly related to the image is added as a variable to the shape feature vector of the face image, and the principal component analysis is performed to closely match the apparent age change. It was shown that it is possible to extract the shape features involved.

  The present invention can be used for synthesizing face images having different age impressions from original images, and can be used for various purposes such as criminal investigations, search for interrogators, and skin care simulations.

It is a block diagram which shows the structure of the face image composition apparatus for performing the face image composition method of this Embodiment. It is a flowchart which shows the blot / wrinkle model creation process by a face image synthesizing | combining apparatus. It is a flowchart which shows a wrinkle model creation process. It is explanatory drawing for demonstrating the matching coefficient P. FIG. It is the schematic diagram which simplified the original image of the subject. It is a figure which shows the wrinkle model created by the blot / wrinkle model creation process and the blot model. It is a flowchart which shows a model application process. (A) is a photograph of an original image of a 20-year-old woman, (b) is a photograph of a result of applying a blot model and a wrinkle model in the 40s to the original image, and (c) is a blot of the 70s. It is a photograph as a result of applying a model and a wrinkle model to an original image. It is a figure which shows an example of an attention pixel and a surrounding pixel. It is a flowchart which shows the preparation process of a bright area image. It is a flowchart which shows the preparation process of a bright area image. It is a flowchart which shows the preparation process of a dark area image. It is a flowchart which shows the preparation process of a dark area image. It is a figure which shows an example of an original image. It is a figure which shows an example of a bright area image. It is a figure for demonstrating the extraction and labeling of a blob. It is a figure for demonstrating the extraction and labeling of a blob. It is a figure for demonstrating the reference length of an original image. It is a flowchart which shows a stain and a wrinkle detection process. It is a figure which shows the example of a blob. It is a figure which shows an example of a stain component or a wrinkle component. It is a flowchart which shows the process of the image processing program performed in the image processing apparatus of FIG. It is a flowchart which shows the process of the image processing program performed in the image processing apparatus of FIG. It is the figure which rearranged and plotted the average of the rating position for every face image in descending order. It is a figure which shows the feature point used as a variable of a face feature vector. (A) is a figure which shows the principal component score of the high-order principal component obtained on the condition without attribute value in the male face image, (b) is the high-order principal component obtained on the condition without attribute value in the female face image It is a figure which shows the main component score of. (A) is a figure which shows the principal component score of the high-order principal component obtained by the attribute value addition condition in the male face image, (b) is the high-order principal component obtained by the attribute value addition condition in the female face image It is a figure which shows the main component score of. It is a figure which shows the face shape reconfigure | reconstructed using the 1st main component obtained on the attribute value addition conditions in the male face image. It is a figure which shows the face shape reconfigure | reconstructed using the 2nd main component obtained on the attribute value addition conditions in the male face image. It is a figure which shows the face shape reconfigure | reconstructed using the 3rd main component obtained on the attribute value addition conditions in the male face image. It is a figure which shows the face shape reconfigure | reconstructed using the 4th main component obtained on the attribute value addition conditions in the male face image. It is a figure which shows the face shape reconfigure | reconstructed using the 1st main component obtained on the attribute value addition conditions in the female face image. It is a figure which shows the face shape reconfigure | reconstructed using the 2nd main component obtained on the attribute value addition conditions in the female face image. It is a figure which shows the face shape reconfigure | reconstructed using the 3rd main component obtained on the attribute value addition conditions in the female face image. It is a figure which shows the face shape reconfigure | reconstructed using the 4th main component obtained on the attribute value addition conditions in the female face image. It is a figure which shows the result of having synthesize | combined the face image from which age differs by mapping an age shape feature to a subject's face image.

Explanation of symbols

50 Image Processing Device 100 Original Image 200 Peripheral Area 300 Pixel of Interest 501 CPU
502 ROM
503 RAM
504 Input device 505 Display device 506 External storage device 507 Recording medium drive device 508 Printing device 509 Recording medium 601 602 601A 602A B1 B2 B3 B4 B5 B6 B7 Blob 700 Processing area

Claims (4)

A face image synthesis device for synthesizing a face image of a specified age or age group from a face image of a subject,
Storage means for storing face images of a plurality of people in advance;
Detecting means for detecting spots or wrinkles of a plurality of persons from the face images of the plurality of persons stored in the storage means;
Generating means for generating an age shape feature that is an average face shape corresponding to an age or an age group from a plurality of face images stored in the storage means;
Creating means for creating a blot model or wrinkle model corresponding to an age or age group from a plurality of spots or wrinkles detected by the detection means;
Extraction means for extracting a face shape having no blotches or wrinkles from the face image of the subject;
Deformation means for deforming the face shape of the subject extracted by the extraction means based on an age shape feature corresponding to a specified age or age group among the age shape characteristics generated by the generation means;
Of the spot model or wrinkle model corresponding to the age or age group created by the creating means, the spot model or wrinkle model corresponding to the specified age or age group is converted into the face shape of the subject deformed by the deforming means. And applying means for synthesizing a face image corresponding to a specified age or age group by applying the deformed blot model or wrinkle model to the face image of the subject after deformation to match .
The creating means extracts the number or area of stains from a plurality of persons for each of the face image divided into a plurality of areas, and arranges the number or area of the extracted spots for each area in ascending or descending order for each of the areas. The face image area having the number or area of the blots located at the approximate center is selected, and a combination of the blots of the selected areas is created as a blot model to create a wrinkle component of the face images of multiple persons. Is combined with the age shape feature generated by the generating means, the position of the wrinkle component is shifted so that the overlapping degree of the wrinkle component is maximized, the luminance of the wrinkle component after the shift is integrated, and the integrated wrinkle component A face image synthesizing apparatus is characterized in that an area where the obtained luminance exceeds a reference value is set as a wrinkle of the wrinkle model . The detection means extracts a small blob based on the difference between the luminance of each pixel and the luminance of the surrounding area in each face image stored in the storage means, and detects a spot or a wrinkle from the extracted blob. The face image synthesizing device according to claim 1 . A face image synthesis method for synthesizing a face image of a specified age or age group from a face image of a subject,
Storing a plurality of face images in advance;
Detecting blots or wrinkles of a plurality of persons from the stored face images of the plurality of persons;
Generating an age shape feature that is an average face shape corresponding to an age or age group from the stored face images of the plurality of persons;
Creating a blot model or wrinkle model corresponding to an age or age group from the detected multiple spots or wrinkles;
Extracting a face shape having no spots or wrinkles from the face image of the subject;
Transforming the extracted facial shape of the subject based on an age shape feature corresponding to a specified age or age group among the generated age shape features;
After deforming the blot model or wrinkle model corresponding to the specified age or age group among the created blot model or wrinkle model corresponding to the age or age group so as to match the face shape of the deformed subject Synthesizing a face image corresponding to a specified age or age group by applying a deformed blot model or wrinkle model to the face image of the subject;
The step of creating extracts the number or area of stains from a plurality of persons for each of the face image divided into a plurality of areas, and calculates the number or area of the extracted spots for each area in ascending or descending order for each of the areas When arranged, a face image area having the number or area of the blots located approximately in the center is selected, and a combination of the blots of the selected areas is created as a blot model, and wrinkles of the face images of multiple people are created. Synthesize the component with the generated age shape feature, shift the position of the wrinkle component so that the overlapping degree of the wrinkle component is the largest, add the luminance of the wrinkle component after the shift, and integrate the luminance of the wrinkle component Dividing the area by the number of persons and setting a region where the obtained luminance exceeds a reference value as a wrinkle of the wrinkle model . A face image synthesis program that can be executed by a computer and synthesizes a face image of a specified age or age group from a face image of a subject,
A process for storing face images of a plurality of people in advance;
A process of detecting spots or wrinkles of a plurality of persons from the stored face images of the plurality of persons;
Processing to generate an age shape feature that is an average face shape corresponding to an age or age group from the stored face images of a plurality of persons;
A process of creating a blot model or wrinkle model corresponding to an age or age group from the detected spots or wrinkles of a plurality of persons;
A process of extracting a face shape having no spots or wrinkles from the face image of the subject person;
A process of deforming the extracted face shape of the subject based on an age shape feature corresponding to a specified age or age group among the generated age shape features;
After deforming the blot model or wrinkle model corresponding to the specified age or age group among the created blot model or wrinkle model corresponding to the age or age group so as to match the face shape of the deformed subject , Causing the computer to execute a process of synthesizing a face image corresponding to the specified age or age group by applying the deformed blot model or wrinkle model to the face image of the subject ,
The process of creating is to extract the number or area of stains from a plurality of people for each area divided into a plurality of face images, the number or area of the extracted spots for each area in ascending or descending order for each area When arranged, a face image area having the number or area of the blots located approximately in the center is selected, and a combination of the blots of the selected areas is created as a blot model, and wrinkles of the face images of multiple people are created. Synthesize the component with the generated age shape feature, shift the position of the wrinkle component so that the overlapping degree of the wrinkle component is the largest, add the luminance of the wrinkle component after the shift, and integrate the luminance of the wrinkle component A face image composition program comprising: dividing an area by the number of persons and setting a region where the obtained luminance exceeds a reference value as a wrinkle of a wrinkle model .

Images