JP2005284775A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize a simple and low cost cosmetic or beauty simulation system. <P>SOLUTION: Means for acquiring shape information comprise a plurality of stereo image pickup cameras positioned to image a three-dimensional object to be imaged from different angles, and pattern projectors disposed for the cameras respectively. Means for acquiring color information comprise a color pickup camera positioned to image the three-dimensional object to be imaged from a position to observe shape imaging patterns. The patterns projected by the pattern projectors are projected by infrared light. The stereo image pickup cameras are cameras for imaging infrared light, and the color pickup camera is a camera for imaging visible light. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for simultaneously acquiring information on the shape and color of a three-dimensional object such as a human body.

  The applicant takes a color image of the subject's face with a camera, and transforms the photographed image into makeup data representing the change in the face due to the type of makeup desired by the subject so as to match the face image of the subject. A patent application was made for a makeup advice system that synthesizes and displays the facial appearance of the subject when makeup is applied (see Patent Document 1).

  In addition to the color information of the subject's face, the applicant also obtains shape information at the same time, and based on that information, 3D object measurement to realize a makeup advice system that displays more realistic images The device was developed and a patent application was filed (see Patent Document 2).

As another technique for acquiring the shape information and color information of the entire three-dimensional object, for example, there is a technique disclosed in Patent Document 3. In this technique, information on the shape and color of the entire three-dimensional object can be acquired by arranging a plurality of sets of shape information imaging cameras, pattern projection apparatuses, and color information imaging cameras.
JP 2000-194835 A JP 2003-57017 A JP 2001-82940 A

  In Patent Document 1, output display is intended only for still images, and it is not considered to process moving images.

  Patent Document 2 proposes to extract information on the color and shape of a three-dimensional object using two types of cameras, but the color information, shape information, and shape displacement amount of a three-dimensional object to be measured. Since it is necessary, it is not always a simple system, and it is difficult to reduce the cost and installation space of the system.

  Patent Document 3 proposes to use two types of cameras to extract information on the shape and color of a three-dimensional object. However, a combination of a shape information photographing camera, a pattern projection device, and a color information photographing camera is proposed. Since a plurality of the devices are arranged, it is not necessarily a simple system as in Patent Document 2. In addition, since there is no means for generating an image signal having position information of a three-dimensional object corresponding to each pixel for each pixel of the image acquired by the color information photographing camera, a moving image based on the acquired color information and shape information It is difficult to process in real time.

  On the other hand, makeup advice systems are often installed in a relatively small space such as a department store's cosmetics department, and a large number of cosmetic advice systems need to be installed in consideration of sales activities nationwide. It is important that it be simple and inexpensive.

  In addition, when performing a makeup simulation at a cosmetics department store in a department store or the like, it is necessary to perform it in a dialogue with a customer, and it is important that face image processing is performed in real time.

  The present invention has been made under such a background, and provides an image processing apparatus capable of realizing a makeup or beauty simulation system that displays a moving image in real time with a small size, simple and low cost. The purpose is to do.

  According to the present invention, a means for photographing a three-dimensional object and obtaining shape information thereof, a means for photographing a three-dimensional object and obtaining color information thereof, color information based on the obtained shape information and color information, An image processing apparatus comprising: means for generating an image signal having the positional information of the three-dimensional object corresponding to each pixel in units of pixels of an image acquired by a photographing camera.

  Here, the feature of the present invention is that the means for acquiring the shape information includes a plurality of stereo image photographing cameras provided at positions for photographing the three-dimensional object to be photographed from different angles, and A pattern projecting device provided for each camera, and the means for acquiring the color information includes a part or all of a pattern for shape photographing projected on the three-dimensional object to be photographed on the three-dimensional object. Is provided with a color information photographing camera provided at a position for photographing from any position observable within the angle of view photographed by the camera for all patterns projected onto the three-dimensional object, and projected by the pattern projecting device The pattern is insensitive to the color information photographing camera.

  The pattern projected by the pattern projection device is projected by infrared light, the stereo image photographing camera is a camera for photographing infrared light, and the color information photographing camera is a camera for photographing visible light. preferable. The three-dimensional object is, for example, a human face or a part of a body.

  The shape information is three-dimensional position information of the target object, and may be point sequence data in addition to an image, and the format is not limited.

  As described above, in the present invention, the most important feature is that the stereo image photographing camera is used. The stereo image photographing camera has two small camera devices built in one casing in advance, and even if a person who does not have specialized knowledge makes settings, the stereo image photographing performance is not affected. Therefore, for example, accurate setting can be performed even in a place where there is no person who has specialized knowledge about stereo image shooting, such as a department store cosmetics department or an esthetic salon.

  In addition, since two small camera devices are built in one housing, the occupied space can be reduced as compared with the case where two camera devices are individually installed. Further, such a stereo image photographing camera is generally commercially available and can be obtained at low cost.

  In addition, since the projection of the pattern for obtaining the three-dimensional shape information is performed with infrared light, the subject does not recognize that the pattern is projected on his / her face or body part, and the face or body Some images of can be taken.

  A total of four stereo image shooting cameras are provided at positions for shooting a human face or body part from the top, bottom, left, and right, respectively, and the color information camera is configured to capture a human face or body part. It is desirable that one unit be provided at a position for photographing from the inside of a rectangle surrounded by the stereo image photographing camera.

  Alternatively, a total of four stereo image shooting cameras are provided at positions where the human face is shot from the top, bottom, left, and right, respectively, and the color information shooting camera is surrounded by the stereo image shooting camera. It is desirable that one unit be provided at a position for photographing from the inside of the rectangle.

  Such camera settings can be provided to the user in a state in which the camera is pre-installed in the frame, or the user can set the setting. Even when the user himself / herself makes the setting, each stereo image capturing camera already has two small camera devices built into one housing, so precise setting is unnecessary and setting due to lack of user knowledge. Failure can be avoided.

  Further, in the present invention, since the subject's face or part of the body is partially photographed from a short distance by a plurality of stereo image photographing cameras, the subject's face or part of the body is photographed by one stereo image photographing camera. Compared with the case where the whole is photographed from a long distance, the resolution of each camera can be lowered. Therefore, since the number of pixels is small, the speed at which the three-dimensional shape information is calculated can be increased. For example, if it is possible to shoot 30 frames per second, this can correspond to a moving image.

  The present invention captures the movement and shape change of a dynamically changing three-dimensional object in real time, and simultaneously measures the surface color, and applies a computer graphics (hereinafter referred to as CG) technology to these virtual images. It is used in a simulation apparatus that creates an image. The present invention was developed for use in a makeup or beauty simulation apparatus, and is an image that creates a virtual composite image by performing various image processing on captured object images, such as movies and game characters. It can also be applied to processing technology.

  Further, according to the present invention, makeup or beauty simulation using moving images can be performed in real time using a small, simple and low-cost apparatus.

  An image processing apparatus according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is an overall configuration diagram of the image processing apparatus according to the present embodiment, and is a view seen from the front (rear part of the subject). FIG. 2 is an overall configuration diagram of the image processing apparatus according to the present exemplary embodiment, as viewed from the side.

  The image processing apparatus according to the present embodiment captures a three-dimensional object and acquires its shape information, means for capturing a three-dimensional object and acquires its color information, the acquired shape information and the color information. An image processing apparatus comprising: means for generating an image signal having position information of the three-dimensional object corresponding to each pixel in units of pixels of an image acquired by a color information photographing camera based on As shown in FIG. 1 and FIG. 2, the means for acquiring the shape information includes a plurality of stereo image photographing cameras 1 provided at positions for photographing the three-dimensional object to be photographed from different angles, and the camera 1. A pattern projection device 2 provided for each, and the means for acquiring the color information includes a part or all of a shape photographing pattern in which the three-dimensional object to be photographed is projected onto the three-dimensional object. All patterns projected onto the three-dimensional object are provided with a color information photographing camera 3 provided at a position for photographing from an arbitrary position that can be observed within an angle of view photographed by the camera. The stereo image photographing camera 1 is a camera for photographing infrared light, and the color information photographing camera 3 is a camera for photographing visible light.

(First Example)
In the first embodiment, the three-dimensional object is a human face.

  A total of four stereo image photographing cameras 1 are provided at positions where the face is photographed from the top, bottom, left and right, respectively, and one color information photographing camera 3 is provided at a position where the face is photographed from the front.

  FIG. 3 is a diagram showing a random pattern. The pattern projected by the pattern projector 2 is a random pattern shown in FIG. The stereo image capturing camera 1 captures a projected random pattern from two different positions with two built-in cameras. The arithmetic unit 4 recognizes the three-dimensional shape of the face by taking in random patterns photographed from different positions by the two cameras and analyzing the difference in appearance. That is, three-dimensional shape information can be calculated by analyzing random pattern matching.

  At this time, the four stereo image capturing cameras 1 individually project random patterns, but there is no problem in the recognition of the three-dimensional shape even if the projection timings overlap. Alternatively, it is easy to perform control to shift the timing by a signal from the arithmetic unit 4 so that the individual projection timings do not overlap.

  This embodiment is characterized in that the face is photographed from above, below, left, and right. This is because, when a three-dimensional shape is photographed from one direction, a portion that cannot be seen due to shielding occurs, but by photographing the face from the top, bottom, left, and right, the portion that cannot be seen due to shielding can be eliminated. In particular, when performing a makeup simulation, there is a tendency to confirm the state of makeup over details while moving the face as a test subject's behavior. At this time, taking pictures from the top, bottom, left and right is effective for taking a picture of the lower part of the chin and the side of the face, etc., and avoiding blind spots.

  In addition, if a single high-definition stereo image shooting camera is used, it is possible to shoot the face and the camera device sufficiently apart to a distance where an invisible part does not occur due to shielding, but a high-definition stereo image Since the photographing camera is expensive and has a large number of pixels, it takes a long time to calculate the three-dimensional shape, which is not suitable for the purpose of reducing the cost of the present invention or the purpose of displaying a moving image in real time. On the other hand, in the present invention, by reducing the distance between the face and the stereo image capturing camera 1, there is an advantage that a high-definition stereo image capturing camera is not required. Moreover, there is an advantage that the installation space can be reduced by shortening the distance between the face and the stereo image photographing camera 1.

  Next, an example of a makeup simulation using the image processing apparatus of this embodiment will be described with reference to FIGS. 4 and 5. The subject's face is photographed by the plurality of stereo image photographing cameras 1 and the color information photographing camera 3 (S1). The images captured by the stereo image capturing camera 1 are respectively taken into the arithmetic device 4, and three-dimensional shape information is generated for each stereo image capturing camera 1, and further, based on the plurality of three-dimensional shape information. Generate three-dimensional shape information of the subject's face. Thereby, the three-dimensional shape information acquired by each of the four stereo image capturing cameras 1 is collected into one piece of information. At this time, the position directly in front of the color information photographing camera 3 becomes the reference point of the three-dimensional shape information.

  This three-dimensional shape information is taken into the image composition device 6. Further, the image photographed by the color information photographing camera 3 is taken into the image composition device 6 as color information (S2). Moreover, the image image | photographed with the color information imaging | photography camera 3 is an image by visible light, and can be displayed on the display part 8 as it is in the state before the test simulation of a test subject's face as it is. Furthermore, the image photographed by the color information photographing camera 3 is recognized as a subject of the makeup simulation inside the image composition device 6. At this time, the image composition device 6 can three-dimensionally recognize the face of the subject based on the three-dimensional shape information.

  Next, desired makeup data is selected from among a large number of makeup data stored in the memory 7 by operating the input device 5 (S3), and the face image of the subject is based on the three-dimensional shape information in which the makeup data is taken. (S4). The makeup data is data relating to colors such as lipstick and foundation. The makeup data processed to match the face image of the subject in this way is synthesized (S5) and displayed on the display unit 8 (S6).

  Note that an image photographed by the stereo image photographing camera 1 is sufficient for one frame, so that the photographing can be performed in a very short time. In the case of the stereo image capturing camera 1 used in the present embodiment, since 30 frames can be captured per second, the time required for capturing is about 1/30 seconds. Even if the calculation time of the three-dimensional shape information by the calculation device 4 is added to this, the face image of the subject can be processed almost in real time. Even if the desired makeup data is changed by operating the input device 5, the face image of the subject can be processed almost in real time.

  This makes it possible to perform makeup simulation with moving images in response to the movement of the subject's face in real time during conversations with customers, such as at a department store's cosmetics department. The simulation can be displayed.

(Second embodiment)
In a second embodiment, the three-dimensional object is a part of a human body. For example, when slimming or shaping, it can be used to make a subject compare the current body shape with the body shape after slimming or shaping. For example, when slimming, the subject can be made to compare the current body shape of the abdomen and the body shape after slimming in a pseudo manner.

  The photographing method and procedure are the same as the face photographing method and procedure described in the first embodiment, and a detailed description thereof will be omitted.

  An example of a beauty simulation using the image processing apparatus of this embodiment will be described with reference to FIGS. A part of the body of the subject is photographed by the plurality of stereo image photographing cameras 1 and the color information photographing camera 3 (S11). Images captured by the stereo image capturing camera 1 are respectively captured by the arithmetic device 4, three-dimensional shape information is generated for each stereo image capturing camera 1, and further, based on the plurality of three-dimensional shape information. Three-dimensional shape information of a part of the subject's body is generated. Thereby, the three-dimensional shape information acquired by each of the four stereo image capturing cameras 1 is collected into one piece of information. At this time, the position directly in front of the color information photographing camera 3 becomes the reference point of the three-dimensional shape information.

  This three-dimensional shape information is taken into the image composition device 6. Further, the image photographed by the color information photographing camera 3 is taken into the image composition device 6 as color information (S12). Moreover, the image image | photographed with the color information imaging | photography camera 3 is an image by visible light, and can be displayed on the display part 8 as it is in the state before the simulation start of a part of test subject's body as it is. Furthermore, the image photographed by the color information photographing camera 3 is recognized as a test subject for the beauty simulation inside the image composition device 6. At this time, the image composition device 6 can three-dimensionally recognize a part of the body of the subject based on the three-dimensional shape information.

  Next, desired body shape data is selected from among a large number of shaping data stored in the memory 7 by operating the input device 5 (S13), so that the captured body image data and shape data can be matched. (S14). The body type data is data relating to a shape such as a deformation of a skeleton or a human body tissue. The body image data processed so as to match the image of the part of the subject's body is synthesized (S15) and displayed on the display unit 8 (S16).

  Note that an image photographed by the stereo image photographing camera 1 is sufficient for one frame, so that the photographing can be performed in a very short time. In the case of the stereo image capturing camera 1 used in the present embodiment, since 30 frames can be captured per second, the time required for capturing is about 1/30 seconds. Even if the calculation time of the three-dimensional shape information by the calculation device 4 is added to this, an image of a part of the body of the subject can be processed almost in real time. Even if the desired body shape data is changed by operating the input device 5, an image of a part of the body of the subject can be processed almost in real time.

  This makes it possible to perform beauty simulations with moving images in response to movements of the subject's body in real time during dialogues with customers at esthetic salons, etc., and respond to changes in the body part Can be displayed.

  According to the present invention, since a makeup or beauty simulation by moving images can be performed in real time using a small, simple and low-cost device, it is suitable for a narrow space such as a department store cosmetic store or an esthetic salon. Further, even when a large number of sales bases are provided, the cost when a makeup or beauty simulation system is installed at each base can be reduced. In addition, since the makeup or beauty simulation can be performed in real time, the makeup or beauty simulation can be performed naturally in the dialogue with the customer.

1 is a front view of an image processing apparatus according to an embodiment. 1 is a side view of an image processing apparatus according to an embodiment. The figure which shows a random pattern. The whole block diagram of the makeup | decoration simulation system of a 1st Example. The flowchart which shows the procedure of the makeup | decoration simulation of a 1st Example. The flowchart which shows the procedure of the beauty simulation of 2nd Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereo image photographing camera 2 Pattern projector 3 Color information photographing camera 4 Arithmetic device 5 Input device 6 Image composition device 7 Memory 8 Display part

Claims (5)

Means for photographing a three-dimensional object and obtaining shape information thereof;
Means for photographing a three-dimensional object and obtaining color information thereof;
Means for generating, based on the acquired shape information and the color information, an image signal having the positional information of the three-dimensional object corresponding to each pixel in units of pixels of the image acquired by the color information photographing camera. An image processing apparatus,
The means for acquiring the shape information includes:
A plurality of stereo image photographing cameras provided at positions for photographing the three-dimensional object to be photographed from different angles;
And a pattern projection device provided for each camera,
The means for acquiring the color information is a method for acquiring the three-dimensional object to be photographed for all patterns in which a part or all of the pattern for shape photographing projected onto the three-dimensional object is projected onto the three-dimensional object. , Equipped with a color information shooting camera provided at a position to shoot from any position observable within the angle of view taken by the camera,
The pattern projected by the pattern projection device is insensitive to the color information photographing camera.   The pattern projected by the pattern projection device is projected by infrared light, the stereo image photographing camera is a camera for photographing infrared light, and the color information photographing camera is a camera for photographing visible light. The image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the three-dimensional object is a human face or a part of a body.   A total of four stereo image shooting cameras are provided at positions for shooting a human face or body part from the top, bottom, left, and right, respectively, and the color information camera is configured to capture a human face or body part. The image processing apparatus according to claim 3, wherein one image processing apparatus is provided at a position for photographing from the inside of a quadrangle surrounded by the stereo image photographing camera.   A total of four stereo image shooting cameras are provided at positions for shooting a human face from above, below, left, and right, respectively, and the color information shooting camera has a rectangular shape surrounded by the stereo image shooting camera. The image processing apparatus according to claim 3, wherein one image processing apparatus is provided at a position for photographing from the inside.

Images