JP6519853B2 - Image processing apparatus, stylus, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, a stylus, and an image processing method, and more particularly to an image processing apparatus, a stylus, and an image processing method for superposing a makeup image on a photographed image of a face.

  Conventionally, there is a so-called virtual makeup technique in which makeup is virtually performed by superimposing an image such as a lipstick or a cheek on an image obtained by capturing a face (see, for example, Patent Document 1).

  The technology described in Patent Document 1 (hereinafter referred to as “conventional technology”) displays a photographed image of a face and an operation area for receiving specification of makeup content on a touch panel screen. And a prior art superimposes a makeup image on the picked-up image of a face based on the touch pen operation with respect to a screen. According to such a conventional technique, it is possible to simulate and display the state of the face when makeup is performed.

JP 2005-92588 A International Publication No. 2013/140776

  By the way, it is well known that children play makeup by imitating adults. Some children feel great pleasure not only in getting a face with makeup, but also in imitating adult makeup movements, such as sliding a lipstick on their lips. Therefore, it is desirable that virtual makeup can be realized by a makeup operation (gesture) such that a makeup image of the lipstick is superimposed on the face image when imitating drawing a lipstick.

  However, according to the prior art, virtual makeup is realized not by the makeup operation but by the touch panel operation on the screen.

  An object of the present invention is to provide an image processing apparatus, a stylus, and an image processing method capable of performing virtual makeup by a makeup operation.

  An image processing apparatus according to an embodiment of the present disclosure includes an imaging unit configured to capture an image, a display unit configured to display the captured image on a screen, and a face position acquisition unit configured to acquire the position of a face included in the image from the image. When a portion of a predetermined marker color is included in the image, a stylus detection unit that detects the portion as a stylus from the image, and the face portion of the image on condition that the stylus is detected And a makeup image superposition unit for superposing a makeup image associated with the predetermined marker color in advance.

  The stylus of the present disclosure is a stylus used in the image processing apparatus, and is a rod-like member, and the predetermined marker color is applied to a region including the tip of the rod-like member.

  The image processing method according to the present disclosure includes a step of shooting a video, a step of displaying the shot video on a screen, a step of acquiring the position of a face included in the video from the video, and Detecting the portion as a stylus from the image when the portion of the marker color is included, and the predetermined marker color for the portion of the face of the image on condition that the stylus is detected And V. superimposing a makeup image associated with in advance.

  According to the present disclosure, virtual makeup can be performed by the makeup operation.

A diagram showing an example of an outline of an image processing apparatus according to the present embodiment The figure which shows an example of the change of the image in this Embodiment A diagram showing a first example of the appearance of a stylus according to the present embodiment Block diagram showing an example of the configuration of the image processing apparatus according to the present embodiment A diagram showing an example of the contents of a makeup image table in the present embodiment Flowchart showing an example of the operation of the image processing apparatus according to the present embodiment A figure showing an example of various virtual makeup in this embodiment The figure which shows an example of the external appearance of the stylus which concerns on the modification 1 of this Embodiment A figure showing an example of contents of a makeup image table in modification 1 of this embodiment The figure which shows an example of the external appearance of the stylus which concerns on the modification 2 of this Embodiment A flowchart showing an example of the operation of the image processing apparatus according to the second modification of the present embodiment A figure showing an example of various virtual makeup in modification 2 of this embodiment A diagram showing an example of a stylus registration screen in the present embodiment A diagram for explaining the definition of a gesture in the present embodiment

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

<Overview of device>
First, an outline of the image processing apparatus according to the present embodiment will be described.

  FIG. 1 is a view showing an example of an outline of an image processing apparatus according to the present embodiment.

  As shown in FIG. 1, the image processing apparatus 100 has, for example, an outer shape that simulates a folding-type compact mirror. The image processing apparatus 100 includes a liquid crystal display 101 in a portion corresponding to a mirror of a compact mirror, and further includes a small digital camera 102 in the vicinity of the liquid crystal display 101. The liquid crystal display 101 may have a touch panel. The digital camera 102 may also be a stereo camera capable of stereo photography.

  The image processing apparatus 100 is used in combination with the stylus 200. The user 300 using the image processing apparatus 100 and the stylus 200 is, for example, a primary school girl. The image processing apparatus 100 horizontally flips an image captured by the digital camera 102 and displays the image on the liquid crystal display 101. The user 300 looks like a makeup tool such as a tip of a lipstick or an eye shadow and performs an operation imitating adult makeup while looking at his face projected on the liquid crystal display 101.

  When the user 300 performs the makeup operation, the image processing apparatus 100 superimposes the makeup image on the area of the image to be displayed, which is the target of the makeup operation.

  FIG. 2 is a view showing an example of a change of an image displayed on the liquid crystal display 101. As shown in FIG.

  As shown in FIG. 2A, the image 410 displayed on the liquid crystal display 101 includes the face 411 of the user. As shown in FIG. 2B, it is assumed that the user 300 performs an operation of tracing the lips with the tip of the stylus 200. Then, as shown in FIG. 2C, the image processing apparatus 100 generates and displays an image 410 in which the makeup image 412 is superimposed on the lips. As a result, the image processing apparatus 100 can give the user 300 a feeling that he or she did the real makeup.

  However, in order to realize such an operation, the image processing apparatus 100 needs to detect this when the user 300 performs the makeup operation.

  Therefore, the image processing apparatus 100 acquires the position of the face from the image captured by the digital camera 102, and when the image includes a portion of a predetermined marker color, detects the portion as the stylus 200 from the image. . Thereby, if a predetermined marker color is given to the stylus 200, the image processing apparatus 100 detects that when the user 300 brings the stylus 200 close to the face, that is, when the makeup operation is performed. Can.

  FIG. 3 is a view showing an example of the appearance of the stylus 200. As shown in FIG.

  As shown in FIG. 3, the stylus 200 is, for example, a rod-like member having a pen-like outer shape. A predetermined marker color such as red is given to the area 210 including the tip of the stylus 200. The marker color is preferably applied to the surface in the direction of 360 degrees around the axis of the stylus 200 so that it is easier to detect from the captured image. The marker color area 210 can be, for example, a paint applied to a rod-like member, a seal attached to the rod-like member, or a colored member constituting the rod-like member.

  The image processing apparatus 100 may detect a predetermined single color as a marker color, or may detect each of a plurality of predetermined colors as different marker colors. In the present embodiment, the image processing apparatus 100 detects each of a plurality of predetermined colors as different marker colors.

<Device configuration>
Next, the configuration of the image processing apparatus 100 will be described.

  FIG. 4 is a block diagram showing an example of the configuration of the image processing apparatus 100. As shown in FIG.

  In FIG. 4, the image processing apparatus 100 includes an information storage unit 110, an imaging unit 120, a face position acquisition unit 130, a stylus detection unit 140, a makeup image superposition unit 150, and a display unit 160.

  The information storage unit 110 includes face feature information for detecting a person's face and face parts included in the image from the image, and a makeup image table for generating a makeup image and superimposing it on the face portion of the video. It is stored in advance.

  The face feature information includes, for example, image features such as shapes of face parts such as eyes and lips generated by learning based on a large number of face image samples, and image features of the entire face such as arrangement of face parts and color histogram of face It is information to show. Each of the face parts includes face feature points such as the corner of the eye and the corner of the eye.

  The makeup image table is a table in which the contents of the makeup image (hereinafter referred to as "makeup image data") are described in advance for each of a plurality of predetermined marker colors. In the present embodiment, the makeup image is generated in units of continuous areas (hereinafter referred to as “target areas”) to which makeup is applied, such as lip areas, eyebrow areas and cheek bone areas. I assume.

  FIG. 5 is a view showing an example of the contents of the makeup image table.

  As shown in FIG. 5, the makeup image table 510 describes a makeup area 512, a makeup color 513, and a makeup density 514 in association with the marker color 511. The makeup area 512, the makeup color 513, and the makeup density 514 constitute makeup image data for each target area.

  The marker color 511 describes the color to be detected as the color representing the stylus by the image processing apparatus 100 for each color that is recognized as the stylus of the same type. Although the marker color 511 is represented by the name of the color here, it is actually a range of coordinate values in a predetermined color space such as RGB (Red, Green, Blue). That is, the marker color 511 is a color range having a certain width.

  The makeup area 512 is, for example, an area of a makeup image (that is, a makeup application area) defined based on the position of a face feature point such as a corner of the mouth. Although the makeup area 512 is represented here by the name of the area of the face, in actuality, it is a coordinate value, a function or the like in the space based on the position of the face feature point.

  The makeup color 513 is the color of the makeup image. In addition, although the makeup color 513 is represented by the name of color here, it is actually a coordinate value in a predetermined color space such as RGB (Red, Green, Blue), HSV (Hue, Saturation, Value), etc. is there. The makeup color 513 does not necessarily have to be the same as the marker color 511, but it is desirable that the makeup color 513 be a color close to the marker color 511. Thus, the user 300 can intuitively perform virtual makeup in a color close to the desired color.

  The makeup density 514 is the density (transparency) of the color of the makeup image when superimposed on the photographed image. Although the cosmetic density 514 is expressed here as the transparency, it is actually a coefficient such as a composition ratio with respect to the pixel value when superposing the cosmetic image on the photographed image by α blending or the like.

  That is, the makeup image indicated by the makeup image data described in makeup image table 510 is an image representing the state of makeup when makeup is applied to an area (target area) predetermined based on the face feature point. is there. In addition, the makeup area 512 is a target area to which makeup is applied.

  The imaging unit 120 in FIG. 4 includes, for example, the above-described digital camera 102, and captures an image of the face of the user 300 (see FIG. 1). Then, the imaging unit 120 reverses the captured video in the left-right direction, and outputs the image to the face position acquisition unit 130, the stylus detection unit 140, and the makeup image superposition unit 150.

  The face position acquisition unit 130 sequentially acquires the position of the face included in the video from the input video. More specifically, the face position acquisition unit 130 performs pattern matching of an image feature of a face or a face part on each portion of an image frame constituting a video based on face feature information. Thereby, the face position acquisition unit 130 extracts the position of each face feature point included in the image frame from each image frame. Then, the face position acquisition unit 130 outputs information indicating the position of the extracted face feature point to the stylus detection unit 140 and the makeup image superposition unit 150.

  When one of the marker colors 511 (see FIG. 5) described in the makeup image table 510 is included in the input video, the stylus detection unit 140 detects the portion as a stylus from the video. As a detection method of the marker color part, for example, a detection method based on color histogram information described in Patent Document 2 can be adopted.

  The stylus detection unit 140 refers to the makeup area 512 (see FIG. 5) of the makeup image table 510 to determine whether a makeup operation (see FIG. 2) for bringing the stylus close to any target area has been performed. . Such determination is performed based on the position of the face feature point indicated by the input information and the position of the detected stylus. Then, the stylus detection unit 140 outputs information indicating the detected marker color to the makeup image superposition unit 150, on the condition that the makeup operation has been performed.

  The position of the face part and the position of the stylus are respectively the position of the image in the two-dimensional space, the position of the real world in the three-dimensional space, and the two-dimensional image of the image in the two-dimensional space Any of the positions in the three-dimensional space may be used.

  For example, when the digital camera 102 is a stereo camera, the face position acquisition unit 130 and the stylus detection unit 140 can acquire the three-dimensional position of a face part or a stylus by calculating the parallax between stereo images. Further, it is assumed that the average value of the size of the face and the average value of the size of the stylus in the image are obtained in advance. In this case, the face position acquisition unit 130 and the stylus detection unit 140 can acquire the three-dimensional position of the face part or the stylus by detecting the actual size of the face and the size of the stylus in the image.

  Under the condition that the stylus is detected, the makeup image superposition unit 150 superimposes a makeup image, which is associated in advance with the detected marker color, on the face portion of the input image. More specifically, when the information indicating the marker color is input, the makeup image superposition unit 150 refers to the makeup image table 510 (see FIG. 5) and acquires makeup image data corresponding to the marker color. The makeup image superimposing unit 150 generates a makeup image based on the acquired makeup image data, and superimposes the generated makeup image on the input video. Then, the makeup image superposition unit 150 outputs, to the display unit 160, an image on which the makeup image is appropriately superimposed.

  The display unit 160 has, for example, the liquid crystal display 101 (see FIG. 1), and displays an input image (hereinafter referred to as “virtual makeup image”) on which the makeup image is appropriately superimposed on the screen of the liquid crystal display. Do.

  Although not illustrated, the image processing apparatus 100 includes, for example, a central processing unit (CPU), a storage medium such as a ROM (Read Only Memory) storing a control program, and a working memory such as a RAM (Random Access Memory). In this case, the functions of the above-described units are realized by the CPU executing a control program.

  When the user 300 brings the stylus 200 close to the face, that is, when the makeup operation is performed, the image processing apparatus 100 having such a configuration detects this and the user 300 on which the corresponding makeup image is superimposed. You can display the image of the face of.

<Operation of device>
Next, the operation of the image processing apparatus 100 will be described.

  FIG. 6 is a flowchart showing an example of the operation of the image processing apparatus 100.

  In step S1100, the shooting unit 120 starts shooting a video. Such an image may include the face of the user 300 and the stylus 200. Note that the image processing apparatus 100 sequentially performs each of the subsequent processes for each image frame that constitutes a video.

  In step S1200, when the image includes a face, the face position acquisition unit 130 detects the position of the face feature point from the image.

  In step S1300, the stylus detection unit 140 detects a marker color from the image, and determines whether or not there is a marker color part in the image. The stylus detection unit 140 advances the process to step S1400 when the marker color portion is present in the image (S1300: YES). Further, when the marker color portion does not exist in the image (S1300: NO), the stylus detection unit 140 advances the process to step S1700 described later.

  In step S1400, the stylus detection unit 140 determines whether a makeup operation has been performed on the target area (makeup area 512, see FIG. 5) associated with the detected marker color. That is, the stylus detection unit 140 determines whether or not the detected marker color portion (stylus 200) is close to the corresponding target area. Note that proximity means that, for example, the distance between the position of the stylus 200 and the target area determined by the position of the face feature point is equal to or less than a predetermined value.

  If the makeup operation has been performed (S1400: YES), the stylus detection unit 140 advances the process to step S1500. Further, when the makeup operation is not performed (S1400: NO), the stylus detection unit 140 advances the process to S1700 which will be described later.

  The stylus detection unit 140 may limit the marker color to be detected for each target area based on the makeup image table 510 (see FIG. 5). In this case, steps S1300 and S1400 can be one process. In addition, when the face position acquisition unit 130 detects a face area, the stylus detection unit 140 may detect each marker color limited to the detected face area.

  Also, the user 300 may erroneously bring the stylus 200 close to the target area even though the user 300 does not intend to perform the makeup operation. Therefore, the stylus detection unit 140 may determine that the makeup operation has been performed on the condition that the state in which the stylus 200 is in proximity to the target area lasts for a predetermined time such as one second or more.

  In step S1500, the makeup image superposition unit 150 determines whether or not the makeup image associated with the detected marker color has been superimposed. The makeup image is an image generated by makeup image data (512 to 514, see FIG. 5) corresponding to the detected marker color. If the makeup image has not been superimposed (S1500: NO), makeup image superposition unit 150 advances the process to step S1600. In addition, when the makeup image has been superimposed (S1500: YES), the makeup image superposition unit 150 advances the process to step S1700 described later.

  In step S1600, the makeup image superposition unit 150 generates a makeup image associated with the detected marker color, and starts superposition on the photographed image. That is, the makeup image superposition unit 150 continues the superposition of the makeup image in the subsequent images. The continuation of the superposition of the makeup image may be, for example, regenerating the makeup image based on the latest position of the face feature point or deforming the generated makeup image so as to fit the latest position of the face feature point. It is done by

  It takes a few seconds or so from the user 300 starting using the stylus 200 to completing the desired makeup operation. Therefore, it is preferable that the makeup image superimposing unit 150 provide a time difference of about several seconds after detection of a predetermined marker color until start of superimposing the corresponding makeup image.

  In step S1700, the display unit 160 displays the generated virtual makeup image. The virtual makeup image includes not only an image on which a makeup image is superimposed but also an image on which a makeup image is not superimposed.

  In step S1800, makeup image superimposing unit 150 determines whether an instruction to end the process of generating and displaying a virtual makeup image has been issued by a user operation or the like. If the makeup image superimposing unit 150 is not instructed to finish the process (S1800: NO), the process returns to step S1200, and shifts to the process for the next image frame or the latest image frame. Further, when an instruction to end the process is given (S1800: YES), the makeup image superposition unit 150 ends the series of processes.

  By such an operation, the image processing apparatus 100 generates a virtual makeup image in which a makeup image is superimposed on the area when the user 300 performs a makeup operation to make the stylus 200 approach the target area such as a lip. It can be displayed.

  The difference in marker color (that is, the difference in the stylus 200) appears as a difference in the area and color of the makeup image in the virtual makeup image.

  FIG. 7 is a view showing an example of the difference between virtual makeup images by the stylus 200, and corresponds to FIG. 2C. The parts corresponding to those in FIG. 2C are assigned the same reference numerals and descriptions thereof will be omitted.

FIG. 7A is an example in the case of using a stylus whose marker color is red. FIG. 7B is an example in the case of using a stylus whose marker color is pink. FIG. 7C is an example in the case of using a stylus whose marker color is light blue. The marker colors of the styluses 200 ₁ to 200 ₃ shown in FIGS. 7A to 7C are red, pink, and light blue in order.

As shown in FIG. 7A, if the stylus 200 ₁ Red is used, the image processing apparatus 100, based on the makeup image table 510 (see FIG. 5), the lips partial superposing decorative image 412 ₁ red. Similarly, the image processing apparatus 100 includes, as shown in FIG. 7B, when the pink stylus 200 ₂ was used, to superimpose coral pink makeup image 412 ₂ lips portions. The image processing apparatus 100 includes, as shown in FIG. 7C, if a light blue stylus 200 ₃ were used, to superimpose the decorative image 412 ₃ blue gray lips portions.

  Thus, the image processing apparatus 100 causes different makeup images to be superimposed on the image of the user's face according to the marker color of the stylus 200.

<Effect of this embodiment>
As described above, the image processing apparatus 100 according to the present embodiment obtains the position of the face included in the video from the captured video, and when the video includes a portion of a predetermined marker color, the portion Is detected as a stylus. Then, the image processing apparatus 100 according to the present embodiment displays the captured video, but at that time, on the condition that the stylus is detected, a predetermined marker color is applied to the face portion of the video. The makeup image associated in advance is superimposed. Thus, the image processing apparatus 100 can perform virtual makeup by the makeup operation.

  Further, as shown in the makeup image table 510 of FIG. 5, the image processing apparatus 100 limits the makeup target area for each marker color. As a result, even if the stylus 200 approaches a portion other than the target area defined for each of the stylus 200, the image processing apparatus 100 does not superimpose the cosmetic image on the portion. Thus, the image processing apparatus 100 can avoid, for example, a situation where a lipstick is accidentally applied to the eyelid in a virtual makeup image.

<Modification 1: Another Example of Stylus Identification Method>
The identification method of the stylus 200 is not limited to the above-described example. For example, when the image processing apparatus 100 includes a portion of a predetermined color arrangement pattern of a plurality of predetermined colors, the portion may be detected as the above-described marker color from the image. In this case, the color arrangement pattern can be used as the identification code of the stylus 200, and the detection accuracy of the stylus 200 can be improved, and the number of the stylus 200 identified can be increased.

  FIG. 8 is a view showing an example of the appearance of a stylus having a color arrangement pattern, which corresponds to FIG.

As shown in FIG. 8, the stylus 200a are each of the first to fourth regions ₂₁₀ 1-210 ₄ which is continuous from the distal end, the color is applied. Between the first to fourth regions ₂₁₀ 1-210 _4, the color to be applied may be different or may be identical. However, between the first to fourth regions ₂₁₀ 1-210 least adjacent region 210 of the _four, it is desirable that the color is different. The color that approximates to the first to at least in part of the fourth region 210 _{1-210 4} cosmetic colors associated with these colors arrangement pattern, it is desirable that coated.

The combination and order of the color which has been subjected to the first to fourth regions ₂₁₀ 1-210 ₄ constitutes a color arrangement pattern applied to the stylus 200a. The number of areas constituting the color arrangement pattern is not limited to four.

  FIG. 9 is a view showing an example of the contents of the makeup image table when the color arrangement pattern is used as a marker color, and corresponds to FIG. The parts corresponding to those in FIG. 5 are given the same reference numerals, and the description thereof is omitted.

  As shown in FIG. 9, the makeup image table 510a describes the makeup area 512, the makeup color 513, and the makeup density 514 in association with the marker color 511a which is a color arrangement pattern. Here, the marker color 511a is represented by a picture of a color arrangement pattern here, but is actually information in which the range of coordinate values in a predetermined color space is described according to the arrangement order.

For example, the stylus detection unit 140 first searches the image for a region of a color corresponding to the first region 210 ₁ (see FIG. 8), and when the region is detected, the second detection is performed around the region. searching for a color region of which corresponds to the area 210 _2. If the area of the color corresponding to the second region 210 ₂ is detected, the stylus detection unit 140 is further in the peripheral region of the detected direction, a third region 210 _3, and the fourth region 210 ₄ Search in order.

  When the stylus detection unit 140 can distinguish and detect N types of colors, the stylus 200 of the marker color consisting of a single color shown in FIG. 3 can identify the N styluses 200, and N types of makeup images are displayed. It can be used.

On the other hand, in the stylus 200a using the color arrangement pattern of four regions as shown in FIG. 8 as the marker color, for example, N ⁴ styluses 200 can be identified, and N ⁴ types of makeup images can be used. it can. That is, by adopting color arrangement patterns of a plurality of colors as marker colors, the image processing apparatus 100 can superimpose various types of makeup images, and can broaden the variation of virtual makeup.

  Furthermore, when the marker color is a color arrangement pattern of a plurality of colors, the possibility that a portion other than the stylus 200 is erroneously detected as the stylus 200 can be reduced as compared with a case where the marker color is a single color. On the other hand, in the case where the marker color is a color arrangement pattern of a plurality of colors, a part of the marker color is concealed by fingers, etc., and the possibility of a detection failure becomes high.

Therefore, it is preferable that the stylus detection unit 140 track the stylus 200a detected in a certain image frame in the subsequent image. Such tracking is performed, for example, by tracking an image feature such as the color and shape of the tip of the stylus 200a using a known target tracking technique for an image. In this case, it is desirable that the color of the area at the tip end of the stalas 200 a (first area 210 ₁ , see FIG. 8) be a predetermined single color.

<Modification 2: Another example of color determination method>
Moreover, the determination method of the color (make-up color) of a makeup image is not limited to the above-mentioned example. The image processing apparatus 100 may determine the color of the makeup image based on, for example, the color applied to the stylus separately from the marker color.

  FIG. 10 is a view showing an example of the appearance of a stylus having a color other than the marker color, corresponding to FIGS. 3 and 8.

As shown in FIG. 10, the stylus 200b are each of the fifth and sixth regions ₂₁₀ 5, 210 ₆ continuous from the distal end, the color is applied. The color of area 210 ₆ The sixth is a color selected from one or more predetermined marker colors.

Colors to be applied to the fifth region 210 _5, it is desirable that colors other than the predetermined marker colors. Incidentally, the fifth region 210 _5, for example, paint applied to the rod-shaped member, affixed seal the rod-like member, or can be a colored member constituting the rod-like member. In addition, it is desirable that the marker color is not frequently used for makeup, such as green, and is a color different from the color of the face.

When using such a stylus 200b, the stylus detection unit 140 specifies a color other than the color of the face included in the peripheral region from the peripheral region of the detected predetermined marker color portion in the image, as the application designation color As detected. That is, the stylus detection unit 140, the color of the fifth region 210 ₅ is detected as a coating designated color. Then, the makeup image superimposing unit 150 generates a makeup image with a color that approximates the detected application designation color.

  The makeup image table 510 (see FIG. 5) does not need to be described for the makeup color 513. Furthermore, when the marker color is one color and the target area is not limited to each stylus 200b, the makeup image table in which the content of the makeup image is previously described for each marker color is not necessarily required. Further, the image processing apparatus 100 may use a makeup image table in which marker colors and makeup image data other than makeup color are described for each target area, instead of the table.

  FIG. 11 is a flowchart showing an example of the operation of the image processing apparatus 100 when using the stylus 200b coated with the marker color and the application designation color, and corresponds to FIG. The same parts as those in FIG. 6 are given the same reference numerals, and the description thereof is omitted.

  If it is determined that the marker color part is present in the image (S1300: YES), the stylus detection unit 140 advances the process to step S1310b.

  In step S1310b, the stylus detection unit 140 detects colors other than the face color in the peripheral region of the marker color portion, and determines whether or not there is a color portion other than the face color. More specifically, for example, when the face position acquisition unit 130 detects a face area, the stylus detection unit 140 detects skin, lips, eyes, eyebrows, and eyebrows based on the color histogram information of the detected face area. Estimate the color of each part of the face including the part. Then, the stylus detection unit 140 detects the color of the portion different from the estimated color and having a predetermined area as a color other than the face color.

For example, in the case where the image includes the fifth and sixth regions 210 ₅ and 210 ₆ of the stylus 200 b shown in FIG. 10, the stylus detection unit 140 sets the fifth region 210 ₆ to a fifth color. detecting color areas 210 _5.

  The stylus detection unit 140 advances the process to step S1320b when there is a color portion other than the face color in the peripheral region of the marker color portion (S1310b: YES). Furthermore, if there is no color other than the face color in the peripheral area of the marker color (S1310b: NO), the stylus detection unit 140 advances the process to step S1410b described later.

  In step S1320b, the stylus detection unit 140 acquires the color detected in step S1310b as the application designation color.

  The stylus detection unit 140 may track the marker color part in the image, and determine that the color that moves in the same manner as the marker color is a color other than the face color described above.

  In step S1410b, the stylus detection unit 140 determines whether the application designation color has been acquired and whether a makeup operation has been performed on any target region. That is, the stylus detection unit 140 determines whether or not the detected marker color portion (stylus 200b) is close to any of the predetermined target areas.

  The stylus detection unit 140 advances the process to step S1510b when the application designation color has been acquired and the makeup operation on the target area has been performed (S1410b: YES). At this time, the stylus detection unit 140 outputs information indicating the application designation color and the target region targeted for the makeup operation to the makeup image superposition unit 150 instead of the information indicating the marker color described above. The stylus detection unit 140 advances the process to step S1700 if the application designation color is not obtained or if the makeup operation on the target area is not performed (S1410b: NO).

  In step S1510b, the makeup image superposition unit 150 determines, based on the input information, whether or not a makeup image of a color close to the application designated color has already been superimposed on the target area. When the makeup image of the color approximating the application designated color is superimposed on the target area (S1510b: YES), the makeup image superposition unit 150 advances the process to step S1700. In addition, when the makeup image of the color approximating the application designated color is not superimposed on the target area (S1510b: NO), the makeup image superposition unit 150 advances the process to Step S1610b.

  In step S1610b, the makeup image superposition unit 150 generates a makeup image of a color approximating the application designated color, starts superposition on the photographed image, and advances the process to step S1700.

  More specifically, the makeup image superimposing unit 150, for example, stores a plurality of divided color spaces obtained by dividing the color space, stored in advance in the information storage unit 110, and a representative color such as an average color in each divided color space. Refer to the information shown. Then, the makeup image superimposing unit 150 specifies the divided color space to which the application designation color belongs, and determines the representative color of the specified divided color space as the makeup color of the makeup image.

  In addition, the makeup image superposition unit 150 may adopt the acquired application designation color as it is as a makeup color, or may make a color obtained by performing predetermined color conversion on the acquired application designation color as makeup. It may be adopted as a color. Furthermore, the makeup image superimposing unit 150 may acquire ambient environment information such as the illumination state, and determine the content of color conversion based on the acquired information.

  FIG. 12 is a view showing an example of the difference in virtual makeup image depending on the application designation color of the stylus 200b, corresponding to FIG. 2C and FIG. The parts corresponding to those in FIG. 2C and FIG. 7 are assigned the same reference numerals and descriptions thereof will be omitted.

As shown in FIG 12A~ Figure 12C, by using each of the stylus 200b ₁ ~200b ₃ different coating designated color, respectively is applied, the cosmetic operation is performed on the lips. In this case, the image processing apparatus 100, the color makeup image 412b ₁ ~412b ₃ of approximating the coating designated color of each stylus 200b, is superimposed on the lip area as an object cosmetic operations.

  When the cosmetic color is determined based on the marker color, the type of cosmetic image is limited according to the number of distinguishable marker colors (color arrangement patterns), and the correspondence between the marker color and the cosmetic color is registered in advance. Need to be

  On the other hand, when the stylus 200b as shown in FIG. 10 is detected based on the marker color and the application designation color is acquired from the detected stylus 200b, there is no limitation described above, and the correspondence relationship between the marker color and the makeup color There is no need to set in advance. That is, by adopting the combination of the marker color and the application designation color, the image processing apparatus 100 can generate a makeup image rich in color flexibility, and can extend the variation of virtual makeup.

<Other Modifications>
Further, the association between the marker color (color arrangement pattern) and the makeup image performed by the image processing apparatus 100 is not limited to the above-described example. For example, the makeup area 512 and the makeup density 514 may not necessarily be limited to each marker color. As a result, the user 300 can perform virtual makeup on a plurality of or all target areas with one stylus 200.

  Further, the shape of the stylus 200 used in the image processing apparatus 100, the arrangement of the marker color and the application designation color in the stylus 200, and the configuration of the color arrangement pattern are not limited to the above-described example. For example, the stylus 200 may have a shape or material closer to various cosmetics, and may have, for example, a flat brush shape or material.

  The image processing apparatus 100 may also receive an input of a marker color from the user 300. In this case, the stylus detection unit 140 displays, for example, a stylus registration screen for acquiring a portion of the stylus 200 from the photographed image on the liquid crystal display 101 prior to the implementation of virtual makeup. Then, the stylus detection unit 140 acquires the color of the portion of the stylus 200 acquired via the stylus registration screen as the marker color attached to the stylus 200.

  FIG. 13 is a diagram showing an example of a stylus registration screen.

  As shown in FIG. 13, the stylus registration screen 610 includes, for example, a position designation image 612 for designating the position of the stylus 200 in the photographed image 611 of the digital camera 102. The stylus registration screen 610 also includes a message 613 instructing the user 300 to hold the stylus 200 so that the position of the stylus 200 matches the position designation image 612 in the captured image 611.

  The stylus detection unit 140 detects whether or not the stylus 200 is aligned with the position designation image 612 by detecting from the captured image 611 the outline of the stylus 200 and the code pattern attached to the stylus 200 registered in advance. . Then, when the stylus 200 is adjusted to the position designation image 612, the stylus detection unit 140 acquires the color of the portion to which the marker color is applied (for example, the tip portion of the stylus 200) from the captured image 611, and the acquired color Is acquired as a marker color.

  In addition, the makeup image superimposing unit 150 may use the acquired color as it is as the makeup color, or may use the color previously associated with the acquired color as the makeup color.

  In addition, information other than color information may be used in combination for identification of the stylus 200 of the image processing apparatus 100. For example, when the stylus 200 has a RFID (Radio Frequency Identification) tag mounted on the stylus 200, the stylus detection unit 140 uses the stylus 200 based on the combination of the information indicated by the wireless signal from the RFID tag and the color information of the stylus 200. May be identified.

  Moreover, the superposition of the makeup image performed by the image processing apparatus 100 may not be limited to the above-described target area. That is, the image processing apparatus 100 may recognize each part of the entire face as a target of the makeup operation.

  In this case, when it is desired to give the user 300 a feeling closer to actual makeup, the stylus detection unit 140 detects whether or not the user 300 has touched the stylus 200 to the surface of the face, and a portion where the stylus 200 contacts. It is desirable to determine as a makeup area. Such detection can detect, for example, the three-dimensional position of each part of the surface of the face and the three-dimensional position of the tip of the stylus 200 with high accuracy, or sensor information of a pressure sensor mounted on the tip of the stylus 200 It can be performed by receiving wirelessly.

  Alternatively, the stylus detection unit 140 further analyzes the movement of the stylus 200 (the region of the marker color) in the vicinity of the target region, and the more detailed positional relationship between the target region and the stylus 200, and performs the makeup operation on the target region. It may be determined whether or not it has been received. Specifically, for example, the stylus detection unit 140 registers a predetermined movement pattern (gesture) of the stylus 200 in advance as the makeup operation for each target area, and detects the gesture to perform the makeup operation. Make a decision.

  The gesture can be defined, for example, using a plurality of reference points set on the face.

  FIG. 14 is a diagram for explaining the definition of a gesture.

  As shown in FIG. 14, in the face 620, a plurality of reference points (621) A to F and X to Z are defined. The position of the reference point is determined based on the face feature point. In addition, one or more reference points are defined in each of the makeup target areas.

  For example, the information storage unit 110 stores in advance, for each target area, a gesture table in which a temporal change in relative positional relationship between one or more reference points and the stylus 200 is described as a makeup operation for the target area. ing.

  For example, the makeup operation of the upper lip may be performed by the stylus 200 in the order of the reference point A, the reference point B (or the midpoint between the reference point B and the reference point D), the reference point C, or the reverse order. Movement of the stylus 200 (a gesture of the user 300) as it approaches the Also, the makeup operation of the lower lip may be performed by the stylus 200 in the order of the reference point A, the reference point F (or the midpoint between the reference point F and the reference point E), the reference point C, or the reverse order. Movement of the stylus 200 as it approaches a point.

  Also, the makeup operation of the right cheek is a movement of the stylus 200 such that the stylus 200 rotates several times around the reference point Z. The makeup operation of the foundation on the entire face is a movement of the stylus 200 such that the stylus 200 approaches these reference points in the order of the reference point X, the reference point Y, the reference point Z, or the reverse order.

  Whether the makeup operation has been performed on each target area by comparing the time change of the position of the stylus 200 detected from the image with the content of each makeup operation described in the gesture table. Determine

  On the contrary, the image processing apparatus 100 is a photographed image of a makeup image that is associated in advance with the type of the stylus 200, on condition that the stylus 200 has just approached the face or that the stylus 200 has simply been detected from the image. You may start superposition on.

  Moreover, the content of the makeup image which the image processing apparatus 100 changes by another of the stylus 200 is not limited to the above-mentioned example. That is, the content of the makeup image described by the makeup image table 510 (see FIG. 5) is, for each marker color (color arrangement pattern), continuation of the makeup image corresponding to the image effect such as the gloss of makeup and holding of the makeup agent. The time may be different.

  Further, the information storage unit 110 may access a predetermined server or the like on the Internet to acquire or update the contents of the makeup image table 510 (see FIG. 5). The information storage unit 110 performs such acquisition or update of the table periodically or at timing when an instruction from a user or a server is received. Thus, the image processing apparatus 100 can flexibly cope with the increase and change of the stylus 200, and can provide the user 300 with virtual makeup reflecting the fashion trend.

  Moreover, the stylus detection part 140 may detect the part of this color as an eraser stylus, when the color predetermined as a color for wiping off makeup is contained in imaging | video. Then, the stylus detection unit 140 may end the superposition of the makeup image on condition that the operation of causing the eraser stylus to approach the portion on which the makeup image is superimposed is performed.

  In addition, the concept of makeup targeted by the image processing apparatus 100 includes application of foundation to the entire face, drawing on the face, wearing of glasses, wearing of color contacts, and hair dyeing of eyebrows and hair, etc. Good. In any case, it is desirable that the stylus imitate a tool used for the corresponding operation, such as a glasses-type plate or a brush-type tool.

  Further, the device to which the image processing device 100 is applied is not limited to the above-described example. The image processing apparatus 100 may be applied to, for example, a smartphone, a tablet terminal, a personal computer, or a digital camera.

  In addition, a part of the configuration of the image processing apparatus 100 may be disposed on a server or the like on the network so as to be separated from the other parts of the configuration of the image processing apparatus 100. In this case, each of these parts needs to be provided with a communication unit for communicating with each other.

<Summary of this disclosure>
An image processing apparatus according to an embodiment of the present disclosure includes an imaging unit configured to capture an image, a display unit configured to display the captured image on a screen, and a face position acquisition unit configured to acquire the position of a face included in the image from the image. When a portion of a predetermined marker color is included in the image, a stylus detection unit that detects the portion as a stylus from the image, and the face portion of the image on condition that the stylus is detected And a makeup image superposition unit for superposing a makeup image associated with the predetermined marker color in advance.

  In the image processing apparatus, the stylus detection unit acquires the position of the stylus, and based on the acquired position of the face and the position of the stylus, the user brings the stylus close to the face. The makeup image superimposing unit may superimpose the makeup image on condition that the makeup operation has been performed.

  Further, in the image processing apparatus, the makeup image is an image representing a state of the makeup when makeup is applied to an area predetermined based on a face feature point, and the face position acquisition unit The position of the face feature point of the face is acquired, and the makeup image superposing unit performs the makeup operation for bringing the stylus close to the area based on the acquired position of the face feature point and the position of the stylus. And the makeup image may be superimposed.

  In the image processing apparatus, the stylus detection unit detects the portion of the predetermined marker color from the image, detects the position of the stylus based on the detected portion, and detects the detected position of the stylus. May be tracked in the video.

  In the image processing apparatus, the stylus detection unit applies a color other than the color of the face included in the peripheral area from the peripheral area of the part of the detected predetermined marker color in the image. The makeup image may be generated as a color that is detected as a designated color, and the makeup image superposition unit approximates the detected application designation color.

  In the image processing apparatus, the makeup image superposition unit superimposes a plurality of predetermined marker colors with reference to a makeup information table in which the contents of the makeup image are previously described for each of the predetermined marker colors. The makeup image may be determined, and the contents of the makeup image described by the makeup table may be different in at least one of the color, the density, and the area of the makeup image for each of the predetermined marker colors.

  In the image processing apparatus, the predetermined marker color may be a predetermined color arrangement pattern including a plurality of predetermined colors.

  The stylus of the present disclosure is a stylus used in the image processing apparatus, and is a rod-like member, and the predetermined marker color is applied to a region including the tip of the rod-like member.

  The image processing method according to the present disclosure includes a step of shooting a video, a step of displaying the shot video on a screen, a step of acquiring the position of a face included in the video from the video, and Detecting the portion as a stylus from the image when the portion of the marker color is included, and the predetermined marker color for the portion of the face of the image on condition that the stylus is detected And V. superimposing a makeup image associated with in advance.

  The present invention is useful as an image processing apparatus, a stylus, and an image processing method capable of performing virtual makeup by a makeup operation.

100 image processing apparatus 101 liquid crystal display 102 digital camera 110 information storage unit 120 imaging unit 130 face position acquisition unit 140 stylus detection unit 150 makeup image superposition unit 160 display unit 200, 200a, 200b stylus

Claims (9)

A shooting unit for shooting a video;
A display unit for displaying the photographed image on a screen;
A face position acquisition unit for acquiring the position of the face included in the video from the video;
A stylus detection unit configured to detect the portion as a stylus from the image when the image includes a portion of a predetermined marker color;
And a cosmetic image superimposing unit that superimposes a cosmetic image, which is previously associated with the predetermined marker color, on the face portion of the image, on condition that the stylus is detected.
Image processing device. The stylus detection unit
The position of the stylus is acquired, and based on the acquired position of the face and the position of the stylus, it is determined whether the user has performed a makeup operation to bring the stylus close to the face.
The makeup image superposition unit is
Superimposing the makeup image on condition that the makeup operation has been performed,
The image processing apparatus according to claim 1. The makeup image is an image representing the state of the makeup when makeup is applied to an area predetermined based on a face feature point,
The face position acquisition unit
Get the position of the face feature point of the face,
The makeup image superposition unit is
The makeup image is superimposed on the condition that a makeup operation for bringing the stylus close to the area is performed based on the acquired position of the face feature point and the position of the stylus.
The image processing apparatus according to claim 1. The stylus detection unit
Detecting a portion of the predetermined marker color from the image, detecting a position of the stylus based on the detected portion, and tracking the position of the detected stylus in the image;
The image processing apparatus according to claim 1. The stylus detection unit
The color other than the color of the face included in the peripheral area is detected as the application designation color from the peripheral area of the detected part of the predetermined marker color in the video.
The makeup image superposition unit is
Generating the makeup image with a color approximating the detected application color;
The image processing apparatus according to claim 1. The makeup image superposition unit is
For the plurality of predetermined marker colors, the makeup image to be superimposed is determined with reference to a makeup information table in which the contents of the makeup image are described in advance for each of the predetermined marker colors,
The content of the makeup image described by the makeup table differs in at least one of the color, the density, and the area of the makeup image for each of the predetermined marker colors.
The image processing apparatus according to claim 1. The predetermined marker color is a predetermined color arrangement pattern consisting of a plurality of predetermined colors.
The image processing apparatus according to claim 1. A stylus used in the image processing apparatus according to claim 1, wherein the stylus is
A rod-like member, wherein the predetermined marker color is applied to a region including the tip of the rod-like member;
stylus. Step of taking a picture,
Displaying the photographed image on a screen;
Acquiring the position of the face included in the video from the video;
Detecting the portion as a stylus from the image when the image includes a portion of a predetermined marker color;
Superimposing a makeup image previously associated with the predetermined marker color on the face portion of the image, on condition that the stylus is detected.
Image processing method.

Images