JP5203271B2 - Display information forming apparatus and system - Google Patents

Description

  The present invention relates to a display information forming apparatus and system, and more particularly, to a display information forming apparatus and system capable of controlling information displayed on a display unit by estimating a spatial positional relationship between an imaging unit and an imaging target. is there.

  In recent years, nail art has become widespread that allows you to enjoy fashionable fashion with glazedly decorated artificial nails.

  Nail art is not only skillful, but it is also difficult to decorate the nails of the dominant hand, so it is generally decorated at a nail art salon. Nail art design samples are presented in various magazines and on the Internet, so you will bring them to the salon and order them, but you won't know until you try them out. Moreover, there is a problem that it takes cost and trouble to go to the salon. Against this background, several methods for casually enjoying nail art have been proposed.

  In Patent Document 1, an apparatus for printing nail art on nails using an inkjet printer has been developed.

  In Patent Document 2, the design can be easily switched by attaching a display using electronic paper technology to a nail with an adhesive.

  In Patent Document 3, a nail art design image can be acquired from a server with a mobile terminal, and in Patent Document 4, a nail art design can be edited.

  In Patent Document 5, it is possible to actually test with a nail by cutting and pasting a nail art sheet.

Japanese Patent Laid-Open No. 2002-165632 “Nail Art Printer” Japanese Patent Application Laid-Open No. 2008-168062 “Electronic Nail and Electronic Nail Changing Device” JP 2008-165316 A "Nail Art Content Display Device and Nail Art Content Display System" Japanese Patent Application Laid-Open No. 2008-130054 “Nail Art Game Software Program and Method for Providing Game Software Using a Mobile Site” Japanese Patent Application Laid-Open No. 2009-11863 “Simple Nail Art Sheet”

  The above prior art has the following problems.

  In Patent Document 1, if the printer and the nail are misaligned, it cannot be redone. Also, since it takes time until the nail art ink dries, it is difficult to change the ink in a short time.

  In Patent Document 2, a thick display equipped with an electronic circuit must be attached to a nail, so that it is not suitable for nail art that places importance on design.

  In Patent Literature 3 and Patent Literature 4, although the design of nail art can be selected or edited, the design image cannot be actually tried with his / her finger.

  In Patent Document 5, it takes time and effort to cut and paste a sheet according to the shape of the nail.

  In view of the above problems, the present invention provides a display information forming apparatus and system capable of displaying a video in which an arbitrary image is superimposed on an image captured by an imaging unit of an information terminal on a display unit of the information terminal. For the purpose. For example, an object of the present invention is to provide a display information forming apparatus and system capable of displaying an image as if nail art is applied on a display unit of an information terminal by superimposing a nail art design on a nail. .

In order to achieve the above-described object, the present invention provides a light source unit that irradiates a fingernail portion with a light amount, an imaging unit that outputs a finger region image photographed at regular time intervals, and information displayed on the nail unit. And a finger region image output from the imaging unit , and based on the light amount difference and edge detection obtained by the analysis, the position , orientation, and root (base) of the nail unit are determined. an estimation unit for at least estimating, the read information to be displayed on the claw portion from the storage unit, the position of the claw portion estimated by the display information read the said estimation unit, and adapted to the orientation and root, below the display There is a feature in that a control unit that creates information to be displayed on the unit and a display unit that displays at least the image output from the imaging unit and the information created by the control unit are provided.

  Further, according to the present invention, the estimation unit includes a difference calculation unit that extracts a specific region from the image output from the imaging unit, a region formation unit that shapes the region extracted by the difference calculation unit, and the region There is another feature in that it includes posture estimation means for estimating the position, orientation, and number of target areas from the area shaped by the forming means.

Further, the present invention is created by a light source unit that irradiates a fingernail unit with a light amount, an imaging unit that outputs a finger region image captured at a predetermined time interval, an image output from the imaging unit, and a control unit described later. an information terminal device having at least a display unit for displaying at least the information, to analyze the output finger region image from the imaging unit, based on the light amount difference and the edge detection obtained by the analysis, the pawl portion An estimation unit that at least estimates the position , orientation, and root (base) of the device, a storage unit that holds information to be displayed on the nail unit , and information to be displayed on the nail unit is read from the storage unit, and the read estimated position by said estimating unit to display information, be adapted to the orientation and fundamentally consists of at least comprises a server and a control unit for creating information to be displayed on the display unit, in the information terminal device Transmitting the shadow image to the server, there are other features that create information to be displayed on the display unit on the server.

  According to the present invention, at least the position and orientation of the shooting target (subject) are estimated from the input image of the information terminal without using special hardware such as an acceleration sensor or a special device, and are stored in the storage unit. It is possible to superimpose existing information on a specific portion of the input video to be photographed.

  In addition, the user can easily enjoy watching an image superimposed on a specific portion of the input video. In addition, since an arbitrary image stored in the storage unit can be selected as the image to be superimposed on the specific portion, it can be enjoyed with various images and can be enjoyed for a long time. As a specific example, by superimposing a nail art design on the nail according to the relative position of the fingertip with respect to the information terminal, it is possible to display an image as if the nail art was applied on the display unit of the information terminal, and the user Can provide great fun.

It is a functional block diagram which shows the structure of the outline of one Embodiment of this invention. It is a flowchart which shows the rough process sequence of this embodiment. It is a schematic diagram which shows a mode that a finger | toe and a nail | claw are image | photographed using an information terminal device. It is a figure which shows the example of distribution of the normal vector of the nail | claw surface. It is a figure which shows the finger | toe and nail | claw as a to-be-photographed object. It is explanatory drawing of the example which calculates | requires the position, direction, and edge of a nail | claw. It is a figure which shows the example which superimposed information on the nail | claw.

  Hereinafter, the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing a schematic configuration of an embodiment of the present invention.

  A display information forming device 10 (information terminal device) of the present embodiment includes an imaging unit 1, a light source unit 2, an estimation unit 3, a control unit 4, a storage unit 5, and a display unit 6. In the following, an example in which a nail is used as a subject and a nail tip is used as an image to be superimposed will be described, but the present invention is not limited to this, and an image can be superimposed on the surface of an object other than a nail. In the present embodiment, the present invention will be described by taking the application to a mobile phone as an example. However, any information terminal including an imaging unit and a light source can be similarly applied to a PDA, a computer, or the like.

  The imaging unit 1 continuously shoots a subject at a predetermined sampling period and outputs the captured image. As the imaging unit 1, a digital camera provided as a standard in a mobile phone can be used. In this embodiment, the subject is a hand, and in particular, a photograph is taken so that a fingernail is captured.

  The estimation unit 3 estimates the position, orientation, number, etc. of the captured nails. The estimation unit 3 performs the difference calculation process (S2), area formation process (S3), and posture estimation process (S4) in FIG. The control unit 4 processes the information to be superimposed based on the estimation result of the input image, and controls the display content of the display unit 6. A plurality of nail chip information is accumulated in the storage unit 5 in advance, and an arbitrary nail chip can be superimposed on the nail according to the user's selection. Alternatively, a new nail chip can be obtained or updated via the network.

  FIG. 2 shows the processing procedure of this embodiment. In the image capturing process (S1), first, the image capturing unit 1 captures an image when the light amount is increased in conjunction with the light source 2. Subsequently, the light source is turned off or the amount of light is reduced to capture an image when the amount of light is reduced. At this time, the two times of photographing are continuously performed to minimize positional deviation due to camera shake, and two images having the same coordinate correspondence are sent to the difference calculation process (S2). FIG. 3 shows an example of a state where a nail is photographed by the display information forming device 10 (information terminal device).

  In the difference calculation process (S2), a region having a large change amount is extracted from the two images. Since the range in which the light reaches is limited to the subject close to the imaging unit 1, the background region can be excluded by comparing the two images. In addition, reflected light from an adjacent subject increases in an image when the amount of light is large, but the nail often obtains a larger luminance value due to the difference in reflectance between the skin of the finger and the nail. Therefore, the difference calculation process (S2) calculates a difference from an image when the light amount is small from an image when the light amount is large, and an area where a large difference value equal to or larger than a preset threshold value is generated is a nail region candidate. Set.

  Alternatively, as another embodiment, the influence of the light source can be highlighted by color space conversion. That is, the two input images are converted from the RGB color space to the HSV color space (brightness V, saturation S, hue H), respectively.

  In the nail region, since the blood color is darker than the skin region, the R (red) value is large. In addition, since the G (green) value of the nail region is larger than the B (blue) value under a general fluorescent lamp, the hue H of the HSV color space takes a positive value. Here, when the light of a cellular phone is illuminated as a light source, since pale light is strongly reflected in the nail region, the magnitude relationship between G and B is often reversed in the nail region. On the other hand, since the reflectance is not relatively high in the skin region, the reverse is unlikely to occur. Therefore, since only the hue H of the nail region tends to be a negative value, the negative number is set to a positive number by adding 360, and then a pixel having a large difference between them is determined as a nail region candidate. The reason why 360 is added is that, as is well known, the hue H is generally expressed as a positive number from 0 to 360 °. For example, the hue H when the light is turned on is about 330 (value after adding 360) for the nail and about 10 for the skin, and the hue H when the light is turned off is about 10 for the nail and about 10 for the skin. 30. Therefore, the difference between the two when the light is turned on and off is significantly larger for the nail than for the skin, so that the nail and the skin can be clearly distinguished. As described above, in determining whether there is a change in the amount of light, the color component of the light emitted from the light source unit 2 is taken into consideration, and after changing the color space in which the amount of change appears to be large, the image is illuminated by the light source unit ON And the difference between the image without irradiation due to OFF can be calculated.

  Note that the color of the nail itself is easily affected by individual differences, health conditions, illumination light, etc., but since the active light quantity change due to the known light source color causes a clear difference in the nail area, the difference is calculated. By doing so, it is possible to suppress the influence of the above individual differences and the like. In addition, by setting a small color component for the subject as a light component of the light source unit, it is possible to easily emphasize the amount of change in the target region having a high reflectance.

  In the area formation process (S3), the nail area candidates are shaped. First, a partial defect of the nail region candidate is compensated by an existing morphological filter. Next, the finger region and the background region are separated by skin color detection. Any method can be applied to skin color detection. As an example, the skin color and the non-skin color are learned by a Gaussian mixture model (GMM), and are determined by comparing the skin probability of the input pixel with the non-skin probability. Finally, the edge area is calculated from the finger area image by Canny edge detection, and the area is grown from the nail area candidates so as not to exceed the edge area. Specifically, when the difference between the adjacent pixel value and the pixel value is within a certain range and the distance between the nearest edge region pixel and the pixel is greater than or equal to a certain distance, the adjacent pixel is incorporated into the nail region candidate.

  The posture estimation process (S4) estimates the direction of the nail, the width of the nail, and the base (base) coordinates of the nail as information necessary for superimposing the nail tip. First, in order to estimate the width of the nail, the direction of the nail is estimated. Since the maximum luminance value in an image with a large amount of light often occurs inside a nail region having a high reflectivity, a small region composed of the maximum luminance value and its neighboring colors is generated. As shown in FIG. 4, the nail surface is configured with substantially the same normal toward the fingertip. Therefore, as shown in FIG. 5, the shape of the reflected light is a region 7 long in the fingertip direction. (The small area). The region 7 is also a region where the light source is reflected.

  Therefore, as shown in FIG. 6, the center of gravity of the region and the two axes of the circumscribed rectangle are calculated for each nail region candidate formed in the region, and the circumscribed rectangle 12 of the small region 11 (for example, the reflected light region 7) is calculated. The length of the axis of the minor axis 13 or the axis close to the minor axis direction is determined as the width of the nail. Next, in order to estimate the coordinates of the base (base) 15 of the nail, the edge distribution by the finger is used. The screen is divided into two by the short axis 13 of the high-luminance small area, and the area where there are many edges is determined as the direction of the nail root 15 based on the edge detection result calculated in the area forming process (S3). A boundary where the skin area calculated in the area forming process (S3) intersects with a line obtained by extending the axis of the nail candidate area in the root direction is defined as a nail root coordinate.

  In the information superimposing process (S5), information to be superimposed on the nail is acquired from the storage unit 5, and the information is deformed according to the nail width calculated in the posture estimation process (S4). Next, as shown in FIG. 7, the superposed position is adjusted so that the deformed superposition information matches the base coordinates of the nail calculated in the posture estimation process (S4).

  In another embodiment applied to video, the estimation unit 3 applies the series of processes only once at the first time. For the input image that follows, by keeping track of the nail root coordinates and the both-end coordinates of the axis indicating the nail width for each nail region by motion search, information on the video is reduced while reducing the load of the estimation process. It is possible to continue superimposing. For example, if an image of a finger with movement is input as the subsequent input image, an image is placed on the fingernail with movement, so it is possible to test how the superimposed information looks dynamically. it can.

  Further, by superimposing information unique to CG such as 3D or animation in addition to the nail chip of a still image, it is possible to provide an expression that could not be realized with existing nail art. Alternatively, interactive art can be realized by changing the information to be superimposed according to the direction, size, and coordinates of the target area.

  The above embodiment is an example in which all of the processing of FIG. 2 is performed inside the display information forming apparatus 10 (information terminal apparatus), but only the image photographing (S1) and display are performed by the information terminal apparatus. You may perform a process (S2-S5) with a server. That is, the captured image information is sent from the information terminal device to a server connected to a network (such as the Internet) by wire or wirelessly, and the server side performs the difference calculation process (S2), the area formation process (S3), and the attitude estimation process ( The information superimposing process (S5) and the information superimposing process (S5) may be performed and the information superimposing process result may be received by the information terminal device and displayed on the display unit 6. In other words, the estimation unit 3, the control unit 4, and the storage unit 5 of FIG. 1 may be placed on the server side to perform the above-described processing, and the processing result may be sent back to the information terminal device.

  DESCRIPTION OF SYMBOLS 1 ... Imaging part, 2 ... Light source, 3 ... Estimation part, 4 ... Control part, 5 ... Memory | storage part, 6 ... Display part, 7 ... Reflected light area, 10 ... display information forming device (information terminal device), 11 ... small region, 12 ... circumscribed rectangle 12, 13 ... short axis, 15 ... base of nail (base).

Claims (12)

A light source for irradiating the fingernail with light,
An imaging unit that outputs a finger region image captured at regular time intervals;
A storage unit for holding information to be displayed on the nail unit;
Analyzing the finger region image output from the imaging unit , and based on the light amount difference and edge detection obtained by the analysis, an estimation unit that estimates at least the position , orientation, and root (base) of the nail unit ;
Read the information to be displayed on the claw portion from the storage unit, the position of the claw portion estimated by the display information read the said estimation unit, and adapted to the orientation and fundamentally, generates information to be displayed on the later display unit A control unit ,
A display information forming apparatus comprising: a display unit configured to display at least an image output from the imaging unit and information generated by the control unit .   The display information forming apparatus according to claim 1, wherein the imaging unit includes means for continuously acquiring a plurality of images having different light amounts in conjunction with the light source unit. The estimation unit includes
Difference calculating means for extracting a specific region from the image output from the imaging unit;
Area forming means for shaping the area extracted by the difference calculating means;
2. The display information forming apparatus according to claim 1, further comprising posture estimation means for estimating a position, orientation, and root of the target area from the area shaped by the area forming means.   The display information forming apparatus according to claim 3, wherein the difference calculation unit extracts a region in which a light amount change is caused by light emitted from the light source unit as a subject in a range where the light reaches.   The difference calculation means calculates a difference between an image with irradiation by turning on the light source unit and an image without irradiation by turning off, and compares the difference value with a threshold to determine whether there is a change in light amount, and in a plurality of areas of the subject The display information forming apparatus according to claim 3, wherein a target area is extracted using a difference in reflectance.   The difference calculation means considers the color component of the irradiation light of the light source unit in determining whether there is a change in the amount of light, changes the color space to a large amount of change, and then irradiates when the light source unit is turned on. The display information forming apparatus according to claim 3, wherein a difference is calculated between the image and an image that is not irradiated by turning off.   4. The difference calculating means makes it easy to emphasize the amount of change in a target region having a high reflectivity by setting a small color component for the subject as an irradiation light component of the light source unit. The display information forming device described in 1.   The region forming means extracts a region and a boundary surface adjacent to the target region, and shapes the shape of the target region based on a distance threshold so as not to cross the region even if the boundary surface has a partial defect. The display information forming apparatus according to claim 3, wherein:   The posture estimation means extracts a region where the light source itself is reflected from the image obtained by the imaging unit, and estimates the orientation of the target region necessary for processing of the superimposed information from the shape of the region. The display information forming apparatus according to claim 3.   The display information forming apparatus according to claim 3, wherein the posture estimation unit estimates a direction of a target region necessary for processing of superimposed information from the edge feature amount obtained by the region forming unit.   The control unit inputs information on the coordinates, orientation, and size of the target area estimated by the estimation unit, processes information stored in the storage unit according to the information, and is obtained by the imaging unit The display information forming apparatus according to claim 1, wherein the display information forming apparatus is superimposed on an image. At least a light source unit that irradiates a fingernail portion with a light amount, an imaging unit that outputs a finger region image captured at regular time intervals, an image output from the imaging unit, and information created by a control unit described later are displayed at least . An information terminal device having at least a display unit;
Analyzing the finger region image output from the imaging unit , and estimating at least the position , orientation, and root (base) of the nail based on the light amount difference and edge detection obtained by the analysis ; a storage unit for holding information to be displayed on said pawl portion, the read information to be displayed on the claw portion from the storage unit, the position of the display information read said estimated by the estimation unit, adapted to the orientation and fundamentally And a server including at least a control unit that creates information to be displayed on the display unit,
An image captured by the information terminal device is transmitted to the server, and information to be displayed on the display unit on the server is created.

Images