JP7151753B2 - Nail contour detection device, printing device, nail contour detection method and program - Google Patents

Description

The present invention relates to a nail contour detection device, a printing device, a nail contour detection method, and a program.

2. Description of the Related Art Conventionally, a printing device (nail printing device) that prints a nail design on a fingernail or the like is known.
When printing a nail design on a nail, for example, the contour of the nail is detected from an image of the finger including the nail, and the area inside the contour of the nail is printed as the print target area.

For this reason, it is important to distinguish the nail region from surrounding finger (skin) portions and accurately detect the contour of the nail in order to obtain a high-quality nail print that does not run out or remain unpainted.
In this regard, a technique is known in which a static image of a target is segmented using a Markov Random Field (MRF) model, and preprocessing is performed for subsequent image processing such as object extraction. (See Patent Document 1, for example).
It is conceivable to apply such a technique to the case where only the nail region is detected by distinguishing between the nail and the rest (skin portion such as a finger).

JP-A-2001-297329

However, in the case of printing on the nails, in order to improve the color development of the ink, etc., there are cases in which a base is applied to the nail region in advance. In this case, the background color is not always the same.
When the color of the base applied to the nail is different from the color learned in advance, the area segmentation using a computational model such as the MRF model as in the method described in Patent Document 1 does not work well, and the nail contour is correctly calculated. The problem is that it is not possible to detect
For example, a learning model trained to detect a white portion cannot correctly detect the contour of the nail when the nail is coated with a base other than white, such as light pink or light blue.

In this respect, it is conceivable to learn for each base color and recreate a learning model corresponding to each color, but recreating a plurality of learning models from scratch is troublesome and time consuming.
In addition, as a method to cope without re-learning, color correction (white balance) is performed on the captured image so that the background color of the captured image becomes the color that has been learned in the learning model (for example, white). It is also possible.
However, in this case, not only the color of the area where the base is applied but also the color of the area other than the nail, such as the skin, is corrected, resulting in a decrease in detection accuracy.

SUMMARY OF THE INVENTION The present invention has been made in view of the circumstances as described above, and provides a nail contour detecting device, a printing device, and a nail contour detecting device capable of appropriately detecting the nail contour regardless of the color of the base applied to the nail. It is an advantage to provide a method and program.

In order to solve the above problems, the nail contour detection device of the present invention includes:
a pixel information obtaining means for obtaining values relating to pixels in a detection area detected from an image including a nail to which a foundation is applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. learning model generation means for generating a second learning model for detection;
detection means for detecting, when the learning model generating means generates the second learning model, applying the second learning model to detect, as a nail contour, a contour of an area where the base is applied from the image;
characterized by comprising

According to the present invention, it is possible to appropriately detect the contour of the nail regardless of the color of the base applied to the nail.

1 is a diagram showing a schematic configuration of a printing apparatus according to this embodiment; FIG. 3 is a block diagram of main parts showing a control configuration of a printing apparatus and a terminal apparatus cooperating with the printing apparatus according to the present embodiment; FIG. 4 is a flowchart showing print processing according to the embodiment; FIG. 4 is a diagram showing an example of a detection region detected as a region corresponding to a nail; FIG. 10 is a diagram showing a modified example of a detection region detected as a region corresponding to a nail;

An embodiment of a nail contour detection device, a printing device, a nail contour detection method, and a program according to the present invention will be described with reference to FIGS. 1 to 4. FIG.
Although various technically desirable limitations are attached to the embodiments described below for carrying out the present invention, the scope of the present invention is not limited to the following embodiments and illustrated examples.
Further, in the following embodiments, a nail printing apparatus in which the printing apparatus prints on fingernails as an object to be printed will be described as an example, but the printing object of the printing apparatus in the present invention is limited to fingernails. Instead of objects, for example, toe nails or the like may be printed.

FIG. 1 is a perspective view showing the appearance of a printing device (nail printing device) according to this embodiment. Further, in this embodiment, a case in which a printing device prints on a nail in cooperation with a terminal device will be exemplified. FIG. 2 is a block diagram showing a main control configuration of a printing device and a terminal device cooperating therewith.
In the following embodiments, up and down, left and right, and front and back refer to directions shown in FIG. Also, the X direction and the Y direction refer to the directions shown in FIG.

In this embodiment, as shown in FIG. 1, the printing apparatus 1 has a housing 11 that is substantially box-shaped.
An operation unit 12 is installed on the upper surface (top plate) of the housing 11 .
The operation unit 12 is an operation unit through which a user performs various inputs.
The operation unit 12 includes operation buttons for performing various inputs, such as a power switch button for turning on the power of the printer 1, a stop switch button for stopping operation, and a print start button for instructing start of printing. there is
When the operation section 12 is operated, an operation signal corresponding to the operation is output to the control device 30, and the control device 30 performs control according to the operation signal to operate each section of the printing apparatus 1. FIG.
Instead of the operation unit 12, each unit of the printer 1 may operate according to an operation signal input from an operation unit 71 of the terminal device 7, which will be described later.

The printing device 1 also includes a communication unit 14 (see FIG. 2). The communication unit 14 is configured to be able to transmit and receive information to and from the terminal device 7 .
Communication between the printing device 1 and the terminal device 7 is performed by, for example, a wireless LAN. The communication between the printing device 1 and the terminal device 7 is not limited to this, and any method may be used. The communication unit 14 includes an antenna chip and the like compatible with the communication method of the terminal device 7 .
The communication unit 14 is connected to a communication control unit 311 (see FIG. 2) of the control device 30 to be described later, and is controlled by the communication control unit 311 .

On the front side of the housing 11 of the printer 1 (the front side in the Y direction in FIG. 1) and approximately in the center in the horizontal direction of the device (the X direction in FIGS. 1 and 3), there is a A finger insertion opening 15, which is an opening into which the finger U is inserted, is formed.
The interior of the housing 11 is vertically partitioned by a partition plate 16, and a finger U inserted during printing is placed and held at a position above the partition plate 16 corresponding to the finger insertion opening 15. A finger holding portion 2 is provided.

The finger holding part 2 holds the finger U in a state where the surface of the nail T is exposed. A finger support member 21 is provided for receiving the abdomen. Further, a nail placing portion 25 (see FIG. 4) on which the tip of the nail is placed is provided on the back side of the finger holding portion 2 in the finger insertion direction (rear side of the printing apparatus 1 in FIG. 1).
In this embodiment, the finger U is held by the finger holding portion 2 and the nail T of the finger U is positioned at a predetermined position by placing the tip of the nail on the nail placing portion 25 .

Further, inside the housing 11, there are provided a printing unit 40 for printing on the nail T of the finger U (the surface of the nail T), an imaging unit 50 for acquiring an image of the finger U including the nail T, and the like. .

The printing unit 40 is a printing unit that prints an area inside the nail contour TL detected by the control device 30 as a nail contour detecting device, which will be described later, as a print target region.
The printing unit 40 moves the print head 41 in the X direction (the X direction in FIG. 1 and the like, the left and right direction of the printing device 1) and the Y direction (the Y direction in FIG. 1 and the like, the front and back direction of the printing device 1). A head moving mechanism 49 (see FIG. 2) and the like are provided for movement.
The print head 41 is supported by an X-direction movement stage for moving the print head 41 in the X direction and a Y-direction movement stage for moving the print head 41 in the Y direction (both not shown). The moving mechanism 49 includes an X-direction moving motor 46, a Y-direction moving motor 48, and the like, which serve as drive units for appropriately moving the print head 41 in the X and Y directions.

The print head 41 of this embodiment has, for example, an ink ejection surface 410 having a plurality of nozzle openings (not shown) for ejecting ink on the surface facing the nail surface. It is an inkjet head of an inkjet system that performs printing by spraying ink directly onto the surface of a nail, which is the surface to be printed (nail T), from the surface.
The print head 41 can eject ink of each color such as cyan (C; CYAN), magenta (M; MAGENTA), and yellow (Y; YELLOW). Note that the type of color ink that can be ejected by the print head 41 is not limited to this, and ink of other colors may be ejected. For example, it may be possible to eject white ink or the like that can serve as a base for a design.

The print head 41 in the printing unit 40, and the X-direction movement motor 46 and the Y-direction movement motor 48 as driving units for driving the print head 41 are connected to a print control unit 315 (see FIG. 2) of the control device 30, which will be described later. It is controlled by the print control unit 315 .

As shown in FIG. 2, the imaging unit 50 includes an imaging device 51 and an illumination device 52 .
The imaging device 51 is a compact camera including a solid-state imaging device having pixels of, for example, two million pixels or more, a lens, and the like. Also, the lighting device 52 is, for example, a lighting lamp such as a white LED.
The photographing unit 50 illuminates the nail T of the finger U held by the finger holding unit 2 with the illumination device 52 . Then, the finger U is photographed by the photographing device 51 to obtain a nail image that is an image of the nail T of the finger U (image of the finger including the nail image, see FIG. 4).
The photographing unit 50 is connected to a photographing control unit 312 (see FIG. 2) of the control device 30 to be described later, and is controlled by the photographing control unit 312 .
Note that the image data of the image captured by the imaging unit 50 may be stored in the storage unit 32, which will be described later. Also, the nail image acquired by the imaging unit 50 may be sent to the terminal device 7 via the communication unit 14 .

The photographing unit 50 may be provided at a position where the finger U and the nail T held by the finger holding unit 2 can be photographed, and the specific arrangement is not particularly limited. In this embodiment, the photographing device 51 and the lighting device 52 are fixed at a position inside the top surface of the housing 11 so as to be able to face the nail T (surface of the nail T) of the finger U held by the finger holding portion 2. are placed.
Note that the imaging unit 50 may be configured to be movable in the XY directions by a head moving mechanism 49 that moves the print head 41 .

The control device 30 mounted on the printing apparatus 1 includes a control unit 31 (see FIG. 2) configured by a processor such as a CPU (Central Processing Unit) (not shown), ROM (Read Only Memory) and RAM (Random Access Memory). etc. (none of which are shown) and a storage unit 32 (see FIG. 2).
In this embodiment, the control device 30 has a control unit 31 functioning as sampling means, pixel information acquisition means, learning model generation means, and detection means, as will be described later, and detects the nail contour TL from the nail image. It functions as a nail contour detector.

The storage unit 32 stores various programs and data for operating the printing apparatus 1 .
Specifically, the storage unit 32 includes a program storage area 321 storing various programs such as a nail contour detection program for performing nail contour detection processing and a print program for performing print processing. A learning model storage area 322 for storing a learning model for detecting contour information (hereinafter referred to as "nail contour"), a nail information storage area 323 for storing various information about the nail T, and the like are provided.
The programs stored in the program storage area 321 are executed by the control unit 31, so that each unit of the printing apparatus 1 is centrally controlled.

A learning model for nail contour detection is stored in the learning model storage area 322 .
The learning model (existing learning model, first learning model) stored in the learning model storage area 322 is, for example, one that has already been generated and stored at the time of shipment of the printing apparatus 1 (learning model in the initial state). .
A first learning model (learning model in an initial state) is generated by previously learning a color distribution. Specifically, a plurality of images of the nail T to which the base is applied are prepared, and from these images, the region (foreground) of the nail T to which the base is applied and the other regions (skin portion such as the finger U, etc.) (background) pixel values (pixel values in this embodiment, as will be described later) are sampled. Then, from the sampled pixels, the color distribution is modeled for each of the region of the nail T to which the foundation is applied (foreground) and other regions (skin, etc., background).

In the present embodiment, the first learning model (learning model in the initial state) is a learned model that has learned about the case where white is applied as the base.
In the present embodiment, a case in which a mixed normal distribution (gaussian mixture distribution) model (GMM: Gaussian Mixture Model) is adopted as a learning model for nail contour detection will be described as an example. Note that the learning model is not limited to the mixed normal distribution model, and various models suitable for segmentation (clustering) of regions can be appropriately adopted.
In this embodiment, even when a new learning model is generated as described later, the learning model in the initial state is saved without being overwritten. A learning model of 1 (initial state learning model) is applied as a default.

The control unit 31 includes a communication control unit 311, an imaging control unit 312, a nail information detection unit 313, a print data generation unit 314, a print control unit 315, and the like from a functional point of view. Functions of the communication control unit 311, the photographing control unit 312, the nail information detection unit 313, the print data generation unit 314, the print control unit 315, etc. are stored in the CPU of the control unit 31 and the program storage area 321 of the storage unit 32. This will be realized in collaboration with the program.

The communication control section 311 controls the operation of the communication section 14 . In this embodiment, communication with the terminal device 7 is controlled, and when data such as nail design data is transmitted from the terminal device 7, it is received.
Moreover, when the nail image is acquired by the photographing unit 50 , the nail image data may be transmitted to the terminal device 7 .

The photographing control unit 312 controls the photographing device 51 and the lighting device 52 of the photographing unit 50 so that the photographing device 51 captures a finger image including an image of the nail T of the finger U held by the finger holding unit 2 (hereinafter referred to as “nail (refer to Fig. 4).
The image data of the nail image acquired by the imaging unit 50 may be stored in the storage unit 32 or may be transmitted to the terminal device 7 via the communication unit 14 .

The nail information detection unit 313 detects nail information about the nail T of the finger U based on the image of the nail T of the finger U captured by the imaging device 51 and held by the finger holding unit 2 .
Here, the nail information includes, for example, the contour of the nail T (the nail contour TL represented by the XY coordinates of the horizontal position of the nail T (see FIG. 4, etc.)), the length of the nail T in the longitudinal direction, the length of the nail T , the inclination angle of the surface of the claw T with respect to the XY plane (inclination angle of the claw T, claw curvature), and the like. Note that if the height of the nail T (vertical position of the nail T) can be acquired from an image or the like captured by the imaging device 51, the height of the nail T is also included in the nail information.

In particular, in this embodiment, the nail information detection unit 313 functions as detection means for detecting the nail contour TL, as well as sampling means, pixel information acquisition means, and learning model generation means.
A nail information detection unit 313 as a detection means applies a learning model for nail contour detection to detect the contour of the area where the foundation is applied from the image of the nail T as the nail contour TL (see FIG. 4). It is assumed that the base is applied to the nail T by hand by a user or the like, but the method of applying the base is not limited to this.
As will be described later, when the nail information detection unit 313 generates a second learning model as a learning model generation unit, the nail information detection unit 313 as a detection unit applies the second learning model to obtain a nail contour. TL detection is performed.

A nail information detection unit 313 as sampling means detects (samples) an area corresponding to the nail T from an image including the nail T to which the foundation is applied as a detection area Ar.
Here, the “region corresponding to the nail T” is a region in the nail image that is determined to be the nail T with a high possibility. When recognizing the area of the nail T, it is difficult to strictly detect the boundary between the nail T and the finger U (skin or the like). In particular, in this embodiment, when sampling the detection area Ar, the first learning model (the learning model in the initial state stored in the learning model storage area 322 of the storage unit 32) trained for the case where the background is white is used. ) is used. Therefore, when the color of the base applied to the nail T is other than white (for example, pale pink), it is difficult to accurately detect every corner of the area where the base is applied.
Therefore, in the present embodiment, by applying the first learning model, the nail information detection unit 313 detects a region that is likely to be the nail T (the inner region of the “provisional nail contour TLe” shown in FIG. 4). Let the predetermined area be "the area corresponding to the nail T". As a result, even if the nail information detection unit 313 as a sampling means cannot strictly detect the boundary portion of the nail T, and the temporary nail contour TLe deviates from the actual nail contour TL as shown in FIG. An area that is likely to be the nail T can be set as the detection area Ar.

The nail information detection unit 313 as pixel information acquisition means acquires values related to pixels in the detection area Ar detected as sampling means.
Here, the "value regarding a pixel" is, for example, a pixel value represented by RGB, YUV, etc., a luminance value, or the like. Note that the values relating to the pixels in the detection area Ar are used for detecting the area where the base is applied, and more accurate detection can be performed if color information is included. Therefore, pixel values such as RGB are preferable to luminance values. Note that the luminance value may be considered together with the pixel value as the “value regarding the pixel”. In the present embodiment, a case where the "value regarding a pixel" is a pixel value will be described below as an example.

A nail information detection unit 313 as a learning model generation unit generates values (pixel values) related to pixels in the base coating region in the first learning model stored in the learning model storage area 322 of the storage unit 32 for nail contour detection and A second learning model for nail contour detection is generated by correcting the first learning model based on the difference from the values (pixel values) of the pixels in the detection area Ar. The generated second learning model is stored in the learning model storage area 322 .

Note that in this embodiment, even if a second learning model is newly generated, the learning model already stored in the learning model storage area 322 (existing learning model, first learning model) is left without being overwritten. It's like For example, a learning model for nail contour detection (initial state learning model) stored at the time of shipment of the printing apparatus 1 distinguishes a white portion from the color of the skin of the finger U, etc., and treats it as an area where the base is applied. learned to detect. By storing such an initial state learning model as a default, when a new nail image is acquired, detection is started from white, and the difference from the color actually applied to the nail T as the base is calculated. you can look at
Note that the first learning model (existing learning model) may be overwritten and updated by the second learning model when it is generated. In this case, when the second learning model is generated by the learning model generating means, the control unit 31 stores the second learning model as the first learning model in the storage unit 32 (learning model storage area 322). function as memory control means for storing in Alternatively, only the learning model in the initial state learned in white may be left as a default, and the second learning model generated thereafter may be appropriately discarded and updated each time a new learning model is generated. .

In the present embodiment, the nail information detection unit 313 as learning model generation means acquires an average value of values (pixel values) related to pixels in the detection area Ar, and compares the average value with the first learning model for nail contour detection. A difference from the average value of the expected values of the normal distributions forming the mixed normal distribution of the base coating region is calculated and used as a correction value for correcting the first learning model.
Note that the expected value and the average value of each normal distribution that constitutes the mixed normal distribution of the base coating area in the first learning model may be calculated when calculating the average value of the pixel values of the detection area Ar. , may be calculated in advance and stored in the storage unit 32 in association with the first learning model.
In addition, in calculating the average value of the mixed normal distribution of the base coating region in the first learning model, the average value is calculated after weighting according to the level (contribution rate) of each normal distribution that constitutes the mixed normal distribution. can be calculated.
A specific process for the nail information detection unit 313 to generate the second learning model and detect the nail contour TL will be described in detail later.

The print data generation unit 314 sets a print target area by the printing unit 40 based on the detection result of the nail information detection unit 313, and generates print data.
As described above, in the present embodiment, the photographing device 51 of the photographing unit 50 photographs the nail T to be printed, and the nail information detection unit 313 detects nail information such as the nail contour TL for the nail image. The print data generation unit 314 sets an area (print target area) to be printed by the printing unit 40 in the area inside the nail contour TL based on the detection result, and generates print data.
Specifically, the print data generation unit 314 cuts out the image data of the nail design selected by the user to be printed according to the nail contour TL, scales the image data appropriately, and generates print data.
When the curvature of the nail T is detected by the nail information detection unit 313, the print data generation unit 314 may perform curved surface correction on the print data based on the curvature of the nail T. When curved surface correction is performed, print data more suitable for the shape of the nail T can be generated.
Note that when the design selected as the design to be printed on the nail T is not printed on the entire nail region, but is, for example, a French nail that is printed only on the tip of the nail, the print data generation unit 314 The design is matched with the nail contour TL detected by the information detection unit 313 to generate printing data of a partial design having a shape and range matching the shape of the user's nail T. FIG.

The print control unit 315 is control means for controlling the print head 41 . Specifically, the print control unit 315 outputs a control signal to the print unit 40 based on the print data generated by the print data generation unit 314, and prints the nail T according to the print data. The X-direction movement motor 46, the Y-direction movement motor 48, the print head 41, etc. of the printing unit 40 are controlled so as to perform the printing.

As described above, in this embodiment, the printing device 1 operates in cooperation with the terminal device 7 .
The terminal device 7 is, for example, a mobile terminal device such as a smart phone. Note that the terminal device 7 is not limited to a smart phone. For example, it may be a tablet personal computer (hereinafter referred to as "PC"), a notebook PC, a stationary PC, a game terminal device, or the like.
As shown in FIG. 2, the terminal device 7 includes an operation section 71, a display section 72, a communication section 73, a control device 80, and the like.

The operation unit 71 can perform various inputs and settings according to user's operations, and is, for example, a touch panel integrally provided on the surface of the display unit 72 . When the operation portion 71 is operated, an input signal corresponding to the operation is transmitted to the control portion 81 .
Various operation screens are displayed on the touch panel configured in the display unit 72 according to the control of the display control unit 812, which will be described later, and the user can perform various operations such as input and setting by touching the touch panel. .
Note that the operation unit 71 for performing various input/setting operations is not limited to a touch panel, and various operation buttons, a keyboard, and the like may be provided as the operation unit 71, for example.

In this embodiment, the user operates the operation unit 71 to output various instructions such as printing start from the terminal device 7 to the printing device 1 . It also functions as an operation unit.
Also, the user can select a nail design to be printed on the nail T by operating the operation unit 71 .

The display unit 72 is composed of, for example, a liquid crystal display (LCD), an organic EL display, other flat displays, or the like.
Note that, as described above, a touch panel for performing various inputs may be integrally formed on the surface of the display unit 72 . In this case, the touch panel functions as the operation unit 71 .
In this embodiment, the nail design input/selected by the user through the operation unit 71, various guide screens, warning display screens, and the like can be displayed on the display unit 72. FIG.

The communication section 73 is configured to be able to communicate with the communication section 14 of the printing apparatus 1 .
The communication between the printing device 1 and the terminal device 7 may be, as described above, either a wireless connection method or a wired connection method, and the specific method is not limited. The communication unit 73 can communicate with the printing apparatus 1, and the communication standard of the communication unit 14 of the printing apparatus 1 is applied.
The communication unit 14 is connected to a communication control unit 811 (see FIG. 2) of the control device 80 to be described later, and is controlled by the communication control unit 811 .

As shown in FIG. 2, the control device 80 of the terminal device 7 of the present embodiment includes a control unit 81 configured by a processor such as a CPU (Central Processing Unit) (not shown), a ROM (Read Only Memory) and a RAM (not shown). (random access memory) or the like.

The storage unit 82 stores various programs, various data, and the like for operating each unit of the terminal device 7 .
Specifically, in the ROM or the like of the present embodiment, in addition to an operation program 821a for controlling each part of the terminal device 7, a nail print application program 821b (hereinafter referred to as a nail print application program 821b) for performing nail printing using the printing device 1 is stored. "Nail print AP") and other various programs are stored. Each part of the device 7 is controlled in an integrated manner.

Further, the storage unit 82 of the present embodiment is provided with a design storage area 822 and the like for storing data of nail designs (also simply referred to as designs).
Note that the nail design (design) stored in the design storage area 822 may be an existing design prepared in advance or a design created by the user himself/herself. In addition, when the terminal device 7 can be connected to various networks, it may be possible to import nail designs stored in a network-connectable server device (not shown) or the like.

The control unit 81 of the terminal device 7 includes a communication control unit 811, a display control unit 812, etc. from a functional point of view. The functions of the communication control unit 811, the display control unit 812, and the like are realized by cooperation between the CPU of the control unit 81 and the programs stored in the ROM of the storage unit 82. FIG. Note that the functions provided by the control unit 81 of the terminal device 7 are not limited to this, and may be provided with various other functional units.
The communication control section 811 controls the operation of the communication section 73 . In this embodiment, communication with the printing device 1 is controlled, and nail design data and the like are transmitted to the printing device 1 .
The display control unit 812 also controls the display unit 72 to display various display screens on the display unit 72 .

Next, with reference to FIG. 3, a nail contour detection method by the control device 30 as the nail contour detection device and the printing apparatus 1 including the control device 30 according to the present embodiment will be described.
When performing nail printing using the printing apparatus 1 of the present embodiment, the user operates the operation unit 12 of the printing apparatus 1 to turn on the power.
Also, the power of the terminal device 7 is turned on, and execution of nail print processing is selected from the operation section 71 of the terminal device 7 . This activates the nail print AP 821b.
When the nail print AP 821b is activated, the display control unit 812 of the terminal device 7 displays an instruction screen for instructing the display unit 72 to place the finger U including the nail T coated with the base on the finger holding unit 2 of the printing apparatus 1. display.

When the finger U is placed on the finger holding part 2, the finger U is photographed by the photographing part 50 as shown in FIG. is obtained (step S1). Then, the nail information detection unit 313 as sampling means detects (samples) an area corresponding to the nail T from the nail image acquired by the photographing unit 50 as a detection area Ar (step S2).
Specifically, the nail information detection unit 313 reads out the first learning model from the storage unit 32, applies it to the nail image, and detects the region corresponding to the nail T. FIG. As described above, at this time, the nail information detection unit 313 performs detection using the first learning model (initial state learning model) that recognizes the white portion as the base region. Therefore, when the base applied to the nail T is a color other than white, the temporary nail contour TLe detected by the nail information detection unit 313 is not as accurate as the actual nail contour TL, and is shown in FIG. As you can see, there may be deviations.

That is, when the base color actually applied to the nail T is not white, but is light pink, light green, or the like, the white portion is recognized as the base region. It is difficult for the learning model to accurately distinguish between the nail T and the finger U (skin). Therefore, as shown in FIG. 4, the provisional nail contour TLe extends inside the actual nail contour TL, or extends outside the actual nail contour TL, which is actually the finger U. It is assumed that it may protrude.

The detection area Ar detected by the nail information detection unit 313 as a sampling means is a sample for correcting the learning model according to the color of the base applied to the nail T at present. ) is included, a correct correction value cannot be obtained. For this reason, the detection area Ar is preferably set to an area with relatively high detection accuracy.
Therefore, in the present embodiment, as shown in FIG. 4, of the inner region of the provisional nail contour TLe detected by the nail information detection unit 313 as the sampling means, a predetermined region in the central portion that does not include the vicinity of the boundary portion is defined as a "nail Detected (sampled) as a detection area Ar as being an area corresponding to T.

When the detection area Ar is detected, the nail information detection unit 313 as pixel information obtaining means obtains pixel values of the detection area Ar (step S3). Then, the nail information detection unit 313, as learning model generation means, calculates the average value of the pixel values of the detection area Ar using the following formula (1) (step S4).

式（１）において、「F」はステップＳ２で検出した検出領域Ａｒを示し、「N」は検出領域Ａｒの画素数を示し、「x」は画素値を示している。
式（１）により、検出領域Ａｒの画素値の平均値「m_x」が算出される。
画素値がＲＧＢで表されるものである場合、爪情報検出部３１３は、このような画素値の平均値の算出を、ＲＧＢそれぞれについて算出する。

In equation (1), "F" indicates the detection area Ar detected in step S2, "N" indicates the number of pixels in the detection area Ar, and "x" indicates the pixel value.
The average value “m _x ” of the pixel values in the detection area Ar is calculated by Equation (1).
When the pixel values are represented by RGB, the nail information detection unit 313 calculates the average value of such pixel values for each of RGB.

Further, the nail information detection unit 313 acquires the average value of the expected pixel values of the base application region in the first learning model for nail contour detection (step S5).
The average value of the expected values of the pixel values of the base coating region in the first learning model is calculated by the following formula (2).
The following formula (2) illustrates a case where the contribution rate of the normal distributions forming the mixed normal distribution is taken into consideration when calculating the average value of the expected values. It is not essential to consider the contribution rate of each normal distribution, but if the contribution rate is taken into consideration, it is possible to obtain an average value that more appropriately reflects the tendency of the mixed normal distribution.
As described above, the average value may be calculated in advance and stored in association with the first learning model.

式（２）において、「M」は下地塗布領域の混合正規分布を構成する正規分布の数を示し、「w」は各正規分布の寄与率を示し、「e_x」は各正規分布の期待値を示している。
式（２）により、各正規分布の寄与率が考慮された期待値の平均値「mt_x」が算出される。

In equation (2), “M” indicates the number of normal distributions that make up the mixed normal distribution of the base coating area, “w” indicates the contribution of each normal distribution, and “e _x ” indicates the expected value of each normal distribution. value.
The average value “mt _x ” of the expected values in which the contribution rate of each normal distribution is taken into account is calculated by Equation (2).

Further, the nail information detection unit 313 as a learning model generating means generates an average value “m _x ” of pixel values in the detection area Ar and an average value “mt _x " is calculated by the following formula (3) (step S6).

式（３）により、検出領域Ａｒの画素値の平均値「m_x」と、第１の学習モデルにおける下地塗布領域の画素値の期待値の平均値「mt_x」との差分「d_x」が算出される。

According to Equation (3), the difference "d _x " between the average value "m _x " of the pixel values of the detection area Ar and the average value "mt _x " of the expected pixel values of the base coating area in the first learning model. is calculated.

Note that the difference “d _x ” between the average value “m _x ” of the pixel values of the detection area Ar and the average value “mt _x ” of the expected pixel values of the base coating area in the first learning model is small. , the color of the base applied to the nail T is close to the color (white, etc.) learned in the first learning model (learning model in the initial state), and the first learning model can be used as it is for detection of the nail contour TL, there is no problem with detection accuracy.
Therefore, in the present embodiment, a predetermined threshold value is set for the difference "d _x ", and the nail information detection unit 313 determines whether or not the difference "d _x " exceeds the threshold value (step S7). The degree of the threshold in this case is a matter to be appropriately set.
Then, if the difference " _dx " exceeds the threshold (step S7; YES), the difference " _dx " between the average values " _mx " and " _mtx " is used as a correction value for the first The expected value of the pixel value of the base coating area in the learning model is corrected by the following formula (4) (step S8).

式（４）により、第１の学習モデルにおける下地塗布領域の画素値の期待値「e_xi」が期待値「ec_xi」に補正された新たな学習モデル（第２の学習モデル）が生成される。
そして、爪情報検出部３１３は、検出手段として補正後の学習モデル（第２の学習モデル）を適用して爪画像から下地が塗布された領域の輪郭を爪輪郭ＴＬとして検出する（ステップＳ９）。

A new learning model (second learning model) in which the expected value “e _xi ” of the pixel value of the base coating region in the first learning model is corrected to the expected value “ec _xi ” is generated by Equation (4). be.
Then, the nail information detection unit 313 applies the corrected learning model (second learning model) as a detecting means to detect the contour of the area where the foundation is applied from the nail image as the nail contour TL (step S9). .

On the other hand, if the difference “d _x ” does not exceed the threshold value (step S7; NO), the nail information detection unit 313 applies the first learning model without correction, and the base is applied from the nail image. Then, the contour of the detected area is detected as the nail contour TL (step S10).

By performing such processing, when the first learning model is trained to extract a region where a white base is applied, the color of the base actually applied to the nail T is not white. , it can be handled relatively easily by correcting the expected value without generating a new learning model from scratch.

As described above, according to the present embodiment, the control device 30, which is a nail contour detection device, detects an area corresponding to the nail T from an image including the nail T to which the foundation is applied, and the detected detection area Ar and corrects the first learning model based on the difference between the pixel value of the base coating region and the pixel value of the detection region Ar in the first learning model for nail contour detection. to generate a second learning model for nail contour detection. Then, when the second learning model is generated, this second learning model is applied to detect the nail contour TL from the image. If the second learning model is not generated, the first learning model for nail contour detection is applied to detect the nail contour TL from the image.
As a result, even when the nail T is coated with a base color (eg, pink) different from the color (eg, white) learned by the first learning data, the learning model can be recreated from scratch, or the image can be subjected to color correction. , it is possible to detect the nail contour TL with high detection accuracy without taking time and effort.
That is, the existing learned learning model (first learning model) is not limited to simply recognizing the white portion, and has accumulated information regarding the recognition of the nail T, such as the color appearing slightly darker at the end of the nail T, for example. Contains various types of information. In this regard, if a learning model is recreated from scratch, it takes a considerable amount of time and effort to obtain learning results in such a wide range.
As in the present embodiment, when the base of the nail T is applied with a different color (eg, pink) from the color (eg, white) learned by the first learning model, the first learning model is corrected. By doing so, it is possible to take over such learning results to a new learning model, and improve the detection accuracy and recognition accuracy of the nail contour TL without lowering the overall accuracy.

Further, in the present embodiment, the control unit 31 (nail information detection unit 313) as sampling means detects the area corresponding to the nail T as the detection area Ar using the first learning model for nail contour detection.
Even if the nail T is coated with a base color different from the color (e.g., white) learned by the first learning data, the color is closer to the color of the finger U (skin) than the nail T. Unless otherwise the approximate position of the claw T can be identified. Therefore, by detecting the temporary nail contour TLe and setting the central portion of the inner region of the nail contour TLe as the detection region Ar, it is possible to easily sample an appropriate region that does not include regions other than the nail T. .

Further, in the present embodiment, the control unit 31 (nail information detection unit 313) as learning model generation means acquires an average value of pixel values in the detection area Ar, and uses the average value and the first learning for nail contour detection. A difference from the average value of the expected values of each normal distribution constituting the mixed normal distribution of the undercoating region in the model is calculated and used as a correction value for correcting the first learning model.
Thus, in this embodiment, correction of learning data corrects only the expected value of each normal distribution, and does not correct variance.
When correcting not only the expected value but also the variance, it is desirable to correct the entire region of the nail T. It is difficult to accurately cut out the entire area of the nail T when it is applied to the .
In this regard, in the method of the present embodiment, correction is performed by shifting only the expected value without considering the variance. For this reason, it is possible to correct the learning model simply by cutting out (sampling) the area that is surely considered to be the nail T as the detection area Ar, and the learning model can be efficiently and accurately obtained (the second correction after correction). learning model) can be generated.

Further, when the control unit 31 as learning model generation means generates the second learning model, the control unit 31 functions as storage control means and stores this second learning model in the storage unit 32 as the first learning model. You may let
In this way, when the first learning model is replaced with the second learning model when the second learning model is generated, the number of learning models stored in the storage unit 32 does not increase. can be dealt with.

In addition, in the present embodiment, the control unit 31 as the detection unit determines that the difference between the value regarding the pixels in the base coating region and the value regarding the pixels in the detection region Ar in the first learning model exceeds a predetermined threshold value. In this case, the second learning model is applied to detect the contour of the base-applied area from the image as the nail contour, and the values of the pixels of the base-applied area and the pixels of the detection area Ar in the first learning model are calculated. If the difference does not exceed a predetermined threshold value, the first learning model is applied to detect the contour of the ground-applied region from the image as the nail contour TL.
When the difference between the value of the pixel in the base application region and the value of the pixel in the detection region Ar in the first learning model is small, the color of the base applied to the nail T is different from that of the first learning model ( This is a case where the color is close to the color (white, etc.) learned in the learning model in the initial state), and even if the first learning model is used as it is for detecting the nail contour TL, there is no problem with detection accuracy. In this case, by applying the first learning model as it is without correction, the processing can be simplified and the nail contour detection processing can be performed quickly.

Further, when the nail contour detection device as in the present embodiment is applied to the printing device 1, the printing unit 40 as the printing means detects the inner side of the nail contour TL detected by the control device 30 as the nail contour detection device. Printing is performed using the area as a print target area.
As a result, printing can be performed on the inner region of the nail contour TL that has been detected with high accuracy without taking time and effort, unlike the case of constructing a new learning model from scratch.
Therefore, regardless of the color of the base applied to the nail T, a high-quality nail print can be applied to the nail T without any undercoating or protrusion.

Although the embodiments of the present invention have been described above, the present invention is not limited to such embodiments, and it goes without saying that various modifications are possible without departing from the scope of the invention.

For example, in the present embodiment, when the nail information detection unit 313 as the sampling means samples (detects) the detection area Ar, the first learning model learned for the case where the background is white is applied as a default. Although an example is shown, the learning model applied when sampling (detecting) the detection area Ar is not limited to this.
For example, if it is known in advance that a pink undercoat or a blue undercoat has been applied, a learning model newly generated corresponding to these is used for sampling the detection area Ar. may The user may input information about the color of the base applied to the nail T from the operation unit 12 or the like.

Further, in the present embodiment, the nail information detection unit 313 as sampling means detects the provisional nail contour TLe using the first learning model, and detects the region corresponding to the nail T from the inner region of the nail contour TLe. Although the case of detecting (sampling) as the area Ar has been exemplified, the method of detecting (sampling) the detection area Ar is not limited to this. The nail information detection unit 313 as sampling means may detect a predetermined area in the image including the nail T as the area corresponding to the nail T. FIG.
For example, if the approximate position of the nail T is known, the detection area Ar may be set in a region of the nail image where the nail T is likely to be placed.
Specifically, when the finger U is held by the finger holding portion 2 and the nail T is arranged so that the tip of the nail rests on the nail placing portion 25, as shown in FIG. A nail image is photographed so that the finger U including the nail T in the vicinity is accommodated. Then, a vertical center line Dc passing through the center of the nail T in the width direction and a horizontal center line Hc dividing the nail image vertically are set on the nail image, and above the center line Hc (that is, in the direction of the fingertip) A predetermined area passing through the center line Dc is detected (sampled) as a detection area Ar.
In this case, the detection area Ar can be cut out only from the image without performing processing using the first learning model (learning model in the initial state) or the like.
For this reason, for example, when the base applied to the nail T is a color that cannot be handled by the first learning model (learning model in the initial state) (for example, when the first learning model is learning in white, Even when a blue undercoat is applied, the area in which the nail T coated with the undercoat is positioned is appropriately detected (sampled) as the detection area Ar, and the nail contour detection processing of the present embodiment is performed. be able to.

Further, in the present embodiment, the case where the user or the like applies the base to the nail T by hand is illustrated, but the base is not limited to the case where the base is applied by hand.
For example, the base may be automatically applied by a printing device for base printing that is separate from the printing device 1 that prints the nail design.
Further, the printing apparatus 1 may include, as the print head 41, a color printing head for printing the nail design and a base printing head for printing the base.
If the printing apparatus 1 is equipped with a base printing head, and if information on the type and color of the liquid material for the base printing can be grasped on the side of the printing apparatus 1, data close to the color of the base is obtained based on the information. A provisional nail contour TLe may be detected using the learning model learned in 1, and the region corresponding to the nail T may be detected (sampled) as the detection region Ar from the inner region of the nail contour TLe.

Further, in this embodiment, a mixed normal distribution model is used as a learning model for nail contour detection, and the learning model is corrected by moving the expected value of the mixed normal distribution according to the color of the base applied to the nail T. However, the learning model for nail contour detection is not limited to the mixed normal distribution model.
For example, a histogram may be used as a learning model for nail contour detection. In this case, a correction method such as moving according to the color of the base applied to the nail T based on the mode value of the histogram can be used.

Further, in the present embodiment, the control device 30 of the printing apparatus 1 functions as a nail contour detection device, and includes a nail information detection unit 313 that acquires nail information such as the nail contour TL from the nail image and corrects the learning model. Although the case where all the processing is completed in the printing device 1 has been illustrated, the nail contour detection device is not limited to the control device 30 of the printing device 1 .
For example, the control device 80 of the terminal device 7 may function as a nail contour detection device and have a nail information detection unit that acquires nail information such as the nail contour TL from the nail image and corrects the learning model. In this case, a nail contour detection program for detecting the nail contour is stored in the storage section 82 (the program storage area 821 of the storage section 82) of the control device 80 of the terminal device 7. FIG.
Furthermore, the control device 80 of the terminal device 7 may have a functional section such as a print data generation section.
In this case, the nail image acquired by the imaging unit 50 of the printing device 1 is sent to the terminal device 7, and the control device 80 of the terminal device 7 corrects the learning model, detects nail information, and generates printing data. etc. are performed.
When the print data is generated on the terminal device 7 side, the generated print data is sent to the printer 1, and the printing unit 40 of the printer 1 performs printing based on this.
By performing control operations other than printing and photographing on the terminal device 7 side in this way, the configuration of the printing device 1 can be simplified.

Although several embodiments of the present invention have been described above, the scope of the present invention is not limited to the above-described embodiments, and includes the scope of the invention described in the claims and equivalent ranges thereof. .
The invention described in the scope of claims originally attached to the application form of this application is additionally described below. The claim numbers in the appendix are as in the claims originally attached to the filing of this application.
[Appendix]
<Claim 1>
a pixel information obtaining means for obtaining values relating to pixels in a detection area detected from an image including a nail to which a foundation is applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. learning model generation means for generating a second learning model for detection;
detection means for detecting, when the learning model generating means generates the second learning model, applying the second learning model to detect, as a nail contour, a contour of an area where the base is applied from the image;
A nail contour detection device comprising:
<Claim 2>
2. A nail contour detection apparatus according to claim 1, further comprising sampling means for detecting an area corresponding to said nail from said image as said detection area.
<Claim 3>
3. The nail contour detecting device according to claim 2, wherein said sampling means detects the region corresponding to said nail using said first learning model for nail contour detection.
<Claim 4>
3. The nail contour detection apparatus according to claim 2, wherein said sampling means detects a predetermined area in an image including said nail as an area corresponding to said nail.
<Claim 5>
The learning model generating means obtains an average value of the values of the pixels in the detection area, and obtains the average value and each normal that constitutes a mixed normal distribution of the base application area in the first learning model for nail contour detection. 5. The method according to any one of claims 1 to 4, wherein a difference between an expected value of the distribution and an average value is calculated and used as a correction value for correcting the first learning model. Nail contour detector.
<Claim 6>
A memory control means for storing the second learning model in the storage unit as the first learning model when the second learning model is generated by the learning model generating means. The nail contour detection device according to any one of claims 1 to 5.
<Claim 7>
The detection means is
When the difference between the value of the pixel in the base coating region and the value of the pixel in the detection region in the first learning model exceeds a predetermined threshold value, the second learning model is applied to the image from the The contour of the area where the base is applied is detected as the nail contour,
When the difference between the value of the pixel in the base coating region and the value of the pixel in the detection region in the first learning model does not exceed a predetermined threshold value, the first learning model is applied to the image from the 7. The nail contour detecting device according to any one of claims 1 to 6, wherein the contour of the area where the foundation is applied is detected as the nail contour.
<Claim 8>
a nail contour detection device according to any one of claims 1 to 7;
printing means for printing an area inside the nail contour detected by the nail contour detecting device as a print target area;
A printing device comprising:
<Claim 9>
a pixel information acquisition step of acquiring values relating to pixels in the detection area detected from the image including the nail with the base applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. a learning model generation step of generating a second learning model for detection;
a detection step of applying the second learning model to detect, as a nail contour, a contour of an area where the foundation is applied from the image when the second learning model is generated in the learning model generating step;
A nail contour detection method, comprising:
<Claim 10>
to the computer,
a pixel information acquisition function for acquiring values related to pixels in a detection area detected from an image including a nail with a base applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. a learning model generation function that generates a second learning model for detection;
a detecting function for detecting, when the second learning model is generated by the learning model generating function, applying the second learning model to detect, as a nail contour, a contour of an area where the base is applied from the image;
A program characterized by realizing

1 printing device 7 terminal device 31 control unit 32 storage unit 40 printing unit 313 nail information detection unit 314 print data generation unit 315 print control unit 322 learning model storage area 323 nail information storage area Ar detection area TL nail contour TLe temporary nail contour T Nail U Finger

Claims (10)

a pixel information obtaining means for obtaining values relating to pixels in a detection area detected from an image including a nail to which a foundation is applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. learning model generation means for generating a second learning model for detection;
detection means for detecting, when the learning model generating means generates the second learning model, applying the second learning model to detect, as a nail contour, a contour of an area where the base is applied from the image;
A nail contour detection device comprising: 2. A nail contour detection apparatus according to claim 1, further comprising sampling means for detecting an area corresponding to said nail from said image as said detection area. 3. The nail contour detecting device according to claim 2, wherein said sampling means detects the region corresponding to said nail using said first learning model for nail contour detection. 3. The nail contour detection apparatus according to claim 2, wherein said sampling means detects a predetermined area in an image including said nail as an area corresponding to said nail. The learning model generating means obtains an average value of the values of the pixels in the detection area, and obtains the average value and each normal that constitutes a mixed normal distribution of the base application area in the first learning model for nail contour detection. 5. The method according to any one of claims 1 to 4, wherein a difference between an expected value of the distribution and an average value is calculated and used as a correction value for correcting the first learning model. Nail contour detector. A memory control means for storing the second learning model in the storage unit as the first learning model when the second learning model is generated by the learning model generating means. The nail contour detection device according to any one of claims 1 to 5. The detection means is
When the difference between the value of the pixel in the base coating region and the value of the pixel in the detection region in the first learning model exceeds a predetermined threshold value, the second learning model is applied to the image from the The contour of the area where the base is applied is detected as the nail contour,
When the difference between the value of the pixel in the base coating region and the value of the pixel in the detection region in the first learning model does not exceed a predetermined threshold value, the first learning model is applied to the image from the 7. The nail contour detecting device according to any one of claims 1 to 6, wherein the contour of the area where the foundation is applied is detected as the nail contour. a nail contour detection device according to any one of claims 1 to 7;
printing means for printing an area inside the nail contour detected by the nail contour detecting device as a print target area;
A printing device comprising: a pixel information acquisition step of acquiring values relating to pixels in the detection area detected from the image including the nail with the base applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. a learning model generation step of generating a second learning model for detection;
a detection step of applying the second learning model to detect, as a nail contour, a contour of an area where the foundation is applied from the image when the second learning model is generated in the learning model generating step;
A nail contour detection method, comprising: to the computer,
a pixel information acquisition function for acquiring values related to pixels in a detection area detected from an image including a nail with a base applied;
Nail contour detection by correcting the first learning model based on the difference between the value of the pixel in the base coating area and the value of the pixel in the detection area in the first learning model for detecting the nail contour stored in the storage unit. a learning model generation function that generates a second learning model for detection;
a detecting function for detecting, when the second learning model is generated by the learning model generating function, applying the second learning model to detect, as a nail contour, a contour of an area where the base is applied from the image;
A program characterized by realizing

Images