JP2021002047A - Image processing device, image processing system, user terminal, image processing method, and program - Google Patents

Abstract

To provide an image processing device, an image processing system, a user terminal, an image processing method, and a program for displaying different hairstyles.SOLUTION: An image processing device is provided, comprising a hairstyle extraction unit configured to extract a partial image corresponding to a hairstyle from a first image of a user including the hairstyle and the face; and a display unit configured to replace a hairstyle of the user in a second image different from the first image with the partial image and display the resultant image.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to image processing devices, image processing systems, user terminals, image processing methods and programs.

In the beauty salon and the barber shop, a mirror is installed in front of the seated user. The practitioner performs the treatment while checking the user's hairstyle through a mirror. However, in conventional beauty salons and hairdressers, it is necessary for the user to verbally convey the image of the hairstyle, and it is difficult to accurately convey the hairstyle preferred by the user to the practitioner.

In response to the above concerns, an image display device including a mirror-type display in which a mirror and a transparent display are integrated, an image input unit, and an image storage unit for storing image data is known. The mirror display of this image display device is designed so that both the current image and the past user are in the user's vision. The past image displayed together with the current image is extracted based on the appearance characteristics of the past image and the current image. However, when this image display device is applied to a beauty salon and a barber shop, the current image at the time of visiting the store and the past image do not always match, and the past image desired by the user may not be displayed. Therefore, there is a demand for an image display method in which an image including a past hairstyle is appropriately discriminated even if the hairstyle is different, and the current hairstyle is replaced with the past hairstyle and displayed.

Japanese Patent No. 5434250 Japanese Patent No. 4473754

An embodiment of the present invention provides an image processing device, an image processing system, a user terminal, an image processing method, and a program for displaying different hairstyles.

In order to achieve the above problems, the image processing apparatus of the embodiment includes a hairstyle extraction unit that extracts a partial image corresponding to the hairstyle from the first image of the user including the hairstyle and the face, and a second image different from the first image. The hairstyle of the user of the image is replaced with the partial image and displayed.

The figure which shows the outline of the image display method of 1st Embodiment. The block diagram which shows the image display system which executes the image display method which concerns on 1st Embodiment. The figure which shows the flowchart which replaces the hairstyle of the image display method which concerns on 1st Embodiment with a partial image and displays. The figure which shows the flowchart which stores an image in the storage part used in the image display method which concerns on 1st Embodiment.

Hereinafter, the image display method according to the embodiment will be described with reference to FIGS. 1, 2, and 3. FIG. 1 is a diagram showing an outline of an image display method of the first embodiment, FIG. 2 is a block diagram showing an image display system, and FIG. 3 is a flowchart of an image display method according to the embodiment. ..

(First Embodiment)
The image display system according to the present embodiment automatically recognizes the input current image (FIG. 1 (a)) and selects similar images (FIG. 1 (b)), and the hairstyle (partial image) of the selected image. ) Is extracted (FIG. 1 (c)) and displayed on the user terminal 10 in place of the hairstyle of the current image (FIG. 1 (d)).

As shown in FIG. 2, the image display system according to the present embodiment includes a user terminal 10 and an image processing server 20. The user terminal 10 includes an image input unit 15, a display unit 16, and a terminal-side transmission / reception unit 17.

The image input unit 15 refers to a photographing device such as a camera, and inputs the current image (referred to as image A) of the user including the hairstyle and face to the user terminal 10. Here, the current image is an image input from the image input unit 15 at a certain current time t by the user.

When displaying the image A, the display unit 16 replaces the current hairstyle with the hairstyle (referred to as image D) transmitted from the image processing server 20 and displays the image A. At this time, the position and size of the image D transmitted from the image processing server 20 are adjusted.

The terminal-side transmission / reception unit 17 transmits the image A to the image processing server 20, while receiving the image D output from the image processing server 20.

The image processing server 20 includes a server-side transmission / reception unit 25, an image processing unit 30, and a storage unit 40.

The storage unit 40 includes an image DB 41 and a feature vector DB 42. The image DB 41 stores in advance the past images of the user including the i-th (i = 1, 2, ..., I) image C. I may be the number of images of a specific user stored in the image DB 41, or may be the total number of images of a plurality of users. The feature vector DB 42 stores each feature vector αi (i = 1, 2, ..., I) corresponding to these past images.

The server-side transmission / reception unit 25 transmits / receives images to and from the terminal-side transmission / reception unit 17.

The image processing unit 30 includes an image analysis unit 31, a feature vector calculation unit 32, a similarity selection unit 33, and a hairstyle extraction unit 34. The image analysis unit 31 analyzes the image A transmitted from the server-side transmission / reception unit 25, and extracts only a part thereof, for example, the face (the extracted image is referred to as an image B). The feature vector calculation unit 32 calculates the feature vector αb of the image B.

The similarity selection unit 33 has a similarity Hi between the feature vector αb calculated by the feature vector calculation unit 32 and the feature vector αi (i = 1, 2, ..., I) stored in the feature vector DB 42. Calculate (i = 1, 2, ..., I).

Further, the similarity selection unit 33 compares the similarity Hi (i = 1, 2, ..., I) with the predetermined threshold values HO to determine whether or not they are similar.

The similarity selection unit 33 instructs the hairstyle extraction unit 34 to select the image C corresponding to the similarity Hi determined to be similar. The image C may have the latest saved year and month, or may have the latest 3 months.

The hairstyle extraction unit 34 extracts an image C corresponding to the similarity Hi determined to be similar from the storage unit 40, extracts only a partial region of the hairstyle to generate an image D, and transmits the image D to the server-side transmission / reception unit 25.

Next, the flow until the image display system 1 replaces the hairstyle of the current image A with the partial image D of the past image C and displays it with reference to FIG. 2 will be described in detail.

The image display system 1 has two operation modes.

First, with reference to FIG. 2, a first operation mode in which the hairstyle of the image A is replaced with the image D and displayed will be described.

S1 ... The image input unit 15 inputs the image A after detecting that the user terminal 10 and the user have confronted each other via a signal from the outside. The image input unit 15 indicates a photographing device such as a camera, and at least an image may be captured. The image A input from the image input unit 15 is transmitted from the image input unit 15 to the server-side transmission / reception unit 25 of the image processing server 20 via the terminal-side transmission / reception unit 17.

S2 ... The server-side transmission / reception unit 25 transmits the received image A to the image analysis unit 31. After performing image recognition of the image A, the image analysis unit 31 extracts the face of the image A to generate the image B. The image B is transmitted from the image analysis unit 31 to the feature vector extraction unit 32. The image recognition is to recognize the positional relationship of each part such as the hairstyle, eyes, and nose of the image A.

S3 ... The feature vector extraction unit 32 detects the feature points of the image B and calculates the feature vector αb of the image B. The calculated feature vector αb is transmitted to the similarity sorting unit 33.

S4 ... The similarity selection unit 33 calculates the similarity Hi between the received feature vector αb and the feature vector αi (i = 1, 2, ..., I) stored in the feature vector DB 42. The similarity may be calculated, for example, by calculating the inner product of αb and αi.

The feature vector αi stored in the feature vector DB 42 is calculated in advance for the i-th image C, and is the same as the feature vector calculated by the feature vector calculation unit 32.

S5 ... The similarity selection unit 33 compares a preset threshold value of similarity (referred to as H0) with the similarity Hi and selects a past image similar to image B. In the present embodiment, when the relationship of Hi ≧ H0 is established, it can be evaluated as “similar”. When the similarity selection unit 33 evaluates as "similar", that is, when the relationship of Hi ≥ H0 holds, the similarity selection unit 33 is associated with the feature vector αi exceeding the threshold value among the arbitrary feature vectors αi. A signal instructing the hairstyle extraction unit 34 to transmit the image C is transmitted to the image DB 41. On the other hand, when the similarity selection unit 33 evaluates as "not similar", the similarity selection unit 33 continuously calculates the similarity between αb and the other feature vector αi. In this way, for example, when there are a plurality of items exceeding the threshold value HO, the largest one may be determined to be "similar", or the plurality of items may be determined to be "similar".

Finally, when the feature vector αi exceeding the threshold value does not exist (Hi <H0) or the feature vector αi is not stored in the similarity DB 42, the similarity selection unit 33 is the server side transmission / reception unit 25 and the terminal. An image instructing the display unit 16 to display "no similar image" is transmitted via the side transmission / reception unit 17.

S6 ... The hairstyle extraction unit 34 divides the image C evaluated as "similar" into a plurality of regions, extracts a partial image of only the hairstyle region, and generates an image D. The image D generated in this way is transmitted to the display unit 16 via the server-side transmission / reception unit 25 and the terminal-side transmission / reception unit 17.

S7 ... The display unit 16 adjusts the position and size of the received image D, replaces the hairstyle of the image A with the image D, and displays the image D. When the user looks at the display unit 16 and wishes to perform the treatment on the past hairstyle image D, the treatment can be performed in that way.

Next, with reference to FIG. 3, a second operation mode in which the storage unit 40 stores an image including the hairstyle after the treatment is completed will be described. For example, after the treatment is completed, the user in the beauty salon or the barber is asked if the image can be saved if he / she likes the hairstyle treated this time, and if the user understands, the image including the hairstyle after the treatment is included. By newly storing the image in the storage unit, the user can select the image as a past image when he / she visits the beauty salon or the barber shop next time.

S11 ... After the treatment, the image input unit 15 inputs the image A1 including the hairstyle after the treatment of the user. The image A1 input from the image input unit 15 is transmitted from the image input unit 15 to the server-side transmission / reception unit 25 of the image processing server 20 via the terminal-side transmission / reception unit 17.

S12 ... The server-side transmission / reception unit 25 transmits the received image A1 to the image analysis unit 31 and the image DB 41. The image analysis unit 31 extracts the face of the image A1 and generates the image B1. The image B1 is transmitted from the image analysis unit 31 to the feature vector extraction unit 32.

S13 ... The feature vector extraction unit 32 detects the feature points of the image B1 and calculates the feature vector αI + 1 of the image B. The calculated feature vector αI + 1 is transmitted to the feature vector DB42.

S14 ... The storage unit 40 stores the image A1 in the image DB 41 as the I + 1th past image, and stores the feature vector vector αI + 1 of the image B1 in the feature vector DB 42.

As a result, since a new image is added and stored in the storage unit 40, the number of images that can be selected by the user increases. By repeating the above-mentioned two operation modes, past images and feature vectors are gradually accumulated in the storage unit 40.

According to the above-described embodiment, the image display system extracts and compares facial feature vectors from the past user's images stored in the storage unit, identifies an image having a close feature vector, and identifies an image having a close feature vector to the past image. Only the hairstyle is extracted, and the hairstyle of the current image is replaced with the past hairstyle and displayed on the display unit. In this way, the finished image when the user has received hair beauty and hairdressing before is memorized, and if you visually convey the image of the hairstyle you want this time while using that image, it is better than telling it in words. I can tell you exactly.
(Second Embodiment)
The points different from the first embodiment will be mainly described.

In the image display system according to the first embodiment, the similarity selection unit 33 automatically selects one of the plurality of images C determined to be similar and instructs the hairstyle extraction unit 34. In the image display system according to the embodiment, the user can select and instruct the hairstyle extraction unit 34 from among the plurality of images C determined to be similar by the similarity selection unit 33. Therefore, it is possible to select the hairstyle of another user or model that is close to the user's feature vector.

Specifically, the similarity selection unit 33 displays a plurality of images C corresponding to a plurality of αi determined to be similar to the user via the display unit 16 in which the relationship of Hi ≧ H0 is established. The user selects from the plurality of displayed images C and displays the hairstyle.

As described above, if the hairstyles of another user or model close to the user's feature vector are selected and displayed on the display unit 16, another hairstyle that the user may look good on can be proposed. Therefore, the range of proposals by hairdressers and barbers and the range of user choices are widened.

Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the description. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

10 ... User terminal, 15 ... Image input unit, 16 ... Display unit, 17 ... Terminal side transmission / reception unit, 20 ... Image processing server, 25 ... Server side transmission / reception unit, 30 ... Image processing unit, 31・ ・ Image analysis unit, 32 ・ ・ Feature vector calculation unit, 33 ・ ・ Similarity selection unit, 34 ・ ・ Hairstyle extraction unit, 40 ・ ・ Storage unit, 41 ・ ・ Image DB, 42 ・ ・ Feature vector DB.

Claims (17)

A hairstyle extraction unit that extracts a partial image corresponding to the hairstyle from the user's first image including the hairstyle and face,
A display unit that replaces the user's hairstyle of the second image different from the first image with the partial image and displays the hairstyle.
An image processing device comprising. The image processing apparatus according to claim 1, wherein the first image and the second image are images generated at different times. The image processing apparatus according to claim 1 or 2, wherein the first image is a past image, and the second image is a current image. The image processing apparatus according to any one of claims 1 to 3, wherein the first image and the second image are images of the same user. The item according to any one of claims 1 to 4, further comprising a sorting unit for selecting the first image of the user from an image database in which the first image of one or more users is stored based on the facial features of the user. The image processing apparatus described. The feature quantity is a feature vector.
The sorting unit calculates the degree of similarity between the feature vector calculated from the user's face and the feature vector in the feature vector database in which the feature vector corresponding to each of the first images in the image database is stored. The image processing apparatus according to claim 5, wherein the first image is selected based on the similarity. The image processing apparatus according to claim 6, wherein the sorting unit sorts the first image when the similarity is equal to or higher than a threshold value. An image processing system that acquires an image including a hairstyle and a face, has a storage unit for storing a past image including the hairstyle and the face, and displays a partial image.
An image processing unit that compares the face included in the acquired image with the face included in the past image in the storage unit and extracts a partial image of a hairstyle portion of a similar past image.
A display unit that superimposes and displays the extracted partial image of the hairstyle portion on the acquired image, and
An image processing system equipped with. A storage unit for storing a past image including a hairstyle and a face is provided, and a hairstyle of a similar past image is compared with a face included in an image acquired by a user terminal and a face included in a past image in the storage unit. A user terminal connected to an image processing server that extracts a partial image of a part.
An image input unit for inputting the user's image and
A display unit that superimposes a partial image of the extracted hairstyle portion on the user's image and displays it.
User terminal with. The user terminal according to claim 9, wherein the display unit has at least a mirror surface. An image processing system consisting of a terminal and an image processing server.
The terminal
A display unit that displays images and
A transmitter / receiver that communicates with the image processing server
Image input section and
Have and
The image processing server
A transmitter / receiver that sends / receives images
Image processing unit and
Has at least a memory
The image processing unit
A feature amount calculation unit that calculates a first feature amount from the user's face from the user's third image including a hairstyle and a face, and a feature amount calculation unit.
A similarity selection unit that calculates the similarity between the first feature amount and the second feature amount based on the image stored in the storage unit and selects the fourth image,
It has a hairstyle extraction unit that extracts a fifth image, which is a partial image corresponding to the hairstyle of the fourth image.
An image processing system in which the fifth image is displayed on the display unit. A server provided with an image database that stores past images including hairstyles and faces for one or more users, and a feature amount database that stores facial features corresponding to each of the past images in the image database. And in an image processing system having a user terminal having a display unit
An image processing method in which when the similarity between the current image and the past image is equal to or greater than a threshold value, an image of the past hairstyle portion is superimposed on an image including the user's face and displayed on the display unit. A server provided with an image database that stores past images including hairstyles and faces for one or more users, and a feature amount database that stores facial features corresponding to each of the past images in the image database. In an image processing system including a user terminal including a transmission / reception unit and a display unit that communicate with the server.
The post-treatment image including the hairstyle after the treatment is saved in the image database.
An image processing method for displaying at least a part of a post-treatment image on the display unit as a past image. It has the image processing system according to claim 11.
The fourth image is an image of a model different from that of the user.
An image processing method in which a hairstyle that the user may look good on is displayed as a proposal on the display unit. An image database that stores past images including the user's hairstyle and face,
A feature database that stores the past features of the face calculated from the past images, and
When the similarity between the current facial features calculated based on the user's current facial image and the past features is equal to or greater than the threshold value, the portion of the past image corresponding to the hairstyle is extracted. Image processing unit and
An image processing system equipped with. The image input unit that captures the current facial image and
A display unit that superimposes and displays a portion corresponding to the hairstyle extracted by the image processing unit on the current facial image, and
The image processing system according to claim 15. A step of extracting a partial image corresponding to the hairstyle from the user's first image including the hairstyle and face, and
A program that causes a computer to perform a step of replacing a user's hairstyle of a second image different from the first image with the partial image and displaying the hairstyle.

Images