JP2021043797A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

To identify a user's specific area to be processed and apply image processing preferable for the user.SOLUTION: An image processing device includes: a photographing unit (220) that photographs a user to generate image data; an extraction unit (303) that extracts a first area which is an outline area of a predetermined portion of the user's face from the image data using a predetermined image extraction method; a generation unit (304) that uses the first area to generate a second area including at least part of the first area and identified as an image processing object; and an image processing unit (305) that processes the second area of the image data by a predetermined method.SELECTED DRAWING: Figure 10

Description

The present invention relates to an image processing device, an image processing method, and an image processing program that process a specific part of a face appearing in a captured image.

Conventionally, there is known a photo sticker creating device that performs predetermined image processing on a captured image and provides it to a user. Specifically, there is a device that recognizes the lip portion and performs image processing on the lip image. (See, for example, Patent Document 1).

In Patent Document 1, for example, a region of a predetermined color existing in a lower region in a face image is recognized as a lip, a mask image corresponding to the recognized lip portion is generated, and the lip image is generated. Described to generate.

Japanese Patent No. 5549727

In Patent Document 1, since the lip portion is specified by the position and color, there is a possibility that the lips cannot be accurately specified depending on the orientation of the face, the color of the lips, the presence of red acne, the presence of obstacles, and the like. In addition, for example, by wearing eyeglasses, the eyes may not be accurately identified, and other parts may not be accurately identified.

The present invention provides an image processing apparatus, an image processing method, and an image processing program that solves the above problems, identifies a specific area of a user to be processed, and performs image processing preferable to the user. With the goal.

A first aspect of the image processing apparatus according to the present invention is an imaging unit that photographs a user and generates image data, and a contour of a predetermined portion of the user's face using a predetermined image extraction method from the image data. An extraction unit that extracts a first region, which is a region, and a generation unit that uses the first region to generate a second region that includes at least a part of the first region and is specified as an image processing target. And an image processing unit that processes the second region of the image data by a predetermined method.

As a result, it is possible to specify a specific area of the user to be processed and perform image processing preferable to the user.

A second aspect of the image processing apparatus according to the present invention is a photographing unit that photographs a user and generates image data, and a contour region of a predetermined portion of the user's face using semantic segmentation from the image data. An extraction unit for extracting a region, a generation unit for generating a second region including at least a part of the first region using the first region, and a second region of image data are defined. It is provided with an image processing unit that processes by the above-mentioned method.

As a result, it is possible to specify a specific area of the user to be processed and perform image processing preferable to the user.

The extraction unit extracts a first region of the user's mouth from the upper lip, the lower lip, and other than the lips, and the generation unit generates a second region corresponding to each first region and images. The processing unit may perform a treatment for the upper lip on the region of the upper lip and a treatment for the lower lip on the region of the lower lip.

As a result, it is possible to specify an area of the user's mouth and perform image processing preferable to the user.

The extraction unit extracts at least one of the tongue, teeth, gingiva or mucous membrane as the first region other than the lips, and the generation unit generates a second region corresponding to each first region and performs image processing. The unit may execute processing for each area generated by the generation unit.

As a result, it is possible to specify an area inside the user's mouth and perform image processing preferable to the user.

The upper lip and lower lip areas of the second area can be the preferred areas for applying lipstick.

As a result, it is possible to perform image processing preferable to the user with respect to the aesthetic appearance of the user's mouth.

The extraction unit extracts the first region of the black eye for the user's eye, the generation unit generates the second region corresponding to the first region of the black eye, and the image processing unit generates the second region of the black eye. The processing for black eyes may be executed for the area of.

As a result, it is possible to specify an area for the user's black eye and perform image processing preferable to the user.

The extraction unit extracts the first region of the white eye for the user's eye, the generation unit generates the second region corresponding to the first region of the white eye, and the image processing unit generates the second region of the white eye. On the other hand, the process for white eyes may be executed.

As a result, it is possible to specify an area for the white of the user and perform image processing preferable to the user.

The second region can be a preferred range for changing at least either size or color.

As a result, it is possible to perform image processing preferable to the user with respect to the aesthetic appearance of the user's eyes.

The extraction unit further extracts the first region of the eyelashes, the generation unit generates a second region corresponding to the first region of the eyelashes, and the image processing unit generates the second region of the eyelashes with respect to the second region of the eyelashes. Processing for eyelashes may be performed.

Thereby, the area of the eyelashes of the user can be specified, and image processing preferable to the user can be performed.

The generation unit may generate a second region using a machine learning model learned using the learning data generated in advance.

As a result, it is possible to perform image processing preferable to the user by performing image processing on the generated area.

The image processing apparatus may further include a printing processing unit that prints the processed image data on the sticker paper.

Thereby, the image-processed image data can be provided as a sticker paper.

The first aspect of the image processing method according to the present invention is a step of photographing a user and generating image data, and a contour region of a predetermined portion of the user's face using a predetermined image extraction method from the image data. A step of extracting the first region, and a step of using the first region to generate a second region including at least a part of the first region and specified as an image processing target, and an image. It has a step of processing a second area of data in a defined manner.

As a result, it is possible to specify a specific area of the user to be processed and perform image processing preferable to the user.

The first aspect of the image processing program according to the present invention is to use a computer, a photographing unit that photographs a user and generates image data, and a predetermined image extraction method from the image data to determine the user's face. Using the extraction unit that extracts the first region, which is the contour region of the portion, and the first region, a second region that includes at least a part of the first region and is specified as an image processing target is generated. It functions as a generation unit to be generated and an image processing unit that processes a second area of image data by a predetermined method.

As a result, it is possible to specify a specific area of the user to be processed and perform image processing preferable to the user.

According to the present invention, image processing can be performed by specifying a specific area of a user to be processed.

It is a perspective view of the photographic sticker making apparatus which is one Embodiment of this invention. It is a perspective view of the photograph sticker making apparatus. It is a figure for demonstrating the space movement of a user in a photo sticker making game. It is a front view of the preselection part. It is a front view of a photographing part. It is a front view of the background part on the shooting space side. It is a front view of the editing space side of an editing unit. It is a left side view of an editing unit. It is a block diagram which shows the internal structure of the photographic sticker making apparatus. It is a block diagram which shows the functional block of the photograph sticker making apparatus. This is an image example for explaining the area extraction of the mouth portion in the photo sticker making device. This is an image example for explaining the area extraction of the eye portion in the photo sticker making device. This is an example of a characteristic mouth image taken by a photo sticker making device. This is another example of a characteristic mouth image taken by a photo sticker making device. This is another example of a characteristic mouth image taken by a photo sticker making device. This is another example of a characteristic mouth image taken by a photo sticker making device. This is an example of an image of a characteristic eye portion taken by a photo sticker making device. This is another example of an image of a characteristic eye portion taken by a photo sticker making device. This is another example of an image of a characteristic eye portion taken by a photo sticker making device. This is another example of an image of a characteristic eye portion taken by a photo sticker making device. It is a flowchart which shows a series of operations about a photo sticker making game by a photo sticker making apparatus. It is a flowchart which shows the detail of a synthesis process. This is an example of an editing screen used in the editing process.

Hereinafter, specific embodiments of the present invention will be described with reference to the accompanying drawings. Members having the same function are designated by the same reference numerals, and the description thereof will be omitted as appropriate. Furthermore, the shape of the configuration described in each drawing and the dimensions such as length, depth and width do not reflect the actual shape and dimensions, and are appropriately changed for the sake of clarity and simplification of the drawings. ing.

The image processing apparatus according to the present invention will be described as an example photo sticker creating apparatus. The photo sticker creating device to which the present invention is applied is a game device in which a user is made to perform shooting and editing as a game, and the photographed and edited image is provided to the user as a photo sticker and data. The photo sticker creating device 1 is installed in, for example, a game center, a shopping mall, a store in a tourist spot, or the like.

In the game provided by the photo sticker making device, the user takes a picture of himself / herself or the like by using the camera provided in the photo sticker making device. The user synthesizes a foreground image and / or a background image with the captured image, and uses an editing function (doodle editing function) such as pen image input or stamp image input as a composite image for editing. Edit and design the captured image to make it colorful. After the game is over, the user receives a photo sticker or the like on which the edited image is printed as a deliverable. Alternatively, the photo sticker creating device provides the edited image to the user's mobile terminal, and the user can also receive the deliverable by the mobile terminal.

(Configuration of photo sticker making device)
1 and 2 are perspective views showing a configuration example of the appearance of the photo sticker making device 1. The photo sticker creating device 1 is a game machine that provides a photographed image and an edited image. The photo sticker creating device 1 provides an image to a user by printing the image on a sticker paper or transmitting the image to a server in order to make the image viewable on the user's mobile terminal. The users of the photo sticker making device 1 are mainly young women such as junior high school girls and high school girls. In the photo sticker creating device 1, a plurality of users, such as mainly two or three people, can enjoy the game per group. Of course, in the photo sticker creating device 1, one user can enjoy the game.

In the photo sticker creating device 1, the user takes a picture by himself / herself as a subject. By editing, the user synthesizes a composite image such as handwritten characters or a stamp image with an image selected from the captured images obtained by shooting. As a result, the captured image is edited into a gorgeous image. The user receives a sticker paper on which the edited image, which is an edited image, is printed, and ends the series of games.

As shown in FIG. 1, the photo sticker creating device 1 is basically configured by being installed in a state where the photographing unit 11 and the editing unit 12 are in contact with each other.

The photographing unit 11 is composed of a preselection operation unit 20, a photographing operation unit 21, and a background unit 22. The preselection operation unit 20 is installed on the side surface of the shooting operation unit 21. The space in front of the preselection operation unit 20 is the preselection space in which the preselection process is performed. Further, the photographing operation unit 21 and the background unit 22 are installed at a predetermined distance. The space formed between the shooting operation unit 21 and the background unit 22 is the shooting space where the shooting process is performed.

As the preselection process, the preselection operation unit 20 provides guidance for introducing the game provided by the photo sticker creation device 1, and causes the user to select various settings in the shooting process performed in the shooting space. Or do. The preselection operation unit 20 is provided with a coin slot for the user to insert a price, a touch panel monitor used for various operations, and the like. The preselection operation unit 20 appropriately guides the user in the preselection space to the shooting space according to the availability of the shooting space.

The shooting operation unit 21 is a device for shooting a user as a subject. The shooting operation unit 21 is located in front of the user who has entered the shooting space. A camera, a touch panel monitor used for various operations, and the like are provided in front of the shooting operation unit 21 facing the shooting space. Assuming that the left side surface is the left side surface and the right side surface is the right side surface when viewed from the user facing the front in the shooting space, the left side surface of the shooting operation unit 21 is composed of the side panel 41A, and the right side surface is formed by the side panel 41B. It is composed. Further, the front surface of the photographing operation unit 21 is composed of the front panel 42. It is assumed that the above-mentioned preselection operation unit 20 is installed on the side panel 41A. The preselection operation unit 20 may be installed on the side panel 41B, or may be installed on both the side panels 41A and 41B.

The background portion 22 is composed of a back panel 51, a side panel 52A, and a side panel 52B. The back panel 51 is a plate-shaped member located on the back side of the user facing the front. The side panel 52A is a plate-shaped member attached to the left end of the back panel 51 and having a width narrower than that of the side panel 41A. The side panel 52B is a plate-shaped member attached to the right end of the back panel 51 and having a width narrower than that of the side panel 41B.

The side panel 41A and the side panel 52A are provided on substantially the same plane. The upper portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A which is a plate-shaped member. The lower part of the side panel 41A and the side panel 52A are connected by a connecting portion 23A'which is, for example, a metal member provided on the floor surface. Similarly, the side panel 41B and the side panel 52B are provided on substantially the same plane. The upper portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B. The lower part of the side panel 41B and the side panel 52B is connected by the connecting portion 23B'.

A green chroma key sheet is attached to the surface of the back panel 51 on the shooting space side, for example. The photo sticker creating device 1 takes a picture with a sheet for chroma key as a background, and performs chroma key composition in a shooting process and an editing process. As a result, the background image desired by the user is combined with the sheet portion.

The opening formed by being surrounded by the side panel 41A, the connecting portion 23A, and the side panel 52A serves as an entrance / exit of the photographing space. Further, an opening formed by being surrounded by the side panel 41B, the connecting portion 23B, and the side panel 52B also serves as an entrance / exit of the photographing space.

Above the shooting space, a ceiling surrounded by the front surface of the shooting operation unit 21, the connecting portion 23A, and the connecting portion 23B is formed. A ceiling strobe unit 24 is provided on a part of the ceiling. One end of the ceiling strobe unit 24 is fixed to the connecting portion 23A, and the other end is fixed to the connecting portion 23B. The ceiling strobe unit 24 has a built-in strobe that irradiates light into the shooting space in accordance with shooting. Inside the ceiling strobe unit 24, a fluorescent lamp is provided in addition to the strobe. As a result, the ceiling strobe unit 24 also functions as lighting for the shooting space.

The editing unit 12 is a device for editing a captured image. The editing unit 12 is connected to the shooting unit 11 so that one side surface is in contact with the front panel 42 of the shooting operation unit 21.

Assuming that the configuration of the editing unit 12 shown in FIG. 1 is the configuration on the front side, the configuration used in the editing work is provided on each of the front side and the back side of the editing unit 12. With this configuration, two sets of users can perform editing work at the same time.

The front side of the editing unit 12 is composed of a surface 61 and a slope 62 formed above the surface 61. The surface 61 is perpendicular to the floor surface and substantially parallel to the side panel 41A of the photographing operation unit 21. A monitor with a built-in tablet and a touch pen are provided on the slope 62 as a configuration used for editing work. On the left side of the slope 62, a columnar support portion 63A that supports one end of the lighting device 64 is provided. On the right side of the slope 62, a columnar support portion 63B that supports the other end of the lighting device 64 is provided. A support portion 65 for supporting the curtain rail 26 is provided on the upper surface of the support portion 63A.

A curtain rail 26 is attached above the editing unit 12. The curtain rail 26 is composed of three rails 26A, 26B, and 26C in combination. The three rails 26A, 26B, and 26C are combined so that the shape when viewed from above is U-shaped. One ends of the rail 26A and the rail 26B provided in parallel are fixed to the connecting portion 23A and the connecting portion 23B, respectively, and the other ends of the rail 26A and the rail 26B are joined to both ends of the rail 26C, respectively.

A curtain is attached to the curtain rail 26 so that the space in front of the front and the space in front of the back of the editing unit 12 cannot be seen from the outside. The space in front of the front and the space behind the back of the editing unit 12 surrounded by the curtain are the editing spaces in which the user performs editing work.

Further, as will be described later, a discharge port for discharging the printed sticker paper is provided on the left side surface of the editing unit 12. The space in front of the left side of the editing unit 12 is a printing waiting space for the user to wait for the printed sticker paper to be ejected.

(Movement of users)
Here, the flow of the image creation game and the movement of the user accompanying it will be described. FIG. 3 is a diagram for explaining the spatial movement of the user during the photo sticker creation game.

First, the user inserts the price into the coin slot in the preselection space A0, which is the space in front of the preselection operation unit 20. Next, the user makes various settings according to the screen displayed on the touch panel monitor. The user performs, for example, selection of a course related to shooting performed in the shooting space as a preselection operation.

The user who has completed the pre-selection work has a shooting space formed between the shooting operation unit 21 and the background unit 22 from the entrance / exit G1 between the side panel 41A and the side panel 52A, as shown by the white arrow # 1. Enter A1. Then, the user performs the shooting work by using the camera or the touch panel monitor provided in the shooting operation unit 21.

The user who has completed the shooting work either exits the shooting space A1 from the entrance / exit G1 and moves to the editing space A2-1 as indicated by the white arrow # 2, or moves to the editing space A2-1 as indicated by the white arrow # 3. Exits the shooting space A1 and moves to the editing space A2-2.

The editing space A2-1 is an editing space on the front side of the editing unit 12. On the other hand, the editing space A2-2 is an editing space on the back side of the editing unit 12. Whether the user moves to the editing space A2-1 or the editing space A2-2 is guided by the screen display of the touch panel monitor of the shooting operation unit 21 or the like. For example, the vacant one of the two editing spaces is guided as the destination. The user who has moved to the editing space A2-1 or the editing space A2-2 starts the editing work. The user of the editing space A2-1 and the user of the editing space A2-2 can perform editing work at the same time.

After the editing work is completed, printing of the edited image is started. When printing is started, the user who has completed the editing work in the editing space A2-1 moves from the editing space A2-1 to the print waiting space A3 as indicated by the white arrow # 4. Further, the user who has completed the editing work in the editing space A2-2 moves from the editing space A2-2 to the print waiting space A3 as indicated by the white arrow # 5.

The user who has moved to the print waiting space A3 waits for the end of printing the image. When the printing is completed, the user receives the sticker paper discharged from the discharge port provided on the right side surface of the editing unit 12, and ends the series of image creation games.

(Structure of preselection section)
Next, the configuration of each device will be described. FIG. 4 is a front view of the preselection operation unit 20.

A touch panel monitor 71 is provided on the upper side of the preselection operation unit 20. The touch panel monitor 71 is composed of a monitor such as an LCD (Liquid Crystal Display) and a touch panel laminated on the monitor. The touch panel monitor 71 has a function of displaying various GUIs (Graphical User Interfaces) and accepting a user's selection operation. On the touch panel monitor 71, selection of a course related to shooting, selection of a background image in the image to be edited to be edited, layout of the created image, BGM (Back Ground Music) flowing during the image creation game, sound and sound A screen used for the pre-selection process for selecting at least one of them and inputting the user's name is displayed.

There are two courses for shooting, a course for two people who shoot with two users, and a course for a large number of people who shoot with three or more people. In addition, a couple course may be prepared in which a couple of men and women take pictures.

A speaker 72 is provided below the touch panel monitor 71. The speaker 72 outputs guidance voice, BGM, sound effect, etc. for preselection processing. Further, a coin insertion / return port 73 for inserting coins by the user is provided adjacent to the speaker 72.

(Structure of shooting section)
FIG. 5 is a front view of the photographing operation unit 21 as the photographing unit. The photographing operation unit 21 is configured to be surrounded by the side panel 41A, the side panel 41B, and the front panel 42.

A camera unit 81 is provided in the center of the front panel 42. The camera unit 81 includes a camera 91 and a touch panel monitor 92 as a display unit.

The camera 91 is, for example, a single-lens reflex camera, and is mounted inside the camera unit 81 so that the lens is exposed. The camera 91 has an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and photographs a user in the photographing space A1. The moving image captured by the camera 91 (hereinafter, also referred to as a live view display image) is displayed on the touch panel monitor 92 in real time. A still image captured by the camera 91 at a predetermined timing such as when shooting is instructed is saved as a shot image.

The touch panel monitor 92 is provided below the camera 91. The touch panel monitor 92 is composed of a monitor such as an LCD and a touch panel laminated on the monitor. The touch panel monitor 92 has a function as a live view monitor that displays a moving image captured by the camera 91, and a function that displays various GUIs and accepts a user's selection operation. Specific examples of the selection operation include selection of a shooting course, instructions for starting and ending shooting (shooting control instructions), selection of eye deformation and color, and selection of skin color correction, and a background image in the created image. The selection of the image and the selection of the BGM (sound / voice) in the image creation game can be mentioned. A moving image (live view image) or a still image (photographed image) captured by the camera 91 is displayed on the touch panel monitor 92.

Above the camera unit 81, an upper strobe 82 with a curved light emitting surface facing the user is installed. The upper strobe 82 irradiates the user's face and upper body with light from above the front surface.

Further, below the camera unit 81, a foot strobe 85 that irradiates the lower body and feet of the user with light is provided. The upper strobe 82 and the foot strobe 85 include a strobe and a fluorescent lamp.

Although not shown in FIGS. 1 and 5, a speaker 93 is provided near the ceiling of the front panel 42, for example. The speaker 93 outputs guidance voice, BGM, sound effect, etc. of the shooting process by the voice signal output from the control unit 201.

(Structure of background part)
FIG. 6 is a front view of the background portion 22 on the shooting space A1 side.

A strobe 101 on the back surface is installed above the back panel 51. The strobe 101 on the back surface irradiates the user with light from above the back surface.

In the figure, the rear left strobe 102 is installed on the left side of the rear panel 51. The rear left strobe 102 irradiates the user from the rear right side. In the figure, the rear right strobe 103 is installed on the right side of the rear panel 51. The rear right strobe 103 irradiates the user from the rear left side.

Further, the chroma key sheet 121 may be attached to the surface of the back panel 51 on the shooting space A1 side (front side in the drawing). The color of the chroma key sheet 121 is, for example, green.

Although not shown, a chroma key sheet may be attached to the surface of the side panels 52A and 52B on the photographing space A1 side as well as the chroma key sheet 121.

(Organization of editing unit)
FIG. 7 is a front view of the editing unit 12 on the editing space A2-1 side.

A tablet built-in monitor 131 is provided at substantially the center of the slope 62. A touch pen 132A is provided on the left side of the tablet built-in monitor 131. A touch pen 132B is provided on the right side of the tablet built-in monitor 131. The tablet built-in monitor 131 is configured by providing the tablet so as to expose the display. The tablet enables operation input using the touch pen 132A or the touch pen 132B. The tablet built-in monitor 131 displays, for example, an editing screen used for editing work. For example, when two users perform editing work at the same time, the touch pen 132A is used by the user on the left side of the tablet built-in monitor 131, and the touch pen 132B is used by the user on the right side of the tablet built-in monitor 131. Used.

FIG. 8 is a left side view of the editing unit 12.

A sticker paper discharge port 161 is provided on the lower side of the left side surface of the editing unit 12. A printer as an output unit is provided inside the editing unit 12. The printer prints an image of the user of the editing space A2-1 or an image of the user of the editing space A2-2 on the sticker paper in a predetermined layout and discharges the image from the sticker paper discharge port 161.

(Internal configuration of photo sticker making device)
FIG. 9 is a block diagram showing an example of the internal configuration of the photo sticker creating device 1. In FIG. 9, the same components as those described above are designated by the same reference numerals. Duplicate explanations will be omitted as appropriate.

The control unit 201 is composed of a CPU (Central Processing Unit) and the like. The control unit 201 executes a program stored in the ROM (Read Only Memory) 206 or the storage unit 202, and controls the overall operation of the photo sticker creation device 1. A storage unit 202, a communication unit 203, a drive 204, a ROM 206, and a RAM (Random Access Memory) 207 are connected to the control unit 201. Each configuration of the preselection operation unit 20, the shooting operation unit 21, the background unit 22, the editing operation units 27A and 27B, and the print operation unit 28 is also connected to the control unit 201.

The storage unit 202 is a non-volatile storage medium such as a hard disk or a flash memory. The storage unit 202 stores various setting information and the like supplied from the control unit 201. The information stored in the storage unit 202 is appropriately read out by the control unit 201.

The communication unit 203 is an interface of a network such as the Internet. The communication unit 203 communicates with an external device according to the control by the control unit 201. The communication unit 203 transmits, for example, a photographed image or an edited image selected by the user to the server. The image transmitted from the communication unit 203 is allocated and stored in a predetermined storage area in the server, and is displayed or downloaded to the mobile terminal that has accessed the server.

A removable media 205 made of an optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 204. The programs and data read from the removable media 205 by the drive 204 are supplied to the control unit 201, and are stored or installed in the storage unit 202.

The ROM 206 stores programs and data executed by the control unit 201. The RAM 207 temporarily stores data and programs processed by the control unit 201.

The preselection operation unit 20 realizes the preselection process for the user in the preselection space A0. The preselection operation unit 20 includes a touch panel monitor 71, a speaker 72, and a coin processing unit 74.

The touch panel monitor 71 displays various selection screens according to the control by the control unit 201, and accepts the user's operation on the selection screen. An input signal representing the content of the user's operation is supplied to the control unit 201, and various settings are made.

The coin processing unit 74 detects the insertion of coins into the coin insertion / return port 73. When the coin processing unit 74 detects that a predetermined amount of coins has been inserted, the coin processing unit 74 outputs an activation signal instructing the start of the game to the control unit 201.

The shooting operation unit 21 realizes shooting processing for the user in the shooting space A1. The photographing unit 220 includes an upper strobe 82, a left strobe 83, a right strobe 84, a foot strobe 85, a camera 91, a touch panel monitor 92, and a speaker 93.

The upper strobe 82, the left strobe 83, the right strobe 84, and the foot strobe 85 are arranged in the photographing space A1 and emit light according to the illumination control signal supplied from the control unit 201.

The camera 91 shoots according to the shutter control by the control unit 201, and outputs the captured image (image data) obtained by the shooting to the control unit 201.

The editing operation unit 27A realizes an editing process for a user in the editing space A2-1. The editing operation unit 27A includes a tablet built-in monitor 131, touch pens 132A and 132B, and a speaker 133. The editing operation unit 27B realizes editing processing for the user in the editing space A2-2, and has the same configuration as the editing operation unit 27A. Hereinafter, when the editing operation units 27A and 27B are not particularly distinguished, they are simply referred to as the editing operation unit 27.

The tablet built-in monitor 131 displays the edit screen according to the control by the control unit 201, and accepts the user's operation on the edit screen. An input signal representing the content of the user's operation is supplied to the control unit 201, and the captured image to be edited is edited.

The print operation unit 28 realizes a printing process of providing a sticker paper on which the created image has been printed to the user in the print waiting space A3. The print operation unit 28 is configured to include a printer 140. A sticker paper unit 141 is attached to the printer 140.

The printer 140 prints the edited image on the sticker paper 142 housed in the sticker paper unit 141 based on the print data supplied from the control unit 201, and discharges the edited image to the sticker paper discharge port 161.

(Functional block of photo sticker making device)
FIG. 10 is a block diagram showing a functional block of the photo sticker creating device 1. The photo sticker creating device 1 functions as a preselection unit 210, a photographing unit 220, an editing unit 230, and a printing unit 240. Further, the control unit 201 executes the photo sticker creation program of the present invention to execute the preselection processing unit 301, the shooting processing unit 302, the extraction unit 303, the generation unit 304, the image processing unit 305, the composition unit 306, and the editing process. It functions as a unit 307 and a print processing unit 308.

The preselection unit 210 includes the preselection operation unit 20 described above and the preselection processing unit 301. The preselection processing unit 301 performs the preselection processing by controlling the touch panel monitor 71, the speaker 72, and the coin processing unit 74 in the preselection operation unit 20. The preselection processing unit 301 causes the touch panel monitor 71 to display a selection screen or the like for selecting a course related to shooting performed in the shooting space A1. Further, the preselection processing unit 301 accepts a user's operation input to the touch panel monitor 71. Specifically, the preselection processing unit 301 accepts selection operation input for the selection screen displayed on the touch panel monitor 71, input of the user's name, and the like. Further, the preselection processing unit 301 controls the output of guidance for explaining various selection operations. The preselection processing unit 301 displays a screen explaining various selection operations on the touch panel monitor 71, and outputs sounds explaining various selection operations from the speaker 72.

The photographing unit 220 includes the above-mentioned photographing operation unit 21 and an imaging processing unit 302. The shooting processing unit 302 performs shooting processing by controlling the camera 91, the touch panel monitor 92, and the speaker 93 in the shooting operation unit 21.

The shooting processing unit 302 receives the user's operation input to the touch panel monitor 92. For example, the shooting process in the shooting space A1 is started when the shooting processing unit 302 accepts a user's contact operation with the touch panel monitor 92 as an input.

The shooting processing unit 302 controls the camera 91 and shoots the user as a subject. There are two types of shooting: moving images and still images. By controlling the display of the touch panel monitor 92, the shooting processing unit 302 displays a moving image captured by the camera 91 in a live view on the touch panel monitor 92, or displays a still image as a shooting result as a shooting image. ..

In addition, the shooting processing unit 302 causes the touch panel monitor 92 to display an instruction screen explaining the number of shots, the standing position of the user, the sample pose, a message about the line of sight, the shooting timing, and the like. Further, the narration voice corresponding to each instruction screen and the BGM are output from the speaker 93.

The extraction unit 303 extracts the user's face area included in the image data photographed by the photographing unit 220 by using a predetermined face recognition technique. When the image data includes a plurality of users, the extraction unit 303 extracts the face area of each user.

In addition, the extraction unit 303 extracts the contour region of the predetermined portion of the user's face as the first region from each face region extracted from the image data by using a predetermined image extraction method. For example, the extraction unit 303 extracts the user's eye region and mouth region as the first region by using, for example, a color difference from the surroundings. At this time, the extraction unit 303 not only extracts the outline of the eye region and the mouth region as a whole, but also subdivides the region into the first one according to the specific configuration of the eyes and the mouth, as will be described later. Extract the area.

Here, the extraction unit 303 can use a semantic segmentation technique as an extraction method for each region. By extracting each region by the segmentation technology, it is possible to extract each region in detail on a pixel-by-pixel basis. As a result, the generation unit 304 in the subsequent stage can generate a range of image processing for each area, and the image processing unit 305 can realize image processing finely set according to each area. Can be done.

The generation unit 304 uses the first region extracted by the extraction unit 303 to generate a second region including at least a part of the first region and set as a target of image processing. For example, the second region is a range that is preferred for the image processing target or a range that is set according to the range that is preferred for image processing. The first region and the second region may be the same. Further, even in the second region, image processing is not always performed. In particular, in the range where image processing is not performed in this way, the first region and the second region may be the same. For example, the preferred range for image processing is, in the case of lips, a range generally preferred for applying lipstick.

The generation unit 304 generates each second region, which will be described later, from the first region by using the machine learning model learned in advance from the training data. Specifically, the machine learning model is learned using learning data including a plurality of sets of "image data of a face region" and "region information of a predetermined portion of a face". At this time, each image data included in the learning data includes image data of a plurality of variations, so that the accuracy of processing in the generation unit 304 is improved. In particular, the photo sticker creating device 1 is often regarded as an amusement device, and in such a case, the user often takes an interesting image. Therefore, since face images of various orientations and facial expressions are taken, it is preferable that each region of image processing can be specified regardless of the face image of any orientation and facial expression. Therefore, for example, by using a plurality of variations of image data as described later with reference to FIGS. 11 to 20, it is possible to correspond to various face images.

《Mouth mask》
Extraction of the mouth region will be described with reference to FIG. For example, the extraction unit 303 divides the mouth into regions of "upper lip", "other than lips (for example, in the mouth)", "lower lip", and "other". FIG. 11A shows an example of a rectangular area including a mouth in the user's face area extracted from the image data. When there is image data of such a rectangular region of the mouth, the extraction unit 303 extracts the region of the mouth as a subdivided region as shown in FIG. 11B. Specifically, the extraction unit 303 extracts the upper lip region as the first mouth region A11, the lower lip region as the second mouth region A12, and the mouth as the third mouth region A13 with respect to the rectangular region of the mouth. The area other than the lips is extracted. Further, the region A14 shown in FIG. 11B is a region other than the mouth, for example, a region of the skin of the face.

The generation unit 304 uses each of the first regions extracted by the extraction unit 303 as shown in FIG. 11 (b) to generate a second region as shown in FIG. 11 (c). Specifically, the generation unit 304 generates the fourth mouth region B11, which is the target range of image processing for the upper lip, from the first mouth region A11 by a predetermined method. That is, for the purpose of improving the aesthetic appearance of the mouth, the generation unit 304 generates as the fourth mouth region B11 a range in which it is preferable to perform image processing in which the color of the lips is changed by applying lipstick or the like. .. For example, the preferred area of the upper lip for applying lipstick is generally particularly outside the mouth, and it is rare to apply lipstick to the inside of the mouth. Therefore, such a preferable range for applying lipstick is set as a preferable range for performing image processing such as applying lipstick, and the generation unit 304 is defined from the first mouth region A11 of the upper lip as described above. The outer range is generated as the fourth port region B11. Therefore, even if the upper lip is not applied with lipstick, for example, even if the inside of the upper lip is visible, the inside of the upper lip without applying lipstick does not correspond to the fourth mouth region B11. The fourth opening region B11 shown in FIG. 11C is substantially the same as the first opening region A11 shown in FIG. 11B.

Similarly, the generation unit 304 generates the fifth mouth region B12, which is the target range of image processing for the lower lip, from the second mouth region A12. That is, the generation unit 304 generates as the fifth mouth region B12 a range in which it is preferable to perform image processing in which the color of the lips is changed by applying lipstick or the like on the lower lip for the purpose of improving the aesthetic appearance of the mouth. .. For example, for the lower lip, the preferred area for applying lipstick is generally the outside of the mouth, not the inside of the mouth. Therefore, the generation unit 304 generates the range outside the lower lip defined in this way from the second mouth region A12 as the fifth mouth region B12 as shown in FIG. 11 (c). Therefore, even if the lower lip is not covered with lipstick, for example, even if the inside of the lower lip is visible, the inside of the lower lip that is not covered with lipstick is in the fifth mouth region B12. Not applicable. Further, in the case of an image in which the lower lip is bitten and the color of the lips is lightened, a portion narrower than the region where it is preferable to actually apply lipstick may be extracted to the extraction unit 303 as the second region A12 of the lower lip. is there. In such a case, the generation unit 304 generates the fifth mouth region B12, which is wider than the second mouth region A12 and is enlarged to a range where it is preferable to actually apply lipstick.

Further, as shown in FIG. 11C, the generation unit 304 generates the sixth mouth region B13, which is the target of image processing for the region of the mouth other than the lips, from the third mouth region A13. Here, the generation unit 304 includes, for example, a portion of the mouth region other than the lips excluding the fourth mouth region B11 and the fifth mouth region B12, including the teeth, the tongue, the inside of the lips, and the like. It can be generated as the sixth port region B13. Therefore, as can be seen by comparing FIGS. 11 (b) and 11 (c), the sixth mouth region B13 may include the inner part of the actual lower lip.

In FIG. 11C, an example in which a region other than the lips is specified as the sixth mouth region B13 will be described, but the region may be further subdivided (not shown). Specifically, the inside of the teeth, tongue, and lips may be different regions. Further, it may be made finer and each tooth may be a different region. This makes it possible to perform image processing having a tooth whitening effect, for example.

Region B14 is a region outside the mouth that is not included in regions B11 to B13. Here, of the outer edge portion of the lips, the edge portion to which the lipstick is not applied is included in this region B14. Therefore, this region B14 is a skin region. Further, if some object is shown on the mouth in the image data, the area of this object is also extracted as the area B14. Further, as shown in FIG. 11C, the generation unit 304 may generate a region B14 which is a region other than the mouth, but the image processing related to the mouth is not performed on this region B14. If the region A14 can be used, the region A14 may be used instead of generating the region B14 other than the mouth.

Here, the information including each region of the regions B11 to B13 is used as a mouth mask. In addition, each area specified by the mouth mask is subject to image processing. The extraction unit 303 extracts the first region finely divided about the mouth from the face region, and the generation unit 304 generates a second region to be image processed corresponding to each second region. As a result, the image processing unit 305 in the subsequent stage for each of the second regions can execute the corresponding processing for each region.

Here, an example has been described in which a rectangular region including a mouth portion as shown in the image shown in FIG. 11A is extracted and a mouth mask region is extracted from this rectangular region, but the present invention is not limited to this. For example, the extraction unit 303 may extract the mouth mask region from the entire face region extracted from the image data.

《Eye mask》
Extraction of the eye region will be described with reference to FIG. For example, the extraction unit 303 divides the eyes into regions of "black eyes" and "white eyes" for extraction. FIG. 12A shows an example of a rectangular area including eyes in the user's face area extracted from the image data. When there is image data of such a rectangular region of the eyes, the extraction unit 303 extracts the region of the eyes as a subdivided region as shown in FIG. 12 (b). Specifically, the extraction unit 303 extracts the so-called black eye region as the first eye region A21 and the so-called white eye region as the second eye region A22.

Here, the extraction unit 303 does not specify and extract the portion in a strict wording sense, but considers image processing and considers the ease of extraction, and sets the region. Extract. Specifically, the third eye region B21 is a region of the black eye, but is not a region limited by the color of the black region, but a region including the pupil and the iris. For example, the color of the eyes does not have to be black, and this region is designated as the first eye region A21 even if it is another color depending on the wearing of colored contacts, the natural color, and the like. Similarly, the second eye region A22 is a region of the white eye, which is a white region in the eye, but does not have to be limited to the meaning of the white portion of the eyeball excluding the pupil and the iris, for example. , The edge of the eye such as the caruncle may be included.

Further, the region A23 shown in FIG. 12B is a skin region, for example, a facial skin region. Further, the region A24 shown in FIG. 12B is a region other than the eyes and the skin as other regions, and corresponds to, for example, eyebrows and hair. If the eyelashes can be identified as an area, the eyelashes are also included in this area.

The generation unit 304 generates a second region as shown in FIG. 12 (c) by using the first region as shown in FIG. 12 (b) extracted by the extraction unit 303. Specifically, the generation unit 304 generates the third eye region B21, which is the target range of image processing for the black eyes, from the first eye region A21 by a predetermined method. That is, the generation unit 304 generates, for example, a region of the black eye that is preferable for increasing the size or changing the color as the third area. Assuming that the "catch light" exists by shining the light on the black eye, and the part of the catch light exists on the peripheral edge of the black eye, the part of the catch light is mistakenly included in the second eye area A22. If so, the generation unit 304 processes this portion as the black eye portion and generates it as the third eye region B21. Here, too, the area B21 is set in the image processing unit 305 for the purpose of image processing for improving the aesthetic appearance such as making the eyes look larger or adjusting the color. Therefore, the third region B21 does not specify the portion in a strict wording sense.

Similarly, the generation unit 304 generates the fourth area B22, which is the target range of image processing for the white eyes, from the second area A22. That is, since the white eye portion may actually include a portion other than the white portion of the eye, for example, only the white portion of the eye is generated as the fourth eye region B22 to be image-processed into white without turbidity. .. Here, too, the fourth area B22 is set in the image processing unit 305 for the purpose of image processing for improving the aesthetic appearance such as making the eyes look larger or adjusting the color. Therefore, the fourth region B22 does not specify the portion in a strict wording sense. Therefore, the second eye region A22 is not limited to the meaning of the white part of the eyeball excluding the pupil and the iris.

The region B23 is a region generated from the region A23 and is a skin region excluding the eye region. In FIG. 12C, an example in which the skin is specified as one region B23 will be described, but the region may be further subdivided (not shown). For example, the generation unit 304 may generate different regions of the skin, such as the upper eyelid, lower eyelid, and eyelid edge, which are areas around the eyes that are specified when applying make-up such as eye shadow. In addition, the generation unit 304 may generate a region for specifying the double eyelid.

The region B24 is a region generated from the region A24 and is not included in the regions B21 to B23. For example, eyebrows, hair existing around the eyes, and the like. Further, if some object is shown around the eyes in the image data, the area of this object is also extracted as the area A24. Specifically, when the user wears spectacles, the frame of the spectacles is extracted as region A24.

As shown in FIG. 11C, the generation unit 304 may generate regions B23 and B24 that are regions other than the eyes, but the regions B23 and B24 are not subjected to image processing related to the eyes. When the areas A23 and B24 can be used, the areas A23 and A24 may be used as a range other than the eyes.

Here, the information including each region of the regions B21 and B22 is used as an eye mask. In addition, each area specified by the eye mask is subject to image processing. The extraction unit 303 extracts the first region that is finely divided about the eyes from the face region, and the generation unit 304 generates the second region that is the target of image processing corresponding to each first region. As a result, the image processing unit 305 in the subsequent stage can execute the corresponding processing for each of the second regions. At this time, if the eyelashes can be extracted, for example, by including the eyelashes in the eye mask, it is possible to prevent the eyelashes from being image-processed as a part of the skin, for example, as if the eyelashes were coated with mascara. Image processing that improves the aesthetic appearance of the eyelashes can be performed.

Here, an example has been described in which a rectangular region including an eye portion as shown in the image shown in FIG. 12A is extracted and an eye mask region is extracted from the rectangular region, but the present invention is not limited to this. For example, the extraction unit 303 may extract the eye mask area from the entire face area extracted from the image data.

The image processing unit 305 processes each region extracted from the image data by a predetermined method for each region. For example, the image processing unit 305 executes a process of improving the aesthetic appearance of the mouth on the mouth mask extracted as the mouth region, or a process of improving the aesthetic appearance of the eye on the eye mask extracted as the eye region. Or run.

For example, as shown in FIG. 13A, an example of a face image in which a part of the face is touched, the mouth is open, and the teeth are visible will be examined. In this case, the generation unit 304 accurately serves as the second region from the first region (not shown) extracted by the extraction unit 303, as shown in FIG. 13 (b). The fifth mouth region B12, the sixth mouth region B13 and the region B14 can be extracted. As a result, when the image processing unit 305 performs image processing, as shown in FIG. 13 (c), it is possible to prevent the entire area of the mouth including other than the lips from being image-processed. Specifically, as shown in FIG. 13 (d) of the image processing section, image processing is performed only on the upper lip and the lower lip, which are preferable areas for applying lipstick, and the boundaries between the upper lip and the lower lip are also distinguished. As a result, for example, it is possible to realize image processing that is more preferable for the user than having a three-dimensional effect. Specifically, depending on the three-dimensional shape of the lips, it is possible to brighten the front part of the upper and lower lips in the depth direction of the face and darken it toward the back, which is more natural with three-dimensionality. Image processing can be realized.

<< Extraction of the first region and generation of the second region >>
Subsequently, specific examples of extraction of the first region and generation of the second region in a plurality of patterns will be described. First, an example of a facial image in which the mouth is open and the teeth and tongue are visible in the mouth, as shown in FIG. 14A, will be examined. In this case, as shown in FIG. 14B, the extraction unit 303 can extract the first mouth region A11, the second mouth region A12, the third mouth region A13, and the region A14. That is, even when the image data is acquired with a mouth shape different from such a general mouth shape, the extraction unit 303 can extract each first region from the contours thereof. It becomes. Further, as shown in FIG. 14C, the generation unit 304 generates the fourth port region B11 from the first port region A11 and generates the fifth port region B12 from the second port region A12. In addition, the generation unit 304 generates the sixth port region B13 and the region B14. As a result, the image processing unit 305 can perform image processing on each of the second regions to realize image processing preferable to the user. For example, it is possible to perform image processing such that lipstick is applied only to the area of the lips where it is preferable to apply lipstick.

Next, an example of a face image in which a part of the face is touched, the mouth is opened, and the chin is pinched by the fingers as shown in FIG. 15A will be examined. In this case, as shown in FIG. 15B, the extraction unit 303 can extract the first mouth region A11, the second mouth region A12, the third mouth region A13, and the fourth mouth region A14. That is, even when the image data is acquired with a mouth shape different from such a general mouth shape, the extraction unit 303 can extract each first region from the contours thereof. It becomes. Further, as shown in FIG. 15C, the generation unit 304 generates the fourth port region B11, the fifth port area B12, the sixth port area B13, and the area B14. Therefore, the image processing unit 305 can realize the image processing preferable to the user by executing the image processing for each second area.

Subsequently, an example of a facial image in which a finger is placed on a part of the mouth, the mouth is opened, and the teeth and tongue are visible in the mouth, as shown in FIG. 16A, will be examined. In such a case, depending on the extraction method, as shown in FIG. 16B, the region A15 in which the entire mouth, which is considered to have no finger, can be seen even though the finger is present may be extracted. .. On the other hand, the extraction unit 303 can accurately extract the first mouth region A11, the second mouth region A12, the third mouth region A13, and the region A14 (not shown). That is, even when the image data is acquired with a mouth shape different from the general mouth shape, each region can be extracted. Further, as shown in FIG. 16C, the generation unit 304 generates the fourth port region B11, the fifth port area B12, the sixth port area B13, and the area B14. Therefore, the image processing unit 305 can realize the image processing preferable to the user by executing the image processing for each second area. For example, it is possible to perform image processing such as applying lipstick only to the area where it is preferable to apply lipstick, without performing unnatural image processing such as applying lipstick to the fingers, mouth and teeth. It is possible to provide various images.

For example, as shown in FIG. 17A, an example of a facial image in which the black eye region is large up to the upper eyelid and the lower eyelid and the bangs are close to the upper eyelid will be examined. In this case, as shown in FIG. 17B, the extraction unit 303 can extract the first region A21, the second region A22, the region A23, and the region A24 as the first region. Further, as shown in FIG. 17C, the generation unit 304 generates the third region B21, the fourth region B22, the region B23, and the region B24 as the second region. As a result, the image processing unit 305 executes image processing on the second area, so that the black eye area protrudes to the outside of the upper and lower eyelids in order to make the eyes look larger. Can be prevented. Therefore, it is possible to realize image processing preferable to the user.

Next, an example of a face image in which the vicinity of the outer corner of the eye is lifted upward by a finger and the black eye can hardly be confirmed as shown in FIG. 18A will be examined. The image data as shown in FIG. 18 is acquired by the user for the purpose of interesting image data. In such a case, the eye region may not be extracted depending on the extraction method because it is far from the general eye shape. On the other hand, as shown in FIG. 18B, the extraction unit 303 can extract the first region A21, the second region A22, the region A23, and the region A24 as the first region. Further, as shown in FIG. 18C, the generation unit 304 generates the third region B21, the fourth region B22, the region B23, and the region B24 as the second region. In the example shown in FIG. 18, the first region and the second region are the same. Therefore, when the image processing unit 305 executes image processing on the second region, unnatural image processing is performed such that the black eye region protrudes to the outside of the lower eyelid in order to make the eyes look larger. Can be prevented. In addition, it is possible to prevent the result of image processing from being far from the fun that the user wants, and to provide a natural image that the user wants.

For example, consider an example of a face image in which a part of the eyes is touched, as shown in FIG. 19 (a). In this case, depending on the extraction method, as shown in FIG. 19B, the region A25 in which the entire eye, which is considered to have no hands, can be seen even though the hands are present may be extracted. On the other hand, the extraction unit 303 can accurately extract the first region A21, the second region A22, the region A23, and the region A24 (not shown). Further, as shown in FIG. 19C, the generation unit 304 generates the third region B21, the fourth region B22, the region B23, and the region B24 as the second region. As a result, the image processing unit 305 performs image processing on the second region, so that the black eye region protrudes to the outside of the upper and lower eyelids in order to make the eyes look larger, or the black eye region is the hand. It is possible to prevent unnatural image processing that extends beyond the area. Therefore, it is possible to realize image processing preferable to the user.

For example, as shown in FIG. 20A, an example of a face image in which a frame portion of eyeglasses is applied to a part of eyes will be examined. In this case, depending on the extraction method, as shown in FIG. 20B, there is a case where the region A25 is extracted so that the entire eye can be seen without the frame of the spectacles even though the frame of the spectacles is present. There is also. On the other hand, the extraction unit 303 can extract the first region A21, the second region A22, the region A23, and the region A24 (not shown). Further, as shown in FIG. 20C, the generation unit 304 generates the third region B21, the fourth region B22, the region B23, and the region B24 as the second region. As a result, the image processing unit 305 executes image processing on the second region, so that the black eye region protrudes to the outside of the upper and lower eyelids in order to make the eyes look larger, or the black eye region becomes It is possible to prevent image processing that extends beyond the frame area of the eyeglasses. Therefore, it is possible to realize image processing preferable to the user.

Since the acquired images include images of various forms as shown in FIGS. 13 to 20, by using such various images as learning data, the generation unit 304 can generate the second region. It is possible to improve the accuracy.

<< Image processing of mouth mask >>
The image processing unit 305 can perform image processing on the area of the mouth mask, for example, by applying lipstick. At this time, the image processing unit 305 does not adjust the color of the entire mouth mask, but executes the processing set for each area included in the mouth mask to improve the aesthetic appearance of the user's mouth. Can be done.

Specifically, the image processing unit 305 can process the first port region A11 with a gradation from the upper side to the lower side. For example, the image processing unit 305 processes the upper edge portion of the first mouth region A11 so as to lighten the color and darken the color of the lower edge portion, which is preferable for the user without impairing the three-dimensional appearance of the lips. Image processing becomes possible. Similarly, the image processing unit 305 also impairs the three-dimensional appearance of the lips by processing the second mouth region A12 with, for example, darkening the color of the upper edge portion and lightly gradation the color of the lower edge portion. This enables image processing that is preferable to the user. In this way, since the upper lip and the lower lip are processed separately, natural image processing can be performed and the aesthetic appearance of the user can be improved.

As described above, the areas where lipstick can be applied are set as the first mouth area A11 and the second mouth area A12. Thereby, the third mouth region A13 can be accurately distinguished from the first mouth region A11 and the second mouth region A12. Therefore, the image processing unit 305 prevents the image processing in which the lipstick is applied to the portion where the lipstick is not applied, and enables the image processing preferable for the user. Specifically, when the mouth is opened and the image contains teeth, it is possible to prevent the color of the teeth as well as the lips from being changed. Also, for the purpose of obtaining an interesting image, when the inside of the lips is visible, the color is changed to the inside of the lips, as in the case where the image was taken by deforming the shape of the mouth. Can be prevented. That is, regardless of the variation in the shape of the mouth, it is possible to distinguish between the lips and those other than the lips, image processing preferable to the user can be performed, and the aesthetic appearance of the user can be improved. ..

The image processing unit 305 may perform processing such as highlighting the first port region A11. Specifically, the image processing unit 305 can realize processing having a highlighting effect at the central portion of the upper end of the first mouth region A11, that is, the point where the upper lip overlaps with the philtrum. As a result, the aesthetic appearance of the user can be improved.

The image processing unit 305 may execute different processing depending on whether the mouth is open or closed. For example, the image processing unit 305 performs image processing preferable to the user by changing the degree of image processing according to the ratio of the area of the first port area A11 and the second port area A12 to the area of the third port area A13. The beautification of the user can be improved.

Further, if the tooth region included in the third mouth region A13 can be distinguished, the image processing unit 305 can also realize image processing such as whitening teeth. As a result, the aesthetic appearance of the user can be further improved.

《Image processing of eye mask》
The image processing unit 305 can perform image processing on the area of the eye mask, for example, to make the eyes look larger or to make the eyes lively. At this time, the image processing unit 305 does not adjust the shape and color of the entire eye mask, but executes the processing set for each area included in the eye mask to further improve the aesthetic appearance of the user. be able to.

Specifically, the image processing unit 305 can perform a process of enlarging the black eyes in the first area A21. For example, the image processing unit 305 can process the first area A21 so as to increase the size by a predetermined ratio, so that the eyes can be made clear. At this time, since the first area A21, the second area A22, and the third area A23 can be accurately distinguished, the black eye does not deviate to the second area A22 or the third area A23 for the user. Preferred image processing becomes possible, and the aesthetic appearance of the user can be further improved.

The image processing unit 305 can perform a process of changing the color of the first area A21. For example, the image processing unit 305 can perform processing that makes the user's eyes look larger by clarifying the color of the eyes. Further, for example, the color of the eyes can be processed into a color as if a color contact lens was attached. At this time, since the first area A21, the second area A22, and the third area A23 can be accurately distinguished, the user can use the second area A22 and the third area A23 without changing the color. Preferred image processing becomes possible, and the aesthetic appearance of the user's eyes can be improved.

The image processing unit 305 can perform a process of changing the color of the second area A22. For example, by changing the color of the white of the eyes to a more transparent white color, it is possible to process the user's impression to be more lively. At this time, the image processing unit 305 may process the color of the second area A22 so as to generate a gentle gradation between the outside and the inside of the second area A22, instead of simply unifying the colors. As a result, it is possible to prevent unnatural image processing in which the user's eyes look flat, and to process the image into a three-dimensional and natural impression to improve the aesthetic appearance of the user's eyes.

If it is possible to distinguish areas such as Mongolian folds and caruncles, the image processing unit 305 may realize image processing that emphasizes or inconspicuously adjusts the areas of Mongolian folds and caruncles. Good. Further, when the upper eyelid, the lower eyelid, the eyelid edge, and the like can be distinguished, the image processing unit 305 may change the color around the eyelid or realize image processing in which the eyelid edge is emphasized. Further, when the double can be specified, the image processing unit 305 may realize the image processing in which the double is emphasized or the double width is adjusted. Further, when the eyelashes can be distinguished, the image processing unit 305 can realize image processing in which the eyelashes are emphasized thickly or emphasized for a long time. These processes can also improve the aesthetic appearance of the user.

<< Other processing >>
Further, in addition to the above-mentioned processing related to the mouth mask and the eye mask, the image processing unit 305 may also perform other processing for the purpose of skin-whitening effect, whitening effect, and small face effect on the face. Since these processes are the same as the existing processes, specific description thereof will be omitted here.

The compositing unit 306 uses the processed image as a compositing image, synthesizes the compositing image with the moving image captured by the camera 91, and causes the touch panel monitor 92 to display the compositing image as a live view display image. Therefore, the user can take a picture while checking the finished image in real time.

The editing unit 230 includes the above-mentioned editing operation units 27A and 27B and an editing processing unit 307. The editing processing unit 307 performs editing processing by controlling the tablet built-in monitor 131 and the speaker 133 in the editing operation units 27A and 27B. The editing processing unit 307 receives the user's operation input using the touch pens 132A and 132B to the tablet built-in monitor 131.

Further, the editing processing unit 307 performs predetermined image processing on the captured image as the image to be edited according to the selection operation for the selection screen displayed on the tablet built-in monitor 131, and displays the captured image on the tablet built-in monitor 131. Alternatively, the editing processing unit 307 performs predetermined image processing on the compositing image in response to an input operation on the editing screen displayed on the tablet built-in monitor 131, or generates a new compositing image in response to the input operation. Then, it is combined with the captured image and displayed on the tablet built-in monitor 131.

In addition, the editing processing unit 307 controls the output of guidance explaining how to proceed with editing. For example, the editing processing unit 307 may display a guidance screen explaining how to proceed with editing on the tablet built-in monitor 131, or output a voice of guidance explaining how to proceed with editing from the speaker 133. In addition, the editing processing unit 307 controls the communication unit 203 to perform processing related to communication via a network such as the Internet. Further, the editing processing unit 307 may perform the printing processing by controlling the printer 140 of the printing operation unit 28.

The printing unit 240 includes the above-mentioned printing operation unit 28 and a printing processing unit 308. The print processing unit 308 receives print data from the editing processing unit 307 and performs printing processing by controlling the printer 140 of the printing operation unit 28. Here, as an example of the output unit that outputs the image data, the printing unit 240 that outputs the photo sticker that is the print data has been described, but the output method of the image data is not limited to this. For example, a transmission means for transmitting the image data itself to an external communication terminal using a network or the like may be used as an output unit.

In this way, the photo sticker creation device 1 provides a photo sticker creation game in which various mechanisms (selection of shooting pose, BGM, narration, etc.) are applied in exchange for billing. Therefore, the image created by the photo sticker creating device 1 is an image that brings out the pleasant facial expression of the user or an elaborate and gorgeous image.

In addition, the photo sticker creation device 1 is fully equipped with lighting and other equipment, and image deformation processing (for example, subject eye size, leg length, etc.) and color correction (subject skin whitening processing, etc.) Since the above can be performed with advanced technology, the image created by the photo sticker creating device 1 is an image that gives a good image to the user.

Further, the editing (doodles on the image) can be performed more easily than the image processing (for example, the processing in an application for photo processing) other than the photo sticker creating device 1, and the variation of the editing is rich. From this point as well, it can be said that the image created by the photo sticker creating device 1 is relatively gorgeous as compared with the image taken and image-processed by other than the photo sticker creating device 1.

(Flow of photo sticker creation game)
Next, the flow of the process in which the user plays the photo sticker making game on the photo sticker making device 1 will be described with reference to FIG. FIG. 21 is a flowchart showing a flow of processing from the start of the game in the photo sticker creating device 1 to the creation of the photo sticker in the game.

In the state before the start of the game, the control unit 201, which functions as the preselection processing unit 301 of the photo sticker creation device 1, causes the touch panel monitor 71 of the preselection operation unit 20 to display a message prompting the insertion of coins. Further, as shown in FIG. 21, the control unit 201 determines whether or not a coin has been inserted into the coin insertion / return port 73 based on the presence / absence of an activation signal from the coin processing unit 74 (S1). When the control unit 201 determines that no coin has been inserted into the coin insertion / return port 73 (S1: NO), the control unit 201 continues the determination process of whether or not the coin has been inserted.

A user who intends to start a game inserts coins into the coin insertion / return port 73 in the preselection space A0, which is the space in front of the preselection operation unit 20. When coins are inserted into the coin insertion / return port 73, a start signal instructing the start of the game is output from the coin processing unit 74. When the start signal is input from the coin processing unit 74, the control unit 201 determines that the coin has been inserted into the coin input / return port 73 (S1: YES), and executes the pre-customer service process for the user (S2).

In the pre-customer service process, the control unit 201 causes the touch panel monitor 71 to display a message urging the selection of the course, the input of the name, the selection of the design, and the like. When the user makes various selections or inputs according to a message or the like displayed on the touch panel monitor 71, the control unit 201 sets the shooting course, name, design, print layout, and the like. The control unit 201 displays a plurality of types of composite background images on the touch panel monitor 92 so that the user can select the composite background image.

When the pre-customer service process is completed, the control unit 201 causes the touch panel monitor 71 to display a message or the like urging the user to move to the shooting space A1 and perform shooting. The control unit 201, which functions as the shooting processing unit 302, causes the touch panel monitor 92 of the shooting operation unit 21 to display a message prompting the user to touch the screen. The start button may be displayed together with or in place of this message. Further, the control unit 201, together with the BGM, outputs a narration prompting the user to touch the screen from the speaker 93. When the user who has moved to the shooting space A1 touches the touch panel monitor 92, the control unit 201 reads that the touch panel monitor 92 has been touched and starts the shooting process (S3).

In the shooting process, the control unit 201 causes the touch panel monitor 92 to display guidance regarding lighting, for example, and prompts the user to select a lighting level. When the user selects a lighting level, the control unit 201 sets the lighting level to the selected level.

Further, the control unit 201 displays an instruction screen for explaining the number of shots on the touch panel monitor 92, and outputs the corresponding narration from the speaker 93. In this embodiment, as an example, the number of shots is set to 7.

Next, the control unit 201 displays an instruction screen for guiding the user to the standing position on the touch panel monitor 92, and outputs the corresponding narration from the speaker 93.

After displaying the instruction screen as described above, the control unit 201 performs a live view in which the background image for composition and the moving image acquired by the camera 91 are combined according to the shooting course selected by the user. The display image is displayed on the touch panel monitor 92 as a live view display (S4). As a result, the user can pose while checking the finished image.

The control unit 201 displays the live view until just before the end of the countdown for shooting, and during that time, the control unit 201 displays the sample pose on the touch panel monitor 92 together with the live view display or instead of the live view display. .. The control unit 201 outputs a narration according to the sample pose from the speaker 93.

The control unit 201 manages the time from the start to the end of the live view display, and when a preset predetermined time elapses, the control unit 201 counts down the display of the touch panel monitor 92 and the sound of the speaker 93.

At the end timing of the countdown, the control unit 201 transmits a lighting control signal to the upper strobe 82, the left strobe 83, the right strobe 84, and the foot strobe 85, and also transmits a shutter signal to the camera 91.

As a result, a flash is emitted from the upper strobe 82, the left strobe 83, the right strobe 84, and the foot strobe 85, and the photographed image in which the illuminated user is captured together with the background is acquired by the camera 91. In the present embodiment, as an example, the processes from step S3 to step S4 are repeated a plurality of times to acquire seven captured images. Further, the control unit 201 stores the edit target image in which the captured image is combined with the background image for composition in the storage unit 202.

Here, the example in which the captured image is acquired at preset time intervals has been described, but the acquisition timing of the captured image is not limited to this. For example, when the shooting operation unit 21 has an operation button for shooting, the shot image may be acquired at the timing when the operation button is operated.

After the shooting is completed, the control unit 201 displays a guidance screen on the touch panel monitor 92 prompting the user to move to either the editing space A2-1 or the editing space A2-2, and moves by voice. The guidance is output to the speaker 93.

Then, the control unit 201 executes an editing process that allows the user to edit the image to be edited (S5). More specifically, the control unit 201 displays the image to be edited on the monitor 131 built into the tablet, and allows the user to draw a stamp image, a pen image, or the like with the touch pens 132A and 132B on the image to be edited. Create an edited image.

After that, the control unit 201 displays a guidance screen for urging the user to move to the print waiting space A3 provided with the sticker paper discharge port 161 on the tablet built-in monitor 131, and provides voice movement guidance to the speaker 133. Output.

Further, the control unit 201 arranges the edited image in the print layout selected by the preselection operation unit 20 to create a print image, and executes a print process of printing the print image on the sticker paper 142 (S6). ).

When the printing process is completed, the control unit 201 executes the ejection process of the sticker paper 142 (S7), ejects the printed sticker paper 142 from the sticker paper ejection port 161 and provides the user as a photo sticker for the game. To finish. In this way, the photographed image of the user created by the photo sticker creation device 1 can be output as a photo sticker. Although detailed description is omitted, the photo sticker creating device 1 of the present embodiment can output a photographed image to a mobile terminal or the like by communication.

(Details of synthesis process)
Next, the synthesis process of step S4 of the flowchart of FIG. 21 will be described with reference to the flowchart shown in FIG. As shown in FIG. 22, in the synthesis process of step S4, the control unit 201 first extracts the user's face region included in the captured image (S41). When the captured image includes a plurality of users, the control unit 201 extracts the face areas of all the users.

The control unit 201 further extracts the mouth area and the eye area for the user from which the face area extracted in step S41 has been extracted (S42). Specifically, the control unit 201 extracts regions A11 to A14 with respect to the mouth region. Further, the control unit 201 extracts regions A21 to A24 with respect to the eye region. When the captured image includes a plurality of users, the control unit 201 extracts the mouth area and the eye area for all the users.

The control unit 201 executes a predetermined process for the mouth region with respect to the image data of the regions A11 to A14 extracted in step S42 (S43). For example, the control unit 201 executes a process of adjusting the color of the mouth as a process of improving the aesthetic appearance of the mouth region.

Further, the control unit 201 executes a predetermined process for the eye region for the image data of the regions A21 to A24 extracted in step S42 (S44). For example, the control unit 201 executes a process of adjusting the color of the eyes as a process of improving the aesthetic appearance of the eye region.

Further, the control unit 201 may execute other image processing on the image data of the user's face region (S45). For example, the control unit 201 processes a small face or a fair face according to a predetermined method. At this time, the control unit 201 may automatically process or may process according to the operation of the user.

The control unit 201 synthesizes an image using the images processed in steps S43 to S45 (S46). The combined image is acquired while being confirmed by the user by the live view display.

(Display example)
FIG. 23 is an example of the editing screen 400 displayed on the tablet built-in monitor 131. As shown in FIG. 23, the editing screen 400 includes a first editing unit 401 and a second editing unit 402.

The first editing unit 401 includes a first thumbnail unit 403, a first editing target image display unit 405, a first operation button display unit 407, and a first palette 409. Similarly, the second editing unit 402 includes a second thumbnail unit 404, a second editing target image display unit 406, a second operation button display unit 408, and a second palette 410.

The first thumbnail unit 403 and the second thumbnail unit 404 display thumbnails of a plurality of captured images Im1 to Im5 to be edited. The thumbnails displayed in the first thumbnail unit 403 and the second thumbnail unit 404 can be selected with the touch pen 132A or 132B. In the example shown in FIG. 21, the images to be edited Im1 and Im3 are selected, respectively. Although not shown, the characters "selected" may be displayed on the thumbnail of the selected image to be edited. Further, when seven images are taken in step S3 of the flowchart of FIG. 19, seven images to be edited are displayed in the thumbnail units 403 and 404, respectively, but FIG. 23 shows five edits for convenience. An example in which the target image is displayed will be described.

The edit target image selected by the first thumbnail unit 403 is displayed on the first edit target image display unit 405. Further, the edit target image selected by the second thumbnail unit 404 is displayed on the second edit target image display unit 406. When the edit target image selected in the edit target image displayed on the thumbnail unit 403 or 404 is changed, the edit target image displayed on the edit target image display unit 405 or 406 is also changed.

The first palette 409 is used to select the type of content or the like when decorating the image to be edited displayed on the first edit target image display unit 405 with characters, patterns, colors, or the like which are contents. Further, the second palette 410 is used to select the type of content and the like when decorating the edit target image displayed on the second edit target image display unit 406 with the content. As will be described in detail later, in the example shown in FIG. 21, each palette 409 and 410 includes tabs for "impersonator", "face graffiti", "item" and "makeup".

As described above, in the present embodiment, by extracting each region, it is possible to execute the processing assumed for each region, and it is possible to realize image processing preferable to the user.

<Modification example>
The above embodiment is an example, and various modifications can be made without departing from the scope of the present invention. For example, the above-described embodiments may be combined as appropriate. For example, in the seventh embodiment, the mode of selecting whether to count for 9 seconds or 7 seconds according to the gender of the user has been described, but the seventh embodiment is combined with the first embodiment or the second embodiment. May be good. In this case, if the user's gender is male, the shooting interval is fixed at 17 seconds, and if the user is female, 9-second count or 7-second count is selected according to the number of shots. May be good.

Alternatively, the shooting interval may be switched manually. For example, a self-shutter button, a slide bar, and the like are provided, and the self-shutter button is configured to determine the shooting timing instead of switching the shooting interval. Then, by operating the slide bar or the like, the shooting interval after the operation of the self-shutter button may be changed.

(Example of realization by software)
The functional block of the photoseal creation device 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit). Good.

In the latter case, the photo sticker creating device 1 is a CPU that executes instructions of a program that is software that realizes each function, and a ROM (Read Only Memory) in which the program and various data are readablely recorded by a computer (or CPU). ) Or a storage device (these are referred to as "recording media"), a RAM (Random Access Memory) for developing the program, and the like. Then, the computer (or CPU) reads the program from the recording medium and executes it, thereby achieving the object of the present invention. As the recording medium, a "non-temporary tangible medium", for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, the program may be supplied to the computer via an arbitrary transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Is also included in the technical scope of the present invention.

1 Photo sticker making device (image processing device)
301 Preselection processing unit 302 Shooting processing unit 303 Extraction unit 304 Generation unit 305 Image processing unit 306 Synthesis unit 307 Editing processing unit 308 Printing processing unit (output unit)

Claims (13)

A shooting unit that shoots the user and generates image data,
An extraction unit that extracts a first region, which is a contour region of a predetermined portion of the user's face, from the image data using a predetermined image extraction method.
A generation unit that uses the first region to generate a second region that includes at least a part of the first region and is specified as an image processing target.
An image processing unit that processes the second region of the image data by a predetermined method, and an image processing unit.
An image processing device comprising. A shooting unit that shoots the user and generates image data,
An extraction unit that extracts a first region, which is a contour region of a predetermined portion of the user's face, from the image data by using semantic segmentation.
A generation unit that uses the first region to generate a second region that includes at least a part of the first region.
An image processing unit that processes the second region of the image data by a predetermined method, and an image processing unit.
An image processing device comprising. The extraction unit extracts the first region of the user's mouth, including the upper lip, the lower lip, and other than the lips.
The generation unit generates the second region corresponding to each of the first regions.
The image processing apparatus according to claim 1 or 2, wherein the image processing unit executes a process for the upper lip on a second region of the upper lip and a process for the lower lip on a second region of the lower lip. The extraction unit extracts at least one of the tongue, teeth, gingiva or mucous membrane as the first region other than the lips.
The generation unit generates the second region corresponding to each of the first regions.
The image processing apparatus according to claim 3, wherein the image processing unit executes processing for each region generated by the generation unit. The image processing apparatus according to claim 3 or 4, wherein the upper lip and lower lip regions of the second region are preferable ranges for applying lipstick. The extraction unit extracts the first region of the black eye with respect to the user's eye.
The generation unit generates the second region corresponding to the first region of the black eye.
The image processing apparatus according to claim 1 or 2, wherein the image processing unit executes processing for black eyes on the black eye region. The extraction unit extracts the first region of the white eye for the user's eye.
The generation unit generates the second region corresponding to the first region of the white of the eye.
The image processing apparatus according to claim 6, wherein the image processing unit executes processing for white eyes on the white eye region. The image processing apparatus according to claim 6 or 7, wherein the second region is a preferable range for changing at least one of size and color. The extraction unit further extracts the first region of the eyelashes.
The generator generates a second region corresponding to the first region of the eyelashes.
The image processing apparatus according to any one of claims 6 to 8, wherein the image processing unit executes eyelash processing on the second region of the eyelashes. The image processing apparatus according to any one of claims 1 to 9, wherein the generation unit generates a second region using a machine learning model trained using learning data generated in advance. The image processing apparatus according to any one of claims 1 to 10, further comprising a printing processing unit for printing the processed image data on a sticker paper. Steps to take a picture of the user and generate image data,
A step of extracting a first region, which is a contour region of a predetermined portion of the user's face, from the image data by using a predetermined image extraction method.
A step of using the first region to generate a second region that includes at least a part of the first region and is specified as an image processing target.
A step of processing the second region of the image data in a predetermined manner, and
Image processing method having. Computer,
A shooting unit that shoots the user and generates image data,
An extraction unit that extracts a first region, which is a contour region of a predetermined portion of the user's face, from the image data using a predetermined image extraction method.
A generation unit that uses the first region to generate a second region that includes at least a part of the first region and is specified as an image processing target.
An image processing unit that processes the second region of the image data by a predetermined method, and an image processing unit.
An image processing program that works.

Images