JP7339515B2 - Image processing device, image processing method, and program - Google Patents

Description

The present technology particularly relates to an image processing device, an image processing method, and a program capable of changing eye impressions in a more natural manner.

Conventionally, a photo sticker creating device is known. A photo sticker creating apparatus photographs a user, makes the user edit the photographed image, and prints the edited image on sticker paper to provide the edited image. A photo sticker creating device is installed in an amusement facility or the like.

The flow of one game provided by the photo sticker creating device is that after taking a picture of the user in the shooting space as the subject, the user is moved to the editing space, and the image is displayed according to the operation performed in the editing space. After editing, the edited image is printed on sticker paper. At the time of photographing in the photographing space, photographing is performed in a state in which a catch light is reflected in the iris of the user's eye by irradiating strobe light.

For example, Japanese Patent Application Laid-Open No. 2002-200000 discloses a technique of painting out the iris of a user in a photographed image with a predetermined color, and drawing an image representing a catch light on the iris of the user whose color has been changed.

JP 2014-182672 A JP 2013-055372 A

When the color of the black part of the eye is changed by image processing in order to change the impression of the eye, the black part of the eye may look unnatural after the change.

The present technology has been developed in view of such circumstances, and is capable of changing the impression of eyes in a more natural manner.

An image processing apparatus according to one aspect of the present technology includes a detection unit that detects the iris of the user in a photographed image obtained by photographing the user as a subject; and an image processing unit for changing the density of the edge of the iris.

In one aspect of the present technology, the iris of the user is detected in a captured image obtained by photographing the user as a subject, and the color of the iris and the edge of the iris are determined according to the user's operation. The density is changed respectively.

According to this technology, it is possible to change the impression of the eyes in a more natural way.

It is a perspective view showing an example of composition of appearance of a photograph sticker production device of this art. FIG. 10 is a perspective view of the appearance of the photo sticker creating apparatus as seen from another angle; It is a figure explaining a user's movement. It is a figure which shows the structural example of a prior selection part. It is a perspective view which shows the structural example of an imaging part. It is a perspective view which shows the structural example of an imaging part. It is a figure which shows the example of angle adjustment of a camera. It is a figure which shows the structural example of a background part. FIG. 3 is a diagram showing a configuration example of an operation unit; FIG. 4 is a diagram showing a configuration example of the front side of an editing unit; It is a figure which shows the installation example of a curtain. It is a figure which shows the design example of the back surface of a curtain. It is a figure which shows the example of utilization of a mirror surface part. FIG. 4 is a diagram showing a configuration example of a side surface of an editing unit; FIG. 4 is a diagram showing an example of a layer structure of an edited image; It is a block diagram which shows the structural example of a photo sticker production apparatus. 2 is a block diagram showing a functional configuration example of a photo sticker creating device; FIG. 18 is a block diagram showing a configuration example of an imaging processing unit in FIG. 17; FIG. 18 is a block diagram showing a configuration example of an edit processing unit in FIG. 17; FIG. FIG. 20 is a block diagram showing a configuration example of an editing image generation unit in FIG. 19; FIG. 10 is a diagram illustrating an example of detection of regions; 21 is a block diagram showing a configuration example of an eyelash processing unit in FIG. 20; FIG. FIG. 10 illustrates an example of removing upside down eyelashes; It is a figure which shows the example of deformation|transformation of an eyelash image. It is a figure which shows the example of the synthesis|combination of an eyelash image. 4 is a flowchart for explaining the operation of the photo sticker creating device; FIG. 27 is a flowchart for explaining editing processing performed in step S3 of FIG. 26; FIG. FIG. 10 is a diagram showing a display example of a guide screen for self-portrait before graffiti. FIG. 10 is a diagram showing an example of an edit screen; FIG. 28 is a flowchart for explaining editing image generation processing performed in step S42 of FIG. 27; FIG. FIG. 31 is a flow chart describing eyelash processing performed in step S64 of FIG. 30. FIG. FIG. 11 is an enlarged view showing an edit palette being selected by a makeup tool; FIG. 10 is a diagram showing an example of an image generated by an eye color tool; FIG. FIG. 10 is a diagram showing an example of a mask image used for image processing when an eye color tool is selected; FIG. 10 is a diagram showing an example of changing the color of the black eye area; FIG. 10 is a diagram showing an example of adjusting the density of the outline of the eye; FIG. 10 is a diagram showing another example of adjusting the density of the contour of the eye;

<Appearance of photo sticker making device>
1 and 2 are perspective views showing configuration examples of the appearance of the photo sticker creating apparatus 1. FIG.

The photo sticker creation device 1 is a game machine that provides photographed images and edited images. The photo sticker creating apparatus 1 provides images to users by printing the images on sticker paper or making the images viewable on the user's portable terminal. The photo sticker creating device 1 is installed in an amusement facility such as a game center or a store.

Users of the photo sticker creating apparatus 1 are mainly high school girls and young women. In the photo sticker creating device 1, a plurality of users, mainly two or three, can enjoy the game per group. Of course, one user can also enjoy the game in the photo sticker creating apparatus 1 .

In the photo sticker creating apparatus 1, the user himself/herself becomes a subject and performs photographing work. Through editing work, the user combines an image for composition such as handwritten characters or a stamp image with an image selected from the photographed images obtained by photographing. As a result, the photographed image is edited into a colorful image. The user receives the seal paper on which the edited image, which is the edited image, is printed, and ends the series of games.

The photo sticker creation device 1 can be called a photo creation game machine that provides a photo creation game.

As shown in FIG. 1, the photo sticker creating apparatus 1 is basically configured by installing a photographing unit 11 and an editing unit 12 in contact with each other. The photographing unit 11 and the editing unit 12 are electrically connected.

The photographing unit 11 has a box-shaped housing large enough for a person to enter, and is composed of a pre-selection section 20 , a photographing section 21 and a background section 22 . The pre-selection unit 20 is installed on the side of the imaging unit 21 . The space in front of the pre-selection unit 20 becomes the pre-selection space where the pre-selection process is performed. Also, the photographing unit 21 and the background unit 22 are placed apart from each other by a predetermined distance. A space formed between the photographing section 21 and the background section 22 is a photographing space in which photographing processing is performed.

As the pre-selection process, the pre-selection unit 20 performs guidance for introducing games provided by the photo sticker creation device 1, and performs various settings in the photographing process performed in the photographing space. The pre-selection unit 20 is provided with a coin slot into which the user inserts money, a touch panel monitor used for various operations, and the like. The pre-selection unit 20 guides the user in the pre-selected space to the photographic space according to the availability of the photographic space.

The photographing unit 21 is a device for photographing a user as a subject. The photographing unit 21 is positioned in front of the user entering the photographing space. A camera, a touch panel monitor used for various operations, and the like are provided in front of the photographing unit 21 facing the photographing space.

Assuming that the left side and the right side are the left side and the right side, respectively, as viewed from the user facing the front in the imaging space, the left side of the imaging unit 21 is composed of the side panel 41A, and the right side is composed of the side panel 41B. be done. Further, a front panel 42 constitutes the front of the photographing unit 21 . The preselection section 20 described above is installed on the side panel 41A. Note that the preselection unit 20 may be installed on the side panel 41B.

The background portion 22 is composed of a back panel 51, side panels 52A, and side panels. The back panel 51 is a plate-like member located on the back side of the user facing the front. The side panel 52A is a plate-like member attached to the left end of the rear panel 51 and having a width narrower than that of the side panel 41A. The side panel 52B is a plate-like member attached to the right end of the rear panel 51 and having a width narrower than that of the side panel 41B.

The side panel 41A and the side panel 52A are provided substantially on the same plane. Upper portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A, which is a plate-like member. The lower portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A' which is provided on the floor and is a member made of metal, for example. Similarly, the side panel 41B and the side panel 52B are provided substantially on the same plane. Upper portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B. The lower portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B'.

The side panel 41A and the side panel 52A are members surrounding the left side of the photographing space, and the side panel 41B and the side panel 52B are members surrounding the right side of the photographing space.

A floor material 27, which is a plate-like member, is laid on the floor of the imaging space.

An opening formed by being surrounded by the side panel 41A, the connecting portion 23A, and the side panel 52A serves as an entrance to the imaging space. An opening formed by being surrounded by the side panel 41B, the connecting portion 23B, and the side panel 52B also serves as an entrance to the imaging space.

A ceiling sheet 24 made of a vinyl member or a plate-shaped member is provided on a part of the ceiling of the photographing space.

The editing unit 12 is a device for editing a captured image. The editing unit 12 is connected to the photographing unit 11 so that one side thereof contacts the front panel 42 of the photographing section 21 .

If the configuration of the editing unit 12 shown in FIG. 2 is taken as the configuration on the front side, the configuration used in the editing work is provided on each of the front side and the back side of the editing unit 12 . This configuration allows two sets of users to perform editing work at the same time.

The front side of the editing unit 12 is composed of a vertical surface 61 and an inclined surface 62 formed above the surface 61 . A built-in tablet monitor and a touch pen are provided on the slope 62 as components used for editing work. A columnar support portion 63A that supports one end of the lighting device 64 is provided on the right side of the slope 62 . A columnar support portion 63B that supports the other end of the lighting device 64 is provided on the left side of the slope 62 . A support portion 65 for supporting the curtain rail 26 is provided on the upper surface of the support portion 63A.

A curtain rail 26 is attached above the editing unit 12 . The curtain rail 26 is configured by combining three rails 26A to 26C.

The three rails 26A to 26C are combined to form a U-shape when viewed from above. One ends of rails 26A and 26B, which are provided in parallel, are fixed to connecting portions 23A and 23B, respectively, and the other ends of rails 26A and 26B are fixed to both ends of rail 26C, respectively.

A curtain is attached to the curtain rail 26 so that the space in front of the front and the space in front of the back of the editing unit 12 are not visible from the outside. The space in front of the front and the space in front of the back of the editing unit 12 surrounded by the curtains attached to the curtain rails 26 serve as an editing space in which the user performs editing work.

A side face of the editing unit 12 is provided with a discharge port through which the printed sticker paper is discharged. A space in front of the side surface of the editing unit 12 becomes a print waiting space where the user waits for the printed sticker paper to be discharged.

<Regarding the movement of users>
Here, the flow of the photo sticker creation game and the movement of the user associated therewith will be described.

FIG. 3 is a top plan view of the photo sticker creating apparatus 1. FIG.

The user inserts money into the coin slot in the pre-selection space A0, which is the space in front of the pre-selection section 20. FIG. Next, the user performs pre-selection according to the screen displayed on the touch panel monitor. For example, the user selects a course related to shooting in the shooting space as a pre-selection task.

After completing the pre-selection work, the user enters the imaging space A1 formed between the imaging unit 21 and the background unit 22 from the entrance G1 between the side panel 41A and the side panel 52A, as indicated by the white arrow #1. to go into. A user uses a camera, a touch panel monitor, or the like provided in the photographing unit 21 to perform photographing work.

The user who has completed the photographing work exits the photographing space A1 from the doorway G1 as indicated by the white arrow #2 and moves to the editing space A2-2, or enters the doorway G2 as indicated by the white arrow #3. , exits the photographing space A1 and moves to the editing space A2-1.

The editing space A2-1 is an editing space on the front side of the editing unit 12. FIG. On the other hand, the editing space A2-2 is the editing space behind the editing unit 12. FIG. To which of the editing space A2-1 and the editing space A2-2 the user should move is guided by the screen display of the touch panel monitor of the photographing unit 21 or the like. For example, the empty one of the two editing spaces is guided as the destination.

The user who has moved to the editing space A2-1 or the editing space A2-2 starts editing work. The user of the editing space A2-1 and the user of the editing space A2-2 can perform editing work at the same time. When there is no need to distinguish between the editing space A2-1 and the editing space A2-2, they are collectively referred to as the editing space A2 as appropriate.

After the editing work is completed, printing of the edited image is started. After completing the editing work in the editing space A2-1, the user moves from the editing space A2-1 to the print waiting space A3 as indicated by the white arrow #5. Also, the user who has finished the editing work in the editing space A2-2 moves from the editing space A2-2 to the print waiting space A3 as indicated by the white arrow #4.

The user who has moved to the print waiting space A3 waits for the end of printing. When the printing is finished, the user receives the sticker paper ejected from the ejection port provided on the side of the editing unit 12, and finishes the series of photo sticker making games.

Next, the configuration of each device will be described.

<Configuration of pre-selection unit>
FIG. 4 is a diagram showing a configuration example of the pre-selection unit 20. As shown in FIG.

A touch panel monitor 71 is provided above the pre-selection section 20 . The touch panel monitor 71 is composed of a monitor such as an LCD (Liquid Crystal Display) and a touch panel stacked thereon. The touch panel monitor 71 has a function of displaying various GUIs (Graphical User Interfaces) and receiving a user's selection operation. The touch panel monitor 71 displays a screen used for selecting the number of users.

A speaker 72 is provided below the touch panel monitor 71 . The speaker 72 outputs a guide voice for pre-selection processing, BGM (Background Music), sound effects, and the like. Next to the speaker 72, a coin insertion/return opening 73 into which the user inserts coins is provided.

<Structure of shooting unit>
5 and 6 are perspective views showing configuration examples of the photographing unit 21. FIG.

The imaging unit 21 is configured to be surrounded by the side panel 41A, the side panel 41B, and the front panel 42. As shown in FIG.

A box-shaped camera unit 81 is provided substantially in the center of the front panel 42 . A camera 91 and a touch panel monitor 92 are provided in front of the camera unit 81 . A smartphone holder 93 is provided on the right side of the camera 91 .

The camera 91 is, for example, a single-lens reflex camera, and is attached inside the camera unit 81 so that the lens is exposed. The camera 91 has an imaging device such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 91 functions as a photographing unit that photographs a user in the photographing space A1 as a subject.

The camera 91 also has a vertical swing function (angle adjustment function).

FIG. 7 is a diagram showing an example of angle adjustment of the camera 91. As shown in FIG.

As shown in FIG. 7, the orientation of the optical axis of the camera 91 can be adjusted vertically by changing the posture inside the camera unit 81 . In FIG. 7, the dashed arrow indicates the direction of the optical axis of the lens.

As shown in FIGS. 7A to 7C, for example, the angle of the camera 91 can be adjusted within a range from a direction substantially horizontal to the ground surface of the photo sticker creating apparatus 1 as an upper limit to a predetermined downward angle as a lower limit. It is possible.

Such adjustment of the angle of the camera 91 is electrically performed according to the operation of the user who is taking the picture, as will be described later. The user can adjust the angle of the camera 91 to a desired angle and take a picture.

A live view image generated based on the moving image captured by the camera 91 is displayed on the touch panel monitor 92 . A still image captured by the camera 91 at a predetermined timing is saved as a photographed image.

A touch panel monitor 92 is provided below the camera 91 . The touch panel monitor 92 is composed of a monitor such as an LCD and a touch panel laminated thereon. The touch panel monitor 92 has a function as a live view monitor that displays a live view image captured by the camera 91, and a function that displays various GUIs and receives a user's selection operation.

On the right side of the camera 91, a smart phone holder 93 is provided for placing a smart phone. The smartphone holder 93 is formed to be long enough to place the smartphone sideways.

The user can place his/her own smartphone on the smartphone holder 93 and use the smartphone to capture a video or still image of himself/herself during the game.

Above the camera unit 81, an upper strobe unit 82 is installed with the position of the camera unit 81 (the position of the camera 91) as a reference and biased to the left. The upper strobe unit 82 is provided so as to cover substantially the entire upper and upper left portions of the camera unit 81 .

The front surface of the upper strobe unit 82 is composed of a panel tilted forward to face obliquely downward, and a panel provided below and forming a substantially vertical plane. The front surface of the upper strobe unit 82 is configured as a light-emitting surface using a translucent milky-white plate-like member. The upper strobe unit 82 emits light in synchronization with photographing by the camera 91, and irradiates mainly the user's face and upper body with strobe light from the upper left front of the user.

Below the camera unit 81, a foot strobe unit 83 is provided that mainly irradiates the user's lower body and feet with strobe light.

The foot strobe unit 83 is formed without a gap between the side panels 41A and 41B so that the entire front surface, which is a horizontally long rectangle when viewed from the front, serves as a light emitting surface. The upper surface of the foot strobe unit 83 is substantially horizontal to the installation surface of the photo sticker creating apparatus 1, and is used as a baggage storage area for the user who is performing the photographing work to place his/her baggage.

Inside the upper strobe unit 82 and the foot strobe unit 83, a constantly lit light bulb and a strobe arc tube are provided. The upper strobe unit 82 and the foot strobe unit 83 illuminate the shooting space at all times by emitting light bulbs, and irradiate the user, who is the subject, with strobe light by emitting light from the strobe arc tube in accordance with the shooting timing. .

A speaker is provided near the ceiling of the front panel 42 . From the speaker, guidance voice, BGM, sound effects, etc. for the shooting process are output. A speaker may be provided in the background portion 22 .

FIG. 8 is a diagram showing a configuration example of the background portion 22 on the photographing space A1 side.

A sheet of a predetermined color such as green is attached to the inner side of the rear panel 51 . Chromakey processing is performed by detecting a green area appearing in the photographed image, and a person's area is detected.

An operation unit 53 is provided inside a side panel 52B attached to the left end of the rear panel 51 .

As shown in FIG. 9, the operation unit 53 includes an angle adjustment button 53A that is pressed to adjust the orientation of the camera 91 upward or downward, and a shooting stop button 53B that is pressed to stop shooting. consists of

Photographing by the camera 91 is continuously performed at predetermined intervals such as 10-second intervals. The shooting stop button 53B is pressed to temporarily stop the continuous shooting. The user can adjust the angle of the camera 91 by pressing the angle adjustment button 53A while the continuous shooting is temporarily stopped.

Since the operation unit 53 is provided on the side panel 52B, the standing position of the user facing the camera 91 is unconsciously slightly shifted to the right side in the photographing space A1. By securing a distance from the upper strobe unit 82 provided on the left side of the front diagonally of the operation unit 53 in the photographing space A1 to the standing position, an appropriate amount of light is emitted from the upper strobe unit 82 to the user. can be irradiated.

The operation unit 53 provided on the side panel 52B also has a function of guiding the user's standing position unconsciously. The operation unit 53 may be provided on the side panel 52A side.

<Structure of editing unit>
FIG. 10 is a diagram showing a configuration example of the front side of the editing unit 12 (on the editing space A2-1 side in FIG. 3). A similar configuration is also provided on the rear side (editing space A2-2 side in FIG. 3).

A tablet built-in monitor 131 is provided approximately in the center of the slope 62 . A touch pen 132A is provided on the left side of the tablet built-in monitor 131 . A touch pen 132B is provided on the right side of the tablet built-in monitor 131 .

The tablet built-in monitor 131 displays, for example, an editing screen used for editing work. When two users edit at the same time, the touch pen 132A is used by the user on the left side of the tablet built-in monitor 131, and the touch pen 132B is used by the user on the right side of the tablet built-in monitor 131. .

An operation using the touch pen 132A and an operation using the touch pen 132B are distinguished. As appropriate, the display on the left side of the tablet built-in monitor 131 can only be operated using the touch pen 132A, and the display on the right side can only be operated using the touch pen 132B. Hereinafter, when there is no need to distinguish between the touch pen 132A and the touch pen 132B, the touch pen 132A and the touch pen 132B are collectively referred to as the touch pen 132.

Below the illumination device 64 and between the columnar support portions 63A and 63B that support the illumination device 64, a mirror surface portion 66 made of an acrylic mirror plate or the like is provided. The mirror surface portion 66 forms a substantially vertical surface.

In FIG. 10, the range of the mirror surface portion 66 that becomes the mirror surface is shown in color. The width of the mirror portion 66 in the horizontal direction is wider than the width of the tablet built-in monitor 131 in the horizontal direction. The mirror surface portion 66 reflects the inside of the editing space A2-1, including the vicinity of the user's face in the editing space A2-1.

FIG. 11 is a diagram showing an example of curtain attachment.

As shown in FIG. 11, a curtain is attached to the curtain rail 26 above the editing unit 12 so as to hide the editing space A2 from the outside. In the example of FIG. 11, a curtain 111 is attached to rail 26C, and a curtain 112 is attached to rail 26B on the front side of editing unit 12. In FIG. A curtain similar to the curtain 112 is attached to the rail 26A on the rear side of the editing unit 12 as well.

A predetermined design is applied to the surfaces of the curtains 111 and 112 made of vinyl material so that the photo sticker creating apparatus 1 stands out. Further, of the curtain 111 and the curtain 112, at least the curtain 112 has a predetermined design on not only the front surface but also the back surface (the surface on the editing space A2-1 side).

FIG. 12 is a diagram showing a design example of the back surface of the curtain 112. As shown in FIG.

As shown in FIG. 12, the reverse side of the curtain 112 is designed by printing characters and letters.

In the example of FIG. 12, a design consisting of an illustration of a bird and an illustration of a flower, which are characters, is applied. Above the curtain 112, the characters "Happy!" surrounded by flowers are printed in a horizontally reversed state. The design of the back side of the curtain 112 is optional, such as printing other illustrations or photographs of landscapes or people.

A user in the editing space A2-1 stands with his/her back facing the curtain 112 designed as shown in FIG. The design of the curtain 112 is reflected as the user's background on the mirror surface portion 66 in front of the user.

FIG. 13 is a diagram showing a usage example of the mirror surface portion 66. As shown in FIG.

As shown in FIG. 13, the user in the editing space A2-1 can point the camera of his/her smart phone toward the mirror surface portion 66 and take an image of himself reflected on the mirror surface portion 66. As shown in FIG. Although only one user is shown in the example of FIG. 13, when two people are playing the game, both of them can take pictures.

In other words, the mirror section 66 is prepared for the user in the editing space A2-1 to take a so-called self-portrait (self-portrait) through the mirror.

Since the back of the curtain 112 has the design as described above, the user can take a selfie with the design on the curtain 112 as his or her background. Also, the characters on the curtain 112 printed in a left-right reversed state are reflected in the mirror surface portion 66 in a normal reversed state, as shown in FIG. The characters printed on the curtain 112 appear in the image obtained by the self-portrait in a reversed state.

The user can upload the self-portrait image taken in the editing space A2-1 to the site of the SNS (Social Networking Service) used by the user and show it to his/her friends.

As described above, the photo sticker creating apparatus 1 is a device capable of taking pictures not only in the photographing space A1 but also in the editing space A2. As will be described later, at the start of editing, the user is advised to take a selfie. For the user who has finished shooting and moved to the editing space A2, time for taking a selfie is secured before editing.

The design of the rear surface of the curtain 112 attached to the editing space A2-1 side may be different from the design of the rear surface of the curtain 112 attached to the editing space A2-2 side. The user can enjoy which design of the curtain 112 is guided to the editing space A2 in which the curtain 112 is installed.

As the design of the back side of the curtain 112, an advertisement for goods or services may be printed.

FIG. 14 is a diagram showing a side configuration example of the editing unit 12. As shown in FIG.

A seal paper discharge port 161 is provided on the lower side of the editing unit 12 . A printer is provided inside the editing unit 12 . A printer prints an image of a user who has performed editing work in the editing space A2-1 or an image of a user who has performed editing work in the editing space A2-2 on a sticker paper in a predetermined layout, and a sticker is printed. It is discharged from the paper discharge port 161 .

<Layer structure of edited image>
FIG. 15 is a diagram showing an example of the layer structure of an edited image.

The edited image P1 at the left end is an image generated by editing. The edited image P1 is basically composed of images of layers of a background image P11, a person image P12, and a foreground image P13. Layers to be subjected to various types of image processing are also set appropriately.

In the example of FIG. 15, the background image P11 is a monochrome image. The foreground image P13 is an image including an image of handwritten characters "Love my friend" and a heart-shaped image. The handwritten character image of "Love my friend" is a pen image input by the user using the pen tool. The heart-shaped image is a stamp image input by the user using the stamp tool.

Thus, the edited image P1 is generated by superimposing the person image P12, which is the image of the person's area extracted from the photographed image by chromakey processing, on the background image P11, and superimposing the foreground image P13 thereon. It is also possible to input a pen image or a stamp image onto the background image P11 and display it in the background of the subject. A foreground image or a background image may be combined with a photographed image, or may not be combined depending on a user's operation even after an editing process.

In the example of FIG. 15, an image in which a range from the vicinity of the user's chest to the top is largely captured is shown as the person image P12. In the photographing process, a close-up image, which is a photographed image in which the face is enlarged in this way, is photographed. Shooting of the close-up image is performed, for example, multiple times. A full-body image of the user's whole body may be captured.

<Internal configuration of photo sticker creation device>
FIG. 16 is a block diagram showing a configuration example of the photo sticker creating apparatus 1. As shown in FIG. In FIG. 16, the same reference numerals are given to the same configurations as those described above. Duplicate explanations will be omitted as appropriate.

As shown in FIG. 16, the photo sticker creating apparatus 1 has an external input/output interface 303 for a PC (Personal Computer) section 301, a prior selection section 302, a photographing section 304, editing sections 305A and 305B, and a printing section 306. It is configured by being connected via A PC section 301, which is an information processing section, is mounted inside the housing of the editing unit 12, for example.

A CPU (Central Processing Unit) 311 , a ROM (Read Only Memory) 312 , and a RAM (Random Access Memory) 313 constituting the PC section 301 are interconnected by a bus 314 .

A CPU 311 executes a predetermined program and controls the overall operation of the photo sticker creating apparatus 1 . The ROM 312 stores programs and data executed by the CPU 311 . A RAM 312 temporarily stores data and programs processed by the CPU 311 .

An input/output interface 315 is further connected to the bus 314 . A storage unit 316 , a communication unit 317 and a drive 318 are connected to the input/output interface 315 .

The storage unit 316 is a nonvolatile storage medium such as a hard disk or flash memory. The storage unit 316 stores various setting information and the like supplied from the CPU 311 . Information stored in the storage unit 316 is appropriately read by the CPU 311 .

A communication unit 317 is a network interface such as the Internet. A communication unit 317 communicates with an external device under the control of the CPU 311 . The communication unit 317 transmits the image selected by the user to an image management server managed by the manufacturer of the photo sticker creating apparatus 1, for example. Images transmitted from the communication unit 317 are stored in a predetermined storage area, displayed on the mobile terminal that has accessed the image management server, or downloaded to the mobile terminal.

A removable medium 319 such as an optical disk or a semiconductor memory is mounted in the drive 318 as appropriate. Programs and data read from the removable medium 319 by the drive 318 are supplied to the CPU 311 and stored in the storage unit 316 or installed.

The external input/output interface 303 is connected to the input/output interface 315 . Each unit is controlled by the PC unit 301 via the external input/output interface 303 .

The pre-selection unit 302 performs pre-selection processing for users in the pre-selection space A0. The preselection unit 302 is composed of a touch panel monitor 71 , a speaker 72 and a coin processing unit 321 .

The touch panel monitor 71 displays various selection screens under the control of the PC unit 301 and accepts user operations on the selection screens. An input signal representing the content of the user's operation is supplied to the PC unit 301, and various settings are performed.

The coin processing unit 321 detects insertion of coins into the coin insertion/return port 73 . When the coin processing unit 321 detects that a predetermined amount of coins has been inserted, the coin processing unit 321 outputs an activation signal instructing the start of the game to the PC unit 301 .

The photographing unit 304 performs photographing processing for users in the photographing space A1. The photographing unit 304 is composed of the operation unit 53 , the camera 91 , the camera driving unit 91A, the touch panel monitor 92 , the illumination device 331 and the speaker 332 .

The operation unit 53 outputs a signal representing the user's operation to the PC unit 301 according to the user's operation on the angle adjustment button 53A and the shooting stop button 53B.

The camera 91 performs photographing according to shutter control by the PC section 301 and outputs a photographed image (image data) obtained by photographing to the PC section 301 .

The camera drive unit 91A adjusts the vertical angle of the camera 91 under the control of the PC unit 301 .

The lighting device 331 is each strobe unit in the shooting space A1 and emits light according to the lighting control signal supplied from the PC section 301 . The lighting device 331 includes an upper strobe unit 82 and a foot strobe unit 83 .

The speaker 332 is composed of speakers provided at respective positions in the imaging space A1.

The editing unit 305A performs editing processing for users in the editing space A2-1. The editing unit 305A is composed of a tablet built-in monitor 131, touch pens 132A and 132B, and a speaker 341. FIG. The editing section 305B performs editing processing for users in the editing space A2-2, and has the same configuration as the editing section 305A. Hereinafter, the editing units 305A and 305B are collectively referred to as an editing unit 305 as appropriate.

The tablet built-in monitor 131 displays an edit screen under the control of the PC unit 301 and accepts user's operations on the edit screen. An input signal representing the content of the user's operation is supplied to the PC unit 301, and the captured image to be edited is edited.

Printing unit 306 is configured to include printer 351 . A sticker paper unit 352 is attached to the printer 351 .

The printer 351 drives a built-in head 361 and prints an image on a roll-shaped sticker paper 363 stored in the sticker paper unit 352 based on print data supplied from the PC section 301 . The printer 351 also cuts the sticker paper 363 on which the image is printed by the cutter 362 at a predetermined cut position, and discharges the cut paper to the sticker paper outlet 161 .

<Example of functional configuration of photo sticker creation device>
FIG. 17 is a block diagram showing a functional configuration example of the photo sticker creating apparatus 1. As shown in FIG. At least some of the functional units shown in FIG. 17 are implemented by the CPU 311 executing a predetermined program.

In the photo sticker creating apparatus 1, a pre-selection processing unit 401, a photographing processing unit 402, an editing processing unit 403, and a printing processing unit 404 are implemented.

The pre-selection processing unit 401 executes pre-selection processing by controlling each unit of the pre-selection unit 302 . Selection of the number of users and the like are performed by the pre-selection process. Information representing the content selected by the user is supplied to the imaging processing unit 402 .

The imaging processing unit 402 performs imaging processing by controlling each unit of the imaging unit 304 . The shooting processing unit 402 outputs the data of the shot image obtained by performing the shooting processing to the editing processing unit 403 . The shooting processing unit 402 supplies the editing processing unit 403 with data of a plurality of captured images obtained by performing shooting a plurality of times.

The editing processing unit 403 performs editing processing by controlling each unit of the editing unit 305 . The editing processing includes image processing for adjusting the size of the user's eyes in the captured image, image processing for editing the captured image, and the like.

After editing the captured image, the edit processing unit 403 displays various screens on the tablet built-in monitor 131, such as a screen used for selecting the number of divisions and a screen used for selecting an image to be transmitted to the image management server. The edit processing unit 403 generates print data by arranging each edited image according to the number of divisions selected by the user, and outputs the print data to the print processing unit 404 . Also, the edit processing unit 403 transmits the edited image selected by the user to the image management server.

The print processing unit 404 outputs print data to the printer 351 to print the edited image.

<Detailed Configuration of Shooting Processing Unit>
FIG. 18 is a block diagram showing a configuration example of the imaging processing unit 402. As shown in FIG.

The shooting processing unit 402 is composed of a shooting control unit 411 , an image processing unit 412 , a display control unit 413 and an image storage unit 414 .

The shooting control unit 411 shoots a moving image by controlling the camera 91 and outputs the image obtained by shooting to the image processing unit 412 . The moving image output from the shooting control unit 411 is used to generate a live view image.

The shooting control unit 411 also shoots a still image in time with the countdown, and outputs the still image obtained by shooting to the image processing unit 412 . A still image output from the imaging control unit 411 is used to generate a captured image.

The image processing unit 412 cuts out a rectangular region from each frame of the moving image supplied from the imaging control unit 411 by trimming, and generates a moving image made up of the cut out images. The moving image generated by the image processing unit 412 is used as a live view image displayed on the live view screen.

Further, the image processing unit 412 generates a photographed image having a predetermined angle of view by cutting out a rectangular area from the still image supplied from the photographing control unit 411 by trimming. The captured image generated by the image processing unit 412 is supplied to the image storage unit 414 and stored therein.

The display control unit 413 controls display of various screens of the touch panel monitor 92 such as a live view screen.

The image storage unit 414 stores captured images generated by the image processing unit 412 . When the shooting process ends, the shot image stored in the image storage unit 414 is supplied to the editing processing unit 403 .

<Detailed configuration of editing processing unit>
FIG. 19 is a block diagram showing a configuration example of the edit processing unit 403 of FIG. 17. As shown in FIG.

The edit processing unit 403 includes a selection reception unit 431 , an edit image generation unit 432 , an edit control unit 433 , a display control unit 434 , a print data generation unit 435 and a communication control unit 436 . The edit control section 433 includes a iris processing section 433A and a contour adjustment section 433B.

The selection accepting unit 431 accepts a user's operation on the screen displayed on the tablet built-in monitor 131 . Information representing the content of the user's operation received by the selection receiving unit 431 is supplied to the editing control unit 433 and the display control unit 434 .

The editing image generation unit 432 generates a shot image for editing based on the shot image supplied from the shooting processing unit 402 . For example, each image after trimming by the image processing unit 412 (FIG. 18) is used as an original image, and an image obtained by performing various processes such as retouching on the original image becomes the photographed image for editing. The photographed image for editing generated by the editing image generation unit 432 is supplied to the editing control unit 433 and the display control unit 434 .

The editing control unit 433 edits the photographed image for editing supplied from the editing image generating unit 432 based on the information output from the selection receiving unit 431 . Editing is image processing for synthesizing an image for synthesis, such as handwritten characters or a stamp image, with a photographed image to be edited.

While the retouching process performed by the editing image generation unit 432 is a process of collectively applying image processing according to the content selected by the user to each of a plurality of captured images, editing is a process of performing image processing according to the user's selection. This is a process of applying image processing according to the content to a photographed image that is being edited at that time. At the time of retouching processing, the eye size and the like may be selected for each captured image.

The editing control unit 433 outputs information representing the editing result to the display control unit 434 and reflects it in the display of the editing screen.

When editing is completed, the editing control unit 433 outputs an edited image, which is a photographed image obtained by editing, to the print data generation unit 435 and the communication control unit 436 .

The iris processing unit 433A of the editing control unit 433 performs iris processing, which is processing for changing the color of the iris of each user appearing in the captured image, according to the user's operation on the editing screen.

The contour adjusting unit 433B of the editing control unit 433 performs image processing according to the user's operation on the editing screen, and emphasizes a specific part of the user appearing in the photographed image for editing. For example, the contour adjustment unit 433B performs image processing for emphasizing the entire eye or part of the eye such as the iris.

The display control unit 434 displays an operation screen used for operation by the user on the tablet built-in monitor 131, and controls the display of the operation screen according to the user's operation. The display control unit 434 is supplied with the captured image output from the editing image generation unit 432, the edited image output from the editing control unit 433, and the like. Various images supplied to the display control unit 434 are used for displaying the operation screen.

The print data generation unit 435 generates print data by arranging the edited images supplied from the edit control unit 433 in a predetermined layout. The print data generation unit 435 outputs the print data to the print processing unit 404 to print it on the sticker paper 363 .

The communication control unit 436 controls the communication unit 317 and transmits the edited image supplied from the editing control unit 433 to the image management server.

FIG. 20 is a block diagram showing a configuration example of the editing image generation unit 432. As shown in FIG.

The editing image generation unit 432 includes an area detection unit 451 , a retouch processing unit 452 , a gender determination unit 453 , an eyelash processing unit 454 and a reduction unit 455 .

A photographed image supplied from the photographing processing unit 402 is input to an area detection unit 451 , a retouch processing unit 452 , and a gender determination unit 453 . Processing by each unit of the editing image generation unit 432 is performed on each captured image.

The region detection unit 451 detects the region of the user who is a person in the captured image, and also detects the regions of each part of the user. For example, the region detection unit 451 detects the region of the user's face, the eye, eyelid, nose, lips, cheek region, hair region, etc. as parts of the face.

FIG. 21 is a diagram illustrating an example of region detection.

When a photographed image P101 in which two women are photographed is supplied, detection of the face and regions of each part of the face is performed for each user. For example, for a user photographed on the left side, respective areas including left and right eye areas are detected.

Focusing on the area of the user's right eye shown on the left side, a black eye area A11 and left and right white eye areas A12 and A13 of the black eye area A11 are detected, respectively, as shown in the enlarged left side of FIG.

The white eye region A12 is the white eye region closer to the nose, and the white eye region A13 is the white eye region closer to the ear. The position p1 represents the position of the inner corner of the eye, and the position p2 represents the position of the outer corner of the eye. In the iris area A11, catch lights corresponding to the shapes and sizes of the upper strobe unit 82 and the foot strobe unit 83 are reflected.

Detection of the black eye region and the white eye region is performed based on, for example, a prediction model configured by a neural network or the like. For the area detection unit 451, a prediction model for detecting each area of the eye, which is generated in advance by machine learning, is prepared.

The prediction model prepared in the region detection unit 451 is, for example, a model that receives a captured image as an input and outputs a mask image of each region of the eyes and a mask image of the skin region around the eyes. The mask image output from the prediction model represents the position, size, shape, and the like of each region of the black eye region and the white eye region.

The detection of the black eye region and the white eye region may be performed by matching based on the feature amount of each region extracted by analyzing the image instead of using the prediction model. In this case, the region detection unit 451 prepares data representing feature amounts of each region of the black eye region and the white eye region.

Not only each area of the eyes, but also the entire face and other parts of the face may be detected using a prediction model generated by machine learning. Alternatively, detection may be performed by matching based on feature amounts.

The area detection unit 451 in FIG. 20 outputs the information of each area detected as described above to the retouch processing unit 452 and the eyelash processing unit 454 . The information output from the area detection unit 451 represents the position, size, shape, and the like of the iris area and the white area of the left and right eyes of each user in the captured image. Information representing the detection result by the area detection unit 451 is also supplied to the editing control unit 433 in the subsequent stage.

The retouch processing unit 452 performs retouch processing on the captured image. The retouching process is image processing for a specific part of the user appearing in the captured image, such as adjusting the contour of the face and adjusting the size of the eyes. Each area to be subjected to image processing is specified by information supplied from the area detection unit 451 . The photographed image that has been retouched is supplied to the eyelash processing unit 454 .

Other processing, such as skin brightness adjustment, may be performed as retouch processing. Other image processing such as filtering may be performed on the entire image instead of processing on a specific part.

The gender determination unit 453 determines the gender of each user appearing in the captured image, and outputs information representing the determination result to the eyelash processing unit 454 . The determination by the gender determination unit 453 may also be performed based on a prediction model generated by machine learning, or may be performed by matching based on the feature amount of each region extracted by analyzing the image. can be

The eyelash processing unit 454 performs eyelash processing, which is image processing for emphasizing eyelashes (eyelashes) of each user appearing in the captured image supplied from the retouch processing unit 452 . Based on the determination result of the gender determination unit 453, eyelash processing is performed only for women, for example. The photographed image in which the eyelashes are emphasized by the eyelash processing is supplied to the reduction unit 455 . The eyelash processing performed by the eyelash processing unit 454 will be detailed later.

The reduction unit 455 reduces the captured image supplied from the eyelash processing unit 454 to generate a captured image for editing. Reduction of the captured image is performed, for example, by thinning out pixels to reduce the number of pixels forming the captured image. The shot image for editing generated by the reduction unit 455 is supplied to the editing control unit 433 and the like.

In this way, in the editing image generation unit 432, the retouching process and the eyelashes process are performed on the original image before reduction. It is possible to perform fine processing on captured images with a large amount of data.

FIG. 22 is a block diagram showing a configuration example of the eyelash processing unit 454 of FIG.

The eyelash processing unit 454 is composed of an upside-down eyelash removing unit 461 , a transforming unit 462 , a synthesizing unit 463 , and an eyelash enhancement processing unit 464 .

The retouched photographed image supplied from the retouch processing unit 452 is input to the upside-down eyelash removal unit 461 . Information on each area detected by the area detection section 451 is input to each section of the eyelash processing section 454 . Information representing the result of gender determination by gender determination unit 453 is input to transforming unit 462 and synthesizing unit 463 .

The upside-down eyelash removing unit 461 removes the upside-down eyelashes by image processing when the eyelashes of the user appearing in the photographed image are present. The upside-down eyelashes are the eyelashes that grow downward and cover the white region of the upper eyelashes, and the eyelashes that grow upward and cover the white region of the lower eyelashes. The position of the white-of-the-eye area and the like are specified based on the information supplied from the area detection section 451 .

FIG. 23 is a diagram showing an example of removal of upside down eyelashes.

When the white eye region A13 is covered with the eyelashes as shown by the dashed circle C1 on the left side of FIG. By (interpolating), we remove the upside-down lashes as shown on the right side of FIG.

The eyelashes appearing in the white eye region A13 is obtained by detecting a pixel of a color other than white set as the standard color of the white eye region A13 as an abnormal pixel. identified. Information representing the standard color of the white eye area is set in advance in the upside-down eyelash removal section 461 .

The upside-down eyelash removal unit 461 removes upside-down eyelashes by interpolating abnormal pixels, which are pixels of colors other than the standard color, with surrounding white pixels.

Pixels of colors other than white are detected as abnormal pixels, and the process of interpolating with white pixels is performed by the upside-down eyelash removal unit 461, so that as a result, the congested and reddish portions are also removed. .

The photographed image from which the upside-down eyelashes have been removed as described above is output from the upside-down eyelash removal unit 461 . If there is no upside-down eyelashes on the user's eye in the captured image, the captured image input to the upside-down eyelash removal unit 461 is output as it is.

If the user in the photographed image supplied from the upside-down eyelash removing unit 461 is a female, the transforming unit 462 transforms the eyelash image in accordance with the shape of the upper contour of the user's eye. The deformation unit 462 stores in advance an eyelash image, which is a texture image representing eyelashes. The eyelash image after deformation by the deformation unit 462 is used for synthesizing the user's eye portion in the photographed image.

FIG. 24 is a diagram showing an example of deformation of an eyelash image.

The deformation section 462 has an eyelash image P111 with a default shape as shown in the upper left of FIG. For example, when the shape of the upper contour of the user's right eye E _R in the photographed image is represented by a curved line from position p1 to position p2 shown in the lower left of FIG. - Deform to the shape of a particular segment of the curve between positions p2 to generate the deformed eyelash image P111' shown on the right side of FIG. A curve between the position p1 and the position p2 is identified based on the information supplied from the area detection section 451. FIG.

The transforming unit 462 outputs the eyelash image transformed as described above to the synthesizing unit 463 . A plurality of default eyelash images are prepared, and an eyelash image to be used for deformation may be selected. The selection of the eyelash image may be performed by the user, or may be automatically performed by the eyelash processing section 454 .

The synthesizing unit 463 in FIG. 22 synthesizes the eyelash image deformed by the transforming unit 462 along the upper contour of the female user's eye appearing in the photographed image supplied from the upside-down eyelash removing unit 461 .

FIG. 25 is a diagram showing an example of synthesizing eyelash images.

When a deformed eyelash image P111′ as shown in the upper left of FIG. 25 is supplied, the synthesizing unit 463 transforms the eyelash image P111′ along the upper contour of the right eye E _R as shown in the right side of FIG. Synthesize. The position of the upper contour of the right eye E _R is identified based on the information supplied from the area detection section 451 .

Synthesis of eyelash images as described above is performed in the same manner for the left eye. Synthesis of the eyelash image for the left eye is performed using, for example, the eyelash image P111 prepared for the right eye.

In this case, the deformation unit 462 reverses the eyelash image P111 horizontally, and deforms the shape of the eyelash image P111 after the horizontal reverse to match the shape of the upper side of the left eye. The synthesizing unit 463 synthesizes the eyelash image P111 deformed by the deforming unit 462 along the upper contour of the left eye.

In this way, an image obtained by horizontally reversing the eyelash image prepared for the right eye is used for synthesizing the eyelash image for the left eye. Since the processing for the left eye is performed using the image for the right eye, the number of eyelash images that need to be prepared can be reduced. Conversely, an eyelash image for the left eye may be provided and the processing for the right eye may be performed using the image for the left eye. Of course, eyelash images for the right eye and left eye may be prepared.

Synthesis of eyelash images as described above is performed for each woman appearing in the photographed image. The synthesizing unit 463 outputs the photographed image after synthesizing the eyelash images to the eyelash enhancement processing unit 464 in FIG. 22 .

The eyelash enhancement processing unit 464 enhances the eyelashes appearing in the captured image supplied from the synthesizing unit 463 by image processing. For example, a process of thickening the upper eyelashes and the lower eyelashes one by one and a process of darkening the color are performed. The eyelash enhancement processing unit 464 outputs a photographed image in which the eyelashes are enhanced.

<Operation of photo sticker creation device>
Here, the operation of the photo sticker creating apparatus 1 that provides the photo sticker creating game will be described with reference to the flowchart of FIG. The process of FIG. 26 is started, for example, when a predetermined amount of coins is inserted.

In step S<b>1 , the pre-selection processing unit 401 performs pre-selection processing by controlling the pre-selection unit 302 . By the pre-selection process, for example, selection of the number of users, selection of an imaging course, and the like are performed. Any selection is made using a selection screen displayed on the touch panel monitor 71 .

In step S<b>2 , the imaging processing unit 402 performs imaging processing by controlling the imaging unit 304 .

For example, the imaging processing unit 402 causes the touch panel monitor 92 to display a live view screen based on the moving image captured by the camera 91 after guiding the standing position. Further, after displaying the live view screen, the imaging processing unit 402 performs continuous imaging of still images with the user in the imaging space A1 as the subject.

In step S3, the editing processing unit 403 controls the editing unit 305 corresponding to the editing space to which the user who has completed the shooting process is moved, out of the editing space A2-1 and the editing space A2-2. to perform the editing process.

For example, the editing processing unit 403 causes the tablet built-in monitor 131 to display an editing screen and allows the user to edit the captured image. On the editing screen, a photographed image to be edited and buttons to be operated when selecting a composite image to be composited with the photographed image are displayed. Details of the editing process will be described later with reference to the flowchart of FIG.

In step S<b>4 , the print processing unit 404 performs print processing by controlling the printer 351 . The print processing unit 404 outputs the print data generated by the editing processing unit 403 to the printer 351 to print it on the sticker paper 363 .

When printing is completed, the printer 351 ejects the sticker paper 363 cut by the cutter 362 to the sticker paper outlet 161 in step S5.

Each group of users of the photo sticker creating apparatus 1 is provided with a photo creating game consisting of a series of processes as described above. When a user of another group starts using the photo sticker creating device 1 following a user of a certain group, the photo sticker creating game of the user who started using the photo sticker creating device 1 first, and the user who started using the photo sticker creating device 1 later. The user's photo sticker creation game that has been started will be played in parallel as appropriate. As appropriate, the photo sticker creation game is also simply called a game.

<Editing process>
Next, the editing process performed in step S3 of FIG. 26 will be described with reference to the flowchart of FIG.

In step S41, the display control unit 434 (FIG. 19) of the editing processing unit 403 causes the tablet built-in monitor 131 to display a guide screen for self-shooting before graffiti. Graffiti here means editing of a photographed image. Editing is performed by using the touch pen 132 to scribble on the captured image.

FIG. 28 is a diagram showing a display example of a guide screen for self-portrait before graffiti.

As shown in FIG. 28, on the left side of the pre-doodle selfie guide screen, an image showing a model person taking a selfie through the mirror surface 66 is displayed. On the right side of the guide screen, a sample image of an image obtained by taking a selfie is displayed. Below each image is a message describing the content of the image.

By displaying such a guide screen, it is possible to tell the user in an easy-to-understand manner what kind of self-portrait can be performed using the mirror portion 66 .

The user who has moved to the editing space A2 prepares his/her smart phone according to the guidance screen and takes a selfie through the mirror surface portion 66. - 特許庁

Returning to the description of FIG. 27, in step S42, the editing image generation unit 432 performs editing image generation processing. Editing image generation processing generates a photographed image for editing based on each photographed image.

In the shooting process, for example, up to eight shot images are shot. Since the number of captured images is large, it takes a certain amount of time to generate a captured image for editing based on each captured image. A photographed image for editing is generated using the time reserved for self-portraits before editing. The user can wait without getting bored while the photographed image for editing is being generated.

The details of the editing image generation processing that is performed while the user is taking a selfie will be described later with reference to the flowchart of FIG.

In step S43, the display control unit 434 causes the tablet built-in monitor 131 to display a keep image selection screen. A keep image selection screen is displayed on the tablet built-in monitor 131 instead of the guidance screen of FIG.

A keep image is a photographed image that is actually edited. The number of shot images that can be selected as keep images is set to four, for example. On the keep image selection screen, retouched captured images are displayed side by side. The selection performed using the selection screen is performed by selecting four photographed images from a maximum of eight photographed images.

When a predetermined number of keep images, such as four, are selected, the display control unit 434 causes the tablet built-in monitor 131 to display an edit screen in step S44. The editing screen displays, together with the photographed image to be edited, buttons that are pressed when selecting various editing tools (compositing images) such as stamp images and pen images used for editing.

FIG. 29 is a diagram showing an example of an edit screen displayed on the tablet built-in monitor 131. As shown in FIG.

On the edit screen, edit areas 501L and 501R are arranged side by side. A thumbnail display area 502 is provided between the editing areas 501L and 501R to display thumbnails of captured images selected as keep images.

Edit palettes 503L and 503R are provided below the edit areas 501L and 501R, respectively. Between the editing palettes 503L and 503R, a category selection button group 504 is provided for selecting categories of editing tools displayed on the editing palettes 503L and 503R.

With such a configuration, on the editing screen, two users individually accept editing input of the shot image selected as the editing target.

Hereinafter, when the editing areas 501L and 501R and the editing palettes 503L and 503R are not distinguished from each other, they are simply referred to as the editing area 501 and the editing palette 503, respectively.

An editing area 501 is a display area for a keep image (editing target image) selected as a target for editing input. The user selects an editing tool using the touch pen 132 and edits the keep image displayed in the editing area 501 .

A thumbnail display area 502 is a display area for thumbnails representing keep images. The user selects a keep image for editing input by selecting a thumbnail displayed in the thumbnail display area 502 .

An editing palette 503 is an area where icons and the like used for selecting various editing tools such as a pen tool, a sticking tool, a stamp tool, and a makeup tool are displayed. Note that the sticking tool is an editing tool used when synthesizing a predetermined image at a position based on the face recognition result. Also, the makeup tool is an editing tool that is used when performing image processing that imitates makeup on the eyes, lips, cheeks, and the like.

A category selection button group 504 includes buttons that are pressed when switching the category of editing tools displayed on the editing palette 503 .

In the example of FIG. 29, the pen button of the category selection button group 504 is selected by the user on the left side of the tablet built-in monitor 131 . The editing palette 503L displays an icon for selecting a pen tool and the like.

Also, the make button of the category selection button group 504 is selected by the user on the right side. Editing palette 503R displays buttons for selecting makeup tools.

Under the thumbnail display area 502, two cellphone bonus buttons 505L and 505R imitating the appearance of a smart phone are displayed side by side. When the mobile phone bonus buttons 505L and 505R are operated, buttons used when creating a mobile phone bonus are displayed. The mobile bonus is an image that is assumed to be used on a mobile terminal, such as an image that is displayed on the standby screen of the mobile terminal or uploaded to SNS. The created mobile bonus is transmitted to the image management server as an image exclusively for transmission to the mobile terminal.

A BGM selection area 506 is displayed above the thumbnail display area 502 . A BGM selection area 506 is operated when selecting BGM during editing work and adjusting the volume.

Returning to the description of FIG. 27, in step S45, the editing control unit 433 edits the captured image to be edited in accordance with the user's operation on the editing screen. An edited image generated by performing editing is supplied to the print data generation unit 435 .

When the end button is operated and the editing ends, the display control unit 434 causes the tablet built-in monitor 131 to display a division number selection screen in step S46.

The division number selection screen is a screen used to select a seal layout. The sticker layout is a layout (arrangement pattern) that determines the number, positions, and sizes of edited images to be arranged on one sheet of sticker paper 363 . Edited images arranged according to the sticker layout selected by the user are printed on the sticker paper 363 .

In step S47, the display control unit 434 causes the tablet built-in monitor 131 to display a transmission image selection screen.

A user needs to register as a member in order to use the image management service, and there are paid members and free members. Paying members can acquire all the images transmitted to the image management server and download them to their mobile terminals, while free members can acquire only one image. The transmission image selection screen is a screen used for selecting one image to be acquired by the free member. Note that the image management service is a service for providing users with images created in the game and transmitted from the photo sticker creating device 1 to the image management server.

In step S48, the edit processing unit 403 causes the tablet built-in monitor 131 to display an email address input screen.

The e-mail address input screen is a screen used for inputting the user's e-mail address. The e-mail address input screen displays alphabets and symbols that can be used for e-mail addresses, buttons for selecting carriers of mobile phones such as smartphones, and the like.

In step S49, the communication control unit 436 transmits the edited image to the image management server. A captured image is also transmitted to the image management server as appropriate. That is, not only the edited image but also the photographed image from which the edited image is based are transmitted to the image management server.

Also, the device ID of the photo sticker creating apparatus 1, the date and time of photography, information indicating which image was selected from the transmission image selection screen, and the like are also transmitted to the image management server.

In step S50, the print data generation unit 435 generates print data by arranging the edited image in the layout selected by the user. The print data is supplied to the print processing unit 404 and used to print the edited image (step S4 in FIG. 26).

In step S51, the display control unit 434 displays a guide screen on the tablet built-in monitor 131 to guide the user who has finished the editing work to move to the print waiting space A3. Guidance to move to the print waiting space A3 is also provided by outputting a voice from the speaker 341. FIG.

After that, the process returns to step S3 in FIG. 26 and the subsequent processes are performed.

As described above, in the game of the photo sticker creating apparatus 1, it is possible to take pictures not only in the photographing space A1 but also in the editing space A2. The users can keep a self-portrait of themselves playing the game in the editing space A2.

<Editing image generation processing>
Next, the editing image generation processing performed in step S42 of FIG. 27 will be described with reference to the flowchart of FIG.

In step S61, the sex determination unit 453 determines the sex of each user appearing in the captured image.

In step S62, the retouch processing unit 452 performs retouch processing on the captured image. The retouch processing by the retouch processing unit 452 is performed according to the content selected using the retouch screen displayed on the touch panel monitor 92 during the shooting processing.

For example, the retouch processing unit 452 detects the eyes of the user appearing in the captured image, and performs processing for adjusting the size of the detected eyes for each captured image. The retouch processing unit 452 performs a plurality of types of image processing other than eye size adjustment on each captured image.

In step S63, the area detection unit 451 detects the area of the user's eyes in the captured image.

In step S64, the eyelash processing unit 454 performs eyelash processing, which is image processing for emphasizing the eyelashes appearing in the captured image. Details of the eyelash processing will be described later with reference to the flowchart of FIG.

In step S65, the reduction unit 455 reduces the captured image supplied from the eyelash processing unit 454 to generate a captured image for editing.

In step S66, the editing image generating unit 432 determines whether or not all the captured images have been processed. If it is determined that there is a captured image that has not been processed yet, the processing from step S61 onward is repeatedly executed. do.

If it is determined in step S66 that the processing of all captured images has been completed, the process returns to step S42 in FIG. 27 and the subsequent processing is performed.

<Eyelash treatment>
Next, the eyelash processing performed in step S64 of FIG. 30 will be described with reference to the flowchart of FIG.

In step S<b>81 , the eyelash processing unit 454 determines whether or not the captured image to be processed includes a woman, based on the gender determination result of the gender determination unit 453 .

As described above, eyelash processing, which is image processing for enhancing eyelashes, is performed only for women, not for men. If it is determined in step S81 that no female image is captured in the captured image to be processed, that is, that all the users captured in the captured image are male, the processing after step S82 in FIG. 31 is skipped. be.

This makes it possible to prevent image processing that emphasizes the eyelashes of men.

At least part of the processing after step S82 may be performed for men.

If it is determined in step S81 that the photographed image to be processed includes a woman, the process proceeds to step S82. If a plurality of women are photographed, the process from step S82 onwards is performed while paying attention to each woman.

In step S82, the upside-down eyelash removal unit 461 detects abnormal pixels included in the white eye region of the eye region, and removes the upside-down eyelashes by interpolating with peripheral pixels.

In step S83, the deformation unit 462 deforms the eyelash image according to the contour shape of the upper side of the eye as described with reference to FIG.

In step S84, the synthesizing unit 463 synthesizes the eyelash image transformed by the transforming unit 462 along the upper contour of the eye of the woman to be processed, which appears in the captured image.

Here, if both eyes of the woman to be processed are open, eyelash images are synthesized for both eyes. For example, if the degree of opening of the eyes is lower than the threshold (when the eyes are closed), the process of emphasizing the eyelashes, including the synthesis of the eyelash images, is not performed for the closed eyes. good too.

If eyelashes are emphasized for closed eyes with a wink or the like, it may look unnatural, but this can be prevented.

In step S85, the eyelash enhancement processing unit 464 enhances the eyelashes by image processing. For example, the eyelashes are enhanced by detecting the upper and lower eyelashes and thickening or darkening them one by one. The process of thickening or darkening each eyelash is also applied to each eyelash constituting the eyelash image synthesized as the upper eyelashes.

That is, in this example, the upper eyelashes are subjected to a process of thickening each eyelash, in addition to the synthesis of the eyelash images. Also, the upper eyelashes and the lower eyelashes are emphasized by different processing.

After that, the process returns to step S64 in FIG. 30, and the subsequent processes are performed.

As described above, the eyelashes of the woman in the photographed image are emphasized not by synthesizing the eyelashes image prepared as the template as they are, but by synthesizing the eyelashes image deformed according to the shape of the contour of the eyes. done.

The shape of each user's eyes differs depending on the facial expression at the time of photographing. By synthesizing the eyelash images deformed according to the shape of each user's eyes, it is possible to emphasize the eyelashes in a more natural manner and make the eyes appear larger.

In addition, by emphasizing the upper eyelashes and the lower eyelashes one by one after synthesizing the eyelash images, the eyelashes synthesized as the eyelash image can be blended with the user's own eyelashes.

<Makeup tool function>
Eye Color Tool Editing using the makeup tool provided on the editing screen will now be described.

FIG. 32 is an enlarged view of the edit palette 503R of FIG. 29 during selection of the makeup tool.

Selection buttons 511-1 to 511-5 used for selecting makeup tools are displayed side by side on the editing palette 503R.

The selection button 511-1 is a button that is pressed when selecting a small face tool for applying makeup to make the face look smaller.

The selection button 511-2 is a button that is pressed when selecting an eye color tool for changing the color of the iris. The user can apply makeup as if wearing colored contact lenses by using the eye color tool. In the example of FIG. 32, the selection button 511-2 is pressed to select the eye color tool.

The selection button 511-3 is a button that is pressed when selecting an edge density adjustment tool for adjusting the density of the eye outline.

The selection button 511-4 is a button that is pressed when selecting a lip tool for changing the lip color. The select button 511-5 is a button that is pressed when selecting a cheek tool that changes the color of cheeks.

When the eye color tool is selected, face image area 512 is displayed on the left side of edit palette 503R, as shown in FIG. A face image area 512 is a display area for an image of the user's face that appears in the captured image and is cut out by performing face recognition.

The iris processing unit 433A (FIG. 19) performs iris processing, which is processing for changing the color of iris, for the user whose face is displayed in the face image area 512. FIG. In the example of FIG. 32, the user on the right side of the two users appearing in the photographed image displayed in the editing area 501R in FIG. 29 is displayed as the target user of the eye color tool.

Below the face image area 512 is displayed a switch button 513 that is pressed when switching the target user of the eye color tool.

On the right side of the face image area 512, color selection buttons 514-1 to 514-4, which are buttons used for eye color selection, are displayed. Color selection buttons 514-1 to 514-4, which are circular buttons, are assigned colors of brown, chocolate, gray, and black, respectively.

For example, when the color selection button 514-1 is pressed, the color of the iris area of the user selected as the target of the eye color tool among the users appearing in the photographed image displayed in the editing area 501R is changed to brown. Be changed.

When the color selection button 514-2 is pressed, the color of the iris area of the user selected as the target of the eye color tool among the users appearing in the photographed image displayed in the editing area 501R is changed to chocolate. is changed to

FIG. 33 is a diagram showing an example of an image generated by an eye color tool.

When the photographed image P101 is selected as an object to be edited and the user on the right side is selected to be an object of the eye color tool, the color selection buttons 514-1 to 514-4 are pressed to change the color of the editing area 501R. The photographed images are switched to the photographed images P101-1 to P101-4 in which the color of the iris area of the user on the right side surrounded by the frame F is changed.

Although the same images are shown in the example of FIG. 33, the captured image P101-1 is an image in which the color of the black eye region of the user on the right side is changed to brown, and the captured image P101-2 is an image on the right side. This is an image in which the color of the iris region of the user is changed to chocolate. The photographed image P101-3 is an image in which the color of the iris region of the user on the right is changed to gray, and the photographed image P101-4 is an image in which the color of the iris region of the user on the right is changed to black. be.

Each time the switching button 513 is pressed, the user targeted for the eye color tool is switched in the order of the user on the left, the user on the right, and two users.

For example, when the user targeted by the eye color tool is selected as the user on the left side, the photographed image of the editing area 501R is changed in response to pressing of the color selection buttons 514-1 to 514-4. , is switched to a captured image in which the color of the iris region of the user on the left is changed.

Further, when the users targeted by the eye color tool are all selected as two users, when the color selection buttons 514-1 to 514-4 are pressed, the editing area 501R The photographed images are switched to photographed images in which the colors of the iris areas of the left and right users are changed.

Thus, by using the eye color tool, the user can change the color of the black eye for each photographed image and for each user appearing in the photographed image. A user can also change the eye color of two users at the same time.

At the lower right of the color selection button 514-4 in FIG. 32, an off button 515 is displayed to be pressed when turning off the eye color tool function. When the OFF button 515 is pressed, a captured image without black-eye processing is displayed in the editing area 501R.

FIG. 34 is a diagram showing an example of a mask image used for image processing when an eye color tool is selected.

When generating a photographed image for editing, a mask image representing each region of the eye as shown in FIG. 34B is generated based on the photographed image P101 shown in FIG. 34A. As described above, when generating a shot image for editing, the area detection unit 451 (FIG. 20) of the image generation unit 432 for editing detects each area of the eye using a prediction model generated by machine learning. A mask image M representing is generated.

In the example of FIG. 34B, among the pixels forming the mask image M, the pixels representing the outlines of the eyes, the outlines of the irises of the eyes, and the eyelashes of the two users are represented by white pixels. Using such a mask image M, black-eye processing is performed by the black-eye processing unit 433A.

FIG. 35 is a diagram showing an example of changing the color of the iris area.

When changing the color of the iris with the eye color tool is selected, the iris processor 433A uses the mask image M to extract the iris area of the user targeted for the eye color tool. In the example in the upper part of FIG. 35, the black eye area A11 _R of the right eye E _R and the black eye area A11 _L of the left eye _EL are extracted based on the mask image M.

Further, the iris processor 433A changes the color of the pixels forming the iris regions A11 _R and A11 _L , which are the extracted iris regions, to a color selected by the user from among brown, chocolate, gray, and black. The hatched black eye areas A11 _R and A11 _L in the lower part of FIG. 35 indicate that the colors of the pixels of the black eye areas A11 _R and A11 _L have been changed.

The change in color of the pixels that make up the iris region is done, for example, by changing the hue of the original color by an amount that depends on the selected color. For each color of brown, chocolate, gray, and black, the manner of hue change, including the degree (amount) of change, is set in advance.

In other words, instead of changing to the selected color, the color is changed by adjusting the reference color according to the selected color with reference to the original black eye color.

Instead of hue, the color of the black eye area may be changed by changing other parameters such as lightness, saturation, etc., or by changing a combination of parameters.

The color may be changed by performing filter processing on each pixel constituting the iris region or by adjusting the brightness using a predetermined tone curve.

In this way, the color of the iris area is changed by changing the color of the pixels of the photographic image rather than synthesizing a substantially circular image having the same shape as the iris area on the photographed image. done. Also, based on the original color which is slightly different for each person, the color of the iris area is changed by changing the original color.

Even if the same color is selected from brown, chocolate, gray, and black, the color will be slightly different for each person, so it is possible to change the impression of the black eye area in a state that gives a natural look without discomfort. becomes.

Assuming that the color of the iris region is changed by synthesizing a circular image of a predetermined color with the iris region, depending on the person, the color of the iris region may change significantly. Although the difference in appearance depending on how the light hits may be lost, it is possible to prevent such an unnatural color change.

• Contour Density Adjustment Tool Next, the adjustment of the density of the contour of the eye performed using the contour density adjustment tool will be described.

When the selection button 511-3 is pressed, the editing palette 503R displays, for example, a slide bar used for adjusting the density of the outline of the eyes. A face image area 512 is displayed on the left side of the edit palette 503R.

For the user whose face is displayed in the face image area 512, the contour density adjustment process for changing the density of the contour of the eyes is performed by the contour adjustment unit 433B (FIG. 19). The contour density adjustment process is also performed using the mask image M generated by the area detection unit 451 .

FIG. 36 is a diagram showing an example of adjusting the density of the outline of the eye.

When darkening the eye contour is selected by the contour density adjustment tool, the contour adjustment unit 433B uses the mask image M to extract the contour of the eye, the contour of the iris of the eye, and the eyelashes.

In addition, the contour adjustment unit 433B adjusts the color of the pixels so that each of the extracted portions becomes darker. For example, the contour of the eye, the contour of the iris of the eye, and the eyelashes are adjusted by thickening or darkening the respective extracted portions. In the lower example of FIG. 36, the outline of the eye, the outline of the iris of the eye (left and right edges), and the eyelashes are shown in a darker state than in the upper state of FIG.

FIG. 37 is a diagram showing another example of adjusting the density of the outline of the eye.

When lightening the eye contour is selected by the contour density adjustment tool, the contour adjustment unit 433B uses the mask image M to extract the contour of the eye, the contour of the iris of the eye, and the eyelashes.

In addition, the contour adjustment unit 433B adjusts the color of the pixels so that each extracted portion becomes lighter. For example, the eye contour, eye iris contour, and eyelashes are adjusted by thinning or lightening the respective extracted portions.

In this manner, the eye contour adjustment is performed based on the mask image M by adjusting the eye contour, the iris contour of the eye, and the eyelashes as a unit. A plurality of slide bars may be displayed so that the outline of the eye, the outline of the iris of the eye, and the density of the eyelashes can be adjusted.

Such edge density adjustment processing is performed by selecting a target user of the edge density adjustment tool.

For example, as described with reference to FIG. 32, when the target user of the contour density adjustment tool is selected as the user on the right side, the edit area 501R is displayed in response to the operation of the slide bar. is switched to a photographed image in which the density of the contour of the user's eye on the right side is changed.

Similarly, when the target user of the contour density adjustment tool is selected as the user who appears on the left side, the photographed image of the editing area 501R is changed to that of the user on the left side in response to the operation of the slide bar. The image is switched to a photographed image in which the density of the outline of the eyes has been changed.

Further, when two users are selected as targets of the contour density adjustment tool, the photographed images of the editing area 501R are changed to two users, one on the left side and the other on the right side, in response to the operation of the slide bar. is switched to a photographed image in which the density of the outline of the user's eyes has been changed.

Thus, by using the contour density adjustment tool, the user can change the density of the contour of the eyes for each photographed image and for each user appearing in the photographed image. . The user can also change the darkness of the eye contours of two users at the same time.

By using the eye color tool and the contour density adjustment tool together, the user can individually adjust the color of the black eye for each user and adjust the density of the contour of the eyes of two people collectively. becomes possible. In addition, it is possible for two users to collectively adjust the color of black eyes, and to individually adjust the density of the contours of the eyes for each user.

<Modification>
Examples of Other Makeup Tools Processing using the lip tools and cheek tools prepared as makeup tools may be performed in the same manner as the eye color tools.

For example, when the selection button 511-4 is pressed to select the lip tool, the user's lip color selected as the target of the lip tool is changed to the color selected using the button displayed on the edit palette 503R. changed accordingly. The color of the pixels forming the lip region extracted using the mask image is changed based on the original lip color, and the reference color is adjusted according to the selected color. .

・Switching of processing according to the output destination The iris area whose color is changed by the eye color tool is a narrow area with respect to the entire image. If a plurality of images in which the color of the iris area is changed are printed side by side on sticker paper, the iris reflected in each image is small, and depending on the layout, the difference in the color of the iris area may not be expressed. In addition, the entire iris may appear to have an unnatural color.

A change in the color of the iris area by the eye color tool may be reflected in the image displayed on the edit screen, but not reflected in the image printed on the sticker paper. In this case, processing for generating an image to be displayed on the editing screen and processing for generating an image to be printed on sticker paper are performed for each captured image in accordance with the user's operation. become.

That is, when the photographed image P101 is selected as an object to be edited, when the user instructs to change the color of the iris region using the eye color tool, the photographed image P101 with the changed iris region color is displayed on the editing screen. In addition to being generated as an image to be displayed, a photographed image P101 in which the color of the iris region is not changed is generated as an image to be printed on sticker paper.

As a result, it is possible to generate an appropriate image according to the output destination of the image, and to prevent an unnatural-looking image from being printed on sticker paper.

Note that the photographed image P101 displayed on the edit screen with the color of the iris area changed is edited using various editing tools according to the user's operation. An image that has been edited and the color of the iris area has not been changed is printed on the sticker paper.

Although the image output destination is the editing screen (tablet built-in monitor 131) or sticker paper, the process may be switched according to other output destinations. The output destination includes not only the configuration provided in the photo sticker creating apparatus 1 such as the touch panel monitor 92, but also the monitor of the user's smart phone and the image management server.

Although the case where the print medium of the image is sticker paper has been described, various media such as printing paper, resin cards, and cloth may be used instead of sticker paper.

- Program The series of processes described above can be executed by hardware or by software. When a series of processes is executed by software, a program that constitutes the software is installed in a computer built into dedicated hardware or a general-purpose personal computer.

The program to be installed is provided by being recorded on removable media 319 shown in FIG. Alternatively, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in the ROM 312 or storage unit 316 .

The program executed by the computer may be a program that is processed in chronological order according to the order described in this specification, or may be executed in parallel or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

The effects described herein are only examples and are not limiting, and other effects may also occur.

Embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

For example, the present technology can take a configuration of cloud computing in which a single function is shared by a plurality of devices via a network and processed jointly.

In addition, each step described in the flowchart above can be executed by a single device, or can be shared and executed by a plurality of devices.

Furthermore, when one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared by a plurality of devices.

1 photo sticker creation device, 66 mirror surface unit, 82 upper strobe unit, 91 camera, 92 touch panel monitor, 131 tablet built-in monitor, 301 PC unit, 302 preselection unit, 304 photographing unit, 305 editing unit, 306 printing unit, 401 advance selection processing unit, 402 shooting processing unit, 403 editing processing unit, 404 printing processing unit, 411 shooting control unit, 412 image processing unit, 413 display control unit, 414 image storage unit, 431 selection reception unit, 432 editing image generation unit , 433 edit control unit, 434 display control unit, 435 print data generation unit, 436 communication control unit, 451 area detection unit, 452 retouch processing unit, 453 gender determination unit, 454 eyelash processing unit, 455 reduction unit

Claims (10)

a detection unit that detects the iris of the user in a photographed image obtained by photographing the user as a subject;
and an image processing unit that changes the color of the iris and the darkness of the edge of the iris according to the user's operation. The detection unit detects the iris of each of the plurality of users appearing in the captured image,
The image processing device according to claim 1, wherein the image processing unit changes the color of the iris and the darkness of the edge of the iris according to an operation performed by selecting the target user. a display control unit for displaying the captured image processed by the image processing unit on an editing screen when editing the captured image;
an editing processing unit that edits the captured image in accordance with the user's operation on the editing screen;
3. The image processing apparatus according to claim 1, further comprising: a printing unit that prints the photographed image that has been edited by the editing processing unit and has not been processed by the image processing unit on a printing medium. The image processing device according to any one of claims 1 to 3, wherein the image processing section changes the color of the iris by changing the pixel values of the pixels forming the iris. 5. The image processing device according to claim 4, wherein the image processing unit changes the pixel values of the pixels forming the iris by an amount preset according to the color selected by the user. The detection unit detects each part of the eye including the iris and eyelashes,
6. The image processing device according to any one of claims 1 to 5, wherein the image processing unit changes the darkness of the edge of the iris, the darkness of the eyelashes, and the darkness of the contour of the eye according to the user's operation. . The image processing unit changes the color of the iris of one of the users selected by the user from among the plurality of users appearing in the photographed image, and changes the iris of the irises of all the users. The image processing apparatus according to claim 1, wherein the densities of are collectively changed. The image processing unit changes the darkness of the iris of the eyes of one user selected by the user from among the plurality of users appearing in the photographed image, and The image processing device according to claim 1, wherein the color of the pupil is changed collectively. The image processing device
Detecting the iris of the user in a photographed image obtained by photographing the user as a subject,
An image processing method for changing the color of the iris and the darkness of the edge of the iris according to the user's operation. to the computer,
Detecting the iris of the user in a photographed image obtained by photographing the user as a subject,
A program for executing processing for changing the color of the iris and the darkness of the edge of the iris according to the user's operation.

Images