JP2023164461A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

To provide an image processing apparatus which can be applied to a photo generating game machine for providing a series of games printing a captured image on a seal paper and which can more naturally changes impression of eyes.SOLUTION: An image processing apparatus (photo seal creating apparatus has an edition processing unit 403 which has an iris processing unit 433A that detects an iris of a person in a captured image and can further change thickness of an edge of the iris in response to an operation by a user. The iris processing unit 433A can further change color of the iris.SELECTED DRAWING: Figure 19

Description

The present technology particularly relates to an image processing device, an image processing method, and a program that can change the impression of the eyes in a more natural manner.

Conventionally, photo sticker producing devices are known. A photo sticker creation device photographs a user, allows the user to edit the photographed image, and prints and provides the edited image on sticker paper. Photo sticker making devices are installed at amusement facilities and the like.

The flow of one game provided by the photo sticker creation device is that after taking a picture of a user in the shooting space as the subject, the user is moved to the editing space, and the image is edited according to the operations performed in the editing space. After editing, the edited image is printed on sticker paper. When photographing in a photographic space, a strobe light is irradiated to capture a catchlight in the eyelids of the user's eyes.

For example, Patent Document 1 discloses a technique in which the iris of a user's eyes in a photographed image is filled in with a predetermined color, and an image representing a catch light is drawn on the iris of the user in a different color.

Japanese Patent Application Publication No. 2014-182672 Japanese Patent Application Publication No. 2013-055372

When the color of the iris is changed by image processing to change the impression of the eyes, the iris may look unnatural after the change.

The present technology was developed in view of this situation, and is intended to make it possible to change the impression of the eyes in a more natural manner.

An image processing device according to one aspect of the present technology includes a detection unit that detects the iris of a person in a photographed image, and an image processing unit that changes the darkness of the iris of the iris according to an operation by a user.

In one aspect of the present technology, the iris of a person in a photographed image is detected, and the darkness of the rim of the iris is changed in accordance with a user's operation.

According to this technology, it is possible to change the impression of the eyes in a more natural manner.

FIG. 1 is a perspective view showing an example of the external configuration of a photo sticker creation device of the present technology. FIG. 2 is a perspective view of the appearance of the photo sticker creation device seen from another angle. It is a figure explaining movement of a user. It is a figure showing an example of composition of a pre-selection part. FIG. 3 is a perspective view showing an example of the configuration of an imaging section. FIG. 3 is a perspective view showing an example of the configuration of an imaging unit. It is a figure which shows the example of angle adjustment of a camera. FIG. 3 is a diagram illustrating a configuration example of a background part. It is a figure showing an example of composition of an operation part. FIG. 3 is a diagram illustrating an example of the configuration of the front side of the editing unit. It is a figure which shows the example of attachment of a curtain. It is a figure which shows the example of a design of the back side of a curtain. It is a figure which shows the example of use of a mirror surface part. FIG. 3 is a diagram illustrating an example of a side configuration of an editing unit. FIG. 3 is a diagram showing an example of a layer structure of an edited image. FIG. 1 is a block diagram showing an example of the configuration of a photo sticker creation device. FIG. 2 is a block diagram showing an example of a functional configuration of a photo sticker creation device. FIG. 18 is a block diagram showing an example of the configuration of the imaging processing section in FIG. 17; 18 is a block diagram showing a configuration example of an editing processing section in FIG. 17. FIG. FIG. 20 is a block diagram showing a configuration example of an editing image generation section in FIG. 19. FIG. FIG. 3 is a diagram illustrating an example of area detection. 21 is a block diagram showing a configuration example of an eyelash processing section in FIG. 20. FIG. It is a figure which shows the example of the removal of upside-down eyelashes. It is a figure which shows the example of deformation|transformation of an eyelash image. FIG. 3 is a diagram showing an example of combining eyelash images. It is a flowchart explaining the operation of the photo sticker creation device. 27 is a flowchart illustrating the editing process performed in step S3 of FIG. 26. FIG. FIG. 6 is a diagram illustrating an example of a guide screen for self-portrait before doodling. FIG. 3 is a diagram showing an example of an editing screen. 28 is a flowchart illustrating the editing image generation process performed in step S42 of FIG. 27. FIG. 31 is a flowchart illustrating eyelash processing performed in step S64 of FIG. 30. FIG. FIG. 7 is an enlarged diagram showing an editing palette while a makeup tool is being selected. FIG. 3 is a diagram illustrating an example of an image generated by an eye color tool. FIG. 7 is a diagram showing an example of a mask image used in image processing when selecting an eye color tool. FIG. 7 is a diagram illustrating an example of changing the color of the iris area. FIG. 7 is a diagram illustrating an example of adjusting the darkness of the eye outline. FIG. 7 is a diagram showing another example of adjusting the darkness of the eye outline.

<Exterior of photo sticker making device>
FIGS. 1 and 2 are perspective views showing an example of the external configuration of the photo sticker producing device 1. FIG.

The photo sticker creation device 1 is a game machine that provides photographed images and edited images. The photo sticker creation device 1 provides images to the user by printing the images on sticker paper or making the images viewable on the user's mobile terminal. The photo sticker creation device 1 is installed in an amusement facility such as a game center or a store.

Users of the photo sticker creation device 1 are mainly high school girls and young women. In the photo sticker creation device 1, a plurality of users, mainly two or three users per group, can enjoy the game. Of course, one user can also enjoy the game in the photo sticker creation device 1.

In the photo sticker creation device 1, a user performs a photographing operation with himself/herself as a subject. Through editing work, the user combines images for synthesis, such as handwritten characters and stamp images, with an image selected from among the photographed images obtained by photographing. As a result, the photographed image is edited into a colorful image. The user receives the sticker sheet on which the edited image, which is the edited image, is printed, and completes the series of games.

The photo sticker creation device 1 can be said to be a photo creation game machine that provides a photo creation game.

As shown in FIG. 1, the photo sticker creation device 1 is basically constructed by installing a photographing unit 11 and an editing unit 12 in contact with each other. The photographing unit 11 and the editing unit 12 are electrically connected.

The photographing unit 11 has a box-shaped housing large enough for a person to enter, and is composed of a pre-selection section 20, a photographing section 21, and a background section 22. The pre-selection unit 20 is installed on the side of the imaging unit 21. The space in front of the pre-selection unit 20 becomes a pre-selection space where the pre-selection process is performed. Further, the photographing section 21 and the background section 22 are installed apart from each other by a predetermined distance. The space formed between the photographing section 21 and the background section 22 becomes a photographing space in which photographing processing is performed.

As a pre-selection process, the pre-selection unit 20 provides guidance introducing games provided by the photo sticker creation device 1, and performs various settings in the photo-taking process performed in the photo-taking space. The advance selection section 20 is provided with a coin slot into which the user inserts money, a touch panel monitor used for various operations, and the like. The pre-selection unit 20 guides users in the pre-selected space to the shooting space depending on the availability of the shooting space.

The photographing unit 21 is a device for photographing a user as a subject. The photographing unit 21 is located in front of the user who has entered the photographing space. A camera, a touch panel monitor used for various operations, and the like are provided at the front of the photographing section 21 facing the photographing space.

Assuming that the left side is the left side and the right side is the right side when viewed from the user facing forward in the shooting space, the left side of the shooting unit 21 is constituted by the side panel 41A, and the right side is constituted by the side panel 41B. be done. Further, the front of the photographing section 21 is constituted by a front panel 42. The above-mentioned pre-selection section 20 is installed on the side panel 41A. Note that the pre-selection section 20 may be installed on the side panel 41B.

The background section 22 includes a back panel 51, a side panel 52A, and a side panel. The back panel 51 is a plate-shaped member located on the back side of the user facing the front. The side panel 52A is a plate-shaped member that is attached to the left end of the back panel 51 and is narrower in width than the side panel 41A. The side panel 52B is a plate-shaped member that is attached to the right end of the back panel 51 and is narrower in width than the side panel 41B.

The side panel 41A and the side panel 52A are provided on substantially the same plane. The upper portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A, which is a plate-shaped member. The lower portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A', which is a metal member, for example, provided on the floor surface. Similarly, the side panel 41B and the side panel 52B are provided on substantially the same plane. The upper portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B. The lower portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B'.

The side panel 41A and the side panel 52A serve as members surrounding the left side of the photographing space, and the side panel 41B and the side panel 52B serve as members surrounding the right side of the photographing space.

A floor material 27, which is a plate-shaped member, is laid on the floor of the photographing space.

An opening formed by being surrounded by the side panel 41A, the connecting portion 23A, and the side panel 52A serves as an entrance/exit to the photographing space. Further, an opening formed by being surrounded by the side panel 41B, the connecting portion 23B, and the side panel 52B also serves as an entrance to the photographing space.

A ceiling sheet 24 made of a vinyl material or a plate-like member is provided on a part of the ceiling of the photographing space.

The editing unit 12 is a device for editing a photographed image. The editing unit 12 is connected to the photographing unit 11 so that one side surface is in contact with the front panel 42 of the photographing section 21 .

If the configuration of the editing unit 12 shown in FIG. 2 is the configuration on the front side, the configuration used in editing work is provided on each of the front side and the back side of the editing unit 12. This configuration allows two sets of users to perform editing work at the same time.

The front side of the editing unit 12 is composed of a vertical surface 61 and a slope 62 formed above the surface 61. A monitor with a built-in tablet and a touch pen are provided on the slope 62 as components used for editing work. A columnar support portion 63A that supports one end of the lighting device 64 is provided on the right side of the slope 62. A columnar support portion 63B that supports the other end of the lighting device 64 is provided on the left side of the slope 62. A support portion 65 that supports the curtain rail 26 is provided on the upper surface of the support portion 63A.

A curtain rail 26 is attached above the editing unit 12. The curtain rail 26 is constructed by combining three rails 26A to 26C.

The three rails 26A to 26C are combined in a U-shape when viewed from above. One end of rail 26A and rail 26B provided in parallel is fixed to connecting part 23A and connecting part 23B, respectively, and the other end of rail 26A and rail 26B is fixed to both ends of rail 26C, respectively.

A curtain is attached to the curtain rail 26 to prevent the space in front of the front and the space in front of the back of the editing unit 12 from being seen from the outside. A space in front of the front and a space in front of the back of the editing unit 12 surrounded by the curtain attached to the curtain rail 26 serve as an editing space in which the user performs editing work.

A side surface of the editing unit 12 is provided with a discharge port through which printed sticker paper is discharged.
The space in front of the side of the editing unit 12 becomes a printing waiting space where the user waits for printed sticker sheets to be ejected.

<About user movement>
Here, the flow of the photo sticker creation game and the accompanying movement of the user will be explained.

FIG. 3 is a plan view of the photographic sticker making device 1 viewed from above.

The user inserts money into the coin slot in the pre-selection space A0, which is the space in front of the pre-selection section 20. Next, the user performs a pre-selection operation according to the screen displayed on the touch panel monitor. For example, the user performs a pre-selection task such as selecting a course regarding photography to be performed in the photography space.

After completing the pre-selection process, the user enters the photographing space A1 formed between the photographing section 21 and the background section 22 from the entrance G1 between the side panel 41A and the side panel 52A, as shown by the white arrow #1. to go into. The user performs photographing work using a camera, a touch panel monitor, etc. provided in the photographing section 21.

After finishing the shooting work, the user either exits the shooting space A1 through the doorway G1 and moves to the editing space A2-2, as shown by the outlined arrow #2, or exits the shooting space A1 and moves to the editing space A2-2, as shown by the outlined arrow #3. The user exits the shooting space A1 and moves to the editing space A2-1.

The editing space A2-1 is an editing space on the front side of the editing unit 12. On the other hand, the editing space A2-2 is an editing space on the rear side of the editing unit 12. The user is guided as to which space to move to, editing space A2-1 or editing space A2-2, by the screen display of the touch panel monitor of the photographing unit 21, etc. For example, the user is guided to the vacant one of the two editing spaces as the destination.

The user who has moved to editing space A2-1 or editing space A2-2 starts editing work.
A user of editing space A2-1 and a user of editing space A2-2 can perform editing work at the same time. When there is no need to distinguish between the editing space A2-1 and the editing space A2-2, they are collectively referred to as editing space A2 as appropriate.

After the editing work is completed, printing of the edited image is started. After completing the editing work in the editing space A2-1, the user moves from the editing space A2-1 to the print waiting space A3 as indicated by the outline arrow #5. Further, the user who has finished the editing work in the editing space A2-2 moves from the editing space A2-2 to the print waiting space A3 as shown by the outline arrow #4.

The user who has moved to the print waiting space A3 waits for printing to finish. When printing is completed, the user receives the sticker paper discharged from the discharge port provided on the side of the editing unit 12, and completes the series of photo sticker creation games.

Next, the configuration of each device will be explained.

<Configuration of pre-selection section>
FIG. 4 is a diagram showing an example of the configuration of the pre-selection section 20. As shown in FIG.

A touch panel monitor 71 is provided above the pre-selection section 20 . The touch panel monitor 71 includes a monitor such as an LCD (Liquid Crystal Display) and a touch panel laminated thereon. The touch panel monitor 71 has a function of displaying various GUIs (Graphical User Interfaces) and accepting selection operations from the user. The touch panel monitor 71 displays a screen used to select the number of users, and the like.

A speaker 72 is provided below the touch panel monitor 71. The speaker 72 outputs guidance audio, BGM (background music), sound effects, etc. for the pre-selection process. Next to the speaker 72, a coin insertion/return slot 73 is provided in which the user inserts coins.

<Structure of the photography department>
5 and 6 are perspective views showing an example of the configuration of the imaging section 21. FIG.

The photographing unit 21 is configured to be surrounded by a side panel 41A, a side panel 41B, and a front panel 42.

A box-shaped camera unit 81 is provided approximately at the center of the front panel 42 . A camera 91 and a touch panel monitor 92 are provided in front of the camera unit 81. A smartphone holder 93 is provided on the right side of the camera 91.

The camera 91 is, for example, a single-lens reflex camera, and is attached inside the camera unit 81 so that the lens is exposed. The camera 91 includes an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 91 functions as a photographing unit that photographs a user in the photographing space A1 as a subject.

Further, the camera 91 has a vertical swing function (angle adjustment function).

FIG. 7 is a diagram showing an example of angle adjustment of the camera 91.

As shown in FIG. 7, the direction of the optical axis of the camera 91 can be adjusted in the vertical direction by changing the posture inside the camera unit 81. In FIG. 7, the dashed arrow represents the direction of the optical axis of the lens.

As shown in FIGS. 7A to 7C, for example, the angle of the camera 91 can be adjusted within a range from an upper limit to a direction substantially horizontal to the ground plane of the photo sticker making device 1 to a predetermined downward angle as a lower limit. It is considered possible.

Such adjustment of the angle of the camera 91 is performed electrically according to the operation of the user who is taking the image, as will be described later. The user can take pictures by adjusting the angle of the camera 91 to a desired angle.

A live view image generated based on the moving image captured by the camera 91 is displayed on the touch panel monitor 92. A still image captured by the camera 91 at a predetermined timing is saved as a captured image.

A touch panel monitor 92 is provided below the camera 91. The touch panel monitor 92 is composed of a monitor such as an LCD and a touch panel laminated thereon. The touch panel monitor 92 has a function as a live view monitor that displays a live view image captured by the camera 91, and a function that displays various GUIs and receives selection operations from the user.

A smartphone holder 93 is provided on the right side of the camera 91 to serve as a place to store a smartphone. The smartphone holder 93 is formed to have a length sufficient to allow a smartphone to be placed horizontally.

The user can place his or her smartphone on the smartphone holder 93 and use the smartphone to take a video or still image of himself or herself playing the game.

An upper strobe unit 82 is installed above the camera unit 81 so as to be biased to the left with respect to the position of the camera unit 81 (the position of the camera 91). The upper strobe unit 82 is provided so as to cover substantially the entire upper and upper left side of the camera unit 81.

The front surface of the upper strobe unit 82 is composed of a panel that is tilted forward so as to face diagonally downward, and a panel that is provided below the panel and forms a substantially vertical surface. The front side of the upper strobe unit 82 is configured as a light emitting surface using a translucent milky white plate-like member. The upper strobe unit 82 emits light in accordance with the photographing by the camera 91, and mainly irradiates the user's face and upper body with strobe light from the upper left of the front of the user.

A foot strobe unit 83 is provided below the camera unit 81 to emit strobe light mainly to the user's lower body and feet.

The foot strobe unit 83 is formed without a gap between the side panel 41A and the side panel 41B so that the entire front surface, which is a horizontally long rectangle when viewed from the front, serves as a light emitting surface. The upper surface of the foot strobe unit 83 is a surface that is substantially parallel to the installation surface of the photo sticker making device 1, and is used as a luggage storage area for users who are performing photographic work to place their baggage and the like.

Inside the upper strobe unit 82 and the foot strobe unit 83, a constantly lit light bulb and strobe light emitting tube are provided. The upper strobe unit 82 and the foot strobe unit 83 constantly illuminate the shooting space by emitting light bulbs, and emit strobe light to the user, who is the subject, by emitting strobe light tubes in accordance with the shooting timing. .

Note that a speaker is provided near the ceiling of the front panel 42. The speakers output audio guidance for the shooting process, background music, sound effects, etc. A speaker may be provided in the background section 22.

FIG. 8 is a diagram showing an example of the configuration of the background section 22 on the photographing space A1 side.

A sheet of a predetermined color such as green is pasted on the inside of the back panel 51. Chromakey processing is performed by detecting the green area in the photographed image, and the area of the person is detected.

An operating section 53 is provided inside a side panel 52B attached to the left end of the back panel 51.

As shown in FIG. 9, the operation unit 53 includes an angle adjustment button 53A that is pressed when adjusting the direction of the camera 91 upward or downward, and a shooting stop button 53B that is pressed when stopping shooting. It consists of

Photography by the camera 91 is continuously performed at predetermined intervals, such as every 10 seconds. The shooting stop button 53B is pressed to temporarily stop the continuous shooting. The user can adjust the angle of the camera 91 by pressing the angle adjustment button 53A while continuous shooting is temporarily stopped.

Since the operation unit 53 is provided on the side panel 52B, the standing position of the user facing the camera 91 is unconsciously positioned slightly to the right in the photographing space A1. By ensuring a distance from the upper strobe unit 82, which is installed on the front left side diagonally to the operation unit 53 in the photographing space A1, to the standing position, the upper strobe unit 82 can illuminate the user with an appropriate amount of light. It becomes possible to irradiate.

The operation unit 53 provided on the side panel 52B also has a function of unconsciously guiding the user's standing position. The operation unit 53 may be provided on the side panel 52A side.

<Editing unit configuration>
FIG. 10 is a diagram showing a configuration example of the front side (editing space A2-1 side in FIG. 3) of the editing unit 12. A similar configuration is provided on the back side (editing space A2-2 side in FIG. 3).

A tablet built-in monitor 131 is provided approximately at the center of the slope 62. A touch pen 132A is provided on the left side of the tablet built-in monitor 131. A touch pen 132B is provided on the right side of the tablet built-in monitor 131.

For example, an editing screen used for editing work is displayed on the tablet built-in monitor 131. When two users perform editing work at the same time, the touch pen 132A is used by the user on the left side facing the tablet built-in monitor 131, and the touch pen 132B is used by the user on the right side facing the tablet built-in monitor 131. .

Note that operations using the touch pen 132A and operations using the touch pen 132B are distinguished. As appropriate, the display on the left side of the tablet built-in monitor 131 can only be operated using the touch pen 132A, and the display on the right side can only be operated using the touch pen 132B. Hereinafter, if there is no need to distinguish between the touch pen 132A and the touch pen 132B, they will be collectively referred to as the touch pen 132.

Below the lighting device 64 and between the columnar support portions 63A and 63B that support the lighting device 64, a mirror surface portion 66 made of an acrylic mirror plate or the like is provided. The mirror surface portion 66 forms a substantially vertical surface.

In FIG. 10, the range of the mirror surface portion 66 that is a mirror surface is shown in color. The width of the mirror surface portion 66 in the lateral direction is wider than the width of the tablet built-in monitor 131 in the lateral direction. The interior of the editing space A2-1, including the vicinity of the user's face in the editing space A2-1, is reflected in the mirror surface portion 66.

FIG. 11 is a diagram showing an example of installing a curtain.

As shown in FIG. 11, a curtain is attached to the curtain rail 26 above the editing unit 12 to prevent the editing space A2 from being seen from the outside. In the example of FIG. 11, a curtain 111 is attached to the rail 26C, and a curtain 112 is attached to the rail 26B on the front side of the editing unit 12. A curtain similar to the curtain 112 is also attached to the rail 26A on the rear side of the editing unit 12.

A predetermined design is applied to the surfaces of the curtains 111 and 112 made of vinyl material so that the photo sticker making device 1 stands out. Further, of the curtain 111 and the curtain 112, at least the curtain 112 has a predetermined design not only on the front surface but also on the back surface (the surface facing the editing space A2-1).

FIG. 12 is a diagram showing an example of the design of the back surface of the curtain 112.

As shown in FIG. 12, the back side of the curtain 112 is designed by printing characters and letters.

In the example of FIG. 12, a design is provided that includes an illustration of a bird as a character and an illustration of a flower. Above the curtain 112, the words "Happy!" surrounded by flowers are printed with the left and right sides reversed. The design on the back side of the curtain 112 can be arbitrary, such as printing other illustrations or photos of landscapes or people.

A user in the editing space A2-1 stands with his or her back facing a curtain 112 designed as shown in FIG. 12. The design of the curtain 112 is also reflected in the mirror surface section 66 in front of the user as the user's background.

FIG. 13 is a diagram showing an example of how the mirror surface section 66 is used.

As shown in FIG. 13, the user in the editing space A2-1 can point the camera of his/her smartphone toward the mirror section 66 and take a picture of himself/herself reflected in the mirror section 66.
In the example of FIG. 13, only one user is shown, but if two people are playing the game, two people can take pictures.

That is, the mirror section 66 is provided for the user in the editing space A2-1 to take a so-called self-portrait (self-portrait) through the mirror.

Since the back side of the curtain 112 has the above-described design, the user can take a selfie with the design on the curtain 112 as the background. Furthermore, the characters printed on the curtain 112 in a horizontally reversed state are reflected on the mirror surface portion 66 in a normal rotated state, as shown in FIG. In the image obtained by taking a self-portrait, the characters printed on the curtain 112 will appear rotated in the normal direction.

Users can upload images taken by taking selfies in editing space A2-1 to the SNS (Social Networking Service) site they are using and show them to their friends.

In this way, the photo sticker creation device 1 is a device that can not only take photos in the shooting space A1, but also take photos in the editing space A2. As will be described later, at the start of editing, the user is prompted to take a selfie. For the user who has finished shooting and moved to the editing space A2, time is secured for the user to take a selfie before editing.

Note that the design of the back side of the curtain 112 attached to the editing space A2-1 side may be different from the design of the back side of the curtain 112 attached to the editing space A2-2 side. The user can enjoy seeing which design of the curtain 112 is installed in the editing space A2.

As the design on the back side of the curtain 112, an advertisement for a product or service may be printed.

FIG. 14 is a diagram showing an example of a side configuration of the editing unit 12.

A sticker paper outlet 161 is provided on the lower side of the editing unit 12. A printer is provided inside the editing unit 12. The printer prints an image of the user who performed editing work in editing space A2-1 or an image of the user who performed editing work in editing space A2-2 on sticker paper in a predetermined layout, and creates a sticker. The paper is discharged from the paper discharge port 161.

<Layer structure of edited image>
FIG. 15 is a diagram showing an example of the layer structure of an edited image.

The leftmost edited image P1 is an image generated by editing. The edited image P1 is basically composed of images of each layer: a background image P11, a person image P12, and a foreground image P13.
Layers to be subjected to various types of image processing are also set as appropriate.

In the example of FIG. 15, the background image P11 is a monochrome image. The foreground image P13 is an image that includes an image of the handwritten words "Love my friend" and a heart-shaped image. The handwritten image of "Love my friend" is a pen image input by the user using a pen tool. Further, the heart-shaped image is a stamp image input by the user using the stamp tool.

In this way, the edited image P1 is generated by overlaying the person image P12, which is an image of a person's area extracted from the captured image by chromakey processing, on the background image P11, and then overlaying the foreground image P13 on top of the background image P11. It is also possible to input a pen image or a stamp image onto the background image P11 and display it in the background of the subject. A foreground image and a background image may be combined with a photographed image, or may not be combined depending on the user's operation even after editing.

In the example of FIG. 15, an image showing a large area from approximately the vicinity of the user's chest to above is shown as the person image P12. In the photographing process, a close-up image that shows a large face is photographed. For example, the close-up image is photographed multiple times. A full-body image showing the user's whole body may be taken.

<Internal configuration of photo sticker making device>
FIG. 16 is a block diagram showing an example of the configuration of the photo sticker creating device 1. As shown in FIG. In FIG. 16, the same components as those described above are given the same reference numerals. Duplicate explanations will be omitted as appropriate.

As shown in FIG. 16, in the photo sticker creation device 1, a pre-selection section 302, a photographing section 304, editing sections 305A, 305B, and a printing section 306 are connected to an external input/output interface 303 for a PC (Personal Computer) section 301. configured by being connected via. The PC section 301, which is an information processing section, is mounted inside the housing of the editing unit 12, for example.

A CPU (Central Processing Unit) 311, a ROM (Read Only Memory) 312, and a RAM (Random Access Memory) 313 that constitute the PC section 301 are interconnected by a bus 314.

The CPU 311 executes a predetermined program and controls the overall operation of the photo sticker creation device 1. The ROM 312 stores programs and data executed by the CPU 311. The RAM 312 temporarily stores data and programs processed by the CPU 311.

An input/output interface 315 is further connected to the bus 314 . A storage section 316 , a communication section 317 , and a drive 318 are connected to the input/output interface 315 .

The storage unit 316 is a nonvolatile storage medium such as a hard disk or flash memory. The storage unit 316 stores various setting information and the like supplied from the CPU 311. Information stored in the storage unit 316 is read out by the CPU 311 as appropriate.

The communication unit 317 is an interface for a network such as the Internet. The communication unit 317 communicates with an external device under the control of the CPU 311. The communication unit 317 transmits the image selected by the user to an image management server managed by the manufacturer of the photo sticker creation device 1, for example. The image transmitted from the communication unit 317 is allocated a predetermined storage area and stored, and is displayed on the mobile terminal that has accessed the image management server or downloaded to the mobile terminal.

A removable medium 319 such as an optical disk or a semiconductor memory is appropriately installed in the drive 318 . Programs and data read from the removable medium 319 by the drive 318 are supplied to the CPU 311 and stored or installed in the storage unit 316.

The external input/output interface 303 is connected to the input/output interface 315 . Control of each section by the PC section 301 is performed via an external input/output interface 303.

The pre-selection unit 302 performs pre-selection processing targeting users in the pre-selection space A0. The pre-selection section 302 includes a touch panel monitor 71, a speaker 72, and a coin processing section 321.

The touch panel monitor 71 displays various selection screens under the control of the PC unit 301 and receives user operations on the selection screens. An input signal representing the content of the user's operation is supplied to the PC section 301, and various settings are performed.

The coin processing unit 321 detects the insertion of a coin into the coin insertion/return slot 73 . When the coin processing unit 321 detects that a predetermined amount of coins have been inserted, it outputs an activation signal to the PC unit 301 instructing the start of the game.

The photographing unit 304 performs photographing processing for the users in the photographing space A1. The photographing section 304 includes an operation section 53, a camera 91, a camera drive section 91A, a touch panel monitor 92, a lighting device 331, and a speaker 332.

The operation unit 53 outputs a signal representing the user's operation to the PC unit 301 in response to the user's operation on the angle adjustment button 53A and the shooting stop button 53B.

The camera 91 performs photography according to the shutter control by the PC unit 301 and outputs the photographed image (image data) obtained by photography to the PC unit 301.

The camera drive unit 91A adjusts the vertical angle of the camera 91 under the control of the PC unit 301.

The lighting device 331 is each strobe unit in the photographing space A1, and emits light in accordance with a lighting control signal supplied from the PC section 301. The lighting device 331 includes an upper strobe unit 82 and a foot strobe unit 83.

The speakers 332 are configured by speakers provided at each position in the imaging space A1.

The editing unit 305A performs editing processing for users in the editing space A2-1. The editing section 305A includes a tablet built-in monitor 131, touch pens 132A and 132B, and a speaker 341. The editing section 305B performs editing processing for users in the editing space A2-2, and has the same configuration as the editing section 305A. Hereinafter, the editing departments 305A and 305B will be collectively referred to as the editing department 305 as appropriate.

The tablet built-in monitor 131 displays an editing screen under the control of the PC unit 301 and receives user operations on the editing screen. An input signal representing the content of the user's operation is supplied to the PC section 301, and the photographed image to be edited is edited.

The printing unit 306 is configured to include a printer 351. A sticker paper unit 352 is attached to the printer 351.

The printer 351 drives a built-in head 361 and prints an image on a roll of sticker paper 363 stored in the sticker paper unit 352 based on print data supplied from the PC section 301 . Further, the printer 351 cuts the sticker paper 363 with the image printed thereon at a predetermined cutting position using the cutter 362, and discharges the sticker paper 363 to the sticker paper discharge port 161.

<Example of functional configuration of photo sticker creation device>
FIG. 17 is a block diagram showing an example of the functional configuration of the photo sticker creating device 1. As shown in FIG. At least some of the functional units shown in FIG. 17 are realized by the CPU 311 executing a predetermined program.

In the photo sticker creation device 1, a pre-selection processing section 401, a photographing processing section 402, an editing processing section 403, and a printing processing section 404 are implemented.

The preselection processing unit 401 executes a preselection process by controlling each part of the preselection unit 302. The pre-selection process selects the number of users, etc. Information representing the selection made by the user is supplied to the photographing processing unit 402.

The photographing processing unit 402 performs photographing processing by controlling each part of the photographing unit 304. The photographing processing unit 402 outputs data of a photographed image obtained by performing photographing processing to the editing processing unit 403. Data of a plurality of photographed images obtained by performing photographing a plurality of times is supplied from the photographing processing section 402 to the editing processing section 403.

The editing processing section 403 performs editing processing by controlling each section of the editing section 305. The editing process includes image processing for adjusting the size of the user's eyes in the photographed image, image processing for editing the photographed image, and the like.

After editing the photographed image, the editing processing unit 403 displays various screens on the tablet built-in monitor 131, such as a screen used for selecting the number of divisions and a screen used for selecting the image to be sent to the image management server. Edit processing unit 403 generates print data by arranging each edited image according to the number of divisions selected by the user, and outputs it to print processing unit 404 . Further, the editing processing unit 403 transmits the edited image selected by the user to the image management server.

The print processing unit 404 outputs print data to the printer 351 and prints the edited image.

<Detailed configuration of shooting processing section>
FIG. 18 is a block diagram showing a configuration example of the imaging processing section 402.

The photographing processing section 402 includes a photographing control section 411, an image processing section 412, a display control section 413, and an image storage section 414.

The photographing control unit 411 photographs a moving image by controlling the camera 91, and outputs the image obtained by photographing to the image processing unit 412. The moving image output from the shooting control unit 411 is used to generate a live view image.

Further, the photographing control unit 411 photographs a still image in accordance with the countdown, and outputs the still image obtained by photographing to the image processing unit 412. The still image output from the photographing control unit 411 is used to generate a photographed image.

The image processing unit 412 cuts out a rectangular area by trimming from each frame of the moving image supplied from the shooting control unit 411, and generates a moving image made of the cut out images. The moving image generated by the image processing unit 412 is used as a live view image displayed on the live view screen.

The image processing unit 412 also generates a captured image with a predetermined angle of view by trimming a rectangular area from the still image supplied from the imaging control unit 411. The photographed image generated by the image processing section 412 is supplied to the image storage section 414 and stored therein.

The display control unit 413 controls the display of various screens on the touch panel monitor 92, such as a live view screen.

The image storage unit 414 stores the photographed image generated by the image processing unit 412. When the photographing process is completed, the photographed image stored in the image storage section 414 is supplied to the editing processing section 403.

<Detailed configuration of editing processing section>
FIG. 19 is a block diagram showing a configuration example of the editing processing unit 403 of FIG. 17.

The editing processing section 403 includes a selection reception section 431, an editing image generation section 432, an editing control section 433, a display control section 434, a print data generation section 435, and a communication control section 436. The editing control section 433 includes a pupil processing section 433A and a contour adjustment section 433B.

The selection accepting unit 431 accepts a user's operation on the screen displayed on the tablet built-in monitor 131. Information representing the content of the user's operation accepted by the selection reception unit 431 is supplied to the editing control unit 433 and the display control unit 434.

The editing image generation unit 432 generates a shot image for editing based on the shot image supplied from the shooting processing unit 402. For example, each image after trimming by the image processing unit 412 (FIG. 18) is used as an original image, and an image obtained by performing various processes such as retouching on the original image becomes a photographed image for editing. The shot image for editing generated by the editing image generation section 432 is supplied to the editing control section 433 and the display control section 434.

The editing control section 433 edits the shot image for editing supplied from the editing image generation section 432 based on the information output from the selection receiving section 431. Editing is image processing that combines images for composition, such as handwritten characters or stamp images, with a photographed image to be edited.

While the retouching process performed by the editing image generation unit 432 is a process of performing image processing on each of a plurality of captured images at once in accordance with the user's selection, editing is based on the user's selection. This is a process in which image processing is applied to a single photographed image that is currently being edited, depending on the content. During retouch processing, it may be possible to select the eye size and the like for each photographed image.

The editing control unit 433 outputs information representing the editing result to the display control unit 434, and causes the information to be reflected on the display of the editing screen.

When the editing is completed, the editing control unit 433 outputs an edited image, which is a photographed image obtained by performing the editing, to the print data generation unit 435 and the communication control unit 436.

The iris processing unit 433A of the editing control unit 433 performs iris processing, which is a process of changing the color of the iris of each user's eye in the photographed image, in response to the user's operation on the editing screen.

The contour adjustment unit 433B of the editing control unit 433 performs image processing according to the user's operation on the editing screen, and emphasizes a specific part of the user that appears in the photographed image for editing. For example, the contour adjustment unit 433B performs image processing to emphasize the entire eye or a part of the eye such as the iris.

The display control unit 434 displays an operation screen used for operations by the user on the tablet built-in monitor 131, and controls the display of the operation screen according to the user's operation. The display control unit 434 is supplied with the captured image output from the editing image generation unit 432, the edited image output from the editing control unit 433, and the like. Various images supplied to the display control unit 434 are used to display the operation screen.

The print data generation unit 435 generates print data by arranging the edited images supplied from the editing control unit 433 in a predetermined layout. The print data generation unit 435 outputs the print data to the print processing unit 404 and causes the print data to be printed on the sticker paper 363.

The communication control unit 436 controls the communication unit 317 and transmits the edited image supplied from the editing control unit 433 to the image management server.

FIG. 20 is a block diagram showing a configuration example of the editing image generation section 432.

The editing image generation section 432 includes an area detection section 451, a retouch processing section 452, a gender determination section 453, an eyelash processing section 454, and a reduction section 455.

The photographed image supplied from the photographing processing section 402 is input to the region detecting section 451, the retouching processing section 452, and the gender determining section 453. Processing by each section of the editing image generation section 432 is performed on each captured image.

The area detection unit 451 detects the area of the user, who is a person in the photographed image, and also detects the area of each body part of the user. For example, the area detection unit 451 detects the user's face area, the eyes, eyelids, nose, lips, cheek areas, hair area, etc. as parts of the user's face.

FIG. 21 is a diagram illustrating an example of region detection.

When a captured image P101 of two women is supplied, the faces and regions of each part of the face are detected for each user. For example, for a user on the left side, each area including the left and right eye areas is detected.

When focusing on the right eye area of the user shown on the left side, as shown in an enlarged view on the left side of FIG. 21, the iris area A11 and the white eye areas A12 and A13 on the left and right sides of the iris area A11 are detected, respectively.

The white of the eye area A12 is the area of the white of the eye closer to the nose, and the white of the eye area A13 is the area of the white of the eye closer to the ear. Position p1 represents the position of the inner corner of the eye, and position p2 represents the position of the outer corner of the eye. Catchlights corresponding to the shapes and sizes of the upper strobe unit 82 and the foot strobe unit 83 are reflected in the iris area A11 due to the light emission of the upper strobe unit 82 and the foot strobe unit 83.

Detection of the iris area and the white area of the eye is performed based on a prediction model configured by, for example, a neural network. For the region detection unit 451, a predictive model for detecting each region of the eye is prepared in advance by machine learning.

The prediction model prepared in the area detection unit 451 is, for example, a model that inputs a photographed image and outputs a mask image of each area of the eye and a mask image of the skin area around the eyes. The mask image output from the prediction model represents the position, size, shape, etc. of each area of the iris and white of the eye.

The detection of the iris region and the white eye region may be performed by matching based on the feature amount of each region extracted by analyzing the image, instead of using a prediction model.
In this case, the area detecting unit 451 is provided with data representing the feature amount of each area of the iris and the white of the eye.

Not only each eye region, but also the entire face and other regions of the face may be detected using a predictive model generated by machine learning. Alternatively, detection may be performed by matching based on feature amounts.

The area detection unit 451 in FIG. 20 outputs information about each area detected as described above to the retouch processing unit 452 and the eyelash processing unit 454. The information output from the area detection unit 451 represents the position, size, shape, etc. of the iris area and the white area of the left and right eyes of each user in the captured image. Information representing the detection result by the area detection section 451 is also supplied to the subsequent editing control section 433.

The retouch processing unit 452 performs retouch processing on the photographed image. Retouch processing is image processing for a specific part of the user in a photographed image, such as adjusting the outline of the face or adjusting the size of the eyes. Each region to be subjected to image processing is specified by information supplied from the region detection section 451. The photographed image subjected to the retouching process is supplied to the eyelash processing section 454.

Other processing such as adjustment of skin brightness may be performed as retouch processing. Other image processing such as filter processing may be performed on the entire image instead of processing on a specific region.

Gender determination section 453 determines the gender of each user appearing in the photographed image, and outputs information representing the determination result to eyelash processing section 454 . The determination by the gender determining unit 453 may also be performed based on a prediction model generated by machine learning, or may be performed by matching based on the feature amount of each region extracted by analyzing the image. You may also do so.

The eyelash processing unit 454 performs eyelash processing, which is image processing that emphasizes the eyelashes of each user appearing in the captured image supplied from the retouch processing unit 452. Eyelash processing is performed, for example, only on women, based on the determination result by the gender determination unit 453. The photographed image, in which the eyelashes have been emphasized by the eyelash processing, is supplied to the reduction unit 455. The eyelash processing performed by the eyelash processing section 454 will be described in detail later.

The reduction unit 455 generates a captured image for editing by reducing the captured image supplied from the eyelash processing unit 454. The reduction of the captured image is performed, for example, by thinning out pixels to reduce the number of pixels that make up the captured image. The shot image for editing generated by the reduction unit 455 is supplied to the editing control unit 433 and the like.

In this way, in the editing image generation unit 432, the retouching process and the eyelash process are performed on the original image before reduction. It becomes possible to perform detailed processing on captured images with a large amount of data.

FIG. 22 is a block diagram showing a configuration example of the eyelash processing section 454 in FIG. 20.

The eyelash processing section 454 includes an upside-down eyelash removal section 461, a transformation section 462, a composition section 463, and an eyelash enhancement processing section 464.

The retouched captured image supplied from the retouch processing section 452 is input to the upside down eyelash removal section 461 . Further, information on each region detected by the region detection section 451 is input to each section of the eyelash processing section 454. Information representing the gender determination result by the gender determination unit 453 is input to the transformation unit 462 and the synthesis unit 463.

The upside-down eyelash removal unit 461 removes the upside-down eyelashes by image processing when there are upside-down eyelashes in the user's eyes appearing in the photographed image. Upside-down eyelashes are the upper eyelashes that grow downward and cover the white of the eye, and the lower eyelashes that grow upward and cover the white of the eye. The position of the white eye area and the like are specified based on information supplied from the area detection unit 451.

FIG. 23 is a diagram illustrating an example of removing upside-down eyelashes.

If the white eye area A13 is photographed with eyelashes as shown surrounded by a broken line circle C1 on the left side of FIG. (by interpolation), the upside-down eyelashes are removed as shown on the right side of FIG.

The fact that eyelashes cover the white of the eye area A13 is caused by detecting a pixel of a color other than white, which is set as the standard color for the white of the eye area, as an abnormal pixel when the pixel is in the white of the eye area A13. be identified. Information representing the standard color of the white eye area is set in advance in the upside-down eyelash removal section 461.

In the upside-down eyelash removal unit 461, upside-down eyelashes are removed by interpolating an abnormal pixel, which is a pixel of a color other than the standard color, with surrounding white pixels.

The process of detecting pixels of colors other than white as abnormal pixels and interpolating them with white pixels is performed in the upside down eyelash removal unit 461, so that as a result, areas that are red due to blood congestion are also removed. .

The photographed image from which the upside-down eyelashes have been removed as described above is output from the upside-down eyelash removal section 461. If there are no upside-down eyelashes in the user's eyes in the photographed image, the photographed image input to the upside-down eyelash removal section 461 is output as is.

When the user appearing in the captured image supplied from the upside-down eyelash removing unit 461 is a woman, the transforming unit 462 transforms the eyelash image to match the shape of the upper contour of the user's eyes.
The transformation unit 462 stores in advance an eyelash image that is a texture image expressing eyelashes. The eyelash image transformed by the transformation unit 462 is used for compositing the user's eye portion shown in the photographed image.

FIG. 24 is a diagram showing an example of deformation of an eyelash image.

The modification unit 462 has an eyelash image P111 in a default shape as shown in the upper left of FIG. 24. For example, if the shape of the upper contour of the user's right eye E _R in the photographed image is represented by a curve from position p1 to position p2 shown in the lower left of FIG. - The eyelash image is transformed according to the shape of a specific section of the curve between position p2, and a transformed eyelash image P111' shown on the right side of FIG. 24 is generated. The curve between position p1 and position p2 is specified based on information supplied from area detection section 451.

The transformation unit 462 outputs the eyelash image transformed as described above to the synthesis unit 463. A plurality of default eyelash images may be prepared, and the eyelash image used for transformation may be selected. The selection of eyelash images may be performed by the user or may be performed automatically by the eyelash processing unit 454.

The composition section 463 in FIG. 22 composes the eyelash image transformed by the transformation section 462 along the upper contour of the female user's eyes appearing in the captured image supplied from the upside-down eyelash removal section 461.

FIG. 25 is a diagram showing an example of combining eyelash images.

When the transformed eyelash image P111' as shown in the upper left of FIG. 25 is supplied, the synthesis unit 463 transforms the eyelash image P111' along the upper contour of the right eye E _R as shown in the right side of FIG. Synthesize. The position of the upper contour of the right eye E _R is specified based on information supplied from the area detection unit 451.

The above-described combination of eyelash images is similarly performed for the left eye. The eyelash image for the left eye is synthesized using, for example, the eyelash image P111 prepared for the right eye.

In this case, the deformation unit 462 reverses the left and right sides of the eyelash image P111, and transforms the shape of the eyelash image P111 after the left and right inversion to match the shape of the upper side of the left eye. The composition unit 463 composes the eyelash image P111 transformed by the transformation unit 462 along the upper contour of the left eye.

In this way, to synthesize the eyelash image for the left eye, an image obtained by inverting the left and right eyelash images prepared for the right eye is used. Since the processing for the left eye is performed using the image for the right eye, the number of eyelash images that need to be prepared can be reduced. Conversely, an eyelash image for the left eye may be prepared, and the processing for the right eye may be performed using the image for the left eye. Of course, separate eyelash images for the right eye and for the left eye may be prepared.

The above-described combination of eyelash images is performed for each woman appearing in the photographed image. The combining unit 463 outputs the captured image after combining the eyelash images to the eyelash enhancement processing unit 464 in FIG. 22.

The eyelash enhancement processing section 464 enhances the eyelashes appearing in the photographed image supplied from the composition section 463 through image processing. For example, processing is performed to thicken each of the upper and lower eyelashes, or to deepen their color. The eyelash enhancement processing unit 464 outputs a photographed image in which the eyelashes are emphasized.

<Operation of photo sticker making device>
Here, the operation of the photo sticker creation device 1 that provides the photo sticker creation game will be described with reference to the flowchart of FIG. 26. The process in FIG. 26 is started, for example, when a predetermined amount of coins is inserted.

In step S1, the preselection processing unit 401 performs a preselection process by controlling the preselection unit 302. Through the pre-selection process, for example, selection of the number of users, selection of a photography course, etc. are performed. All selections are made using a selection screen displayed on the touch panel monitor 71.

In step S2, the photographing processing section 402 performs photographing processing by controlling the photographing section 304.

For example, after guiding the user to a standing position, the imaging processing unit 402 displays a live view screen on the touch panel monitor 92 based on the moving image captured by the camera 91. Further, after displaying the live view screen, the photographing processing unit 402 continuously photographs still images using the user in the photographing space A1 as the subject.

In step S3, the editing processing unit 403 controls the editing unit 305 corresponding to the editing space to which the user moves after completing the shooting process, between the editing space A2-1 and the editing space A2-2. Perform the editing process.

For example, the editing processing unit 403 displays an editing screen on the tablet built-in monitor 131 and allows the user to perform editing work on the captured image. On the editing screen, buttons to be operated when selecting a composition image to be combined with the photographed image are displayed along with the photographed image to be edited. Details of the editing process will be described later with reference to the flowchart in FIG.

In step S4, the print processing unit 404 performs print processing by controlling the printer 351. The print processing unit 404 outputs the print data generated by the editing processing unit 403 to the printer 351 to print it on the sticker paper 363.

When printing is completed, the printer 351 discharges the sticker paper 363 cut off by the cutter 362 to the sticker paper discharge port 161 in step S5.

A photo creation game consisting of a series of processes as described above is provided to each group of users of the photo sticker creation device 1. If a user of another group starts using the photo sticker creation device 1 following a user of one group, the photo sticker creation game of the user who started using the photo sticker creation device 1 first and the game of the user who started using the photo sticker creation device 1 first will be played. The photo sticker creation game of the user who started the game will be played in parallel as appropriate. When appropriate, the photo sticker creation game is also simply referred to as a game.

<Editing process>
Next, the editing process performed in step S3 of FIG. 26 will be described with reference to the flowchart of FIG. 27.

In step S41, the display control unit 434 (FIG. 19) of the editing processing unit 403 causes the pre-doodle selfie guide screen to be displayed on the tablet built-in monitor 131. Graffiti here means editing of a photographed image. Editing is performed by doodling on the photographed image using the touch pen 132.

FIG. 28 is a diagram illustrating a display example of a guide screen for self-portrait before doodling.

As shown in FIG. 28, on the left side of the pre-doodle selfie guide screen, an image showing a model person taking a selfie through the mirror surface 66 is displayed. On the right side of the guide screen, sample images of images obtained by self-portraits are displayed. A message explaining the content of the image is displayed below each image.

By displaying such a guide screen, it is possible to clearly tell the user what kind of self-portraits they can take using the mirror section 66.

The user who has moved to the editing space A2 prepares his or her smartphone and takes a selfie through the mirror section 66 according to the guidance screen.

Returning to the explanation of FIG. 27, in step S42, the editing image generation unit 432 performs editing image generation processing. Through the editing image generation process, a photographed image for editing is generated based on each photographed image.

In the photographing process, for example, up to eight photographed images are photographed. Since there are a large number of captured images, a certain amount of time is required to generate captured images for editing based on each captured image. Generation of a shot image for editing is performed using the time reserved for self-portraits before editing. The user can wait without getting bored even while the captured image for editing is being generated.

Details of the editing image generation process performed while the user is taking a selfie will be described later with reference to the flowchart of FIG. 30.

In step S43, the display control unit 434 displays a keep image selection screen on the tablet built-in monitor 131. By displaying a keep image selection screen on the tablet built-in monitor 131 instead of the guide screen in FIG. 28, the user finishes taking a selfie and selects a keep image according to the display on the tablet built-in monitor 131.

The keep image is a captured image that is actually edited. The number of captured images that can be selected as keep images is set to, for example, four. On the keep image selection screen, photographed images that have been subjected to retouching processing are displayed side by side. Selection using the selection screen is performed by selecting four captured images from a maximum of eight captured images.

If a predetermined number of keep images, such as four, are selected, the display control unit 434 causes the tablet built-in monitor 131 to display an editing screen in step S44. On the editing screen, buttons to be pressed to select various editing tools (images for synthesis) such as stamp images and pen images used for editing are displayed along with the photographed image to be edited.

FIG. 29 is a diagram showing an example of an editing screen displayed on the tablet built-in monitor 131.

On the editing screen, editing areas 501L and 501R are provided side by side, respectively. A thumbnail display area 502 is provided between the editing areas 501L and 501R, in which thumbnails of captured images selected as keep images are displayed.

Editing palettes 503L and 503R are provided below the editing areas 501L and 501R, respectively. A category selection button group 504 is provided between the editing palettes 503L, 503R for selecting the category of editing tools displayed on the editing palettes 503L, 503R.

With this configuration, on the editing screen, editing inputs for the photographed image selected as the editing target are individually received by the two users.

In the following, when the editing areas 501L, 501R and the editing palettes 503L, 503R are not distinguished from each other, they are simply referred to as the editing area 501 and the editing palette 503.

The editing area 501 is a display area for a keep image (editing target image) selected as a target for editing input. The user selects an editing tool using the touch pen 132 and edits the keep image displayed in the editing area 501.

The thumbnail display area 502 is a display area for thumbnails representing keep images. The user selects a keep image to perform editing input by selecting a thumbnail displayed in the thumbnail display area 502.

The editing palette 503 is an area where icons used for selecting various editing tools such as a pen tool, a stick tool, a stamp tool, and a makeup tool are displayed. Note that the merging tool is an editing tool used when combining a predetermined image at a position based on the face recognition result. The makeup tool is an editing tool used to perform image processing that simulates makeup on the eyes, lips, cheeks, and the like.

The category selection button group 504 is composed of buttons that are pressed when switching the categories of editing tools displayed on the editing palette 503.

In the example of FIG. 29, the pen button of the category selection button group 504 is selected by the user on the left side when facing the tablet built-in monitor 131. Icons for selecting a pen tool and the like are displayed on the editing palette 503L.

Further, the user on the right side has selected the makeup button in the category selection button group 504. Editing palette 503R displays buttons for selecting makeup tools and the like.

Below the thumbnail display area 502, two mobile bonus buttons 505L and 505R that imitate the appearance of a smartphone are displayed side by side. When the mobile bonus buttons 505L and 505R are operated, buttons used when creating a mobile bonus are displayed. Note that the mobile bonus is an image that is intended to be used on a mobile terminal, such as an image displayed on the standby screen of the mobile terminal or uploaded to SNS. The created mobile bonus is transmitted to the image management server as an image exclusively for transmission to the mobile terminal.

A BGM selection area 506 is displayed above the thumbnail display area 502. The BGM selection area 506 is operated when selecting BGM or adjusting the volume during editing work.

Returning to the description of FIG. 27, in step S45, the editing control unit 433 edits the photographed image to be edited in accordance with the user's operation on the editing screen. The edited image generated by the editing is supplied to the print data generation unit 435.

If editing is completed by operating the end button, etc., the display control unit 434 displays a division number selection screen on the tablet built-in monitor 131 in step S46.

The division number selection screen is a screen used to select a sticker layout. The sticker layout is a layout (arrangement pattern) that determines the number, position, and size of edited images arranged on one sheet of sticker paper 363. Edited images arranged according to the sticker layout selected by the user are printed on the sticker paper 363.

In step S47, the display control unit 434 causes the tablet built-in monitor 131 to display a transmission image selection screen.

Users must register as members in order to use the image management service, and there are paid members and free members. Paid members can acquire all images sent to the image management server and download them to their mobile terminals, while free members can acquire only one image. The transmission image selection screen is a screen used by a free member to select one image to be acquired. The image management service is a service that provides users with images created in the game and transmitted from the photo sticker creation device 1 to the image management server.

In step S48, the editing processing unit 403 causes the tablet built-in monitor 131 to display an email address input screen.

The email address input screen is a screen used to input the user's email address.
The e-mail address input screen displays alphabets and symbols that can be used in e-mail addresses, buttons for selecting the carrier of a mobile phone such as a smartphone, etc.

In step S49, the communication control unit 436 transmits the edited image to the image management server. Photographed images are also sent to the image management server as appropriate. That is, not only the edited image but also the photographed image that is the source of the edited image is sent to the image management server.

Additionally, the device ID of the photo sticker creation device 1, the date and time of photography, and information indicating which image was selected from the transmission image selection screen are also transmitted to the image management server.

In step S50, the print data generation unit 435 generates print data by arranging the edited image in the layout selected by the user. The print data is supplied to the print processing unit 404 and used to print the edited image (step S4 in FIG. 26).

In step S51, the display control unit 434 displays a guide screen on the tablet built-in monitor 131 to guide the user who has finished the editing work to move to the print waiting space A3. The guidance for moving to the print waiting space A3 is also performed by outputting audio from the speaker 341.

Thereafter, the process returns to step S3 in FIG. 26, and subsequent processing is performed.

As described above, in the game of the photo sticker creation device 1, it is possible to take pictures not only in the shooting space A1 but also in the editing space A2. Users can take self-portraits of themselves playing games in the editing space A2.

<Editing image generation processing>
Next, with reference to the flowchart of FIG. 30, the editing image generation process performed in step S42 of FIG. 27 will be described.

In step S61, the gender determination unit 453 determines the gender of each user appearing in the photographed image.

In step S62, the retouch processing unit 452 performs retouch processing on the photographed image. The retouch processing by the retouch processing unit 452 is performed according to the content selected using the retouch screen displayed on the touch panel monitor 92 during the photographing process.

For example, the retouch processing unit 452 detects the user's eyes appearing in the photographed images, and performs processing for adjusting the size of the detected eyes on each photographed image. The retouch processing unit 452 performs multiple types of image processing on each captured image, other than eye size adjustment.

In step S63, the area detection unit 451 detects the area of the user's eyes appearing in the photographed image.

In step S64, the eyelash processing unit 454 performs eyelash processing, which is image processing that emphasizes the eyelashes in the photographed image. Details of the eyelash processing will be described later with reference to the flowchart of FIG. 31.

In step S65, the reduction unit 455 reduces the captured image supplied from the eyelash processing unit 454 to generate a captured image for editing.

In step S66, the editing image generation unit 432 determines whether the processing of all photographed images has been completed, and if it is determined that there is a photographed image that has not yet been processed, it repeatedly executes the processing from step S61 onward. do.

If it is determined in step S66 that all photographed images have been processed, the process returns to step S42 in FIG. 27 and subsequent processing is performed.

<Eyelash treatment>
Next, the eyelash processing performed in step S64 of FIG. 30 will be described with reference to the flowchart of FIG. 31.

In step S81, the eyelash processing unit 454 determines whether a woman is included in the photographed image to be processed, based on the gender determination result by the gender determination unit 453.

As described above, eyelash processing, which is image processing that emphasizes eyelashes, is performed only on women, not men. If it is determined in step S81 that there are no women in the photographed image to be processed, that is, all the users in the photographed image are male, the processing from step S82 onward in FIG. 31 is skipped. Ru.

This makes it possible to prevent image processing that emphasizes a man's eyelashes from being performed.

At least some of the processes after step S82 may be performed for men.

If it is determined in step S81 that a woman is included in the photographed image to be processed, the process proceeds to step S82. If a plurality of women are in the photograph, the processing from step S82 onward is performed, focusing on each woman one by one.

In step S82, the upside-down eyelash removal unit 461 detects an abnormal pixel included in the white of the eye area, and removes the upside-down eyelash by interpolating with surrounding pixels.

In step S83, the deformation unit 462 deforms the eyelash image according to the shape of the upper contour of the eye, as described with reference to FIG. 24.

In step S84, the composition unit 463 composes the eyelash image transformed by the transformation unit 462 along the upper contour of the eye of the woman to be processed, which appears in the photographed image.

Here, if both eyes of the woman to be processed are open, eyelash images are combined for both eyes. For example, if the degree of opening of the eyes is lower than the threshold value (if the eyes are closed), the process to emphasize eyelashes, including compositing eyelash images, will not be performed on the closed eyes. Good too.

If eyelashes are emphasized on eyes that are closed when winking or the like, there is a possibility that the eyelashes will look unnatural, but this can be prevented.

In step S85, the eyelash enhancement processing unit 464 emphasizes the eyelashes through image processing. For example, the eyelashes are emphasized by detecting the upper and lower eyelashes and making them thicker or darker one by one. The process of thickening or darkening each eyelash is also performed on each eyelash that makes up the eyelash image that is synthesized as the upper eyelash.

That is, in this example, in addition to compositing the eyelash images, the upper eyelashes are subjected to processing such as making each eyelash thicker one by one. Furthermore, the upper eyelashes and lower eyelashes will be emphasized through different processes.

Thereafter, the process returns to step S64 in FIG. 30, and subsequent processing is performed.

As mentioned above, to emphasize the eyelashes of women in photographed images, instead of composing eyelash images prepared as templates as they are, we synthesize eyelash images that have been transformed to match the shape of the eye contour. It will be done.

The shape of each user's eyes will differ depending on their facial expressions at the time of shooting. By combining eyelash images transformed to match the shape of each user's eyes, it is possible to emphasize the eyelashes in a more natural manner and make the eyes appear larger.

Further, after the eyelash images are combined, by emphasizing each of the upper and lower eyelashes, it is possible to make the eyelashes combined as an eyelash image blend in with the user's own eyelashes.

<Make tool function>
・Eye Color Tool Here, editing using the makeup tool provided on the editing screen will be explained.

FIG. 32 is an enlarged view of the editing palette 503R of FIG. 29 while the makeup tool is being selected.

Selection buttons 511-1 to 511-5 used for selecting makeup tools are displayed side by side on the editing palette 503R.

The selection button 511-1 is a button that is pressed when selecting a small face tool that applies makeup to make the face look smaller.

The selection button 511-2 is a button that is pressed when selecting an eye color tool that changes the color of the iris. By using the eye color tool, users can apply makeup to look like they are wearing colored contact lenses. In the example of FIG. 32, the selection button 511-2 is pressed and the eye color tool is selected.

The selection button 511-3 is a button that is pressed when selecting an outline density adjustment tool for adjusting the density of the eye outline.

The selection button 511-4 is a button that is pressed when selecting a lip tool that changes the color of lips. The selection button 511-5 is a button that is pressed when selecting a cheek tool that changes the cheek color.

When the eye color tool is selected, a face image area 512 is displayed on the left side of the editing palette 503R, as shown in FIG. 32. The face image area 512 is a display area for an image of the user's face that appears in the captured image and is extracted by performing face recognition.

Eye iris processing, which is processing for changing the color of the iris, is performed by the iris processing unit 433A (FIG. 19) for the user whose face is displayed in the face image area 512. In the example of FIG. 32, the user on the right of the two users in the photographed image displayed in the editing area 501R in FIG. 29 is displayed as the target user of the eye color tool.

Displayed below the face image area 512 is a switching button 513 that is pressed when switching the target user of the eye color tool.

Color selection buttons 514-1 to 514-4, which are buttons used to select eye colors, are displayed on the right side of the face image area 512. Color selection buttons 514-1 to 514-4, which are circular buttons, are assigned the colors brown, chocolate, gray, and black, respectively.

For example, when the color selection button 514-1 is pressed, the color of the iris area of the user selected as the target of the eye color tool among the users appearing in the photographed image displayed in the editing area 501R is changed to brown. Be changed.

In addition, when the color selection button 514-2 is pressed, the color of the iris of the user selected as the target of the eye color tool among the users appearing in the photographed image displayed in the editing area 501R is changed to chocolate. will be changed to

FIG. 33 is a diagram illustrating an example of an image generated by the eye color tool.

When the photographed image P101 is selected as the target for editing and the user on the right side is selected as the target for the eye color tool, in response to pressing the color selection buttons 514-1 to 514-4, the editing area 501R is The photographed images are switched to photographed images P101-1 to P101-4 in which the color of the iris of the user's eye area on the right side surrounded by a frame F has been changed.

Although they are shown as the same images in the example of FIG. 33, the captured image P101-1 is an image in which the color of the eye area of the user on the right side has been changed to brown, and the captured image P101-2 is an image in which the color of the eye area of the user on the right side has been changed to brown. This is an image in which the color of the iris of the user's eyes has been changed to chocolate. The photographed image P101-3 is an image in which the color of the iris of the right user's eye area has been changed to gray, and the photographed image P101-4 is an image in which the color of the iris area of the right user has been changed to black. be.

Each time the switching button 513 is pressed, the users targeted by the eye color tool are switched in the order of the user on the left, the user on the right, and then the two users.

For example, when the target user of the eye color tool is selected as the user on the left side, the captured image in the editing area 501R is , the image is switched to a captured image in which the color of the eye area of the user on the left side has been changed.

In addition, when the target users of the eye color tool are all selected as two users, the editing area 501R is The photographed image is switched to a photographed image in which the colors of the iris areas of the two users on the left and right sides have been changed.

In this way, by using the eye color tool, the user can change the color of the iris for each photographed image and for each user in the photographed image. The user can also change the eye colors of two users at the same time.

At the lower right of the color selection button 514-4 in FIG. 32, an off button 515 that is pressed to turn off the function of the eye color tool is displayed. When the off button 515 is pressed, a captured image that has not been subjected to iris processing is displayed in the editing area 501R.

FIG. 34 is a diagram showing an example of a mask image used in image processing when selecting an eye color tool.

When generating a photographed image for editing, a mask image representing each area of the eye as shown in FIG. 34B is generated based on the photographed image P101 shown in FIG. 34A. As described above, when generating a captured image for editing, the area detection unit 451 (FIG. 20) of the image generation unit 432 for editing uses a predictive model generated by machine learning to detect each area of the eye, etc. A representing mask image M is generated.

In the example of FIG. 34B, among the pixels constituting the mask image M, the pixels representing the outlines of the eyes, the outlines of the black eyes, and the eyelashes of the two users are represented by white pixels. Using such a mask image M, the iris processing unit 433A performs iris processing.

FIG. 35 is a diagram illustrating an example of changing the color of the iris area.

When changing the color of the iris using the eye color tool is selected, the iris processing unit 433A uses the mask image M to extract the iris area of the user as a target of the eye color tool. In the example in the upper part of FIG. 35, the iris area A11 _R of the right eye E _R and the iris area A11 _L of the left eye E _L are extracted based on the mask image M.

Further, the iris processing unit 433A changes the color of the pixels forming the iris areas A11 _R and A11 _L , which are the extracted iris areas, to a color selected by the user from among brown, chocolate, gray, and black. The fact that the iris areas A11 _R and A11 _L are hatched in the lower part of FIG. 35 indicates that the colors of the pixels in the iris areas A11 _R and A11 _L have been changed.

The color of the pixels constituting the iris area is changed, for example, by changing the hue of the original color by an amount corresponding to the selected color. For each of the colors brown, chocolate, gray, and black, the method of changing the hue, including the degree (amount) of the change, is set in advance.

That is, instead of changing to the selected color, the color is changed by using the original black eye color as a reference and adjusting the reference color according to the selected color.

The color of the iris area may be changed not by changing the hue but by changing other parameters such as brightness and saturation, or by changing a combination of multiple parameters.

The color may be changed by applying filter processing to each pixel forming the iris area or adjusting the brightness using a predetermined tone curve.

In this way, the color of the iris area is changed by changing the color of the pixels in the captured image itself, rather than composing a substantially circular image having the same shape as the iris area on top of the captured image. It will be done. Furthermore, the color of the iris area is changed by changing the original color based on the original color, which differs slightly from person to person.

Even if you choose the same color from brown, chocolate, gray, or black, the color will be slightly different for each person, so it is possible to change the impression of the black eye area while still looking natural and natural. becomes.

If the color of the iris area were to be changed by compositing a circular image of a predetermined color onto the iris area, some people might notice that the color of the iris area changes significantly, or that the direction of the eyes or the strobe light changes. Differences in appearance depending on the way the light hits it may be lost, but it is possible to prevent such unnatural color changes.

- Contour Density Adjustment Tool Next, the adjustment of the darkness of the eye contour performed using the contour density adjustment tool will be described.

When the selection button 511-3 is pressed, a slide bar used for adjusting the darkness of the eye outline, for example, is displayed on the editing palette 503R. A face image area 512 is displayed on the left side of the editing palette 503R.

For a user whose face is displayed in the face image area 512, the contour adjustment unit 433B (FIG. 19) performs contour density adjustment processing, which is a process of changing the density of the eye contour. The contour density adjustment process is also performed using the mask image M generated by the area detection unit 451.

FIG. 36 is a diagram illustrating an example of adjusting the darkness of the eye outline.

When darkening the eye contour is selected by the contour density adjustment tool, the contour adjustment unit 433B extracts the eye contour, the iris contour, and the eyelashes using the mask image M.

Further, the contour adjustment unit 433B adjusts the color of the pixels so that each extracted portion becomes darker. For example, the outline of the eyes, the outline of the iris, and the eyelashes are adjusted by making each extracted part thicker or darker in color. In the example in the lower row of FIG. 36, the outline of the eye, the outline of the iris (left and right edges), and the eyelashes are shown in a darker state than in the upper row of FIG.

FIG. 37 is a diagram showing another example of adjusting the darkness of the eye outline.

When thinning the eye contour is selected by the contour density adjustment tool, the contour adjustment unit 433B extracts the eye contour, the black eye contour, and the eyelashes using the mask image M.

Furthermore, the contour adjustment unit 433B adjusts the color of the pixels so that each extracted portion becomes lighter. For example, the outline of the eyes, the outline of the iris of the eye, and the eyelashes are adjusted by making each extracted part thinner or lighter in color.

In this way, the adjustment of the eye contour is performed based on the mask image M by adjusting the eye contour, the black eye contour, and the eyelashes as one unit. A plurality of slide bars may be displayed so that the outline of the eye, the outline of the iris of the eye, and the darkness of the eyelashes can be adjusted.

Such contour density adjustment processing is performed by selecting a target user of the contour density adjustment tool.

For example, as described with reference to FIG. 32, if the target user of the contour density adjustment tool is selected as the user on the right side, the editing area 501R The captured image is switched to a captured image in which the density of the eye outline of the user on the right side has been changed.

Similarly, if the target user of the contour density adjustment tool is selected as the user on the left side, the captured image in the editing area 501R will be changed to the user on the left side in response to the slide bar being operated. The image is switched to a photographed image in which the darkness of the eye outline has been changed.

In addition, if two users are selected as targets for the contour density adjustment tool, the captured image in the editing area 501R will be changed to the two users on the left and right sides in response to the slide bar being operated. The image is switched to a photographed image in which the density of the outline of the user's eyes has been changed.

In this way, by using the contour density adjustment tool, the user can change the darkness of the eye contour for each photographed image and for each user in the photographed image. . The user can also change the darkness of the eye contours of two users at the same time.

By using the eye color tool and contour density adjustment tool together, users can adjust the color of the black eye individually for each user, and adjust the darkness of the eye outline etc. for both users at the same time. becomes possible. Further, the users can adjust the color of the iris of two people's eyes at once, and individually adjust the darkness of the outline of the eyes for each user.

<Modified example>
- Examples of other makeup tools Processing using a lip tool and a cheek tool provided as makeup tools may be performed in the same manner as the eye color tool.

For example, when the selection button 511-4 is pressed and the lip tool is selected, the color of the user's lips selected as the target of the lip tool is changed to the color selected using the button displayed on the editing palette 503R. Changes will be made accordingly. The color of pixels constituting the lip area extracted using the mask image is changed based on the original lip color, and the reference color is adjusted according to the selected color. .

・Switching processing according to the output destination The iris area whose color is changed using the eye color tool is a narrow area compared to the entire image. If a plurality of images with different colors of the iris areas are printed side by side on sticker paper, the iris in each image is small, so depending on the layout, it may not be possible to express the difference in the color of the iris areas. Additionally, the entire iris of the eye may appear to be an unnatural color.

A change in the color of the iris area using the eye color tool may be reflected in the image displayed on the editing screen, but not reflected in the image printed on the sticker paper. In this case, for each captured image, processing to generate an image to be displayed on the editing screen and processing to generate an image to be printed on sticker paper are performed in response to user operations. become.

That is, when the photographed image P101 is selected as an editing target and the user instructs to change the color of the iris area using the eye color tool, the photographed image P101 with the color of the iris area changed is displayed on the editing screen. The captured image P101, which is generated as an image to be displayed and in which the color of the iris area is not changed, is generated as an image to be printed on sticker paper.

As a result, an appropriate image can be generated depending on the destination of the image, and it is possible to prevent an unnatural looking image from being printed on the sticker paper.

Note that the photographed image P101 displayed on the editing screen with the color of the iris area changed is edited using various editing tools according to the user's operations. An image that has been edited but with the color of the iris area unchanged will be printed on the sticker paper.

Although it is assumed that the output destination of the image is the editing screen (tablet built-in monitor 131) or the sticker paper, the processing may be switched depending on other output destinations. Output destinations include not only components provided in the photo sticker creation device 1 such as the touch panel monitor 92, but also the monitor of the user's smartphone and the image management server.

Although the case has been described in which the image printing medium is sticker paper, various media such as printing paper, resin cards, cloth, etc. may be used instead of sticker paper.

- Regarding the program The series of processes described above can be executed by hardware or software. When a series of processes is executed by software, a program constituting the software is installed in a computer built into dedicated hardware or a general-purpose personal computer.

The program to be installed is provided by being recorded on a removable medium 319 shown in FIG. 16, such as an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.) or semiconductor memory. It may also be provided via a wired or wireless transmission medium, such as a local area network, the Internet, or digital broadcasting. The program can be installed in the ROM 312 or the storage unit 316 in advance.

Note that the program executed by the computer may be a program in which processing is performed chronologically in accordance with the order described in this specification, in parallel, or at necessary timing such as when a call is made. It may also be a program that performs processing.

The effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiments of the present technology are not limited to the embodiments described above, and various changes can be made without departing from the gist of the present technology.

For example, the present technology can take a cloud computing configuration in which one function is shared and jointly processed by multiple devices via a network.

Moreover, each step explained in the above-mentioned flowchart can be executed by one device or can be shared and executed by a plurality of devices.

Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

1 Photo sticker making device, 66 Mirror section, 82 Upper strobe unit, 91 Camera, 92 Touch panel monitor, 131 Monitor with built-in tablet, 301 PC section, 302 Pre-selection section, 304 Photography section, 305 Editorial section, 306 Printing section, 401 Preliminary selection processing section, 402 photographing processing section, 403 editing processing section, 404 print processing section, 411 photographing control section, 412 image processing section, 413 display control section, 414 image storage section, 431 selection reception section, 432 image generation section for editing , 433 editing control unit, 434 display control unit, 435 print data generation unit, 436 communication control unit, 451 area detection unit, 452 retouch processing unit, 453 gender determination unit, 454 eyelash processing unit, 455 reduction unit

Claims (8)

A detection unit that detects the iris of a person in a photographed image,
An image processing unit that changes the density of the edge of the iris according to an operation by a user. The image processing device according to claim 1, wherein the image processing unit further changes the color of the iris. A button operated when selecting a tool used for adjusting the color of the iris of the eye and a button operated when selecting a tool used for adjusting the darkness of the edge of the iris are displayed on the same screen together with the photographed image. The image processing device according to claim 2, further comprising a display control section. According to claim 2 or 3, the image processing unit changes the color of the iris of one of the plurality of people appearing in the photographed image, and changes the darkness of the iris of the iris of all of the plurality of people at once. The image processing device described. According to claim 2 or 3, the image processing unit changes the darkness of the edge of the iris of one of the plurality of people in the photographed image, and changes the color of the iris of all of the plurality of people at once. The image processing device described. The image processing device according to claim 1, wherein the image processing unit changes the density of the outline of the eye and the density of the edge of the iris. The image processing device
Detects the black eyes of people in captured images,
An image processing method that changes the density of the edges of the iris according to user operations. to the computer,
Detects the black eyes of people in captured images,
A program that executes processing to change the darkness of the edges of the iris according to user operations.

Images