JP7074994B2 - Photographing game console and image processing method - Google Patents

Description

The present technology relates to a photo-creating game machine and an image processing method, and more particularly to a photo-creating game machine and an image processing method that enable a highly satisfactory image to be provided.

Conventionally, a photo sticker machine installed in an amusement facility or the like is known. The photo sticker machine photographs the user and edits the captured image according to the operation of the user. The photo sticker machine prints the edited photographed image on the sticker paper.

In recent years, in a photo sticker machine, face recognition is performed on a photographed image in order to perform appropriate image processing on the photographed image.

For example, in Patent Document 1, a face is detected and registered from an image obtained by recognition imaging for recognizing a face, and the registered face is collated with a face in a captured image obtained by subsequent imaging. Is disclosed. This makes it possible to perform image processing for changing the size and shape of the eyes on the face of each user in the captured image.

Further, Patent Document 2 discloses that a cosmetic image (makeup stamp) corresponding to an organ of a face detected by face recognition is synthesized in a photographed image.

Japanese Unexamined Patent Publication No. 2013-243648 Japanese Unexamined Patent Publication No. 2015-34836

However, in the configurations disclosed in Patent Document 1 and Patent Document 2, if the recognition of the face or its organ fails, appropriate image processing cannot be applied to the captured image, and a highly satisfying image is provided. There wasn't.

This technique has been made in view of such a situation, and makes it possible to more reliably provide a highly satisfying image.

The photo-creating game machine of the present technology has a recognition unit that recognizes a face or a part thereof in a captured image, and a region of the face or a part thereof that has been successfully recognized , regardless of the user's operation . The first image processing unit that performs the image processing of 1 and the face or a part thereof that has not been subjected to the first image processing due to failure in recognition due to the operation of the user in the captured image. A region specifying unit for specifying an area and a second image processing unit for performing a second image processing on the face or a part of the region specified by the region specifying unit are provided.

The image processing method of the present technology recognizes a face or a part thereof in a captured image, and performs first image processing on the area of the face or a part thereof that has been successfully recognized without any operation by a user. In the captured image, the face or a part of the face to which the first image processing was not performed due to the failure in recognition by the operation of the user is specified, and the specified face or the specified area or a part thereof is specified. A step of performing a second image processing on a part of the area is included.

In the present technique, the face or a part thereof is recognized in the captured image, and the first image processing is applied to the area of the face or a part thereof that has been successfully recognized without any operation by the user. In the captured image, the face or a part of the area to which the first image processing has not been performed due to the failure in recognition is specified by the operation of the user, and the specified face or one thereof. A second image process is applied to the area of the portion.

According to this technique, it is possible to more reliably provide a highly satisfying image.

It is a perspective view which shows the structural example of the appearance of the photograph sticker making apparatus of this technique. It is a perspective view which looked at the appearance of the photograph sticker making apparatus from another angle. It is a figure explaining the movement of a user. It is a figure which shows the structural example of the preselection part. It is a figure which shows the structural example of the photographing part. It is a figure which shows the structural example of the background part. It is a perspective view which shows the floor material. It is a figure which shows the configuration example of the front of an editing unit. It is a figure which shows the configuration example of the side surface of an editing unit. It is a figure which shows the example of a sticker paper. It is a figure which shows the example of the layer composition of an edited image. It is a block diagram which shows the configuration example of the photograph sticker making apparatus. It is a block diagram which shows the functional composition example of the photograph sticker making apparatus. It is a block diagram which shows the functional structure example of an edit processing part. It is a block diagram which shows the functional structure example of a retouch processing part. It is a flowchart explaining the photo sticker making game process. It is a flowchart explaining the editing process. It is a flowchart explaining the automatic retouching process. It is a figure which shows the example of an edit screen. It is a figure which shows the example of the guidance pop-up of a manual retouching process. It is a figure which shows the example of the usage pop-up of the manual retouching process. It is a flowchart explaining the manual retouching process. It is a figure explaining the identification of the area to be the target of the manual retouching process. It is a figure explaining the result of the manual retouching process. It is a figure explaining the identification of the area to be the target of the manual retouching process. It is a figure which shows the other example of an edit screen. It is a figure which shows another example of the usage pop-up of the manual retouching process. It is a figure which shows another example of the usage pop-up of the manual retouching process. It is a figure explaining the identification of the area to be the target of the manual retouching process. It is a figure explaining the angle limitation of the line drawn in the manual retouching process.

<Appearance of photo sticker making device>
1 and 2 are perspective views showing a configuration example of the appearance of the photo sticker creating device 1.

The photo sticker creating device 1 is a game machine that provides a photographed image and an edited image. The photo sticker creating device 1 provides an image to a user by printing the image on a sticker paper or making the image viewable on the user's mobile terminal. The photo sticker creating device 1 is installed in an amusement facility such as a game center or a store.

The users of the photo sticker making device 1 are mainly high school girls and young women. In the photo sticker creating device 1, a plurality of users, such as mainly two or three people, can enjoy the game per set. Of course, in the photo sticker creating device 1, one user can enjoy the game.

In the photo sticker creating device 1, the user takes a picture by himself / herself as a subject. By editing, the user synthesizes a composite image such as handwritten characters or a stamp image with an image selected from the captured images obtained by shooting. As a result, the captured image is edited into a colorful image. The user receives a sticker paper on which the edited image, which is an edited image, is printed, and ends the series of games.

The photo sticker creation device 1 can be said to be a photo creation game machine that provides a photo creation game.

As shown in FIG. 1, the photo sticker creating device 1 is basically configured by being installed in a state where the photographing unit 11 and the editing unit 12 are in contact with each other. The photographing unit 11 and the editing unit 12 are electrically connected.

The photographing unit 11 has a box-shaped housing having a size large enough for a person to enter, and is composed of a preselection unit 20, a photographing unit 21, and a background unit 22. The preselection unit 20 is installed on the side surface of the photographing unit 21. The space in front of the preselection unit 20 is the preselection space in which the preselection process is performed. Further, the photographing unit 21 and the background unit 22 are installed at a predetermined distance from each other. The space formed between the photographing unit 21 and the background unit 22 is the photographing space in which the photographing process is performed.

As the preselection process, the preselection unit 20 provides guidance for introducing the game provided by the photo sticker creating device 1, and makes various settings in the photography process performed in the photography space. The preselection unit 20 is provided with a coin slot for the user to insert the price, a touch panel monitor used for various operations, and the like. The preselection unit 20 appropriately guides the user in the preselection space to the shooting space according to the availability of the shooting space.

The photographing unit 21 is a device for photographing a user as a subject. The photographing unit 21 is located in front of the user who has entered the photographing space. A camera, a touch panel monitor used for various operations, and the like are provided in front of the photographing unit 21 facing the photographing space.

Assuming that the left side surface is the left side surface and the right side surface is the right side surface when viewed from the user facing the front in the shooting space, the left side surface of the photographing unit 21 is composed of the side panel 41A and the right side surface is composed of the side panel 41B. Will be done. Further, the front surface of the photographing unit 21 is composed of the front panel 42. The preselection unit 20 described above is installed on the side panel 41A. The preselection unit 20 may be installed on the side panel 41B.

The background portion 22 is composed of a back panel 51, a side panel 52A, and a side panel 52B. The back panel 51 is a plate-shaped member located on the back side of the user facing the front. The side panel 52A is a plate-shaped member attached to the left end of the back panel 51 and having a narrower width than the side panel 41A. The side panel 52B is a plate-shaped member attached to the right end of the back panel 51 and having a width narrower than that of the side panel 41B.

The side panel 41A and the side panel 52A are provided on substantially the same plane. The upper portions of the side panel 41A and the side panel 52A are connected by a connecting portion 23A which is a plate-shaped member. The lower portions of the side panel 41A and the side panel 52A are connected by, for example, a connecting portion 23A'which is a metal member provided on the floor surface. Similarly, the side panel 41B and the side panel 52B are provided on substantially the same plane. The upper portions of the side panel 41B and the side panel 52B are connected by the connecting portion 23B. The lower portions of the side panel 41B and the side panel 52B are connected by a connecting portion 23B'.

A flooring material 27, which is a plate-shaped member, is laid on the floor of the photographing space.

For example, a green chroma key sheet is attached to the surface of the back panel 51 on the shooting space side. The photo sticker creating device 1 takes a picture with a sheet for chroma key as a background, and performs chroma key composition in a shooting process and an editing process. As a result, the desired background image is combined with the background portion of the user in which the sheet is shown.

An opening formed by being surrounded by the side panel 41A, the connecting portion 23A, and the side panel 52A serves as an entrance / exit of the photographing space. Further, the opening formed by being surrounded by the side panel 41B, the connecting portion 23B, and the side panel 52B also serves as an entrance / exit of the photographing space.

Above the photographing space, a ceiling surrounded by the front surface of the photographing unit 21, the connecting portion 23A, and the connecting portion 23B is formed. A ceiling strobe unit 24 is provided on a part of the ceiling. One end of the ceiling strobe unit 24 is fixed to the connecting portion 23A, and the other end is fixed to the connecting portion 23B.

The ceiling strobe unit 24 has a built-in strobe that irradiates light into the shooting space in accordance with shooting. Inside the ceiling strobe unit 24, a fluorescent lamp is provided in addition to the strobe. As a result, the ceiling strobe unit 24 also functions as lighting for the shooting space.

The editing unit 12 is a device for editing a captured image. The editing unit 12 is connected to the photographing unit 11 so that one side surface is in contact with the front panel 42 of the photographing unit 21.

Assuming that the configuration of the editing unit 12 shown in FIG. 1 is the configuration on the front side, the configuration used for the editing work is provided on each of the front side and the back side of the editing unit 12. With this configuration, two sets of users can perform editing work at the same time.

The front side of the editing unit 12 is composed of a surface 61 and a slope 62 formed above the surface 61. The surface 61 is perpendicular to the floor surface and substantially parallel to the side panel 41A of the photographing unit 21. A monitor with a built-in tablet and a stylus are provided on the slope 62 as a configuration used for editing work. On the left side of the slope 62, a columnar support portion 63A that supports one end of the lighting device 64 is provided. On the right side of the slope 62, a columnar support portion 63B that supports the other end of the lighting device 64 is provided. A support portion 65 for supporting the curtain rod 26 is provided on the upper surface of the support portion 63A.

A curtain rod 26 is attached above the editing unit 12. The curtain rail 26 is composed of a combination of three rails 26A to 26C. The three rails 26A to 26C are combined so that the shape when viewed from above is U-shaped. One ends of the rail 26A and the rail 26B provided in parallel are fixed to the connecting portion 23A and the connecting portion 23B, respectively, and the other ends of the rail 26A and the rail 26B are joined to both ends of the rail 26C, respectively.

A curtain is attached to the curtain rail 26 so that the space in front of the front and the space in front of the back of the editing unit 12 cannot be seen from the outside. The space in front of the front and the space behind the back of the editing unit 12 surrounded by the curtain are the editing spaces in which the user performs editing work.

As will be described later, the left side surface of the editing unit 12 is provided with a discharge port from which the printed sticker paper is discharged. The space in front of the left side of the editing unit 12 is a printing waiting space for the user to wait for the printed sticker paper to be ejected.

<About the movement of users>
Here, the flow of the photo sticker creation game and the accompanying movement of the user will be described.

FIG. 3 is a plan view of the photo sticker creating device 1 as viewed from above.

The user inserts the price into the coin slot in the preselection space A0, which is the space in front of the preselection unit 20. Next, the user makes various settings according to the screen displayed on the touch panel monitor. For example, the user performs preselection work such as selection of a course related to shooting performed in the shooting space.

As shown by the white arrow # 1, the user who has completed the preselection work has a shooting space A1 formed between the shooting unit 21 and the background unit 22 from the entrance / exit G1 between the side panel 41A and the side panel 52A. to go into. The user performs shooting work by using a camera or a touch panel monitor provided in the shooting unit 21.

The user who has completed the shooting work can either leave the shooting space A1 from the entrance / exit G1 and move to the editing space A2-1 as indicated by the white arrow # 2, or move to the editing space A2-1 as indicated by the white arrow # 3. Exit the shooting space A1 and move to the editing space A2-2.

The editing space A2-1 is an editing space on the front side of the editing unit 12. On the other hand, the editing space A2-2 is an editing space on the back side of the editing unit 12. Whether the user moves to the editing space A2-1 or the editing space A2-2 is guided by the screen display of the touch panel monitor of the photographing unit 21 or the like. For example, the vacant one of the two editing spaces is guided as the destination. The user who has moved to the editing space A2-1 or the editing space A2-2 starts the editing work. The user of the editing space A2-1 and the user of the editing space A2-2 can perform editing work at the same time.

After the editing work is completed, printing of the edited image is started. When printing is started, the user who has completed the editing work in the editing space A2-1 moves from the editing space A2-1 to the printing waiting space A3 as indicated by the white arrow # 4. Further, the user who has completed the editing work in the editing space A2-2 moves from the editing space A2-2 to the print waiting space A3 as indicated by the white arrow # 5.

The user who has moved to the print waiting space A3 waits for printing to finish. When printing is completed, the user receives the sticker paper discharged from the discharge port provided on the right side surface of the editing unit 12, and ends a series of photo sticker creation games.

Next, the configuration of each device will be described.

<Structure of preselection unit>
FIG. 4 is a diagram showing a configuration example of the preselection unit 20.

A touch panel monitor 71 is provided on the upper side of the preselection unit 20. The touch panel monitor 71 is composed of a monitor such as an LCD (Liquid Crystal Display) and a touch panel laminated on the monitor. The touch panel monitor 71 has a function of displaying various GUIs (Graphical User Interfaces) and accepting a user's selection operation. The touch panel monitor 71 displays a screen used for selecting the number of users.

A speaker 72 is provided below the touch panel monitor 71. The speaker 72 outputs a guidance voice for preselection processing, BGM (Back Ground Music), a sound effect, and the like. A coin insertion / return port 73 for inserting coins by the user is provided adjacent to the speaker 72.

<Structure of shooting unit>
FIG. 5 is a diagram showing a configuration example of the front surface of the photographing unit 21. The photographing unit 21 is configured to be surrounded by the side panel 41A, the side panel 41B, and the front panel 42.

A camera unit 81 is provided slightly above the center of the front panel 42. A camera 91, a touch panel monitor 92, and a front strobe 93 are provided on the front surface of the camera unit 81.

The camera 91 is, for example, a single-lens reflex camera, and is mounted inside the camera unit 81 so that the lens is exposed. The camera 91 has an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and photographs a user in the photographing space A1. The camera 91 functions as a shooting unit that shoots with the user as a subject.

The moving image captured by the camera 91 (hereinafter, also referred to as a live view image) is displayed on the touch panel monitor 92 in real time. A still image captured by the camera 91 at a predetermined timing such as when shooting is instructed is saved as a shooting image.

The touch panel monitor 92 is provided below the camera 91. The touch panel monitor 92 is composed of a monitor such as an LCD and a touch panel laminated on the monitor. The touch panel monitor 92 has a function as a live view monitor that displays a moving image captured by the camera 91, and a function that displays various GUIs and accepts a user's selection operation.

The front strobe 93 is provided above the camera 91. The front strobe 93 has a crescent-shaped light emitting surface made of a milky white acrylic plate. The front strobe 93 emits light in accordance with the shooting by the camera 91 to illuminate the vicinity of the user's face as a subject from the front.

Above the camera unit 81, an upper strobe unit 82 with a curved light emitting surface facing the user is installed. The upper strobe unit 82 irradiates the user's face and upper body with light from above the front of the user.

Below the camera unit 81, a foot strobe unit 83 that irradiates the lower body and feet of the user with light is provided.

The upper strobe unit 82 and the foot strobe unit 83 are composed of a fluorescent lamp and a lighting device capable of strobe light emission. The upper strobe unit 82 and the foot strobe unit 83 brighten the shooting space, constantly illuminate the shooting space in order to ensure user operability and safety, and improve the image quality of the shot image. The strobe fires at the shooting timing to illuminate the user who is the subject.

A box-shaped portion 84 is provided between the foot strobe unit 83 and the side panel 41A. Further, a box-shaped portion 85 is provided between the foot strobe unit 83 and the side panel 41B. The upper surfaces of the box-shaped portions 84 and 85 form a surface substantially horizontal to the installation surface of the photo sticker making device 1, and are used as a luggage storage place for a user who performs photography work to put baggage and the like.

Although not shown, a speaker is provided near the ceiling of the front panel 42, for example. The speaker outputs guidance voice, BGM, sound effect, etc. for shooting processing.

<Structure of background part>
FIG. 6 is a diagram showing a configuration example of the background portion 22 on the photographing space A1 side.

A strobe 101 on the back surface is installed above the back panel 51. The strobe 101 on the back surface irradiates the user with light from above the back surface.

In the figure, the rear left strobe 102 is installed on the left side of the rear panel 51. The rear left strobe 102 irradiates the user from the rear right side. In the figure, the rear right strobe 103 is installed on the right side of the rear panel 51. The rear right strobe 103 irradiates the user from the rear left side.

Further, the chroma key sheet 121 is attached to the surface of the back panel 51 on the shooting space A1 side (front side in the drawing). The color of the chroma key sheet 121 is, for example, green.

Although not shown, the chroma key sheet is also attached to the lower side (floor surface side) of the surface of the side panels 52A and 52B on the photographing space A1 side in the same manner as the chroma key sheet 121.

FIG. 7 is a perspective view showing the floor material 27. For convenience of explanation, illustrations of the photographing unit 21, the side panel 41A, and the like are omitted.

A horizontally long rectangular floor sheet 27A is attached to a position near the rear of the floor material 27 so that its long side is substantially parallel to the back panel 51. In FIG. 7, the direction in which the back panel 51 is located is behind the photographing space A1, and the direction in which the photographing unit 21 is indicated by the white arrow is in front of the photographing space A1.

Mark figures 28-1 to 28-4 are printed side by side on the front side of the floor sheet 27A, and mark figures 28-5 to 28-8 are printed side by side behind the mark figures 28-1 to 28-4. The marker figures 28-1 to 28-8 arranged in 2 rows × 4 columns are visually different figures, and the whole is filled with black, for example.

The mark figure 28-1 on the first line is a diamond-shaped mark, and the mark figure 28-2 is a heart-shaped mark. The mark figure 28-3 is a four-leaf cloverleaf mark, and the mark figure 28-4 is a square mark.

Further, the mark figure 28-5 on the second line is a flower-shaped mark composed of four petals, and the mark figure 28-6 is a star-shaped mark. The mark figure 28-7 is a circular mark, and the mark figure 28-8 is a tulip petal-shaped mark.

Each mark figure has a size that fits within a range of approximately 30 cm × 30 cm so that one user can put both feet on it. Each size may be different.

As will be described in detail later, the mark figures 28-1 to 28-8 are used as marks for guiding the standing position at the time of shooting for each user.

For example, when the number of users is two and an image showing a large upper body is taken, the four-leaf cloverleaf marker figure 28-is set as the standing position of the user standing on the right side of the camera 91. 3 is designated, and a heart-shaped marker figure 28-2 is designated as the standing position of the user standing on the left side.

The marker figures 28-1 to 28-8 are printed at positions where the optimum standing position can be specified according to the number of users, the angle of view (range) of the image to be captured, and the like. .. The optimum standing position means a position where the light from the strobe is sufficiently irradiated to the subject, a position which does not deviate from the shooting range of the camera 91, a position where the balance between the background and the size of the face is improved, and the like.

As the mark figure, a figure with a color other than black, a figure with a predetermined pattern, or a figure with a color and a pattern may be used. It is also possible to change at least one of the shape, size, pattern, and color for each marker figure. The mark figures 28-1 to 28-8 shown in FIG. 7 have the same color as black, but have different shapes.

<Structure of editing unit>
FIG. 8 is a diagram showing a configuration example of the front side (editing space A2-1 side) of the editing unit 12.

A monitor with a built-in tablet 131 is provided at substantially the center of the slope 62. A touch pen 132A is provided on the left side of the tablet built-in monitor 131. A stylus 132B is provided on the right side of the tablet built-in monitor 131.

The tablet built-in monitor 131 is configured by stacking a tablet and a display. The tablet enables operation input using the touch pen 132A or the touch pen 132B. The tablet built-in monitor 131 displays, for example, an editing screen used for editing work. When two users edit at the same time, the touch pen 132A is used by the user on the left side of the tablet built-in monitor 131, and the touch pen 132B is used by the user on the right side of the tablet built-in monitor 131. ..

The operation using the touch pen 132A and the operation using the touch pen 132B are distinguished. As appropriate, only the operation using the touch pen 132A is possible for the display on the left side of the tablet built-in monitor 131, and only the operation using the touch pen 132B is possible for the display on the right side. Hereinafter, when it is not necessary to distinguish between the touch pen 132A and the touch pen 132B as appropriate, they are collectively referred to as the touch pen 132.

FIG. 9 is a diagram showing a configuration example of the left side surface of the editing unit 12.

A sticker paper discharge port 161 is provided on the lower side of the left side surface of the editing unit 12. A printer is provided inside the editing unit 12. The printer prints an image of the user of the editing space A2-1 or an image of the user of the editing space A2-2 on the sticker paper in a predetermined layout and discharges the image from the sticker paper ejection port 161.

<Example of sticker paper>
FIG. 10 is a diagram showing an example of a sticker paper produced by the photo sticker creating device 1.

When there are two users, for example, two stickers with photographs (edited images) printed in the layout shown in FIG. 10 are created and provided to the users. The layout shown in FIG. 10 is one of a plurality of types of layouts, and the user can select a desired layout.

An edited image printing area 181A having a size of a business card and a horizontally long strip-shaped extra image printing area 181B are formed on one sticker paper. The edited image 191-1 is printed in a large size on the substantially left half of the edited image print area 181A, and four edited images 191-2 to 191-5 are printed on the right side of the edited image 191-1 in two lines with a size smaller than that of the edited image 191-1. × Printed side by side in 2 columns.

The edited images 191-1 to 191-5 are images generated by editing the close-up image, which is an image in which the face and upper body of the subject are largely captured. Actually, a background image which is an image of a predetermined color or pattern is combined with the background of the user as appropriate, and a foreground image such as a handwritten character or a stamp image is combined with the foreground of the user.

In the shooting process, in addition to the close-up image, an image showing a wide range from above the head to near the knee is also shot. This image is used, for example, to preserve the coordination of clothes. Hereinafter, an image having a wider shooting range than the close-up image, which is used to keep the coordination of clothes as appropriate, is referred to as a coordinated image.

The aspect ratio of the close-up image and the coordinated image are different. For example, the ratio of the horizontal length to the vertical length of the close-up image is 1: 1.2, and the ratio of the horizontal length to the vertical length of the coordinated image is 1: 1.5. In this example, the coordinated image is not printed on the sticker paper.

In the bonus image printing area 181B, photographed images 192-1 to 192-5 trimmed into a simple shape similar to the marker figures 28-1 to 28-8 printed on the floor sheet 27A are printed side by side. To. The image printed in the bonus image print area 181B is used by cutting it with scissors, for example.

Instead of the image before editing, the edited image (the same image as the edited images 191-1 to 191-5) may be trimmed and printed as captured images 192-1 to 192-5.

Bonus A sticker ID or the like is printed in the margin of the image print area 181B. The sticker ID is used when viewing an image transmitted from the photo sticker creation device 1 to the server on a mobile terminal such as a smartphone.

The image is transmitted to the mobile terminal via a server that manages an image acquisition site, which is a website for importing (storing) the image transmitted from the photo sticker creating device 1 into the mobile terminal. The user can post the image captured on the mobile terminal to SNS (Social Networking Service) or set it as wallpaper for use. Hereinafter, the server that manages the image acquisition site is referred to as an image acquisition site management server as appropriate.

FIG. 11 is a diagram showing an example of the layer structure of the edited image.

The leftmost edited image P1 is an image generated by editing. The edited image P1 is composed of images of each layer of the background image P11, the person image P12, and the foreground image P13.

In the example of FIG. 11, the background image P11 is an image of the pattern of the wall surface on which bricks are stacked. The foreground image P13 is an image including an image of handwritten characters of "Love my friend" and a heart-shaped image. The handwritten image of "Love my friend" is a pen image input by the user using the pen tool. The heart-shaped image is a stamp image input by the user using the stamp tool.

As described above, the edited image P1 is generated by superimposing the person image P12, which is an image of the area of the person extracted from the captured image by chroma key processing, on the background image P11, and superimposing the foreground image P13 on the background image P11. It is also possible to input a pen image or a stamp image on the background image P11 and display it on the background of the subject. Foreground images and background images may be combined with the captured image, and even after editing, they may not be combined depending on the user's operation.

<Internal configuration of photo sticker making device>
FIG. 12 is a block diagram showing a configuration example of the photo sticker creating device 1. In FIG. 12, the same components as those described above are designated by the same reference numerals. Duplicate explanations will be omitted as appropriate.

As shown in FIG. 12, in the photo sticker creating device 1, the preselection unit 302, the photographing unit 304, the editing units 305A and 305B, and the printing unit 306 are external input / output interfaces 303 with respect to the PC (Personal Computer) unit 301. It is configured by being connected via. The PC unit 301, which is an information processing unit, is housed inside the housing of the editing unit 12, for example.

The CPU (Central Processing Unit) 311, ROM (Read Only Memory) 312, and RAM (Random Access Memory) 313 constituting the PC unit 301 are connected to each other by a bus 314.

The CPU 311 executes a predetermined program and controls the overall operation of the photo sticker creating device 1. The ROM 312 stores programs and data executed by the CPU 311. The RAM 312 temporarily stores data and programs processed by the CPU 311.

An input / output interface 315 is further connected to the bus 314. A storage unit 316, a communication unit 317, and a drive 318 are connected to the input / output interface 315.

The storage unit 316 is a non-volatile storage medium such as a hard disk or a flash memory. The storage unit 316 stores various setting information and the like supplied from the CPU 311. The information stored in the storage unit 316 is appropriately read out by the CPU 311.

The communication unit 317 is an interface of a network such as the Internet. The communication unit 317 communicates with an external device according to the control by the CPU 311. The communication unit 317 transmits the image selected by the user to, for example, an image acquisition site management server managed by the manufacturer of the photo sticker creation device 1. The image transmitted from the communication unit 317 is allocated and stored in a predetermined storage area, and is displayed on a mobile terminal that has accessed the image acquisition site management server, or downloaded to the mobile terminal.

A removable media 319 made of an optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 318. The programs and data read from the removable media 319 by the drive 318 are supplied to the CPU 311 and stored or installed in the storage unit 316.

An external input / output interface 303 is connected to the input / output interface 315. The control of each unit by the PC unit 301 is performed via the external input / output interface 303.

The preselection unit 302 realizes the preselection process for the user in the preselection space A0. The preselection unit 302 includes a touch panel monitor 71, a speaker 72, and a coin processing unit 321.

The touch panel monitor 71 displays various selection screens according to the control by the PC unit 301, and accepts the user's operation on the selection screen. An input signal representing the content of the user's operation is supplied to the PC unit 301, and various settings are made.

The coin processing unit 321 detects the insertion of coins into the coin insertion / return port 73. When the coin processing unit 321 detects that a predetermined amount of coins has been inserted, the coin processing unit 321 outputs an activation signal instructing the start of the game to the PC unit 301.

The shooting unit 304 realizes shooting processing for the user in the shooting space A1. The photographing unit 304 includes a lighting device 331, a camera 91, a touch panel monitor 92, and a speaker 332.

The lighting device 331 is each strobe unit in the photographing space A1 and emits light according to the lighting control signal supplied from the PC unit 301.

The camera 91 shoots according to the shutter control by the PC unit 301, and outputs the captured image (image data) obtained by the shooting to the PC unit 301.

The editorial unit 305A realizes an editing process for a user in the editing space A2-1. The editorial unit 305A includes a tablet built-in monitor 131, touch pens 132A and 132B, and a speaker 341. The editing unit 305B realizes editing processing for the user in the editing space A2-2, and has the same configuration as the editing unit 305A. Hereinafter, when the editorial units 305A and 305B are not particularly distinguished, they are simply referred to as the editorial unit 305.

The tablet built-in monitor 131 displays the edit screen according to the control by the PC unit 301, and accepts the user's operation on the edit screen. An input signal representing the content of the user's operation is supplied to the PC unit 301, and the captured image to be edited is edited.

The printing unit 306 is configured to include a printer 351. A sticker paper unit 352 is attached to the printer 351.

The printer 351 drives the built-in head 361 and prints an image on the roll-shaped sticker paper 363 stored in the sticker paper unit 352 based on the print data supplied from the PC unit 301. Further, the printer 351 cuts the sticker paper 363 on which the image is printed at a predetermined cut position by the cutter 362, and discharges the sticker paper 361 to the sticker paper discharge port 161.

<Example of functional configuration of photo sticker creation device>
FIG. 13 is a block diagram showing a functional configuration example of the photo sticker creating device 1. At least a part of the functional units shown in FIG. 13 is realized by executing a predetermined program by the CPU 311.

In the photo sticker creating device 1, a preselection processing unit 401, a shooting processing unit 402, an editing processing unit 403, and a printing processing unit 404 are realized.

The preselection processing unit 401 executes the preselection processing by controlling each unit of the preselection unit 302. By the pre-selection process, the number of users is selected, the background image is selected, and the like. Information indicating the content selected by the user is supplied to the photographing processing unit 402.

The photographing processing unit 402 performs imaging processing by controlling each unit of the photographing unit 304. The shooting processing unit 402 outputs the data of the shot image obtained by performing the shooting processing to the editing processing unit 403. Data of a plurality of captured images obtained by performing imaging a plurality of times is supplied from the photographing processing unit 402 to the editing processing unit 403.

The editing processing unit 403 performs editing processing by controlling each unit of the editing unit 305. The editing process includes a process of adjusting the size of the user's eyes and the brightness of the skin in the captured image, a process of editing the captured image, and the like.

The editing process in the game provided by the photo sticker creating device 1 also includes a process of generating a moving image using a photographed image. This moving image is composed of each frame image obtained by synthesizing a captured image which is a still image, and is a moving image having a short reproduction time such as 7 seconds.

After the game is over, the user can access the image acquisition site management server, import the moving image into his / her mobile terminal, and post it on SNS for use. Hereinafter, the moving image generated by using the captured image as appropriate is referred to as a portable mini moving image.

After finishing editing of the captured image, the editing processing unit 403 generates print data by arranging the edited image in a predetermined layout and outputs the print data to the print processing unit 404.

The print processing unit 404 outputs print data to the printer 351 and prints the edited image.

<Configuration example of editing processing unit>
FIG. 14 is a block diagram showing a functional configuration example of the editing processing unit 403.

The edit processing unit 403 includes a display control unit 431, an edit control unit 432, a communication control unit 433, and a retouch processing unit 434.

The display control unit 431 controls the display of the tablet built-in monitor 131. For example, the display control unit 431 causes the tablet built-in monitor 131 to display a selection screen for selecting the degree of image processing applied to the captured image and an editing screen for editing work on the captured image.

The editing control unit 432 edits the captured image according to the user's operation on the editing screen. Specifically, the editing control unit 432 synthesizes the compositing image selected by the user on the editing screen at a predetermined position of the captured image.

The communication control unit 433 controls the communication unit 317 and performs processing related to communication via a network such as the Internet. For example, the communication control unit 433 controls the communication unit 317 to transmit the captured image obtained in the photographing process and the edited image obtained in the editing process to the image acquisition site management server.

The retouch processing unit 434 performs retouch processing on the photographed image obtained in the photographing process. The retouching process is an image process for processing or modifying a face or a part thereof in a captured image.

Next, an example of the functional configuration of the retouching processing unit 434 will be described with reference to FIG.

The retouch processing unit 434 includes a recognition unit 451, a first image processing unit 452, an area specifying unit 453, and a second image processing unit 454.

The recognition unit 451 recognizes a face in a captured image or a part of the face including an organ (face part) thereof. A part of the face includes facial parts such as eyes, nose, mouth and ears, as well as cheeks, chin, forehead and hair. The recognition unit 451 recognizes a face or a face part by extracting contour points of a portion to be recognized and determining whether or not a region connecting them is a face or a face part. The number of contour points to be extracted is arbitrary, and may be changed depending on the recognized portion.

Further, the face or the face part may be recognized by detecting the color of the portion to be recognized. For example, the face is recognized by whether or not the skin color is detected, and the eyes are recognized by whether or not white and black are detected. It is sufficient that the portion (for example, eyes) to be finally retouched is recognized, and only the eyes may be detected without detecting the face or other facial parts.

Further, the neck, chest, arms, hands, and the like may be recognized in addition to the face and a part of the face.

The first image processing unit 452 performs image processing that realizes retouching processing on the region of the face or the face part recognized by the recognition unit 451. For example, the first image processing unit 452 performs image processing for enlarging the eyes on the recognized eye area.

Here, the automatic retouching is performed by the image processing performed by the first image processing unit 452, that is, the retouching processing realized by the image processing performed by recognizing the face or the face part regardless of the user's operation. It is called processing.

The area specifying unit 453 identifies the same area as the area of the face or the face part that is the target of the automatic retouching process in the captured image by the operation of the user. For example, when the target of the automatic retouching process is the eye area, the area specifying unit 453 identifies the eye area in the captured image by the user's operation.

The second image processing unit 454 performs image processing that realizes retouching processing on the area of the face or part specified by the area specifying unit 453. For example, the second image processing unit 454 performs image processing for enlarging the eyes on the eye area specified by the area specifying unit 453.

Here, the image processing performed by the second image processing unit 454, that is, the retouching process realized by the image processing performed by specifying the face or the face part by the operation of the user is referred to as a manual retouching process.

The target of image processing by the first image processing unit 452 and the second image processing unit 454 is not limited to one face part, but may be a plurality of face parts. Further, the image processing target may include not only the area of the face part but also the area around the face part. This makes it possible to perform more natural processing on the area including the face part, and even if the position of the recognized / specified face part shifts, the processing is performed without being greatly affected by it. Can be applied.

<Operation of photo sticker making device>
Here, the operation of the photo sticker creation device 1 that provides the photo sticker creation game will be described with reference to the flowchart of FIG.

In step S1, the preselection processing unit 401 determines whether or not a predetermined amount of coins has been inserted based on the activation signal supplied from the coin processing unit 321 and waits until it is determined that coins have been inserted. do.

If it is determined in step S1 that a coin has been inserted, the process proceeds to step S2. In step S2, the preselection processing unit 401 performs the preselection processing by controlling the preselection unit 302. Specifically, the preselection processing unit 401 is obtained by a number of people course that determines the type of game to be played according to the number of users as a course related to shooting performed in the shooting space, or a shooting performed in the shooting space. Various settings are made by letting the user select a background image to be the background of the captured image.

In step S3, the photographing processing unit 402 performs the photographing process by controlling the photographing unit 304. Specifically, the shooting processing unit 402 displays a moving image of the subject captured by the camera 91 in a live view on the touch panel monitor 92, shoots the user in the shooting space A1 as the subject, and generates a shot image.

In step S4, the editing processing unit 403 controls the editing unit 305 corresponding to the editing space of the editing space A2-1 and the editing space A2-2, which is the destination of the user who has completed the shooting process. Then, edit it. Specifically, the editing processing unit 403 generates an edited image by causing the user to perform editing work on the captured image obtained by the photographing process.

In step S5, the print processing unit 404 performs (starts) printing processing by controlling the printer 351. Specifically, the print processing unit 404 outputs the edited image obtained by the editing process to the printer 351 and prints it on the sticker paper. The photographed image obtained by the photographing process may be printed on the sticker paper.

When printing is completed, in step S6, the printer 351 discharges the sticker paper on which the image is printed to the sticker paper discharge port 161 and ends the process.

<Details of editing process>
Next, with reference to the flowchart of FIG. 17, the details of the editing process of step S4 in the series of processes of the above-mentioned photo sticker creation game will be described.

When the editing process is started, in step S31, the display control unit 431 displays the eye size / brightness selection screen on the tablet built-in monitor 131.

The eye size / brightness selection screen is a screen used for selecting the user's eye size and skin brightness in the captured image.

When the eye size and the skin brightness are selected on the eye size / brightness selection screen, the process proceeds to step S32.

In step S32, the display control unit 431 causes the tablet built-in monitor 131 to display the name input screen.

The name input screen is a screen used for a user to input a name. On the name input screen, a button for selecting hiragana that can be used for entering a name and a text box displaying a character string corresponding to the selected button are displayed.

When the name input is completed on the name input screen, the process proceeds to step S33.

In step S34, the display control unit 431 causes the tablet built-in monitor 131 to display the edit screen.

The edit screen is a screen used for editing a photographed image to be edited. On the editing screen, the captured image to be edited and various editing tools (composite images) such as stamp images and pen images used for editing are displayed.

The user uses the touch pens 132A and 132B to superimpose the stamp image, pen image, etc. on a predetermined position on the captured image to be edited, thereby synthesizing the composite image with the captured image to be edited. Will be.

That is, in step S34, the editing control unit 432 edits the captured image to be edited according to the operation of the user.

When the editing work is completed, in step S35, the display control unit 431 displays the division number selection screen on the tablet built-in monitor 131.

The division number selection screen is a screen used for selecting the division number and the seal layout of the sticker paper to be finally output. The sticker layout is a layout (arrangement pattern) that determines the number, position, and size of edited images (photographed images) arranged on one photo sticker. Here, an image in which the edited image is arranged according to the sticker layout selected by the user, that is, an image printed on the sticker paper and finally output is called a layout image.

Therefore, on the division number selection screen, for example, a plurality of layout images in which the edited images edited for each of the five captured images are arranged in different sticker layouts are displayed.

When any of the layout images is selected on the division number selection screen, the process proceeds to step S36.

In step S36, the display control unit 431 causes the tablet built-in monitor 131 to display the mini game screen.

The mini-game screen is a screen for playing a game for hitting (selecting) a moving image transmitted to a mobile terminal owned by a user, in addition to the image for mobile transmission described later. Although detailed description is omitted, this moving image is generated by using a plurality of captured images obtained by the imaging process.

When the game is played on the mini-game screen and the moving image to be transmitted to the mobile terminal is determined, the process proceeds to step S37.

In step S37, the display control unit 431 displays the image selection screen for mobile transmission on the tablet built-in monitor 131.

The image selection screen for mobile transmission is a screen used for selecting an image for mobile transmission to be transmitted to a mobile terminal owned by a user. On the image selection screen for mobile transmission, in addition to the five captured images, an image dedicated to mobile transmission with predetermined editing is displayed on any one of them.

When any of the images is selected on the image selection screen for mobile transmission, the process proceeds to step S38.

In step S38, the display control unit 431 displays the e-mail address input screen on the tablet built-in monitor 131.

The e-mail address input screen is a screen used for inputting the e-mail address of the mobile terminal owned by the user. The email address input screen displays alphabets and symbols that can be used for the email address, buttons for selecting the carrier of the mobile phone, and a text box that displays the character string corresponding to the selected button. ..

When the input of the e-mail address is completed on the e-mail address input screen, the process proceeds to step S39.

In step S39, the communication control unit 433 controls the communication unit 317 to display an image for mobile transmission selected by the user via a network such as the Internet, together with an e-mail address input by the user. Acquisition Send to the site management server.

The image acquisition site management server sends an e-mail containing the URL (Uniform Resource Locator) required for the mobile terminal to access the image acquisition site management server to the e-mail address entered by the user. Then, based on the URL, the mobile terminal owned by the user accesses the image acquisition site management server via the network, so that the mobile transmission image transmitted to the image acquisition site management server is provided to the mobile terminal. Will be done.

Then, in step S40, the editing processing unit 403 guides the user who has completed the editing work to move to the print waiting space A3. Guidance for moving to the print waiting space A3 is performed by displaying a guidance screen on the tablet built-in monitor 131 of the editorial unit 305 or by outputting voice from the speaker 341.

The editing process is performed as described above.

<Automatic retouch processing>
Face parts in the captured image are recognized for the captured image to be edited as described above, and automatic retouching processing is executed according to the result.

Here, the automatic retouching process will be described with reference to the flowchart of FIG. The timing at which this process is executed may be after the shooting process, for example, while the shooting work is finished and the user is moving to the editing space, or while the editing work is being performed.

In step S51, the recognition unit 451 recognizes a face part, specifically an eye, in the captured image.

In step S52, the recognition unit 451 determines whether or not the eye recognition is successful. If it is determined that the eye recognition is successful, the process proceeds to step S53.

In step S53, the first image processing unit 452 performs image processing on the region of the eyes that has been successfully recognized, specifically, processing for enlarging the eyes. For example, this process is performed as a process of changing the size of the eyes reflected in the captured image to the size of the eyes selected on the above-mentioned eye size / brightness selection screen.

On the other hand, if the eye recognition is not successful (eye recognition fails) in step S52, step S53 is skipped and no image processing is performed.

For example, when eye recognition is performed by extracting two points, the outer corner of the eye and the inner corner of the eye, as contour points of the eye, the recognition of the eye may fail when the eye in the captured image is in the following state.
(1) Hair on the outer or inner corners of the eyes (2) Wearing glasses (3) Closed eyes (4) Face is not facing the camera and eyes are not visible

In such a case, even if the user selects the desired eye size on the eye size / brightness selection screen, the captured image is not automatically retouched, and as a result, the degree of satisfaction is satisfied. High images may not be provided.

Therefore, in the present embodiment, it is possible to provide an image with a high degree of satisfaction even in the above-mentioned cases, especially in the cases (1) and (2).

For example, on the edit screen, a process of enlarging the eyes is performed on the area of the eyes that failed to be recognized.

<Example of edit screen>
The edit screen is basically configured by providing the main configurations symmetrically. The left half area is an area used by the user on the left side of the tablet built-in monitor 131. The right half area is an area used by the user on the right side of the tablet built-in monitor 131. With such an editing screen configuration, mainly two users can perform editing work at the same time.

In FIG. 19, the left half area (left half area) of the editing screen is shown.

A thumbnail display area 511 is provided in the upper center of the left half area. The thumbnail display area 511 is a display area for a thumbnail image representing a captured image. The user selects the captured image to be edited by selecting the thumbnail image displayed in the thumbnail display area 511.

In the example of FIG. 19, thumbnail images representing five captured images are displayed side by side in the thumbnail display area 511.

An editing area 512 is provided substantially in the center of the left half area. The editing area 512 is a display area of a captured image selected as an editing target. The user selects an editing tool using the touch pen 132A and edits the captured image displayed in the editing area 512.

The edit palette 513 is displayed at the bottom of the left half area. The editing palette 513 is used to select various editing tools (composite images) such as pen images and stamp images.

In the editing palette 513, each editing tool is prepared for each tab. The user switches the editing tool displayed on the editing palette 513 by selecting each tab. For example, a tab named "Stamp" is a tab that is manipulated when selecting a stamp image, and a tab named "Pen" is manipulated when selecting a pen image. It's a tab.

A manual retouching area 514 and a make palette 515 are provided on the left side of the editing area 512.

The manual retouching area 514 is a display area of a button for selecting the size of the eyes in the captured image to be edited in order to execute the manual retouching process for the eye area.

The manual retouch area 514 is provided with a "small enlargement" button 521, a "large enlargement" button 522, and a "from the beginning" button 523. The "small enlargement" button 521 is a button operated to enlarge the eyes at a predetermined enlargement ratio. The "large enlargement" button 522 is a button operated to enlarge the eyes at a magnification larger than that of the "small enlargement" button 521. The "from the beginning" button 523 is a button operated to return the size of the eyes enlarged by the "small enlargement" button 521 or the "large enlargement" button 522 to the original size.

When the edit screen is first displayed on the tablet built-in monitor 131 (default state), the "from the beginning" button 523 is grayed out, and the "small enlargement" button 521 and the "large enlargement" button 522 are displayed. When any of them is operated, the grayout display is canceled.

In the example of FIG. 19, two buttons, a "small enlargement" button 521 and a "large enlargement" button 522, are provided as buttons operated to enlarge the eyes, but they are simply a predetermined enlargement ratio. There may be only one button operated to magnify the eyes with.

The make palette 515 is a display area for an icon for selecting a make tool. The make-up tool is an editing tool for making the user's face in the photographed image look like a make-up. When the make-up tool is selected, the face parts in the captured image are subjected to image processing corresponding to the selected make-up tool. The image processing corresponding to the make-up tool is performed by the user's selection operation for the make-up palette 515 (specifically, pressing the "two-person simultaneous" button or the "one-person at a time" button provided on the make-up palette 515). It may be applied to each user who appears in the captured image at the same time, or it may be applied to each user.

A mobile bonus button 516 is provided on the right side of the editing area 512. When the mobile phone bonus button 516 is operated, a screen for creating a mobile phone bonus (mobile phone bonus creation screen) is displayed on the editing screen, and the user can create a mobile phone bonus. The mobile bonus is used, for example, as a standby image of a mobile terminal or as an image published on a website such as a blog or SNS.

Now, when either the "smaller enlargement" button 521 or the "larger enlargement" button 522 is operated in the manual retouching area 514, the function of the manual retouching process is performed on the editing area 512 as shown in FIG. A guidance pop-up 530 is displayed to explain.

The guidance pop-up 530 displays words and images explaining the function of the manual retouching process, a "use" button 531 and a "not use" button 532.

In the example of FIG. 20, when the “not used” button 532 is pressed, the guidance pop-up 530 is closed and the editing screen transitions to the display state shown in FIG.

On the other hand, when the "use" button 531 is pressed, or when either the "smaller enlargement" button 521 or the "larger enlargement" button 522 is pressed for the first time in the manual retouch area 514, the edit is performed as shown in FIG. A usage pop-up 540 explaining how to use the manual retouching processing function is displayed on the area 512.

The usage pop-up 540 displays words and images explaining how to use the manual retouching function. Specifically, an explanation that the touch pen 132 touches the left and right of the eye to draw a line and an animation in which the pen tip moves from the inner corner of the eye to the outer corner of the eye in the model image and a line is drawn are displayed.

In the state of FIG. 21, when the close button 541 provided on the upper left of the usage pop-up 540 is pressed or the area other than the usage pop-up 540 on the editing screen is pressed, the usage pop-up 540 is closed and the editing screen is displayed. , The display state shown in FIG. 19 is changed, and the manual retouching processing function can be used.

As described above, according to the manual retouching processing function, by specifying the eye region in the captured image with the touch pen 132, the processing of enlarging the eyes is performed with respect to the region.

<Manual retouching process>
Here, the manual retouching process will be described with reference to the flowchart of FIG.

In step S71, the area specifying unit 453 determines whether or not the area of the face part has been specified. Specifically, the area specifying unit 453 determines whether or not a predetermined area has been designated by the operation of the user's touch pen 132.

The process of step S71 is repeated until a predetermined area is designated by the operation of the user's touch pen 132. Then, when a predetermined area is designated by the operation of the user's touch pen 132, the area specifying unit 453 specifies the designated area as the area of the face part, and the process proceeds to step S72.

In step S72, the second image processing unit 454 performs image processing on the area specified by the area specifying unit 453.

For example, as shown in FIG. 23, it is assumed that a line 551 from the start point SP to the end point EP is drawn by the touch pen 132 on the captured image. When the length of the line 551 is L, the area specifying portion 453 is a line having a length L parallel to the line 551 at a distance of L / 4 in two directions (vertical direction in the figure) perpendicular to the line 551. Set the minutes. Then, the area specifying portion 453 specifies a rectangular area 552 having a long side length L and a short side length L / 2 as a face part area.

Then, the second image processing unit 454 performs an enlargement / movement process of enlarging / moving the pixels of each area divided by the line 551 only in the direction away from the line 551 in the rectangular area 552 in the rectangular area 552. Apply to.

The upper limit of the length of the line 551 that can be drawn by the touch pen 132 is, for example, 15% of the width of the editing area 512 on which the captured image is displayed. This value is a value required when the face is extremely large in the captured image. By setting an upper limit on the length of the drawable line 551, the possibility that an unnatural image is created can be reduced.

Further, since the above-mentioned enlargement / movement processing is not performed on the line 551 having a length that is determined to be a point, the lower limit of the length of the drawable line 551 is set to about several pixels. The upper and lower limits of the length of the drawable line 551 are the shooting range of the shot image, in other words, the shot image is a close-up image or a whole-body image such as a coordinated image showing almost the whole body of a person. It may take different values depending on the type.

By such processing, as shown on the left side of FIG. 24, the area of the eye on which the line 551 is drawn from the inner corner of the eye to the outer corner of the eye by the stylus 132 becomes larger as shown on the right side of FIG. 24. Change.

The enlargement ratio / movement amount of the pixels in the above-mentioned enlargement / movement processing is different between when the "small enlargement" button 521 is pressed and when the "large enlargement" button 522 is pressed. Specifically, the enlargement ratio and the amount of movement of the pixels are larger when the "large enlargement" button 522 is pressed than when the "small enlargement" button 521 is pressed.

According to the above processing, even if the recognition of the face part fails, the image processing is performed by specifying the area of the face part by the touch pen 132, so that the user can be more sure. In addition, it is possible to provide an image with a high degree of satisfaction.

<Regarding the identification of the area>
In the above, the area of the face part is specified by drawing a line with the touch pen 132, but simply, for example, a plurality of points such as two points (two points of the inner and outer corners of the eye) are designated by the touch pen 132. The area of the face part may be specified based on the line segment connecting them.

Further, as shown in FIG. 25, the rectangular area 561 having a diagonal line connecting the point P1 first designated by the touch pen 132 and the point P2 designated next is specified as the area of the face part. You may do so.

Further, the area of the face part may be specified by estimating the position of the face part by tracing the contour of the face in the captured image on the screen with the touch pen 132.

Further, after the area of the face part is specified, the position designated by the touch pen 132 in the captured image on the screen may be the center of the area of the face part. In this case, you may be able to move the area of the face part by dragging at the center of the area, or you can provide a cross key (direction key) on the screen and operate the cross key to move the face part. You may be able to move the area of. Further, an enlargement / reduction button may be provided on the screen so that the area of the face part can be enlarged / reduced.

In addition, a slider having a scale indicating the size of the area of the face part is provided at a predetermined position on the screen, and the size of the area of the face part is changed stepwise by operating the slider. Alternatively, a plurality of buttons displaying a figure indicating the size of the area of the face part may be provided so that the size of the area of the face part changes stepwise according to the selected button. good.

Further, a finger may be used instead of the touch pen 132 to operate the screen. In this case, the size of the area of the face part may be adjusted by an operation such as pinch-in or pinch-out. Specifically, the pinch-out operation increases the area of the face part, and the pinch-in operation reduces the area of the face part. Further, the size of the area of the face part may be adjusted according to the movement direction of the touch pen 132 or the finger on the screen.

The manual retouching process function can also be used for the area of the face part that has been successfully recognized. Therefore, in the captured image on the screen, the area of the face part may be specified by touching the vicinity of the recognized face part with the touch pen 132 or a finger. Further, a button corresponding to each face part may be provided on the screen so that the area of the face part can be specified according to the selected button.

<Other examples of retouching processing>
In the above, an example in which a process of enlarging the eyes is performed as the retouching process has been described, but a process other than the process of enlarging the eyes may be performed. For example, as the retouching process, a process of making the eyes smaller, a process of changing the shape of the eyes into a sagging eye, a hanging eye, or the like, or a process of changing the position of the entire eye may be performed.

Further, when the function of the manual retouching process is used, a process of changing the sharpness of the eye area may be performed in addition to the process of enlarging the eyes. For example, when the "small enlargement" button 521 is pressed, the sharpness is set to be weak, and when the "large enlargement" button 522 is pressed, the sharpness is set to be strong. As a result, it becomes possible to suppress the uncomfortable appearance that occurs only by enlarging the eyes.

Further, if the black eye area can be specified in the eye area, a process of enlarging the black eye area may be further performed. This makes it possible to make the catch light reflected in the black eyes stand out.

In addition to the retouching process for the eyes, the retouching process for the nose, mouth, and face may be performed.

For example, as a retouching process for the nose, a process for highlighting the nasal muscles is performed. In this case, a process of increasing the brightness of the nasal muscle portion may be performed, or a process of synthesizing an image imitating a highlight may be performed on the nasal muscle portion.

In addition, as a retouching process for the mouth, a process for thickening or thinning the lips is performed. In addition, a process may be applied so that the open mouth is changed to the closed mouth.

Further, as a retouching process for the face, a process for making the face smaller is performed. In this case, the process of moving the area of the contour portion of the face to the center of the face is performed.

The manual retouching process (process of FIG. 22) may be executed every time the area of the face part is specified. However, if the eyes become infinitely large, the face appearing in the captured image becomes unnatural, so that an upper limit may be set for the number of processes. In this case, when the line 551 is drawn by the touch pen 132 in excess of the upper limit number of times, or when the "small enlargement" button 521 or the "large enlargement" button 522 is operated in a state where the upper limit number of times is exceeded, the manual retouching process is further performed. Display a pop-up indicating that the function cannot be used.

The maximum number of manual retouching processes may differ depending on the shooting range of the shot image, in other words, whether the shot image is a close-up image or a whole-body image such as a coordinated image that shows almost the entire body of a person. good. Specifically, in the full-body image, the area of the face part in the entire image is smaller and less conspicuous than in the close-up image. May be reduced.

Further, the manual retouching process may not be performed again on the area where the manual retouching process has already been performed (once), or on the area where the automatic retouching process (process of FIG. 18) has already been performed. May not be applied.

<Other examples of manual retouching processing>
(Example 1)
As a method of specifying the area of the face part, the area of the face part may be specified for each face part in the captured image. Specifically, as described above, the buttons corresponding to each of the face parts are displayed on, for example, the edit palette 513. For example, when the button corresponding to the eye is pressed, the eye area is subjected to a process of enlarging the eye. In this case, the eye area shall be identified by successful recognition or estimated from the positional relationship of each face part in the area of the entire face. As a result, regardless of whether the eye recognition is successful or unsuccessful, the manual retouching process can be performed without the trouble of drawing a line connecting the inner and outer corners of the eye with the stylus 132. Become.

(Example 2)
The area of the face part may be specified for each face in the captured image. Specifically, apart from the buttons corresponding to the face parts, a button for manually retouching the target for each user in the captured image and a button for all users are displayed. To do so. When the button for targeting the manual retouching process to each user is pressed, the manual retouching process is performed only for one selected user, and the target of the manual retouching process is set to all users. When the button for is pressed, all users are manually retouched.

(Example 3)
The enlargement ratio of the face parts may be changed according to the proportion of the face in the entire captured image. Specifically, when the face is large in the captured image, the enlargement ratio of the face parts is increased, and when the face is small, the enlargement ratio of the face parts is decreased. As a result, the larger the face in the captured image, the more the size of the eyes that determines the impression of the entire captured image can be emphasized.

When a plurality of users are shown in the captured image, the enlargement ratio of the face parts may be changed according to the average of the sizes of the faces.

Further, as the manual retouching process, in addition to the process of enlarging the eyes, a process of correcting the color tone such as the brightness, saturation, and contrast of the eye area may be performed. In this case, buttons indicating various color tones are provided so that the color tones in the eye area are corrected to the color tones corresponding to the selected buttons. As a result, the captured image can be finished into a more ideal image.

(Example 4)
In the recognition of the eyes, the success of the recognition of both eyes may be judged as the success of the recognition.

Here, if only one eye is successfully recognized, only the other eye that fails to be recognized may be the target of the manual retouching process. In this case, when the "small enlargement" button 521 or the "large enlargement" button 522 is operated in the manual retouch area 514, the area of the eye that has been successfully recognized is displayed in grayscale, and the display of that area is changed. You may do it. This makes it possible for the user to understand that the eye that has been successfully recognized is not the target of the manual retouching process.

Also, if the area of one eye is identified and the other facial parts such as the nose and mouth are successfully recognized when the recognition of both eyes fails, the other is determined from the positional relationship between them. The area of the eye may be specified (estimated). In this state, the lengths between the inner and outer corners of the eyes are compared for both eyes on the captured image, and if the lengths are clearly different, one of the regions may not be the region of the eyes. Therefore, the area may be excluded from the target of the manual retouching process.

(Example 5)
If the recognition of the entire face is performed with the success of the recognition of the face parts, even if the recognition of the face parts fails, the area is specified so that the recognition of the entire face is successful. May be good.

For example, when the recognition of the eyes fails and the recognition of the entire face cannot be performed, it is assumed that the recognition of the entire face is successful by specifying the area of the eyes. As a result, the positions of the face parts other than the eyes are estimated in the entire area of the face, so that it is possible to perform processing such as changing the color of the lips on the face parts other than the eyes, for example, the lips.

<Retouch processing on image acquisition site management server / mobile terminal>
The retouching process may be performed not only on the photo sticker creating device 1 but also on the image acquisition site management server or the mobile terminal.

When the retouching process is performed on the image acquisition site management server, the parameters of the manual retouching process performed on the photo sticker creation device 1 and the captured image without any processing are transmitted to the image acquisition site management server. To do so. The image acquisition site management server performs retouching processing on the captured image using the parameters. At this time, the image acquisition site management server may use parameters different from the parameters from the photo sticker creation device 1.

When the retouching process is performed on the mobile terminal, the retouching process similar to the manual retouching process performed on the photo sticker creating device 1 may be performed, or the manual retouching performed on the photo sticker creating device 1 may be performed. The retouching process for the captured image may be performed independently of the process.

<About the number of people>
The manual retouching process may be executed according to the number of users in the captured image.

For example, if there are four or more users in the captured image, or more than a predetermined number of users, the faces of the users may overlap and the face parts may not be recognized, so the manual retouching process is executed. To do so.

On the contrary, when the number of users in the captured image is less than a predetermined number, such as two or less, the proportion of the face in the captured image becomes high, and the impression of the face affects the appearance of the entire image. The manual retouching process may be executed.

Further, the manual retouching process may be executed only when the number of users shown in the captured image is a specific number such as two.

<About the shooting range>
In the shooting process, when a close-up image showing the face and upper body of a person and a full-body image showing the whole body of a person are generated as shot images, manual retouching is performed according to the shooting range. The process may be executed.

For example, in a full-body image, the proportion of the face in the image is lower than that in the close-up image, so the manual retouching process should not be executed for the full-body image, and the manual retouching process should be executed only for the close-up image. To.

In addition to the close-up image and the whole body image, when the above-knee image showing the person above the knee is generated as the captured image, the manual retouching process is not executed for the above-knee image. It may be executed or it may be executed.

<About special shooting>
In addition to the normal shooting performed in the above-mentioned shooting process, a user can shoot a moving image, shoot to obtain a collage image in which multiple shot images are arranged and combined, and shoot a single shot with multiple users as subjects. Special shooting (special shooting) such as shooting for obtaining a shot image for each shot may be performed.

Such special shooting is intended to give a more interesting or special feeling than to improve the appearance of the face in the image. Therefore, the manual retouching process is not executed for the image obtained by the special shooting, and the manual retouching process is executed only for the captured image obtained by the normal shooting.

<Modification example of automatic retouch processing>
In the above, it is assumed that the automatic retouching process is executed according to the eye size selected in the editing process, but even if the eye size is selected in the preselection process or the shooting process. good.

Here, in the shooting process, for example, five shots are taken, and among the five shot images obtained, the eyes of two users are recognized in the four shot images, and one shot is taken. It is assumed that the eyes of the two users are not recognized in the image. In this case, in one captured image, the recognition of the eyes of at least one user has failed. In this case, for the user who failed to recognize the eyes, the area of the eyes is estimated based on the face parts recognized other than the eyes. As a result, even if the eye recognition fails, the automatic retouching process is performed.

Further, in this case, the editing screen may be displayed so that the user can understand that the captured image has failed to recognize the eyes.

Further, the automatic retouching process may be applied to the live view image displayed at the time of shooting. In this case, guidance may be output to the user so that the face part to be processed is correctly recognized.

In the above, on the edit screen, the processing to enlarge the eyes is applied to the area of the eyes that failed to be recognized. In addition to this, when the sizes of the left and right eyes are different, the left and right eyes are processed. The processing may be performed to make the size of the eyes uniform.

<Other examples of edit screen>
FIG. 26 is a diagram showing another example of the edit screen.

In the edit screen of FIG. 26, the thumbnail display area 611, the edit area 612, and the edit palette 613 have the same functions as the thumbnail display area 511, the edit area 512, and the edit palette 513 in the edit screen of FIG. ..

A manual retouching area 614 is provided on the left side of the editing area 612.

The manual retouching area 614 is a display area of a button for changing the size of the eyes in the captured image to be edited in order to execute the manual retouching process for the eye area.

The manual retouch area 614 is provided with a "fix" button 621 and 622 and an OFF button 623.

The "fix" button 621 is a button operated to make the size of the left and right eyes the same.

When the "fix" button 621 is operated, as shown in FIG. 27, a usage pop-up 630 explaining the function of the manual retouching process for making the left and right eyes the same size is displayed on the editing area 612. ..

The usage pop-up 630 displays words and images explaining how to use the manual retouching function that makes the left and right eyes the same size. Specifically, the message "Please touch the smaller eye!" And the model image with the left and right eyes of the model person enlarged are displayed. The model image of FIG. 27 is displayed as an animation showing how the tip of the touch pen 132 is touched by the smaller eye portion of the left and right eyes of the model.

In the state of FIG. 27, when the close button 631 provided on the upper right of the usage pop-up 630 is pressed, the usage pop-up 630 is closed.

The "fix" button 622 is a button operated to enlarge the eyes.

When the "fix" button 622 is operated, as shown in FIG. 28, a usage pop-up 640 explaining the function of the manual retouching process for enlarging the eyes is displayed on the editing area 612.

The usage pop-up 640 displays words and images explaining how to use the function of the manual retouching process that enlarges the eyes. Specifically, the message "Draw a line on the unrecognized eyes" and the model image with the left and right eyes of the model person enlarged are displayed. The model image of FIG. 28 is displayed as an animation showing how the tip of the touch pen 132 moves from the inner corner of the eye to the outer corner of the eye of the left and right eyes of the model, which fails to recognize, and a line is drawn.

In the state of FIG. 28, when the close button 641 provided on the upper right of the usage pop-up 640 is pressed, the usage pop-up 640 is closed.

As described above, on the editing screen of FIG. 26, two processing functions, a manual retouching process for making the left and right eyes the same size and a manual retouching process for enlarging the eyes that failed to be recognized, can be used. ..

Therefore, in this example, even when the usage pop-up 630 and the usage pop-up 640 are displayed, it is possible to accept operations for realizing each function. As a result, after selecting either the "fix" button 621 or the "fix" button 622, the user can use the touch pen 132 to perform which operation (touch the smaller eye or the eye that fails to recognize). Do you draw a line from the inner corner of the eye to the outer corner of the eye?) Without making a mistake. That is, in the above-mentioned two types of manual retouching processing, the operation for realizing the function is different for each processing. Therefore, by displaying the usage pop-up corresponding to the selected process, the user can perform the operation to realize the desired function without making a mistake while checking the usage pop-up.

In addition, in the state where the usage pop-up 630 and the usage pop-up 640 are displayed, the usage pop-up 630 and 640 are both in the lower center of the editing area 612 so that the operation for the eye area reflected in the captured image of the editing area 612 is possible. It is displayed in the area of.

In addition to the usage pop-up that explains how to use the above-mentioned two types of manual retouching processing functions, a usage pop-up that explains how to use other processing functions executed by operations on the edit screen is displayed. You may do so. For example, a usage pop-up explaining how to use the function of the selected makeup tool for facial parts such as eyes, lips, and cheeks, and a usage pop-up explaining how to use the image processing function that changes the shape of the eyes are displayed. You may do so. In this case, these usage pop-ups are also displayed in an area that does not hide the face or eye area so that the corresponding processing area can be operated.

The OFF button 623 is a button for canceling (invalidating) all the processing performed by operating the "fix" button 621 and the "fix" button 622.

In the default state of the edit screen, the OFF button 623 is grayed out as shown in FIG. 26, and when either the "fix" button 621 or the "fix" button 622 is operated, FIG. 27 or As shown in FIG. 28, the grayout display is canceled.

When the "back" button 615 provided at the lower right of the editing area 612 is operated, the most recent operation is performed by operating the "fix" button 621 and the "fix" button 622. Only one process performed by the button is canceled.

For example, by operating the "fix" button 621, a manual retouching process for making the left and right eye sizes the same for one user appearing in the captured image is performed, and then the "fix" button 622 is pressed. It is assumed that the manual retouching process for enlarging the eyes that failed to be recognized is applied to the other user in the captured image by being operated.

When the OFF button 623 is operated in this state, the manual retouching process for making the left and right eye sizes the same for one user and the recognition for the other user are performed. The manual retouching process that enlarges the failed eye is canceled.

On the other hand, when the "back" button 615 is operated, only the manual retouching process for enlarging the eyes that failed to be recognized by the most recently operated button is canceled. When the "back" button 615 is further operated, only the manual retouching process for making the left and right eyes the same size applied to one user by the previously operated button is released.

The model image displayed in the usage pop-up 630 of FIG. 27 and the usage pop-up 640 of FIG. 28 may be an enlarged image of the user's eye portion cut out from the captured image. In this case, the enlarged image of each eye portion may be displayed, and the user corresponding to the selected enlarged image may be the target of the manual retouching process.

<Details of image processing in manual retouching processing>
Here, the details of the image processing in the above-mentioned two types of manual retouching processing will be described.

In the manual retouching process in which the sizes of the left and right eyes are made the same, the area of the eye recognized in the captured image is touched by the touch pen 132, so that the contour point of the eye is, for example, based on the center point of the eye. Processing is performed to move it away from the center point. Of course, the present invention is not limited to this, and it is sufficient to perform a process of enlarging the area of the eye touched by the touch pen 132, such as a process of enlarging the area of the eye surrounded by connecting the contour points of the eyes. ..

If an area other than the eye area is touched by the touch pen 132 after the "fix" button 621 is operated, no processing is performed.

In the manual retouching process for enlarging the eyes that failed to be recognized, as shown in FIG. 29, a line 651 is drawn from the inner corner of the eye (start point P11) to the outer corner of the eye (end point P12) by the touch pen 132 on the captured image. Then, the process described with reference to FIG. 23 is performed. In the example of FIG. 29, the line 651 is drawn, and the icon 652 imitating the eyes with the start point P11 as the inner corner of the eye and the end point P12 as the outer corner of the eye is displayed.

The line 651 drawn from the start point P11 to the end point P12 is valid only when drawn in the horizontal direction (horizontal direction) on the editing area 612, and is valid in other directions (vertical direction or diagonal direction). If it is drawn, it will be invalid.

However, some users take pictures with their faces tilted, and the photographed images obtained in that state show the user's face and, by extension, their eyes tilted. In such a case, it is not preferable to set the above-mentioned limitation on the drawn line 651 in the manual retouching process for enlarging the eyes that failed to be recognized.

Therefore, as shown in FIG. 30, a limit is set on the angle formed by the horizontal axis (x axis) and the line 651 drawn from the start point P11 toward the end point P12 on the two-dimensional plane with the start point P11 as the origin. Specifically, when the angle formed by the line 651 and the horizontal axis (x axis) is in the range of 0 ° to 40 °, 130 ° to 210 °, 320 ° to 360 °, the line 651 is effective. To do. That is, when the angle formed by the line 651 and the horizontal axis (x axis) is in the range of 41 ° to 129 °, 211 ° to 319 °, the line 651 becomes invalid, and if it is not drawn, the eyes are closed. No enlargement processing is applied.

This makes it possible to use the manual retouching processing function for enlarging the eyes that have failed to be recognized even for the captured image in which the user's eyes are tilted.

Further, also in this example, the upper limit number of manual retouching processes is set. In this case, regardless of the type of manual retouching process (whether the left and right eyes are the same size or the eyes that failed to be recognized are enlarged), if the total number of processes exceeds the upper limit, this is the case. A pop-up indicating that the manual retouching processing function cannot be used is displayed.

Here, the upper limit of the number of manual retouching processes shall differ depending on the number of people course that determines the type of game to be played according to the number of users.

For example, when a two-person course that provides a game suitable for two users is selected as the number of people course, once for each of the left and right eye areas of the two users. The upper limit of the manual retouching process is set to 4 so that the manual retouching process is allowed only. In addition, when a course of 3 or more people who provides a game suitable for 3 or more users is selected as the number of people course, 1 for each of the left and right eye areas of a maximum of 5 users. The upper limit of the manual retouching process is set to 10 so that the manual retouching process is allowed only once.

In the above-described embodiment, the photo sticker creating device 1 is configured to print the obtained image on a sticker paper and send it to the image acquisition site management server to provide it to the user's mobile terminal. .. Not limited to this, it is also possible to adopt a configuration in which the image is only provided to the user's mobile terminal by transmitting the image to the image acquisition site management server without printing it on the sticker paper. On the contrary, it is also possible to adopt a configuration in which the image is only printed on the sticker paper without being transmitted to the image acquisition site management server.

That is, the present technology can be applied not only to a photo sticker making device for making a photo sticker in which an image is printed on a sticker paper, but also to a photo making game machine that simply provides a game for making an image.

The series of processes described above can be executed by hardware or software. When the series of processes described above is executed by software, the programs constituting the software are installed from a network or a recording medium.

This recording medium is, for example, as shown in FIG. 12, a magnetic disk on which the program is recorded, which is distributed to distribute the program to the administrator of the photo sticker creation device 1 separately from the device main body. It is composed of a removable medium 319 including a flexible disk), an optical disk (including a CD-ROM and a DVD), a magneto-optical disk, or a semiconductor memory. In addition to these, the recording medium is composed of a ROM 312 in which a program to be distributed to the administrator is recorded in a state of being preliminarily incorporated in the main body of the apparatus, a hard disk included in the storage unit 316, and the like.

In the present specification, the steps for describing a program to be recorded on a recording medium are not only processed in chronological order in the order described, but also in parallel or in parallel, even if they are not necessarily processed in chronological order. It also includes processes that are executed individually.

In the above, the print medium is not limited to the sticker paper and the photo paper. For example, the image may be printed on a predetermined size of paper or film, a card such as a prepaid card or an IC (Integrated Circuit) card, or a cloth such as a T-shirt. In this case, a sticker layout image in which one or a plurality of captured images are arranged may be printed on these print media.

Further, the embodiment of the present technique is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technique.

1 Photo sticker creation device 401 Preselection processing unit 402 Imaging processing unit 403 Editing processing unit 404 Printing processing unit 431 Display control unit 432 Editing control unit 433 Communication control unit 434 Retouching processing unit 451 Recognition unit 452 First image processing unit 453 Area Specific part 454 Second image processing part

Claims (10)

A recognition unit that recognizes a face or a part of it in a captured image,
A first image processing unit that performs first image processing on the face or a part of the area that has been successfully recognized without any operation by the user .
In the captured image, the area specifying portion that specifies the area of the face or a part thereof that has not been subjected to the first image processing due to the recognition failure due to the operation of the user, and the area specifying portion.
A photo-creating game machine including a second image processing unit that performs a second image processing on the face or a part of the region specified by the region specifying unit. The recognition unit recognizes the face part in the captured image, and the recognition unit recognizes the face part.
The first image processing unit performs the first image processing on the region of the face part that has been successfully recognized.
The area specifying portion identifies the area of the face part in the captured image, and determines the area of the face part.
The photo-creating game machine according to claim 1, wherein the second image processing unit performs the second image processing on the area of the face part specified by the area specifying unit. The photo-creating game machine according to claim 2, wherein the area specifying unit specifies an area designated by the operation of the user in the photographed image as an area of the face part. The photo-creating game machine according to claim 3, wherein the second image processing unit does not perform the second image processing on the area where the second image processing has already been performed. The photo-creating game machine according to claim 3 or 4, wherein the second image processing unit does not perform the second image processing in the area where the first image processing has already been performed. The face parts are eyes,
The photo-creating game machine according to any one of claims 2 to 5, wherein the first and second image processing are processes for enlarging the area of the eyes. The photo-creating game machine according to any one of claims 2 to 6, wherein the area specifying unit specifies an area of the face part for each face in the captured image. The photo-creating game machine according to any one of claims 2 to 6, wherein the area specifying unit specifies an area of the face part for each face part in the photographed image. As the photographed image, a photographing processing unit for generating a close-up image in which the face and upper body of the person is captured and a whole-body image in which the whole body of the person is captured is further provided.
The photo-creating game machine according to any one of claims 2 to 8, wherein the area specifying unit specifies an area of the face part in the photographed image when the photographed image is the close-up image. Recognize the face or a part of it in the captured image
The first image processing is applied to the face or a part of the area that has been successfully recognized, without any operation by the user .
In the captured image, the area of the face or a part thereof , which was not subjected to the first image processing due to the failure in recognition due to the operation of the user, was identified.
An image processing method including a step of performing a second image processing on the identified face or a part thereof.

Images