JP2012142772A - Image processing system and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suitably correct shapes of faces of respective persons when a plurality of persons are in a photographed image.SOLUTION: It is determined whether the respective faces are matched with an overlapping condition. An image matched with the overlapping condition is not made to be an object of an image processing for deforming an outline of the face. An image which is not matched with the overlapping condition is made as the object of the image processing. An image which is matched with the overlapping condition and is not the object of the image processing is selected again. Then, determination whether the image is matched with the overlapping condition is repeated. The overlapping condition is the condition that a maximum outline which is set in a selected face crosses the outline of the other face or not. This invention can be applied to a photograph seal creation device.

Description

  The present invention relates to an image processing device and an image processing method, and in particular, when a plurality of people are shown in a photographed image, an image processing device capable of suitably correcting the shape of each person's face, and The present invention relates to an image processing method.

  2. Description of the Related Art A photo sticker creation apparatus is known that takes a picture of a user, causes the user to edit the photographed image, and prints and provides the edited image on sticker paper. Such a photo sticker creating apparatus is installed in an entertainment facility.

  The flow of one game provided by the photo sticker creation device usually shoots a user in the shooting booth as a subject, moves the user to the editing booth, and edits the image according to the operations performed in the editing booth. Then, the edited image, which is an edited image, is printed on a sticker sheet and discharged. Editing of the photographed image is performed by adding a stamp image to the photographed image using a stamp tool or inputting handwritten characters using a pen tool to decorate the photographed image.

  By the way, in recent years, there is a photo sticker creating apparatus equipped with a function for generating a good-looking shot image by performing image processing. The image processing here is different from the editing processing performed at the editing booth in which some image is added to the photographed image to decorate it, and the process for correcting the person (user of the photo sticker creating apparatus) itself that appears in the photographed image It is.

  The image processing of the photographed image includes a process of changing the size of the eyes to make it look larger and a process of changing the skin color to make the skin look beautiful. In addition, there is a process of deforming the jaw line so as to be a small face, and a process of acquiring a face line and deforming a face area of a portion excluding the eyes based on the line to make a small face.

  For example, the process of changing the size of the eyes and the process of changing the skin color are disclosed in Patent Document 1, and the process of deforming the chin line is disclosed in Patent Document 2. Further, Patent Document 3 discloses a process for deforming a face area excluding eyes.

JP 2005-175580 A JP 2010-50501 A JP 2010-61646 A

  Various image processing functions are provided in order to provide a good-looking photographed image. For example, the technique disclosed in Patent Document 1 is processing for eye size and skin color, Since the face contour is not deformed, the image may be unbalanced when the entire face is viewed. Further, since the user has to perform individual operations, the operability is not excellent.

  Also, depending on the technique disclosed in Patent Document 2, for example, when there are a plurality of users, the jaw line of the face to be subjected to image processing is deformed without considering the relationship with the faces of other users. As a result, when the faces of all the people after image processing are viewed, the image may become unbalanced. Furthermore, how many faces are included in the captured image is acquired by the face detection process, and the process is repeated for the number of acquired persons. In this respect, it cannot be said that a captured image that is satisfactory for all users can be provided.

  Depending on the technique disclosed in Patent Document 3, when there are a plurality of people appearing in a photographed image, both of which are within a predetermined distance between the boundary line of a range where one person is captured and the boundary line of a range where another person is captured Neither is subject to image processing, and no processing is disclosed when the number of people in the captured image is three or more. There are many cases in which a group of three people approach each other to shoot a photo sticker creation device. In such a case, assuming that no one's face is subject to image processing, this is also important for all users. This means that a satisfactory photographed image cannot be provided. Furthermore, since it is not subject to image processing when approaching, it is subject to image processing when leaving, so there is a difference in the face of the same person between multiple captured images generated in the same game. , It may provide images that look bad, or may make the user uncomfortable.

  The present invention has been made in view of such a situation, and in the case where a plurality of people are shown in a photographed image, the shape of each person's face can be suitably corrected. .

  An image processing apparatus according to an aspect of the present invention includes a face detection unit that detects a person's face in a photographed image, and a line that represents a maximum contour size that can be deformed on the person's face detected from the photographed image. When there are a plurality of human faces detected by the face detecting means and a setting means for setting a maximum contour line, the maximum contour line set to a predetermined person's face among the faces of the plurality of persons If it is determined whether or not the contour line intersects with the contour line of the face of another person, and if it does not intersect, the range of the size represented by the maximum contour line for the face of the predetermined person If the image processing for deforming the contour is performed so as to have a predetermined size within the image, and the face deformation processing unit does not perform the image processing and the face deformation processing unit does not perform the image processing, the face of the predetermined person Every time the image processing is performed It characterized by having a selection means for selecting a face of another person as the face of the predetermined person to be processed in the face deformation processing means.

  The face detecting means detects the face of a person in the photographed image, calculates the probability that the object detected as the person's face is a person's face, and the selecting means includes the face detecting means The face of the predetermined person to be processed can be selected in descending order of the probability obtained by the above.

  The selection means determines that the maximum contour line set for the faces of all of the plurality of persons intersects with the contour line of the face of the other person except for the face to be subjected to the image processing. Up to this point, the selection of the face of the predetermined person to be processed can be repeated.

  An image processing method according to an aspect of the present invention detects a person's face in a photographed image, and a maximum contour that is a line representing a maximum contour size that can be deformed on the person's face detected from the photographed image. When a line is set and there are a plurality of detected human faces, the maximum contour line set for the face of the predetermined person among the faces of the plurality of persons is the contour line of the face of another person. Determining whether or not to intersect, if it is determined not to intersect, the contour is deformed so that the face of the predetermined person becomes a predetermined size within the size range represented by the maximum contour line When the image processing is performed and the image processing is performed, and the image processing is performed on the face of the predetermined person, the predetermined person whose face is another person's face is processed each time the image processing is performed. Selecting as a face.

  In one aspect of the present invention, the face of a person appearing in a photographed image is detected, and a maximum contour line that is a line representing the maximum deformable contour size is set on the face of the person detected from the photographed image. Is done. In addition, when there are a plurality of detected human faces, the maximum contour line set for the face of the predetermined person among the plurality of human faces intersects with the contour lines of the faces of other persons. If it is determined whether or not to intersect, the contour of the predetermined person's face is deformed so as to have a predetermined size within the range of the size represented by the maximum contour line. When it is determined that the image processing is performed and the image processing is performed, the image processing is not performed. Each time the image processing is performed on the face of the predetermined person, the face of another person is selected as the face of the predetermined person to be processed.

  According to the present invention, when a plurality of people are shown in a photographed image, the shape of each person's face can be suitably corrected.

It is a perspective view which shows the structural example of a photograph sticker production apparatus. It is the perspective view which looked at the photograph sticker production apparatus from another angle. It is a figure explaining a user's movement. It is a figure which shows the structural example of the front of the front unit of an imaging | photography unit. It is a figure which shows the structural example of the side surface of an edit unit. It is a figure which shows the structural example of the other side of an edit unit. It is a block diagram which shows the example of an internal structure of a photograph sticker production apparatus. It is a block diagram which shows the function structural example of the control part of FIG. It is a block diagram which shows the function structural example of the imaging | photography process part of FIG. It is a figure which shows the example of a picked-up image. It is a figure which shows the other example of a picked-up image. It is a figure which shows the further another example of a picked-up image. It is a figure which shows the example of a picked-up image. It is a figure which shows the example of the position of the face reflected in a picked-up image. It is a figure which shows the other example of the position of the face reflected in a picked-up image. It is a figure which shows the further another example of the position of the face reflected in a picked-up image. It is a figure which shows the example of the position of the face reflected in a picked-up image. It is a figure which shows the other example of the position of the face reflected in a picked-up image. It is a figure which shows the further another example of the position of the face reflected in a picked-up image. It is a flowchart explaining the photograph sticker creation game process of a photograph sticker creation apparatus. It is a flowchart explaining the imaging | photography process performed in step S3 of FIG. It is a flowchart explaining the face deformation process performed in step S16 of FIG. It is a figure which shows the specific example of an image process. It is another figure which shows the specific example of an image process. It is another figure which shows the specific example of an image process. It is a figure explaining the effect of image processing. It is another figure explaining the effect of image processing.

[Configuration of photo sticker creation device]
FIG. 1 is a perspective view showing a configuration example of a photographic sticker creating apparatus 1.

  The photo sticker creating apparatus 1 is a game machine (image processing apparatus) that allows a user to perform operations such as shooting and editing as a game and receives a price from the user instead of providing a shot image or an edited image. The photo sticker creating apparatus 1 is installed in a store such as a game center. There may be one user or a plurality of users.

  In the photo sticker creation game provided by the photo sticker creation device 1, the user enters a price, takes a picture of himself as a subject, selects an image to be edited from the shot images obtained by the shooting, By editing the image to be edited using an editing function for synthesizing a frame image as a background or foreground, a handwritten line drawing, or a stamp image, the photographed image becomes a colorful image. After the game, the user receives a sticker sheet on which the edited image is printed.

  As shown in FIG. 1, the housing 10 of the photo sticker creating apparatus 1 mainly includes a photographing unit 12, an editing unit 13, a ceiling strobe unit 14, and a background curtain unit 15.

  The photographing unit 12 is a unit having a function of photographing a user as a subject. Photographing is performed in a photographing space inside the photographing unit 12. The photographing unit 12 is roughly divided into a front unit 12A and a rear unit 12B.

  The front unit 12A is a unit on the front side of the shooting space, and performs processing related to shooting and receives operations by the user. An external monitor 16, a speaker 17, and touch pens 18a and 18b are provided on the side surface of the front unit 12A. A coin slot 55 is provided below the external monitor 16. The rear unit 12B is a wall-like unit on the rear side of the photographing space.

  The editing unit 13 is a unit provided with a configuration for a user to edit a captured image. A configuration for performing editing or the like is provided on each of the two surfaces of the editing unit 13 so that two sets of users can perform editing work simultaneously. The photo sticker creating apparatus 1 can execute a photo sticker creating game for a certain user and a photo sticker creating game for another user in parallel.

  The ceiling strobe unit 14 provided above the photographing space incorporates a strobe that emits a flash toward the photographing space in accordance with the photographing timing. A fluorescent lamp is arranged inside the ceiling strobe unit 14 and also functions as illumination of the photographing space.

  On the upper part of the rear unit 12B, a background curtain unit 15 in which a movable curtain (background curtain) used for the background is stored is provided. The background curtain unit 15 stores a plurality of winding curtains having different colors or patterns. The background curtain unit 15 operates in conjunction with a photographing function such as a camera or a strobe provided in the front unit 12A. At the time of photographing, for example, the curtain of the color selected by the user is lowered and the other curtains are wound. take. As for the background curtain, a plurality of curtains may be used as an elevating curtain as a single chroma key curtain, and a chroma key curtain is pasted on the back of the shooting space in advance, and a plurality of background images for synthesis are prepared. In the editing process, a background image desired by the user may be combined with the curtain portion.

  The external monitor 16 provided on the side surface of the front unit 12A is used for a user who is waiting outside the shooting space to use the shooting space next time when shooting is performed in the shooting space, or a photo sticker. For those who pass by the creation device 1, a video showing the progress of the game is displayed, a video (demonstration) that guides the flow of the game, how to play, etc. is displayed, and the photo sticker creation device 1 is installed Display advertisements for the stores and areas that are used. The external monitor 16 is a tablet built-in monitor, displays a screen related to pre-service processing, and accepts a selection operation by the user.

  The speaker 17 outputs sound that matches the display on the external monitor 16.

  The touch pens 18a and 18b are used by a user who performs pre-service work. The user can select various items by tapping the external monitor 16 with the touch pen 18 (touch pens 18a, 18b) or moving the external monitor 16 on the external monitor 16. By preparing two touch pens, it becomes possible for two users to simultaneously perform a selection operation in the pre-service operation.

  FIG. 2 is a perspective view of the photo sticker creating apparatus 1 as seen from another angle.

  The editing unit 13 is installed adjacent to the surface of the front unit 12A opposite to the surface on the rear unit 12B side (surface on the imaging space side). On the two side surfaces of the editing unit 13, a first editing interface 13A and a second editing interface 13B are provided for the user to edit the captured image.

  Here, the flow of the photo sticker creation game and the accompanying movement of the user will be described with reference to FIG. FIG. 3 is a plan view of the photo sticker creating apparatus 1 as viewed from above.

  A user (user A) who has paid for the coin insertion slot 55 in the pre-service space 31 in front of the external monitor 16 uses the touch pen 18 according to the screen displayed on the external monitor 16. Set up. In the prior customer service space 31, for example, selection as to whether or not to use a mobile transmission function that is a function of transmitting an image to a server is performed as a prior customer service.

  The user who has completed the customer service operation enters the imaging space 32 formed between the front unit 12A and the rear unit 12B, and performs the imaging operation using a camera or a monitor provided in the front unit 12A (user B).

  The user who has finished the shooting work selects an image to be edited from the shot images, and after selecting the image, for example, according to the movement guide displayed on the monitor of the front unit 12A, from the shooting space 32 to the editing space where the editing work is performed. Moving.

  A first editing space 33A is formed on the right side (in front of the first editing interface 13A) of the editing unit 13 as viewed from the front unit 12A, and a second editing space 33B is formed on the left side (in front of the second editing interface 13B). The Hereinafter, when it is not necessary to distinguish the first editing space 33A and the second editing space 33B as appropriate, they are collectively referred to as an editing space 33.

  The user (user B) of the shooting space 32 is guided to the free one of the two editing spaces when the shooting operation is finished. The user who has moved to the first editing space 33A or the second editing space 33B starts editing work. A user C who is a user of the first editing space 33A and a user C ′ who is a user of the second editing space 33B can simultaneously perform editing work.

  The user who has finished the editing work next performs a print setting work such as setting the number of divisions in the editing space 33. After the print setting work is completed, printing is started. During image printing, if the user has selected to use the mobile transmission function during customer service, the user can select an image to be stored on the server from among the edited images or play a mini game. Or When printing is completed, the user moves to the printed material receiving area 34 to receive the sticker paper, and finishes a series of photo sticker creation games.

  Next, the configuration of each unit will be described.

  FIG. 4 is a diagram illustrating a configuration example of the front unit 12A. A front surface 12 </ b> A- 1 in FIG. 4 is a surface that is positioned forward for a user who performs a photographing operation in the photographing space 32.

  A camera 51 that captures a subject in the imaging space 32 is provided in the approximate center of the front surface 12 </ b> A- 1, and a touch panel monitor 52 is provided below the camera 51.

  The front surface 12A-1 is provided with lighting devices 53-1 to 53-8 that emit light in accordance with the shooting timing of the camera 51, and a baggage storage area for a user who is shooting to put baggage or the like. 54-1 and 54-2 are provided on the left and right with the illumination device 53-7 under the touch panel monitor 52 interposed therebetween. For example, near the ceiling of the front surface 12 </ b> A- 1, a speaker that outputs sound such as guidance sound, BGM (Back Ground Music), and sound effects in the shooting work of the photo sticker creation game is also provided.

  The camera 51 is configured by an image sensor such as a CCD (Charge Coupled Device), and photographs the user in the photographing space 32. A part of the captured image captured by the camera 51 is cut out or an image such as a background is synthesized and displayed on the touch panel monitor 52 in real time. An image captured by the camera 51 at a predetermined timing such as when a user gives an instruction to shoot is saved as a photographic image (still image).

  The touch panel monitor 52 includes a live view monitor such as an LCD (Liquid Crystal Display) and a touch panel laminated thereon. The touch panel monitor 52 has a function as a live view monitor that displays a composite image (moving image) obtained by combining an image such as a background or foreground with a captured image captured by the camera 51, and various GUI (Graphical User Interface) images. And a function of accepting a user's selection operation by a touch panel superimposed on the screen.

  FIG. 5 is a diagram showing a configuration example of the editing unit 13 on the right side when viewed from the photographing unit 12 side (configuration example on the first editing space 33A side).

  The first editing interface 13A is provided with a tablet built-in monitor 62, touch pens 63-1, 63-2, and a speaker 64. A lighting device 61 is also provided above them.

  The tablet built-in monitor 62 is configured by superimposing a tablet capable of operation input using the touch pen 63-1 or 63-2 on a display device such as an LCD. On the tablet built-in monitor 62, for example, an edit screen that is a screen used for editing the image to be edited is displayed. When two people perform editing work simultaneously, the touch pen 63-1 is used by one user, and the touch pen 63-2 is used by the other user. Hereinafter, the touch pen 63-1 and the touch pen 63-2 are collectively referred to as a touch pen 63, as appropriate.

  The speaker 64 outputs voices related to editing work such as guidance voices, sound effects, and BGM. The configuration of the editing unit 13 on the second editing space 33B side is basically the same as the configuration on the first editing space 33A side shown in FIG.

  FIG. 6 is a diagram illustrating a configuration example of the photo sticker discharge unit 13 </ b> C of the editing unit 13.

  A photographic sticker discharge port 71 is provided in the photographic sticker discharge portion 13C. The photo sticker discharge port 71 is obtained by a sticker on which an image obtained by editing by the user of the first editing space 33A is printed and by a user of the second editing space 33B. The sticker paper on which the printed image is printed is discharged. A printer is provided inside the editing unit 13, and an image obtained by editing work is printed by the printer.

  Next, the internal configuration of the photo sticker creating apparatus 1 will be described. FIG. 7 is a block diagram illustrating an internal configuration example of the photo sticker creating apparatus 1. The same code | symbol is attached | subjected to the same structure as the structure mentioned above. The overlapping description will be omitted as appropriate.

  The control unit 101 controls the overall operation of the photo sticker creating apparatus 1. Connected to the control unit 101 are a storage unit 102, a communication unit 103, a drive 104, a ROM (Read Only Memory) 106, a RAM (Random Access Memory) 107, a pre-service unit 111, a photographing unit 112, an editing unit 113, and a printer 114. Is done.

  The storage unit 102 includes a non-volatile storage medium such as a hard disk or a flash memory, and stores various setting information supplied from the control unit 101. Information stored in the storage unit 102 is appropriately read out by the control unit 101.

  A communication unit 103 is an interface of a network such as the Internet, and communicates with an external device according to control by the control unit 101.

  A removable medium 105 made of an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a semiconductor memory, and the like is appropriately attached to the drive 104. The computer program and data read from the removable medium 105 by the drive 104 are supplied to the control unit 101 and stored in the storage unit 102 or installed.

  The ROM 106 stores programs and data executed by the control unit 101. The RAM 107 temporarily stores data and programs processed by the control unit 101.

  The pre-service section 111 is a block that performs pre-service processing, which is processing for causing the user to perform pre-service work, and includes an external monitor 16, a speaker 17, a touch pen 18, and a coin processing section 121.

  The external monitor 16 displays an image that guides the flow of the photo sticker creation game, how to play, and the like according to control by the control unit 101. In addition, the external monitor 16 displays a selection screen for determining whether or not to use the mobile transmission function after the price is inserted. A signal representing the content of the user's operation input to the external monitor 16 is supplied to the control unit 101.

  The coin processing unit 121 detects the insertion of coins into the coin insertion slot. When the coin processing unit 121 detects that a coin has been inserted, the coin processing unit 121 outputs an activation signal indicating that to the control unit 101.

  The photographing unit 112 is a block that performs photographing processing, which is processing for causing a user to perform photographing work, and includes a background curtain unit 15, an illumination control unit 122, a camera 51, a touch panel monitor 52, and a speaker 124. .

  The background curtain unit 15 raises and lowers the background curtain based on the background control signal supplied from the control unit 101. The background may be selected manually by the user.

  The illumination control unit 122 turns on or off the illumination devices 53-1 to 53-8 based on the illumination control signal supplied from the control unit 101.

  The camera 51 shoots according to the control by the control unit 101 and outputs image data obtained by the shooting to the control unit 101. In the photographing process, photographing is performed a plurality of times, and a predetermined number of images selected from the plurality of photographed images are stored in the storage unit 102 as editing target images.

  The editing unit 113 is a block that performs an editing process that is a process for causing the user to perform an editing operation. The editing unit 113 includes a first editing unit 113A corresponding to the first editing interface 13A and a second editing interface corresponding to the second editing interface 13B. 2 editing section 113B. Although the configuration of the first editing unit 113A will be described here, the second editing unit 113B also has the same configuration as the first editing unit 113A.

  The first editing unit 113A includes a tablet built-in monitor 62, touch pens 63-1, 63-2, and a speaker 64.

  The tablet built-in monitor 62 includes an editing monitor 141 and a tablet 142, displays an editing screen according to control by the control unit 101, and detects a user operation on the editing screen. A signal representing the content of the user's operation is supplied to the control unit 101, and the editing target image is edited. On the editing screen, the thumbnail image of the image to be edited stored in the storage unit 102 is displayed, and the image to be edited selected by tapping the thumbnail image is displayed in a larger size than the thumbnail image. The user switches the image to be edited to be displayed as appropriate, and inputs handwritten characters using the pen tool displayed on the editing screen or the stamp tool for the enlarged image to be edited. You can enter an image such as a heart pattern.

  The printer 114 prints the edited image obtained by the editing process on the sticker sheet 162 and discharges it to the photo sticker discharge port 71.

  FIG. 8 is a block diagram illustrating a functional configuration example of the control unit 101. The configuration shown in FIG. 8 is realized by the control unit 101 executing a predetermined program. In the control unit 101, a pre-service processing unit 201, a photographing processing unit 202, an editing processing unit 203, a print processing unit 204, and a post-service service processing unit 205 are realized.

  The pre-service processing unit 201 performs pre-service processing, which is processing related to a price (coin) input when starting the photo sticker creation game, processing related to setting before photographing, and the like. The photographing processing unit 202 performs photographing processing that is processing for photographing the user. The editing processing unit 203 performs editing processing such as editing on the captured image. The print processing unit 204 performs a printing process for printing an image obtained by the editing process on a sticker sheet. The post-service processing unit 205 performs post-service processing that provides a mini game or the like to a user who has finished editing work and is waiting for printing.

[About face deformation processing]
In the photo sticker creating apparatus 1 having the above-described configuration, face deformation processing, which is processing for deforming the face of a person appearing in a captured image, is performed on the captured image. The face deformation process is performed, for example, after shooting is finished and before editing is started. That is, in the editing process, the image after the face deformation process is used as an editing target image. The face deformation process may be performed according to a user instruction, or may be automatically performed by the photo sticker creating apparatus 1. A face deformation process is performed using an image obtained by photographing as an original image, and further, a vertically long shape, a horizontally long shape, and a square shape image are cut out, and the cut out image becomes an editing image.

  FIG. 9 is a block diagram illustrating a functional configuration example of the imaging processing unit 202. The imaging processing unit 202 includes an imaging control unit 211, a face recognition unit 212, a maximum contour line setting unit 213, and an image processing unit 214.

  The imaging control unit 211 captures a user in the imaging space 32 a plurality of times and outputs a captured image selected as an editing target image. The captured image output from the imaging control unit 211 is supplied to the face recognition unit 212, the maximum contour line setting unit 213, and the image processing unit 214, respectively.

  The face recognition unit 212 recognizes a face that appears in the captured image supplied from the imaging control unit 211. The face recognition unit 212 extracts, for example, a feature at each position of the object shown in the photographed image by analyzing the photographed image, and matches the extracted feature with a feature at each position of the human face prepared in advance. Face recognition. The face recognizing unit 212 recognizes an object having a high score representing a matching result among objects appearing in a captured image as a human face.

  FIG. 10 is a diagram illustrating an example of a captured image. In the example of FIG. 10, two persons are shown, and the faces of the persons are recognized as shown surrounded by frames F1 and F2. Hereinafter, the face of the left person shown surrounded by the frame F1 will be described as face 1, and the face of the right person shown surrounded by the frame F2 will be described as face 2.

  The face recognition unit 212 outputs information representing the position of each recognized face and information representing the position of each part of the face such as the face outline, eyes, nose, and mouth to the maximum contour line setting unit 213. Further, the face recognition unit 212 outputs the score obtained by matching to the image processing unit 214. The score (reliability) output to the image processing unit 214 represents the probability that the object recognized as a face is actually a face.

  The maximum contour line setting unit 213 specifies the contour of the face that appears in the captured image based on the information supplied from the face recognition unit 212, and the maximum contour range that can be deformed and enlarged to each face. The maximum contour line, which is a line representing, is set.

  As shown in FIG. 11, the case where the contour of the face 1 is represented by the contour line L1 and the contour of the face 2 is represented by the contour line L2 will be described. In this case, as shown in FIG. 12, the maximum contour line setting unit 213 has a maximum contour line L1max that represents the range of the maximum contour that can be deformed and expanded outside the contour line L1. Is set outside the contour line L2, and the maximum contour line L2max representing the maximum contour range in which the contour of the face 2 can be deformed and enlarged is set.

  For example, the maximum contour line uses the center of the face obtained from each part of the face such as eyes, nose, and mouth as the reference position, and the distance from the reference position to each position on the contour is a predetermined ratio outside the face. Only set by spreading. Instead of expanding the distance from the reference position to each position on the contour at the same rate, the rate of spreading in the vertical direction is larger than the rate of spreading in the horizontal direction so that the deformed face looks smaller. It may be. The maximum contour line setting unit 213 outputs information on the maximum contour line set for each face of the captured image to the image processing unit 214.

  Not only the maximum contour line but also the minimum contour line is set by narrowing the distance from the reference position to each position on the contour by a predetermined ratio inside the face, as indicated by a broken line in FIG. It may be. In this case, the face contour can be reduced not only to the maximum contour line range but also to the minimum contour line range by image processing. In some cases, it may be set to a line that gives a good-looking image that can be a maximum (minimum) deformation line in which a part that is partially inside and outside the face contour line is mixed. . For example, the line length may be set to be slightly outside the contour line above the cheek and below the lip to be slightly inside the contour line so as to be the surface length.

  Also, as shown by the alternate long and short dash line, a line substantially parallel to both lines is set between the contour line and the maximum contour line, and the range of the contour line can be expanded along this line. Also good.

  Based on the reliability supplied from the face recognition unit 212, the image processing unit 214 selects the face with the highest probability of being a human face, and determines whether the selected face meets the overlapping condition. judge.

  The overlapping condition is a condition as to whether or not the maximum contour line set for the selected face intersects with the contour line of another face. When the maximum contour line of the selected face intersects with the contour line of the other face, the image processing unit 214 determines that the overlapping condition is satisfied, and the maximum contour line of the selected face is If it does not intersect with the contour line, it is determined that the overlapping condition is not met.

  When the selected face matches the overlapping condition, the image processing unit 214 selects the face with the next highest reliability without setting the face as the target of image processing for deforming the contour, and sets the overlapping condition. It is determined whether or not it matches. In other words, when a face is deformed to enlarge its face, a face that is closer to another person's face or has a larger overlap with another person's face than before the deformation is subjected to image processing. The process proceeds with another face as the target, not the target.

  Here, for each pattern of the position of the face appearing in the photographed image, it will be described whether each face is an image processing target. Although the case where there are two faces in the captured image will be described, the same applies to the case where there are three or more faces.

  For example, as illustrated in FIG. 14, when the relationship between the face 1 and the face 2 is so far away that the maximum contour lines do not intersect, the image processing unit 214 selects either the face 1 or the face 2. Image processing is also performed on the target. In the example of FIG. 14, the maximum contour line L1max does not intersect with the contour line L2, and the face 1 does not meet the overlapping condition. Further, the maximum contour line L2max does not intersect with the contour line L1, and the face 2 does not meet the overlapping condition.

  As shown in FIG. 15, the relationship between the face 1 and the face 2 is such that the maximum contour lines intersect with each other, but the maximum contour line is so far away that it does not intersect with the other contour line. The processing unit 214 performs image processing on both the face 1 and the face 2. In the example of FIG. 15, the maximum contour line L1max does not intersect with the contour line L2, and the face 1 does not meet the overlapping condition. Further, the maximum contour line L2max does not intersect with the contour line L1, and the face 2 does not meet the overlapping condition.

  As shown in FIG. 16, the maximum contour line L1max intersects with the contour line L2 because the relationship between the face 1 and the face 2 is not vertically aligned but shifted vertically, but the maximum contour line L2max is the contour line L1. The image processing unit 214 performs image processing on the face 2 without setting the face 1 as a target. Face 1 meets the overlap condition, and face 2 does not meet the overlap condition.

  For example, when the contour of the face 2 is reduced as shown in FIG. 17 by the image processing for the face 2 in FIG. 16, the image processing unit 214 selects the face 1 again, and the face 1 is set as the overlapping condition. It is determined whether or not they match. In the example of FIG. 17, the maximum contour line L1max intersects with the contour line L2 set on the deformed face 2, and the face 1 meets the overlapping condition.

  As shown in FIG. 18, when the relationship between the face 1 and the face 2 is such that the maximum contour line L1max intersects with the contour line L2, but the maximum contour line L2max does not intersect with the contour line L1, the image processing unit 214 Image processing is performed on the face 2 without using the face 1 as a target. In the example of FIG. 18, face 1 matches the overlap condition, and face 2 does not match the overlap condition.

  For example, when the image processing on the face 2 in FIG. 18 reduces the contour of the face 2 as shown in FIG. 19, the image processing unit 214 selects the face 1 again, and the face 1 is set as the overlapping condition. It is determined whether or not they match. In the example of FIG. 19, since the maximum contour line L1max does not intersect with the contour line L2 of the deformed face 2 and the face 1 does not meet the overlapping condition, the image processing unit 214 next targets the face 1 as an object. Perform image processing.

  Even if one face meets the overlap condition, it may not meet the overlap condition by changing the size of the other face's outline. In this case, it is determined first that the overlap condition is met. The face that has been used is also subject to image processing.

  Returning to the description of FIG. 9, the image processing unit 214 selects an image processing target as described above, and outputs a captured image in which the contour of the face is deformed. The captured image output from the image processing unit 214 is output to the editing processing unit 203 and used as an editing target image.

  Image processing may not be performed by the image processing unit 214 for all captured images, but may be performed only for captured images selected from a plurality of captured images. In this case, after the photographing process by the photographing control unit 211 is completed, a selection screen is displayed on the touch panel monitor 52, and a photographed image to be subjected to image processing is selected by the user.

[Operation of photo sticker creation device]
Here, with reference to the flowchart of FIG. 20, the process of the photo sticker creation apparatus 1 that provides a photo sticker creation game will be described.

  In step S1, the coin processing unit 121 of the photo sticker creating apparatus 1 determines whether or not a coin has been inserted, and waits until it is determined that the coin has been inserted.

  If it is determined in step S1 that a coin has been inserted, in step 2, the pre-service processing unit 201 performs pre-service processing. For example, whether or not to use the mobile transmission function is selected by the pre-service process, and when the use is selected, an input screen for the mail address of the user's mobile phone is displayed on the external monitor 16. The user enters the mail address of his / her mobile phone.

  In step S3, the imaging processing unit 202 performs imaging processing. The photographed image obtained by the photographing process is displayed on the touch panel monitor 52, and the editing target image selected from the photographed image is supplied to the editing processing unit 203.

  In step S4, the edit processing unit 203 displays an edit screen and performs an edit process in accordance with a user operation. By the editing process, as described above, an image such as a handwritten character or pattern or a heart pattern prepared in advance is input to the image to be edited.

  In step S5, the post-service processing unit 205 performs post-service processing. When the use of the mobile transmission function is selected during the pre-service processing, the post-service processing unit 205 displays the image to be edited on the tablet-incorporated monitor 62, and a predetermined number of images are transmitted from the mobile phone. Let the user select it as a trusted image.

  The image selected as the portable transmission image is transmitted from the photo sticker creating apparatus 1 to the server together with the mail address input by the user, and is stored in a storage medium included in the server. An e-mail including a URL indicating the storage destination is transmitted from the server to the user's mobile phone. When the user operates the mobile phone to instruct access to the URL included in the e-mail, the server is accessed and the mobile transmission image is displayed on the display of the mobile phone.

  In step S <b> 6, the print processing unit 204 controls the printer 114 to perform a printing process for printing the edited image obtained by the editing process on the sticker sheet 162.

  In step S <b> 7, the print processing unit 204 guides the movement forward of the photo sticker discharge port 71 by displaying a predetermined screen on the tablet built-in monitor 62.

  When printing is completed, in step S8, the print processing unit 204 discharges the sticker sheet 162 on which the edited image is printed to the photo sticker discharge port 71, and ends the process.

  Next, with reference to the flowchart of FIG. 21, the imaging process performed in step S3 of FIG. 20 will be described.

  At the start of the shooting process, the live view image captured by the camera 51 is displayed on the touch panel monitor 52, and a start button operated when instructing the start of shooting is displayed at a predetermined position. In step S11, the imaging processing unit 202 determines whether or not a predetermined time until the start of imaging has elapsed. If it is determined that it has not elapsed, in step S12, the imaging processing unit 202 determines whether or not the start button has been pressed. To do.

  If it is determined in step S12 that the start button has not been pressed, the imaging processing unit 202 returns to step S11 and repeats the above processing. On the other hand, if it is determined in step S12 that the start button has been pressed, in step S13, the imaging processing unit 202 displays a countdown indicator on the touch panel monitor 52 and performs imaging.

  In step S14, the shooting processing unit 202 determines whether or not shooting has been performed a predetermined number of times. If it is determined that shooting has not been performed, the shooting processing unit 202 returns to step S13 and repeats shooting.

  On the other hand, if it is determined in step S14 that shooting has been performed a predetermined number of times in advance, in step S15, the shooting processing unit 202 displays a selection screen on the touch panel monitor 52 and allows the user to select an image to be edited. . On the selection screen for the image to be edited, a photographed image obtained by repeatedly photographing is displayed.

  When a captured image to be edited is selected, a face deformation process is performed in step S16 to deform the outline of the face shown in the captured image by image processing. The face deformation process will be described later with reference to the flowchart of FIG.

  In step S <b> 17, the imaging processing unit 202 displays a guidance screen that is a screen for guiding the movement to the editing space 33. For example, it is determined which one of the first editing space 33A and the second editing space 33B can be used, and the movement to the editing space that is not used (vacant) by other users is guided. Is done. Thereafter, the process returns to step S3 in FIG. 20, and the subsequent processing is performed.

  Next, the face deformation process performed in step S16 in FIG. 21 will be described with reference to the flowchart in FIG.

  In step S21, the face recognition unit 212 performs face recognition on the captured image. Face recognition detects faces in the captured image, and a degree of reliability representing the probability that each face is actually a human face is obtained.

  In step S22, the maximum contour line setting unit 213 identifies the contour of the face recognized by the face recognition unit 212, and sets the maximum contour line for each face.

  In step S <b> 23, the image processing unit 214 selects one face having the highest probability of being a face based on the reliability obtained by the face recognition unit 212. In the face selection here, the faces that have already undergone contour transformation are excluded.

  In step S24, the image processing unit 214 determines whether or not the selected face matches the overlapping condition.

  Since it is determined in step S24 that the selected face does not meet the overlapping condition because the maximum contour line does not intersect with the contour line of another face, the image processing unit 214 performs image processing in step S25. Thus, the contour of the selected face is deformed to a target size. The target size may be selected by the user, or may be automatically selected by the image processing unit 214. When the maximum contour line is selected as the target size, the contour is expanded until the contour line overlaps the maximum contour line, and when the minimum contour line is selected as the target size, the contour line is the minimum contour line. The outline is reduced until it overlaps.

  For example, when enlarging the contour until the contour line overlaps the maximum contour line, the image processing interpolates the range from the contour line to the maximum contour line using pixels inside the contour line, and the contour line is Along with the enlargement, the pixel outside the contour line is moved outward. When the contour is reduced until the contour line overlaps with the minimum contour line, the image processing is performed from the contour line to the minimum contour line by using pixels outside the contour line as the contour line is reduced. The range is interpolated.

  When the image processing of the selected face is completed, the image processing unit 214 returns to step S23, selects the next highly reliable face, and repeats the subsequent processing. For example, when image processing of the face with the highest reliability is performed, the face with the second highest reliability is selected and the process is repeated. In addition, when the image processing of the face with the highest reliability is performed for the second and subsequent times, the face with the highest reliability is selected in order, except for the face whose contour has already been deformed, and the process is repeated.

  On the other hand, in step S24, when it is determined that the selected face matches the overlapping condition because the maximum contour line intersects with the contour line of another face, in step S26, the image processing unit 214 detects all the faces. It is determined whether or not it has been selected.

  If it is determined in step S26 that all the faces have not been selected, the process returns to step S23, the face with the next highest reliability is selected, and the subsequent processing is repeated. If it is determined in step S26 that all faces have been selected, the process returns to step S16 in FIG. 21, and the subsequent processing is performed.

  FIG. 23 is a diagram illustrating a specific example of image processing.

  As shown on the left side of FIG. 23, a case will be described in which image processing is performed on a captured image P1 in which three faces are shown. In the captured image P1, among the three faces, the center face is slightly smaller than the left and right faces. Assume that the face of the right person is recognized as the face of ID1, the face of the left person is recognized as the face of ID2, and the face of the central person is recognized as the face of ID3 in descending order of reliability. In this case, the face with ID1 on the right side is selected first.

  In the photographed image P1, since the left end portion of the maximum contour line L1max of the face with ID1 intersects with the contour line L3 of the face with ID3 in the center, the face with ID1 meets the overlapping condition and is not subject to image processing. .

  Next, the face with ID2 on the left is selected. In the photographed image P1, since the right end portion of the maximum contour line L2max of the face with ID2 intersects with the contour line L3 of the face with ID3 in the center, the face with ID2 meets the overlapping condition and is not subject to image processing. .

  Next, the center face with ID3 is selected. In the photographed image P1, the maximum contour line L3max of the face of ID3 does not intersect with both the contour line L1 of the face of ID1 and the contour line L2 of the face of ID2, so the face of ID3 does not meet the overlapping condition, and the arrow As shown before # 1, image processing is performed on the face of ID3. In the photographed image indicated by the tip of arrow # 1, the maximum contour line or the like only on the face of ID3 indicates that image processing is performed on the face of ID3.

  Here, it is assumed that the face of ID3 is reduced to such an extent that the left end portion of the maximum contour line L1max does not intersect the contour line L3 and the right end portion of the maximum contour line L2max does not intersect the contour line L3.

  A photographed image P2 shown on the left side of FIG. 24 is a photographed image after image processing obtained by reducing the face of ID3.

  Since the face ID3 is being processed, in this case, the right face ID1 is selected again. In the photographed image P2, since the maximum contour line L1max of the face with ID1 does not intersect with the contour line L3 of the face with ID3, the face with ID1 does not meet the overlapping condition, and image processing is performed on the face with ID1.

  After image processing is performed on the face with ID1, the face with ID2 on the left side is selected next. In the photographed image P2, since the maximum contour line L2max of the face of ID2 does not intersect with the contour line L3 of the face of ID3, the face of ID2 does not meet the overlapping condition, and as shown by the tip of arrow # 2, the ID2 Image processing is performed on the face.

  FIG. 25 is a diagram illustrating a captured image obtained by performing image processing on each face of ID1, ID2, and ID3 by the above-described series of flows. As described above, it is determined whether or not each face meets the overlap condition. Images that match the overlap condition are not subject to image processing, and images that do not meet the overlap condition are subject to image processing. In the above, by selecting again an image that matches the overlap condition and is not subject to image processing, and repeatedly determining whether or not the overlap condition is met, it is possible to check It is possible to perform image processing on all faces in consideration of the relationship.

  FIG. 26 is a diagram for explaining the effect of selecting an image processing target from an image that does not meet the overlap condition. Here, description will be made using the face of ID1 and the face of ID3.

  26, the maximum contour line L1max of the face of ID1 intersects the contour line L3 of the face of ID3, but the maximum contour line L3max of the face of ID3 is the ID1 face. There is a relationship that does not intersect the contour line L1.

  In this case, as described above, in the process of reducing the contour, the range from the contour line to the minimum contour line is interpolated using the pixels outside the contour line as the contour line is reduced. Therefore, if the face of ID1 is reduced first, a part of the face of ID3 (near the maximum contour line L1max) is the face of ID1 as indicated by the tip of arrow # 11. The face of ID3 may be distorted into a convex shape.

  On the contrary, the process of enlarging the contour interpolates the range from the contour line to the maximum contour line using the pixels inside the contour line, and the outer side of the contour line as the contour line expands. Since a certain pixel is moved to the outside, if the face of ID1 is enlarged first, a part of the face of ID3 extends along the maximum contour line L1max as indicated by the tip of arrow # 12. The face of ID3 may be distorted into a concave shape. By selecting the image processing target as described above, it is possible to prevent the face of ID3 from being distorted.

[Modification]
In the above, the case where the contour of the face is deformed by image processing has been described, but the position and size of each part of the face such as the eyes, nose, mouth, etc. are also corrected along with the deformation of the face contour. It may be.

  In the above description, the face deformation process is performed for a face appearing in an image to be edited selected from the captured images. However, the face deformation process is performed for a face appearing in another image. May be. For example, it may be performed in real time for a face appearing in a captured image, or may be performed for a face appearing in an edited image.

  As shown in FIG. 27, a face whose maximum contour line does not fit within the range of the captured image may be excluded from the target of image processing. In the example of FIG. 27, a part of the maximum contour line L1max of the face with ID1 protrudes from the captured image. Even in this case, the determination of the overlap condition for the face of ID3 is performed by determining whether or not the maximum contour line L3max of the face of ID3 and the contour line L1 of the face of ID1 overlap. Is called. Here, when the maximum contour line L1max of the face of ID1 protrudes from the captured image, it is excluded from the target of image processing. However, the face of ID1 is subjected to image processing without enlarging the target image and removing it from the target. You may make it become object. The captured image is enlarged by, for example, copying and enlarging (extracting) pixels at the end of the captured image until the maximum contour line L1max is accommodated.

  In the above description, it is assumed that the order of determining the overlapping condition is in the order of high reliability. Further, the shape of the face may be determined such as the order of a round face and the order of a fine face by comparing the shape of a template with a contour prepared in advance.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention. For example, in the above, the determination of the overlapping condition is performed based on whether or not the lines intersect, but it may be performed based on whether or not the surfaces overlap.

  DESCRIPTION OF SYMBOLS 1 Photo sticker production apparatus, 2 Mobile terminal, 3 Management server, 12 Shooting unit, 13 Editing unit, 101 Control part, 111 Advance customer service part, 112 Shooting part, 113 Edit part, 201 Advance customer service part, 202 Shooting process part, 203 editing processing unit, 204 printing processing unit, 205 post-service processing unit, 211 photographing control unit, 212 face recognition unit, 213 maximum contour line setting unit, 214 image processing unit

Claims (4)

Face detection means for detecting the face of a person in a captured image;
Setting means for setting a maximum contour line, which is a line representing the size of the maximum contour that can be deformed, on the face of the person detected from the captured image;
When there are a plurality of faces of a person detected by the face detection means, the maximum contour line set for a face of a predetermined person among the faces of the plurality of persons is a contour line of the face of another person If it is determined that it does not intersect, the contour of the predetermined person's face is set to a predetermined size within the range of the size represented by the maximum contour line. If it is determined that the image processing to be deformed and intersect, face deformation processing means that does not perform the image processing;
Each time the image processing is performed on the face of the predetermined person by the face deformation processing means, selection of selecting the face of another person as the face of the predetermined person to be processed by the face deformation processing means And an image processing apparatus. The face detection means detects the face of a person in the captured image, calculates the probability of being a person's face with respect to an object detected as the person's face,
The image processing apparatus according to claim 1, wherein the selection unit selects the face of the predetermined person to be processed in descending order of the probability obtained by the face detection unit. When the maximum contour line set for the faces of all of the plurality of persons intersects with the contour line of the face of the other person, except for the face that is the target of the image processing, the selection unit is configured to deform the face. The image processing apparatus according to claim 2, wherein the selection of the face of the predetermined person to be processed is repeated until the processing means determines. Detect the face of a person in the shot image,
A maximum contour line, which is a line representing the maximum contour size that can be deformed, is set on the face of the person detected from the captured image;
If there are a plurality of detected human faces, whether or not the maximum contour line set for the face of a predetermined person among the faces of the plurality of people intersects with a contour line of the face of another person If it is determined that it does not intersect, image processing is performed for deforming the contour so that the face of the predetermined person becomes a predetermined size within the range of the size represented by the maximum contour line. , If it is determined to cross, the image processing is not performed,
Each time the image processing is performed on the face of the predetermined person, the image processing method includes a step of selecting a face of another person as the face of the predetermined person to be processed.

Images