JP2017208780A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of easily reflecting preference for each user in correction of a photographed image.SOLUTION: An identification processing section 520 specifies a user included in subjects by person detection and person authentication in a photographed image, and a correction processing section 540 corrects the photographed image. When the correction processing section 540 corrects the photographed image including a plurality of users in the subjects, the correction processing section reads a specified value of correction, which is previously stored for each user, for each user specified by the identification processing section, and correction corresponding to the specified value which is read can be performed on an image range of each user among the plurality of users.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus, and more particularly to an apparatus for processing image correction.

  2. Description of the Related Art Conventionally, an imaging device for a game and a game that can shoot a subject including a plurality of users with a camera a plurality of times and output the captured image as a photographic print sheet or send it to an external device via a network Photography services are known. Many such shooting apparatuses have been proposed that can perform, for example, the following editing process according to the user's preference. First, chroma key processing is performed on a photographed subject image to generate a subject image from which the subject portion is extracted. Then, after combining the subject image with the background image selected by the user, editing processing such as correcting the eye color and size, skin color, face outline, etc., and overlaying characters and patterns (Generally also called “graffiti”), the final print image is generated. Also, by providing such editing functions so that multiple users who photographed together can operate with a touch pen at the same time, friends can enjoy together from shooting to editing, which is gaining popularity.

  On the other hand, in recent years, there is an increasing number of application software (hereinafter also referred to as “applications”) that operate on a mobile terminal with a camera function such as a smartphone that can easily take a photograph and process a photographed image. In particular, ordinary smartphones are equipped with a sub-camera for shooting the user side in addition to the main camera, and a shooting method called “self-shooting” or “self-shooting” that takes a picture of itself with the sub-camera is also common. .

JP 2015-2111275 A

  As a result of the widespread use of mobile devices with camera functions such as smartphones, it is easy to take group photos of several people when multiple people gather. It has become desirable to be able to do this (for example, see Patent Document 1). Against this backdrop, the present inventor has come to recognize that a new and highly convenient function that is not available in the game shooting apparatus can be realized by using a function specific to the portable terminal.

  The present invention has been made in view of these problems, and an object of the present invention is to provide an image processing apparatus that can easily reflect the preference of each user in correcting a captured image.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes an imaging unit that captures an image including a user as a subject, an image acquisition unit that acquires a captured image of the user captured by the imaging unit, An identification processing unit that identifies a user included in the subject by person detection and person authentication in the captured image, a correction processing unit that performs correction on the captured image, and a display control unit that displays the captured image on the screen. When performing correction on a captured image including a plurality of users as subjects, the correction processing unit reads a correction default value stored in advance for each user for each user specified by the identification processing unit, and reads the read-out value. Correction according to the predetermined value can be performed on each image range of a plurality of users.

  Here, the “image processing device” may be a mobile terminal such as a camera-equipped mobile phone or a computer terminal that can execute the program. The “captured image” may be either a frame or a still image included in the moving image. The “person detection” may be a technique for detecting a person by detecting a part of a person such as a face by image recognition. “Person authentication” may be a technique for identifying a user by comparing feature information of a part of a person such as a face with feature information stored in advance. “Correction” may be image processing that changes the shape, size, color, texture, and the like of a part included in the face, such as eyes, contours, nose, and skin. According to this aspect, even when a plurality of users are photographed as subjects on one terminal and each face is corrected, not only the correction can be performed with different preference settings for each user, Can be easily read and set. Thus, the preference for each user can be easily reflected in the correction of the captured image.

  The correction processing unit may be able to read a default value for correction from different storage means when there are one user and a plurality of users included in the subject of the captured image. In addition, when there are a plurality of users included in the subject of the captured image, the correction processing unit may be able to read a correction default value from different storage means for each user.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can reflect the preference for every user for correction | amendment of a picked-up image can be provided.

1 is a diagram schematically illustrating an overall configuration of an image processing system. It is a functional block diagram which shows the function structure of a terminal device. It is a hardware block diagram of a terminal device. It is the perspective view which looked at the imaging device for play from the left front side. It is the top view which looked at the imaging device for play shown in FIG. 4 from the top. It is a functional block diagram which shows typically the functional structure of the imaging device for play. It is a functional block diagram which shows typically the functional structure of a management server. It is a flowchart which shows the process in case a user uses the application of this embodiment for the first time. It is a flowchart which shows the terminal user setting process in S20 of FIG. 8 in detail. It is a flowchart which shows the correction process in S26 of FIG. 8 in detail. It is a flowchart which shows the terminal user setting in S30 of FIG. 8 in detail. It is a figure which shows typically the example of a setting screen in the case of performing a terminal user setting before the 1st imaging | photography. It is a figure which shows typically the example of a screen of registering identification information as a terminal user setting. It is a figure which shows typically the example of a screen which makes a user select "taste" as 1st correction | amendment. It is a figure which shows typically the example of a screen as which a user selects "finish of a reflection" as 2nd correction | amendment. It is a figure which shows typically the example of a screen which makes a user select "brightness of skin" as 3rd correction | amendment. It is a figure which shows the example of a screen when a terminal user performs the first imaging | photography without setting. It is a figure which shows typically the example of a screen which detects a person from an image and specifies a user. It is a figure which shows typically the example of the user specific screen in 2nd Embodiment.

(First embodiment)
In the present embodiment, a user includes himself or herself by using a predetermined photography application program (hereinafter also simply referred to as “app”) installed in a terminal device such as a mobile phone with a camera such as a smartphone. Take a photo with multiple people as subjects. The app can correct the image using the individual setting values of the users included as the subject in the captured image. Each user's setting value is registered as a favorite default value on his / her terminal device or management server, and the application refers to the user's default value according to the face authentication result of the person included in the photographed image, and uses the default value. The correction for each user used is executed for each user's image range. The default value of correction for each user may be registered on the terminal device or the management server based on the content set in advance in the application or the execution result of the correction, or a shooting device for games such as a so-called print sticker May be registered on the terminal device or the management server on the basis of the correction result executed in step. Images captured and corrected by the application can be posted and shared with external services such as various SNS (Social Network Service) sites.

  FIG. 1 is a diagram schematically showing the overall configuration of the image processing system 600. The image processing system 600 is a system in which a plurality of terminal devices 500 are connected to the management server 300 via the network 200 such as the Internet. The management server 300 is connected to the game shooting device 100 via the network 200. The plurality of terminal devices 500 can also be connected to the SNS site 400 via the network 200. In the figure, two terminals, the terminal device 500a and the terminal device 500b, are illustrated as a plurality of terminals, but a large number of terminal devices 500 corresponding to the number of users can be connected to the management server 300 via the network 200. Hereinafter, in the present specification, the terminal devices 500 are simply collectively referred to as “terminal devices 500” unless otherwise distinguished.

  An application installed in the terminal device 500 detects a person as a subject from a photograph image taken using a built-in camera, detects face feature information for each person, and performs face authentication based on the feature information for each person. Identify users. A default value registered in advance for each identified user is read from the terminal device 500 or the management server 300, and correction is executed based on the default value. For unregistered users, corrections are provisionally executed based on an average recommended value prepared in advance for applications for unregistered users, and then the desired correction content is used based on the corrected images. Is specified and reflected in the correction. The final correction result is the user's “favorite correction value”, which is registered in the management server 300 as the user's default value, and the user who owns the terminal device 500 on which the application is being executed. The default value is also registered on the application of the terminal device 500.

  The game shooting apparatus 100 is an apparatus such as a so-called print sticker machine including a camera and an electronic device having a user interface for correcting and editing a photographed photograph image. When the user corrects the image by taking a picture with the game shooting device 100, the correction setting value and the facial feature information are linked to the identification information such as the user's mail address input by the game shooting device 100. Is transmitted to the management server 300 and registered in the management server 300. Further, the correction default value and feature information that are executed by the application on the terminal device 500 and registered in the management server 300 can be referred to from the game shooting device 100 based on the identification information. It may be used for user identification or correction. As described above, the correction setting values used in the application are also used in the game shooting apparatus 100, thereby simplifying the user's operation required in the game shooting apparatus 100 and reducing the operation time. .

  FIG. 2 is a functional block diagram illustrating a functional configuration of the terminal device 500. The terminal device 500 includes an imaging unit 510, an identification processing unit 520, an image acquisition unit 530, a correction processing unit 540, a data storage unit 550, an authentication processing unit 560, an input / output control unit 570, and a communication processing unit 580. The imaging unit 510 includes a camera control unit 511 and captures an image including the user as a subject. The camera control unit 511 controls the camera built in the terminal device 500 and images the user.

  The image acquisition unit 530 acquires a user-captured image captured by the imaging unit 510. The image acquisition unit 530 includes a video acquisition unit 531 and an image recording unit 532. The video acquisition unit 531 acquires the video imaged by the imaging unit 510, for example, the video imaged by the imaging unit 510 during standby for capturing a still image or during video recording, and temporarily stores the data in the data storage unit 550 described later. The live view video is transmitted to the unit 551 in real time, and the frames included in the video are buffered. The buffered live view video is displayed on the touch panel screen by the display control unit 571 of the input / output control unit 570 described later, and is subject to person detection or face authentication by the identification processing unit 520.

  The image recording unit 532 acquires an image captured by the imaging unit 510 as a still image at a timing based on a shutter button operation by the user or a shutter-on instruction by a timer, and stores the image in the temporary storage unit 551 of the data storage unit 550. The captured image stored in the temporary storage unit 551 is a target of person detection or person authentication by the identification processing unit 520 and a target of correction processing by the correction processing unit 540. The image after the correction process is stored in an image storage unit 554 described later.

  The identification processing unit 520 identifies a user included in the subject through person detection and person authentication in the captured image. The identification processing unit 520 includes a person detection unit 521, a feature detection unit 522, and a feature matching unit 523.

  The person detection unit 521 detects the image range of the user included in the subject from the image captured by the imaging unit 510 using an image processing technique such as a known person detection technique or a face detection technique. The “image picked up by the image pickup unit 510” here may be a live view video before a photo shoot (hereinafter also referred to as a “main shoot”) in which the shutter button is operated by the user, or a main shoot. After that, it may be a captured image as a still image acquired by actual shooting. The live view video is a video for confirming in real time the subject and angle of view imaged by the image sensor of the built-in camera. The person detection unit 521 may use a method for detecting only the image range of the face, not the whole person, as person detection.

  The feature detection unit 522 detects the image range of the person's face from the image range of the person detected by the person detection unit 521 (however, if the specification is such that the person detection unit 521 detects the image range of the person's face) The detection unit 522 does not need to detect the image range of the face). The feature detection unit 522 detects the feature amount of each human face from the face image range using the existing face recognition technology, thereby detecting the feature amount of each constituent element as face feature information. For example, an area such as a skin color area, an eye, or a nose is extracted to detect a face area, and a feature amount such as a distance, arrangement, or size between the organs such as an eye or nose is detected from the detected face area. The person detection unit 521 and the feature detection unit 522 may detect face feature information by performing person detection or face detection in real time from each frame included in the live view video. Alternatively, from a frame or still image included in a live view video for several seconds buffered in the temporary storage unit 551 or a frame or still image included in a moving image for several seconds stored in the image storage unit 554 The detection process may be performed at an ex-post timing unrelated to the reproduction timing. In this case, it is possible to execute detection processing that takes time and load because it is not real-time detection in parallel with image reproduction.

  The feature matching unit 523 compares the feature information of the face detected by the feature detection unit 522 with the feature information stored in advance in the data storage unit 550, that is, the feature information of the face of the terminal user of the terminal device 500. Then, it is determined whether the terminal user of the terminal device 500 is included in the subject. The feature matching unit 523 specifies a user other than the terminal user by matching feature information through wireless communication with an external device by the communication processing unit 580.

  The feature information is collated by, for example, counting a higher score as the feature value is approximated for each of a plurality of feature items included in the feature information, and adding a total score obtained by weighting the feature items. If it exceeds the reference value, it may be determined as “match”. Alternatively, if there is only one user included in the subject, it is highly probable that the user is a terminal user, and even if the overall score is lower than the predetermined first reference value, the second lower than the first reference value. If it exceeds the reference value, it may be determined as “match”. That is, the strictness of collation may be adjusted by collating feature information using different criteria depending on whether the user is a terminal user or another user. Further, when there are a plurality of users included in the subject, the person who has the highest score in the comparison with the feature information of the terminal user is determined as the terminal user as a criterion for determining which user is the terminal user. Also good. For example, even if a plurality of persons whose scores exceed a predetermined reference value are detected, it can be expected that the possibility of erroneous determination is reduced by determining only the person with the highest score as the terminal user.

  In the present embodiment, the feature matching unit 523 transmits the feature information of users other than the terminal user to the management server 300 via the data communication unit 582 of the communication processing unit 580, and the user feature information stored in the management server 300 To identify the user. In the modified example, the feature information of the terminal user is also stored in the management server 300 instead of the data storage unit 550, and the feature information is transmitted to the management server 300 for all users included in the subject. It is good also as a specification which specifies a user by collation via. In another modification, while the feature information of the terminal user is stored in the data storage unit 550, the feature information of the user previously associated with the terminal user as a “friend” is acquired from the management server 300 and stored. It may be stored in the unit 550. The association as to whether it is a “friend” may be managed by the management server 300 independently as the image processing system 600 based on the history of users who have taken together in the past in the game shooting device 100, or may be an SNS site. It is good also as acquiring the relevance information of "friend" managed in 400 through the authentication of the SNS site 400, and using the relevance. In these cases, the feature matching unit 523 compares the feature information for each user acquired from the management server 300 and stored in the data storage unit 550 with the detected feature information to identify the user. The association as to whether it is a “friend” may be managed by the management server 300 independently as the image processing system 600 based on the history of users who have taken together in the past in the game shooting device 100, or may be an SNS site. It is good also as acquiring the relevance information of "friend" managed in 400 through the authentication of the SNS site 400, and using the relevance. In another modified example, the feature information of all the users stored in the management server 300 is used as the target of matching as the feature information of the partner to be matched with the feature information of the user included in the subject. You may collate, narrowing down the object according to the priority order. This is because the time required for processing may increase if collation is performed in a random order from all feature information stored in the management server 300. For example, as described above, the user's feature information associated with the terminal user as a “friend” is collated with the highest priority. When not detected from among them, you may narrow down the object to collate based on regional information, such as a residence area and an action area. That is, on the premise that regional information such as a residential area and a behavioral area is associated and registered in the management server 300 for each user, the characteristics of the user associated with the regional information of the terminal user or regional information that is the same as or close to the current location It is good also as collating information preferentially. Regional information is automatically associated with feature information based on location information such as GPS acquired by a mobile terminal when taking an image, or location information obtained from a mobile communication base station or wireless LAN communication access point. May be registered in the server.

  The feature matching unit 523 specifies the user included in the subject based on the confirmation process by the user based on the user identification information in the matching between the detected feature information and the feature information for each user. For example, the feature matching unit 523 identifies feature information that matches the detected feature information among the feature information of a plurality of users, and uses the input / output control unit 570 to identify identification information associated with the feature information. Present on the screen. The identification information may be a user name such as a nickname registered for each user. In this case, for example, whether or not this person is the correct user, such as “Is this person Mr. XX?” Displays a character string for asking a question on the screen. If this information is correct, the user operates the touch panel to indicate that it is correct, and if it is incorrect, indicates that it is a misrecognition. Through such confirmation processing, the user is correctly identified when it is recognized that the matching result by the feature matching unit 523 is correct. In this embodiment, the user name as identification information is presented to the user in order to ask whether the collation is correct, such as “Is this person Mr. XX?”. May be displayed on the terminal of another unrelated person. Therefore, from the viewpoint of protecting personal information, it is preferable to register a user name such as a nickname that is not the user's real name. However, as a modified example, there is no problem of leakage of personal information when it is set as a specification for managing friendships between users and compared with a user associated in advance as a “friend” of the terminal user of the terminal device 500. Therefore, it may be a specification that can present a name including the real name.

  As a modification, when a user other than the terminal user is included in the subject, the identification processing unit 520 does not transmit the feature information of the user to the management server 300 but transmits the captured image itself to the management server. It is good also as a specification which makes 300 perform the process from a person detection to a user identification.

  The authentication processing unit 560 performs data communication by associating the facial feature information detected by the feature detection unit 522, the correction default value set by the correction processing unit 540, and the identification information input by the input control unit 573 with each other. The data is transmitted to the management server 300 via the unit 582. In particular, whenever facial feature information is detected by the feature detection unit 522, it is transmitted to the management server 300 and stored in the management server 300. The authentication processing unit 560 identifies or authenticates the user based on the user identification information. In particular, when specifying a user other than the terminal user, in order to authenticate whether or not the user is correct, the user is input identification information, and the input identification information is encrypted and transmitted to the management server 300, The user is confirmed by receiving the authentication result from the management server 300. The authentication processing unit 560 uses, as identification information, one or more pieces of information mainly consisting of character strings such as an email address, a user name, and a password, and biological information such as fingerprint information, iris information, and retina information. Certify. The authentication processing unit 560 encrypts identification information such as a mail address, a user name, a password, and fingerprint information input via the input control unit 573 and transmits the encrypted identification information to the management server 300 via the data communication unit 582. When fingerprint information is used as the identification information, the fingerprint information may be acquired by a fingerprint sensor built in the terminal device 500, or the fingerprint information may be acquired by a touch panel. When using iris information, retinal information, or vein authentication as identification information, images of iris, retina, and vein may be acquired by a camera. When voiceprint authentication is used as identification information, sound may be acquired by a microphone.

  The data storage unit 550 includes a temporary storage unit 551 that is a buffer memory that temporarily stores live view video and still images acquired by the image acquisition unit 530, a feature storage unit 552 that stores user feature information, and a user A default value storage unit 553 that stores a default value for correction of the image, and an image storage unit 554 that stores a user's captured image.

  The feature storage unit 552 stores the feature information of the face of the terminal user of the terminal device 500 detected by the feature detection unit 522 in association with the identification information together with the identification information of the user. The feature information of other users is stored in a data storage unit in the management server 300. However, in the modified example, the feature information of other users, particularly users associated as “friends” of the terminal user, is stored in the management server 300. May be stored in the feature storage unit 552 in association with the identification information of the user. Each time facial feature information is detected from an image in the terminal device 500, the facial feature information is transmitted and accumulated in the feature storage unit 312, thereby increasing the accuracy of matching with the feature information.

  The default value storage unit 553 stores a correction default value corresponding to the terminal user of the terminal device 500 in a form associated with the identification information of the user. The default value of the terminal user's correction can be registered by the terminal user according to his / her own instruction before the first shooting after installing the application, and the setting value when the correction is actually performed after the first shooting is set as the default value. You can also register. In addition, the default value can be updated with the set value based on the instruction of the terminal user every time the captured image is corrected. Alternatively, each time a captured image is corrected, the default value may be automatically updated with the set value, or the default value may be updated with a relatively frequent set value. The correction default values corresponding to other users are stored in the data storage unit in the management server 300 in association with the feature information and identification information for each user. However, in a modified example, the default value storage unit acquires a default value of correction of another user, particularly a user associated as a “friend” of the terminal user, from the management server 300 and associates it with identification information of the user. You may memorize | store in 553.

  When biometric information is used as identification information, in addition to fingerprint information, information for authentication such as iris authentication, retinal authentication, vein authentication, and voiceprint authentication is collected from a dedicated sensor, image, or sound and stored in the feature storage unit 552. You may let them. However, for users who do not like to authenticate with biometric information, which cannot be changed due to its nature from the viewpoint of privacy, only changeable text information such as e-mail address, user name, password and password is used as an alternative. This may be selectable by the user. Alternatively, biometric information and character information such as a password and a personal identification number may be used in combination, and the user may be specified only when both are compatible. In the modification, an application ID that is uniquely assigned when installing an application may be used as the identification information.

  The correction processing unit 540 includes a simple correction unit 541 and a main correction unit 542. The simple correction unit 541 performs simple correction capable of high-speed processing with a relatively low processing load for each of a plurality of frames included in the live view video imaged and temporarily stored in the temporary storage unit 551. Run. The content of the simple correction is a standard set value prepared as a recommended value by the manufacturer for uniform provisional enforcement for all users. This standard set value may be common to all members, or different set values may be prepared in advance for each gender and race, and different corrections may be executed for each user. However, in a modified example, a correction default value registered for each user may be read, and different corrections for each user may be performed on each image range based on the default value.

  The main correction unit 542 performs correction on a captured image that is a still image. The correction for the captured image is a process for changing the color and size of the eyes, the contour of the face, the shape of the nose, the color and texture of the skin. When executing correction on a captured image including a plurality of users in the subject, the main correction unit 542 reads a correction default value stored in advance for each user for each user specified by the identification processing unit 520, and Correction according to the read default value can be performed on each image range of a plurality of users. The main correction unit 542 can read a default value for correction from different storage means when there are one user and a plurality of users included in the subject of the captured image. That is, since it can be estimated that the terminal user of the terminal device 500 has photographed alone when there is only one user, in principle, the default value stored in the default value storage unit 553 is read and correction according to the default value is performed. To the image range of the terminal user included in the. On the other hand, when there are a plurality of users, the default value is read from the default value storage unit 553 for the user collated with the feature information of the terminal user stored in the feature storage unit 552, and the data communication unit for other users. A default value is read from the management server 300 via 582, and correction according to the default value is performed on the image range for each user. As described above, when there are a plurality of users included in the subject of the captured image, the correction default value can be read from the storage means that differs for each user. The corrected image is displayed on the screen of the touch panel by the display control unit 571.

  The input / output control unit 570 includes a display control unit 571, a voice control unit 572, and an input control unit 573. The display control unit 571 displays the display content including the live view video and the captured image on the screen of the touch panel. The audio control unit 572 outputs audio from the speaker. The input control unit 573 accepts a user operation input via a touch panel or buttons.

  The communication processing unit 580 controls communication with the outside. The communication processing unit 580 includes a short-range wireless unit 581, a data communication unit 582, and a position acquisition unit 583. The short-range wireless unit 581 controls data communication with an external device such as the terminal device 500 of another user, for example, by short-range wireless communication such as Bluetooth (trademark) or NFC (Near Field Communication). The data communication unit 582 controls data communication with the management server 300 by wireless LAN communication, cellular phone packet communication, or the like. The position acquisition unit 583 receives the GPS signal and acquires current position information of the terminal device 500.

  FIG. 3 is a hardware block diagram of the terminal device 500. In the terminal device 500, a camera 515, a communication module 585, a touch panel 555, a speaker 556, a GPU 557, a CPU 590, and a memory 575 are connected to each other via a bus 591.

  The camera 515 includes a main camera 516 called a so-called out camera built in the back side of the terminal device 500 and a sub camera 517 called a so-called in camera built in the front side of the terminal device 500. The main camera 516 and the sub camera 517 functionally form part of the imaging unit 510 and are controlled by the camera control unit 511, and a video or an image is acquired by the image acquisition unit 530. Any video and image captured by any camera can be the target of image processing by the application of this embodiment.

  The communication module 585 includes a short-range wireless module 586 that performs short-range wireless communication such as Bluetooth (trademark) and NFC, a wireless LAN module 587 that performs wireless LAN communication, a portable communication module 588 that performs packet communication of a mobile phone, and a GPS signal. The GPS module 589 which acquires the current position information of the terminal device 500 based on it is included. The communication module 585 functionally constitutes a part of the communication processing unit 580.

  The touch panel 555 is a device for screen display and touch input in the terminal device 500. The touch panel 555 functionally constitutes a part of the input / output control unit 570, the display control unit 571 controls display on the touch panel 555, and the input control unit 573 detects operation input to the touch panel 555 and buttons. The The speaker 556 is a device that outputs sound, and functionally constitutes a part of the input / output control unit 570, and the sound control unit 572 controls the sound output by the speaker 556.

  A GPU (Graphics Processing Unit) 557 is a microprocessor that executes image display and image conversion processing, and functionally forms part of an identification processing unit 520, a correction processing unit 540, and a display control unit 571. A CPU (Central Processing Unit) 590 is a central processing unit mainly responsible for processing and calculation other than image processing, and functionally includes an image acquisition unit 530, an identification processing unit 520, a correction processing unit 540, an authentication processing unit 560, A part of the input / output control unit 570 or the like is configured. The memory 575 is a non-volatile storage medium such as a readable / writable flash memory, and functionally forms part of the data storage unit 550.

  FIG. 4 is a perspective view of the gaming apparatus 100 viewed from the left front side. FIG. 5 is a plan view of the gaming apparatus 100 shown in FIG. 4 as viewed from above. In the following, other drawings will also be described with reference to the front, rear, left and right directions shown in FIG. As shown in FIGS. 4 and 5, the gaming device 100 includes a front-side shooting unit 50 and a rear-side editing unit 51, and has a substantially rectangular parallelepiped shape that is long in the front-rear direction. The left side or right side of the photographing unit 50 is provided with a presetting unit 40 for presetting the contents of the coin including the number of coins and the number of photographers. An output unit 52 is provided.

  The photographing unit 50 includes a photographing case 11 in which a camera for photographing a user is incorporated, and a background wall portion 12 disposed behind the photographing case 11 via the photographing space 10. A gap between the side surface of the imaging housing 11 and the side surface of the background wall portion 12 constitutes first entrances 101 a and 101 b that are entrances to the imaging space 10. The pre-setting unit 40 includes a touch panel 41 that displays setting items of shooting contents including the number of photographers and receives instructions from the user, and a coin insertion unit 42 that includes a coin insertion slot and a return slot. Although the figure shows an example in which the housing of the presetting unit 40 is detachably attached to the left side surface of the photographing unit 50, it can be attached to either the left or right side according to the layout of the store where the game photographing device 100 is installed. Good. The user enters the imaging space 10 from the entrance on the side where the presetting unit 40 is attached. The editing unit 51 includes an editing housing 21 disposed adjacent to the back of the background wall 12. The editing casing 21 is provided with a pair of operation surfaces that face in the left and right directions. The user can enter the editing spaces 20a and 20b for performing input operations on the operation surfaces via the second entrances 201a and 201b. The output unit 52 includes an output housing 31 that contacts the rear end surface of the editing housing 21. The operation surface faces the rear end side so that the user operates the output casing 31 from the rear end side of the game photographing apparatus 100. Therefore, the space on the rear end side of the output housing 31 becomes an output space 30 for the user to perform an output operation. As a modification, the game shooting device 100 may be configured without the presetting unit 40, or the game shooting device 100 may be configured without the editing unit 51 or the editing space 20. Good.

  The side surfaces of the upper end portions of the photographing housing 11 and the background wall portion 12 are connected to each other by a ceiling frame 13 extending in the front-rear direction. Inside the frame of the ceiling frame 13, a ceiling illumination unit 14 that covers the imaging space 10 from above, and illuminates and flashes is provided. On the left side or the right side of the photographic housing 11, a posting space 15 is provided for posting instructions including how to use the game shooting device 100. In the upper part of the posting space 15, a display unit 16 is provided to indicate whether or not the photographing housing 11 is in use.

  The display unit 16 includes an indicator lamp (not shown) that emits light or is turned off. The display unit 16 indicates to the user that the photographing unit 50 is in use by light emission of the indicator lamp, and indicates to the user that the photographing unit 50 has become empty by turning off the light. The shooting device 100 for play is formed in a triangular shape in plan view in which the display unit 16 protrudes outward in the left-right direction, but the shape is not particularly limited, and may be other shapes such as a convex curve shape. .

  The background wall portion 12 is formed by a plate-like member having substantially the same width and height as the photographing housing 11, and extends from the left and right ends of the main body plate 121 to the front. And a pair of side plates 122 provided. For this reason, the background wall part 12 is formed in the planar view U shape as a whole. A single color (for example, blue or green) for chroma key processing is colored on the surface of the main body plate 121 facing the photographing housing 11. Regarding the coloring, in addition to coloring the main body plate 121 directly with paint or the like, a colored fabric, plate or the like may be attached. Although not shown, the floor surface of the imaging space 10 is colored with a single color for chroma key processing in a manner similar to the main body plate 121 so as to be continuous with the main body plate 121. As described above, the first entrances 101 a and 101 b for entering and exiting the imaging space 10 are configured from the rear end edge of the imaging casing 11, the lower end edge of the ceiling frame 13, and the front end edge of the side plate 122. .

  The editing case 21 is a face-to-face type that allows two sets of users to edit simultaneously from the left and right sides, and has an operation surface on each of the left and right sides. The editing casing 21 is shorter in the left-right direction than the width of the background wall 12 in the same direction. The editing housing 21 is connected at its front end surface to the center of the back surface of the background wall portion 12. The spaces on the left and right sides of the editing housing 21 become editing spaces 20a and 20b in which two sets of users perform editing operations, respectively. Above the editing housing 21, a substantially U-shaped curtain frame 22 is provided in plan view. The curtain frame 22 includes a pair of side frames 221 extending in the front-rear direction and a rear frame 222 extending in the left-right direction connecting the rear ends of the side frames 221. Although not shown, a computer including a CPU (Central Processing Unit) such as a PC (Personal Computer) is installed inside the editing case 21.

  The front end of each side frame 221 is connected to the rear end of the ceiling frame 13 in the front-rear direction, and the central portion of the rear frame 222 is supported by a support member 23 provided at the rear upper end of the editing housing 21. Is done. The side frames 221 on one side of the left and right sides and the almost half length of the rear frame 222 constitute upper frame portions of the second entrances 201a and 201b for entering and exiting the editing spaces 20a and 20b.

  FIG. 6 is a functional block diagram schematically showing a functional configuration of the game shooting apparatus 100. The game shooting apparatus 100 includes a pre-setting unit 40, a shooting processing unit 130, a display control unit 135, a data generation unit 140, a data storage unit 150, and an output processing unit 160.

  This figure shows a functional configuration for realizing the game photographing apparatus 100, and other configurations are omitted. In this figure, each element described as a functional block for performing various processes can be configured by a camera (not shown), a CPU, a main memory, and other LSI (Large Scale Integrated circuit). In terms of software, it is realized by the CPU executing a program or the like loaded in the RAM. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  Prior to shooting, the presetting unit 40 presets shooting contents such as music to be played in the shooting space 10 during shooting play, background images for synthesis, and the like based on user instructions. The background image preset by the presetting unit 40 is set in such a manner that the user selects one of a plurality of background images corresponding to the music selected by the user. Further, prior to shooting, the presetting unit 40 presets the shooting contents of the user who is a subject, the number of shots, the name and group name, a favorite theme, and the like based on a user instruction. The number of photographers to be preset by the presetter 40 is set in such a manner that the user selects one of “2”, “3”, and “large number of people”.

  The imaging processing unit 130 includes an individual authentication unit 131, an imaging unit 132, and an image acquisition unit 133. The imaging unit 132 sets one or more users as subjects by the front camera 111 or the side camera 112 according to the control by the individual authentication unit 131 or the image acquisition unit 133 based on the play start instruction received from the presetting unit 40. Take an image. The individual authentication unit 131 performs acquisition of a photographed image for authentication, person detection for the acquired photographed image for authentication, and person authentication. The individual authentication unit 131 performs person detection for detecting a person from the acquired photographed image for authentication. The person detection in the present embodiment is, for example, face detection. The individual authentication unit 131 detects a face region by extracting a skin color region or an organ such as an eye or a nose from a photographed image for authentication. The individual authentication unit 131 detects feature quantities such as the distance, arrangement, and size of organs such as eyes and nose from the detected face area, and executes person authentication that identifies the characteristics of each person based on the feature quantities. . The individual authentication unit 131 temporarily stores the captured image for authentication in the data storage unit 150.

  The display control unit 135 displays the live view video being shot by the imaging unit 132 on the screen of the front monitor 113 or the side monitor 114 in real time.

  The image acquisition unit 133 sets the focal length, the angle of view, the trimming range, and the like of the imaging unit 132 (that is, the front camera 111 and the side camera 112), and causes the imaging unit 132 to capture an image. Then, the image acquisition unit 133 displays the live view on the front monitor 113 or the side monitor 114 via the display control unit 135. Further, the image acquisition unit 133 acquires a user's captured image by capturing the subject imaged by the imaging unit 132 a plurality of times. In the present embodiment, the image acquisition unit 133 performs at least five shootings and generates about five shot images.

  The image acquisition unit 133 causes the data storage unit 150 to store a plurality of acquired captured images for main shooting as a group of photographic images. More specifically, the image acquisition unit 133 assigns the same play ID to a plurality of acquired photographed images for actual photographing, and assigns a unique image identifier to each photographed image in order to identify the photographed image. To do.

  The image acquisition unit 133 acquires and acquires a user's moving image for a subject imaged by the imaging unit 132 a plurality of times. The image acquisition unit 133 executes, for example, three moving image shootings, and acquires three moving images.

  Prior to the start of moving image shooting, the display control unit 135 displays on the screen of the front monitor 113 a model moving image for showing the user a model of subject movement during moving image shooting. The display control unit 135 switches to the live view of the user's video imaged by the imaging unit 132 after finishing the display of the model moving image. After switching to the live view, the image acquisition unit 133 captures a moving image of the user. By repeating the process from the display of the model moving image to the shooting of the moving image of the user three times, three moving images are acquired. The image acquisition unit 133 stores the acquired moving image in the data storage unit 150 in association with the play ID.

  The data generation unit 140 includes a background processing unit 141, a correction processing unit 142, an editing processing unit 143, and a moving image processing unit 144. The background processing unit 141 synthesizes the subject and the background image with a known chroma key composition technique for each of the authentication photographed image, the plurality of actual photographed images, and the plurality of moving images stored in the data storage unit 150. To do. In other words, the background processing unit 141 converts the background information (chroma key mask) for distinguishing between the subject in the captured image and the background region other than the binarized image or the grayscale image, or data obtained by compressing them. And an image obtained by extracting the subject from the background using the background information. For the photographed image for authentication, the background processing unit 141 synthesizes an image of only the subject with a predetermined background image, for example, a background image in which a single achromatic color such as white, gray, or black is applied on one side. The background processing unit 141 synthesizes the image of only the subject with the background image of the user's preference set in advance for the captured image for actual shooting. The composite image synthesized with the subject-only image and the background image is stored in the data storage unit 150. With regard to the captured image for the main photographing, a new composite image can be regenerated by compositing another background image with the image of only the subject in the subsequent editing process. The background processing unit 141 may perform processing for extracting a subject in real time from the live view video being shot by the imaging unit 132 and combining the background image and causing the display control unit 135 to display them. The background processing unit 141 similarly performs the above-described chroma key composition on the moving image acquired by the image acquisition unit 133 to generate a combined moving image.

  The correction processing unit 142 performs person detection and person authentication in the captured image for main shooting, and also performs correction for the captured image for main shooting. The person detection in the photographed image for main photographing is, for example, face detection similar to the person detection in the photographed image for authentication. The correction processing unit 142 extracts a skin color region, an organ such as an eye and a nose from a captured image for actual photographing, and detects a face region. The person authentication in the photographed image for main photographing includes the same process as the person authentication in the photographed image for authentication and the person specifying process. More specifically, first, feature quantities for each person such as the distance, arrangement, and size of organs such as eyes and nose are detected from the face area detected by the person detection in the captured image for the main photographing. The feature quantity for each person is collated with the feature quantity for each user specified in advance in the photographed image for authentication and temporarily stored in the data storage unit 150, and based on the collation result, The process of identifying which user is the person included in the captured image is “person identification”. For example, when there are two pieces of feature amount information for each user to be specified and stored in advance, A and B, the feature amount for each person specified from the captured image for actual shooting is relative to either A or B. It is checked whether it approximates to. Thereby, which user is identified for each person included in the captured image for the main photographing.

  The correction for the captured image for the actual shooting is a process of changing the eye color and size, the face outline, the skin color and the texture, etc., and before the editing process (especially the graffiti process) after the actual shooting. It is executed on a captured image for shooting. The correction processing method differs depending on the number of photographers, and also differs depending on whether the image is an up-shot image or a whole-body image. When the preset number of photographers is three or less, individual correction for each person is executed by a correction method selected according to the preference preset for each user. The correction setting value specified for each user and the facial feature information for each user are transmitted to the management server 300 via the transmission output unit 163 and stored in association with the identification information for each user by the management server 300. The When the number of photographers exceeds three, the correction processing unit 142 performs the same correction on the entire person included in the photographed image for actual photographing. In addition, for example, an up-shot image and a whole-body image differ in the size of the part of the eye, including the eyes, and the ratio of the degree of impression of the eyes that affects the impression of the face or the whole body. The rate is slightly different.

  The editing processing unit 143 executes editing processing called graffiti, and plays for identifying a group of photographic images including a plurality of photographed images stored in the data storage unit 150 and a user specified by the individual authentication unit 131. Content data associated with an ID is generated. The content data may include the background information acquired by the background processing unit 141 and the identification information of the gaming device 100. The edit processing unit 143 generates content data every time the image acquisition unit 133 captures a series of photographic image groups (for each play). The content data can be uniquely specified using the play ID generated by the image acquisition unit 133.

  The editing processing unit 143 sets the captured image that has been subjected to the synthesis of the background image by the background processing unit 141 and the correction processing by the correction processing unit 142 as an editing target image, and executes editing processing on the editing target image. The editing process is executed individually for a plurality of editing target images, and can be executed simultaneously for a plurality of editing target images based on respective instructions from a plurality of users.

  Specifically, the editing processing unit 143 displays an editing screen including the composite image corrected by the correction processing unit 142 and editing tools such as characters, patterns, figures, and stamp images that can be superimposed on the composite image as decorations. The image is displayed on the editing monitor 213 and an instruction for editing processing for the composite image is received. In particular, the editing processing unit 143 initially displays an editing tool corresponding to the music received by the pre-setting unit 40 on the editing screen. The instruction of the editing process is input by a user editing operation on the editing monitor 213 using the touch pen 214, for example, a selection operation for selecting an editing tool or an input operation for specifying a position for an image to be edited (composite image). . The editing processing unit 143 performs editing corresponding to the editing tool selected by the user on the editing target image to generate a plurality of edited images, and stores them in the data storage unit 150. The editing processing unit 143 displays a selection screen for determining a printing image to be printed on the sheet paper on the editing monitor 213 and accepts the selection of the printing image.

  The moving image processing unit 144 performs an editing process on a plurality of moving images acquired by the image acquisition unit 133 and stored in the data storage unit 150, and generates a display moving image and a transmission moving image to be output.

  The output processing unit 160 includes a print processing unit 161, a wireless output unit 162, and a transmission output unit 163. The print processing unit 161 prints a printing image based on a plurality of edited images stored in the data storage unit 150 on a sticker sheet by the printer 34. The wireless output unit 162 transmits a plurality of edited images to the user terminal via the non-contact communication unit. The transmission output unit 163 transmits all edited images to the management server 300 via the network. The transmission output unit 163 receives identification information for identifying an individual such as an e-mail address input by the user on a keyboard displayed on the touch panel after the play ends. The transmission output unit 163 can upload the edited image to the management server 300 in association with the input mail address and play ID, and can download the edited image later by the user.

  The transmission output unit 163 transmits the display moving image set by the moving image processing unit 144 to the display control unit 135 to display the display moving image on a display device such as a liquid crystal monitor. The transmission output unit 163 uploads the transmission moving image or the captured moving image set by the moving image processing unit 144 to the management server 300 in association with the input e-mail address and play ID, and causes the user to download later. be able to. The management server 300 stores the edited image and the moving image for transmission (or the captured moving image) in a form corresponding to the e-mail address and the play ID, and the play ID associated with the e-mail address is addressed to the e-mail address. Is transmitted as content access information. The user opens a page on the predetermined image acquisition website where the edited image or the moving image for transmission can be downloaded by accessing the URL of the received content access information with the terminal device 500 such as a mobile phone. Can be downloaded. In addition, a play ID is printed on the sticker paper together with a print image based on the edited image, and the user can input his / her edited image or a moving image for transmission by inputting the play ID into a predetermined image acquisition website. A page where the image can be downloaded can be opened and downloaded. When the user registers an e-mail address as a free member, a page where an image selected by the user among the plurality of edited images uploaded to the management server 300 can be acquired can be opened. When the user registers an e-mail address as a paying member, a page where all of the plurality of edited images uploaded to the management server 300 can be acquired can be opened.

  FIG. 7 is a functional block diagram schematically showing the functional configuration of the management server 300. The management server 300 stores the feature information and correction default values for each user in association with the identification information of the user, as well as user information such as content data acquired from the shooting apparatus for play 100 and a usage history of the photography service. It is a server that manages The content data is data such as a play ID, a group of photographic images, and an e-mail address input by the game shooting apparatus 100. The management server 300 includes a data storage unit 310 that stores user data, a communication processing unit 320 that communicates with the terminal device 500, the game shooting device 100, and the SNS site 400 via the network 200, and an identification process that identifies the user. Unit 330 and an authentication processing unit 340 for authenticating a user.

  The data storage unit 310 includes an identification information storage unit 311, a feature storage unit 312, a default value storage unit 313, and an image storage unit 314. The identification information storage unit 311 stores identification information for each user. The feature storage unit 312 stores feature information for each user in correspondence with identification information. The user identification information may be any of information mainly consisting of character strings such as e-mail addresses, user names, and passwords, and biometric information such as fingerprint information. The default value storage unit 313 stores a correction default value for each user in correspondence with the identification information. The image storage unit 314 stores content data such as a photographic image group transmitted from the game shooting apparatus 100.

  The identification processing unit 330 collates the feature information transmitted from the terminal device 500 with a plurality of feature information stored in the feature storage unit 312, and the authentication processing unit 340 obtains user identification information corresponding to the matching feature information among them. The data is transmitted to the terminal device 500 via the communication processing unit 320. The feature information matching method is the same as the matching in the terminal device 500 described above. When the authentication processing unit 340 receives information indicating that the user identification information is correct from the terminal device 500 via the communication processing unit 320, the default value for the correction of the user is transmitted to the terminal device 500 via the communication processing unit 320. Send.

  FIG. 8 is a flowchart showing a process when the user uses the application of this embodiment for the first time. When using this application, it is necessary to register facial feature information and the like as user settings in the terminal device 500. There are two types of user settings: a method for setting in advance before the first shooting, and a method for setting after shooting first.

  When the user downloads an application from a predetermined site and installs it on the terminal device 500 (S10), the application is ready for use. When the user starts the application (S12), at this time, the application enters a standby state in which shooting can be performed immediately without prompting the user to execute terminal user settings, and display of a live view image captured by the camera 515 is started. (S140). This is because the specifications that cannot be shot without user settings immediately after installation may cause the user to feel psychologically troublesome in setting, and the first shooting can be performed without user settings. The setting is postponed so as to reduce the annoyance. As a modified example, it may be a specification that displays that the terminal user setting can be executed at this time, that is, at the first activation after the application is installed, and that the terminal user setting can be arbitrarily started.

  After starting the live view display, simple correction is performed on the live view video (S16). In this case, the simple correction is performed by detecting the face of one or more users from the live view video, and performing simple correction with the same set value regardless of whether or not the terminal user is included in the user. Apply. However, in a modified example, the terminal user may be specified, and simple correction or full-scale correction may be performed on the terminal user based on a default value associated with the terminal user. In the photographing enabled state, when the user instructs execution of various settings including terminal user settings from a predetermined menu (Y in S18), terminal user settings are executed (S20). Details of the terminal user setting will be described later. While there is no setting execution instruction (N in S18) and there is no shooting operation (N in S22), the process returns to S16 and repeats real-time simple correction on the live view video. Here, when a photographing operation such as a shutter button operation is performed by the user (Y in S22), the main photographing is performed (S24), and correction is performed on the image range for each user included in the photographed image (S24). S26). Details of the correction will be described later. After the correction, if the terminal user is not set (Y in S28), the terminal user setting is executed (S30), and if the terminal user has been set, S30 is skipped and one shooting is performed. The process ends (N in S28). The terminal user setting in S30 will be described later, but the procedure is different from the terminal user setting in S20. Thereafter, the processing from S14 to S30 is repeated until the application is terminated. That is, correction is executed every time one image is taken, and it is not a method in which correction is performed collectively after taking a plurality of photographs. Thus, every time the application is activated, the processing from S14 to S30 is repeated by the number of times of photographing.

  FIG. 9 is a flowchart showing in detail the terminal user setting process in S20 of FIG. First, the terminal user himself is photographed alone by the camera 515 to obtain a still image of the terminal user (S50). However, the specification which acquires a user's image as a live view image | video as a modification may be sufficient. A person (especially a face portion) is detected from the image of the terminal user (S52), and feature information is detected from the face of the person (S54). The user inputs a correction setting value for registration as a correction default value (S56), and the user inputs identification information such as an e-mail address and a user name (S58). While registering with the device 500, the same information is also transmitted to the management server 300 (S60). In S50 to S52, when a plurality of users are included as subjects and a plurality of persons are detected, a dialog for causing the user to select which person to register as a terminal user is displayed and selected by the user. Face feature information is detected for each person, and this is registered with the terminal device 500 and the management server 300 in association with the identification information. In the modification, the specification may be such that the feature information of the terminal user is registered only in the terminal device 500.

  FIG. 10 is a flowchart showing in detail the correction processing in S26 of FIG. First, one or a plurality of persons (particularly a face portion) are detected from the photographed image (S100), and feature information is detected from the face of the person (S102). If the terminal user setting has already been made (Y in S104), the characteristic information of the terminal user is read and collated (S106). If the terminal user is identified from the subject (Y in S108), the face image of the terminal user is displayed. In addition to displaying the enlarged image, the terminal user's correction default value is read and set, and correction based on the set value is executed on the terminal user's image range (S110). If the setting value is changed according to the preference of the terminal user, the correction based on the changed setting value is executed again. If no terminal user is specified in S108, S110 is skipped (N in S108). If the terminal user setting is not made in S104, S106 to S110 are skipped (N in S104).

  If a user other than the terminal user is included, the feature information of the other user is acquired from the management server 300 and collated (S112). If the user can be identified (Y in S114), The uncorrected face image of one person is enlarged and displayed, and the specified correction value of the specified user is acquired from the management server 300 and set (S116), and the correction based on the set value is set in the image range of the user. Execute (S118). If the setting value is changed according to the preference of the user, the correction based on the changed setting value is executed again. If the user is not specified in S114, S116 is skipped (N in S114), and correction is performed based on the set value according to the user's preference (S118). If there are other uncorrected users among the users included in the subject, the process returns to S114 (Y in S120), and the correction processing in S114 to S118 is repeated for the number of users. If there is no uncorrected user, the flow of this figure is ended (N in S120). When the terminal user is not set, correction is also performed for the terminal user in the process of S112 to S120. In that case, if there is an instruction to set the terminal user when the correction for the terminal user is executed in S118, the process proceeds to the flow of FIG. 11 to set the terminal user.

  FIG. 11 is a flowchart showing in detail the terminal user setting in S30 of FIG. The terminal user setting flow in this figure is different from the terminal user setting in FIG. 9 and has already been photographed, and the person and face feature information has already been detected, so these processes are omitted. If a plurality of persons have already been detected, the user is allowed to select which person is set as the terminal user (S150). However, if the detected person is one person, the user is confirmed whether the person can be set as the terminal user. In addition, when there is an instruction to set a terminal user in S118 of FIG. 10, it is clearly indicated which person's correction setting value is used to register the default value of the terminal user, and thus S150 is omitted.

  For the person specified as the terminal user, the user is allowed to input identification information such as an e-mail address and a user name (S152). The characteristic information of the specified person and the set value of correction already made for the person are registered as terminal user settings in the terminal device 500 in association with the identification information, and the same information is also transmitted to the management server 300. (S154).

  FIG. 12 is a diagram schematically illustrating an example of a setting screen when terminal user settings are executed before the first shooting. In the terminal user setting of FIG. 9, the screen of this figure is displayed. First, a live view image captured by the sub camera 517 is displayed in the image frame 610, and the terminal user's face is displayed in an up position. Under the image frame 610, by displaying an instruction to the user such as “Look forward so that the face fits in the image frame and take a picture when the lamp is lit”, face the front to the terminal user. Position adjustment is urged so that a large face appears in the image frame 610. Initially, the shutter button 634 is grayed out so that it cannot be operated. When a face can be detected from the live view video, the preparation lamp 633 is lit in green and the shutter button 634 is displayed in an operable state. When the terminal user operates the shutter button 634, a face picture of the terminal user is taken, and a still image is displayed in the image frame 610. The feature detection unit 522 detects facial feature information from the face photograph of the terminal user, and registers it in the feature storage unit 552 and the management server 300 as the feature information of the terminal user. If the terminal user is not satisfied with the photograph of his / her face, he / she can operate the shutter button 634 again to take a picture again. When he / she is satisfied with the photograph of the image, he / she operates the transition button 635 written “Next”. Then, the screen transitions to the identification information registration screen shown in the next figure. In this case, the feature detection unit 522 detects facial feature information from the recaptured facial photograph, and re-registers it in the feature storage unit 552 and the management server 300 as the official feature information of the terminal user.

  FIG. 13 is a diagram schematically showing a screen example of a screen for registering identification information as terminal user settings. The image frame 610 displays the image taken in the previous figure, and a user name field 636, an address field 637, and a password field 638 are displayed below the image frame 610. The user inputs his / her favorite user name in the user name column 636, inputs his / her e-mail address in the address column 637, inputs an arbitrary password in the password column 638, and then operates the registration button 632. As a result, the user identification information is registered, and the screen shifts to the correction screen in the next figure. Here, the following will be described on the assumption that the user name of the terminal user is registered as “A”. In the present embodiment, the specification is such that the identification information is registered first and then the default correction value is registered. However, the order in which the identification information is registered after the correction is executed first and the correction default value is registered. Also good. When a terminal user who has already been registered as a user reinstalls the application or installs an application in another terminal device 500 due to a model change or the like, the registered user name or Enter your e-mail address and password. The input user name or e-mail address and password are transmitted to the management server 300, and when authenticated as correct information, the registered terminal user feature information and the like are transmitted from the management server 300 and registered in the terminal. In this way, the terminal user's environment can be transferred or reset to the new terminal device 500 or the like.

  FIG. 14 is a diagram schematically illustrating a screen example that allows the user to select “taste” as the first correction. The touch panel screen 601 displays the face image of the terminal user “Mr. A” in the image frame 610, and also displays the nickname of “Mr. A” who is the correction target person as the correction target user name 620. “Mr. A” indicates that correction is in progress. Below the image frame 610, three choices of “cute”, “beautiful”, and “natural” are displayed as a taste selection button 621 to allow the user to select. When the user operates any of the selection buttons, highlighting is performed such that the selection button is surrounded by a thick frame and the color changes, and correction corresponding to the selection button is performed on the image in the image frame 610. At the bottom of the touch panel screen 601, buttons indicated as “correction complete” and “next correction” are displayed as screen switching selection buttons 630. When the user performs a touch operation on “next correction”, the selection of the correction item “taste” on this screen is completed, and the screen is switched to the setting screen for the next correction item. On the other hand, when the user touches “Correction Complete”, the correction contents at the present time are determined without moving to the next correction item, and all correction items are completed.

  FIG. 15 is a diagram schematically illustrating an example of a screen that allows the user to select “finished image” as the second correction. On the touch panel screen 601, an image frame 610 and a correction target user name 620 are the same as those in FIG. 14 on the previous screen. Below the image frame 610, an eye selection button 622, an outline selection button 623, and a nose selection button 624 are displayed. As the eye selection button 622, three options “eye 1”, “eye 2”, and “eye 3” indicating different eye sizes are displayed, and initially “cute” corresponding to the “cute” taste selected on the previous screen is displayed. “Eye 3” is selected and highlighted. At this point, correction based on the set value “eye 3” has already been performed. As the contour selection button 623, three options “contour 1”, “contour 2”, and “contour 3” indicating different contour shapes are displayed, and “contour 1” corresponding to the “cute” taste selected on the previous screen is initially displayed. "Is selected and highlighted. At this point, correction based on the set value “contour 1” has already been performed. As the nostril selection button 624, three options “nostril 1”, “nasal nose 2”, and “nasal nose 3” indicating different nostral shapes are displayed, and “nose 1” corresponding to the “cute” taste initially selected on the previous screen is displayed. "Is selected and highlighted. At this time, the correction based on the set value “small nose 1” has already been performed. When the user touches another button, the highlighted object is switched to the touched button, and the correction applied to the face image in the image frame 610 is also corrected based on the correction setting value corresponding to the item selected by the user. Switch.

  At the bottom of the touch panel screen 601, buttons labeled “return”, “correction complete”, and “next correction” are displayed as screen switching selection buttons 630. When the user touches “RETURN”, the screen returns to the previous screen in FIG. 14, and when “NEXT CORRECTION” is touched, selection of the correction item “Finished” on this screen is completed and the next correction item is set. Switch to the screen. When the user touches “Correction Complete”, all correction items are completed without moving to the next correction item.

  FIG. 16 is a diagram schematically illustrating a screen example that allows the user to select “skin brightness” as the third correction. On the touch panel screen 601, the image frame 610 and the correction target user name 620 are the same as those in FIGS. Below the image frame 610, three choices “Bright Black”, “Recommended”, and “Whitening” indicating different skin brightness are displayed as a skin selection button 625, and initially selected by the user on the screen of FIG. “Whitening” corresponding to the “cute” taste is selected and highlighted. At this point, correction based on the set value “whitening” has already been performed. At the bottom of the touch panel screen 601, buttons labeled “return” and “correction complete” are displayed as screen switching selection buttons 630. When the user touches “Return”, the screen returns to the previous screen of FIG. 15, and when “Correction Complete” is touched, correction is completed for all correction items and terminal user settings are also completed. Identification information, correction default values, and facial feature information registered as terminal user settings can be changed as needed based on user instructions, and once captured face images are stored in the image storage unit 554 as “profile photos”. It is stored and displayed every time the terminal user setting screen is opened, but this picture can be taken again and replaced as shown in FIG. The image stored in the image storage unit 554 as the “profile photo” is also transmitted to the management server 300 and is stored in the management server 300 in association with the identification information.

  In FIGS. 14 to 16, the description has been given of the specification in which options are presented with buttons for a plurality of correction items, and the user is allowed to select. In a modified example, an interface that can be freely deformed or corrected by directly touching a face image of the image frame 610 with a finger instead of a button operation may be used. For example, if you want to change the shape or size of your eyes, contours, nose, etc., you can touch with two fingers to narrow the two fingers, or “pinch out” to spread the two fingers An interface that can adjust the shape and size in multiple stages may be used. In another modified example, an interface may be used in which correction is performed by sequentially switching among a plurality of options each time a touch operation is performed on a portion such as an eye, an outline, or a nose. For example, if the eye part is touched in a state where the correction of “eye 1” is reflected, the correction is switched to the correction of “eye 2”, and “eye 3”, “eye 1”, and “eye 2” are reflected each time the touch is further performed. In this way, correction is performed in a circulating manner. In these modified examples, it is not necessary to display a selection button, so that not only the correction stage but also the multistage adjustment can be made, and the screen configuration can be simplified. Further, it is not necessary to switch the screen as in FIGS. 14 to 16, and the correction can be executed in any order without being restricted by the order of correction.

  FIG. 17 shows an example of a screen when the terminal user executes the first shooting without setting. When the terminal user performs the first shooting without setting, the screen of this figure is displayed, and it is set whether or not the person included in the captured image as the subject is the terminal user. The example shown in the figure is a case where two persons are detected, and the first user frame 611 and the second user frame 612 are displayed so as to surround each face. Initially, a dotted user frame is displayed, and a question such as “Which is the terminal user?” Is displayed to prompt the user to select one of the frames. For example, when the user selects the first user frame 611 on the left side, the person surrounded by the first user frame 611 is specified as the terminal user, and the process proceeds to the correction processing of FIGS. At this time, the specification may be changed to the identification information registration screen shown in FIG. 13 before the correction processing, and the terminal user identification information is registered first. Or it is good also as a specification which makes a user select the registration timing of a terminal user's identification information before and after a correction process like "registration immediately" "registration later". When registering the identification information of the terminal user after the correction processing, even if the correction is completed by operating the “correction completed” button in FIG. 16, the terminal user identification information registration screen in FIG. Good. Alternatively, after waiting for correction for all persons, the terminal user identification information registration screen in FIG. 13 may be displayed. When the correction process for the terminal user is completed on the screen of FIG. 16, the set value for the correction is registered as the default value for the correction of the terminal user. When registration of the terminal user's correction default value or identification information is completed, the screen transitions to the next screen. When the user operates the skip button 640, the correction process of FIGS. 14 to 16 is advanced one by one without specifying the terminal user and other users.

  FIG. 18 is a diagram schematically illustrating a screen example for detecting a person from an image and specifying a user. Since the person on the left has already been specified as the terminal user, the user name “Mr. A” of the terminal user is displayed as the first user name 613 under the first user frame 611. On the other hand, under the second user frame 612 of the right person, the user name obtained by matching the feature information of the person's face by the feature matching unit 523 is a second user with a question mark such as “Mr. Displayed as name 614. When the user selects the second user frame 612, the second user frame 612 is changed to a thick frame highlight to indicate that it is in a selected state, and the authentication processing unit 560 displays “Yes” and “No” buttons (FIG. When the user touches “Yes”, the password field 615 is displayed and “Please enter the password” is displayed. The authentication processing unit 560 encrypts the password input by the user and transmits the encrypted password to the management server 300. When the response indicating that the password matches the password associated with “Mr. B” is received, The person is determined to be “Mr. B”, and the person is shifted to the correction processing of FIGS. In this way, the terminal user and other users differ in whether input of identification information such as a password is requested as identity verification. This is because the possibility of misrecognition is low because the possibility of being included as a subject is always high for terminal users, and there is little problem of personal information protection. The degree of request for identity verification is increased from the viewpoint. In a modification, biometric information such as fingerprint information may be input instead of a password, and the biometric information may be encrypted and transmitted to the management server 300 for authentication.

  As a result of collation for the user in the second user frame 612, when there are a plurality of feature information whose total score exceeds a predetermined reference value, a plurality of user candidates can be displayed one by one in descending order of score. May be. For example, “Mr. B?” Having the highest score is displayed as the second user name 614, and when the user selects “No”, the display is switched to the display of “Mr. C?”, Which is the score of the next point. If "" is selected, the display may be changed to the display of the next score, "Mr. D?". Then, the presentation of candidates may be repeated until the user selects “Yes” to obtain password authentication or displays all of a plurality of user candidates exceeding a predetermined reference value. Further, when the user touches “No”, the display of the user name is once deleted. However, when the user touches the user frame again, the user name of the next candidate may be displayed. Or, when “No” is touched, options such as “re-specify the user” and “manually input the user name” are displayed, and the user can select whether to specify the user automatically or manually. Also good. Alternatively, another user may be specified by a method described later as the second embodiment. When the user selects “respecify user”, the user name of the next candidate is displayed. When the user selects “manually enter user name”, the user name and password input fields are displayed, the user is prompted to enter each of them, and the information is encrypted and transmitted to the management server 300. When the user name of the user to be returned is returned, that user is confirmed. As a modification, when there are a plurality of pieces of feature information whose total score exceeds a predetermined reference value as a result of collation for the user in the second user frame 612, the option for the plurality of user candidates is “Mr. B?”. It is good also as a specification which presents at a time and makes a user select any one like "C-san?" "D-san?". When the selection of the user is confirmed, the process shifts to the correction process of FIGS. When the identification and correction process for one or more users included in the subject is completed, the first shooting is finished. However, the user selects “No” or the like, and the process proceeds to the correction process without specifying some users. In this case, the screen switching selection button 630 may be touched. If the person in the first user frame 611 is a registered user as a terminal user as in the example of FIG. 18 and the person in the second user frame 612 is not registered as a user, the second user frame 612 is used. User registration may be made for this person. For example, when the second user frame 612 is touched, an option such as “register user” is further displayed in addition to the options such as “re-specify the user” and “manually enter the user name”. May be selected, the screen may change to the screens of FIGS. 13 to 16 and user registration and correction values may be set.

  Here, in the case of correction in the shooting device for play, it is necessary to process the correction in as short a time as possible, so when there are a plurality of users included as subjects, it is difficult to set the correction content individually, The larger the number of people, the fewer correction items that can be set. In the application of the present embodiment, since there is no particular time restriction like the shooting device for games, it is configured so that each person can set his / her preference for all correction items regardless of the number of users. Yes. As a correction procedure, a predetermined number of users included in the subject sequentially set a favorite value for each correction item, or the correction procedure is switched when the number of users is equal to or greater than the predetermined number. In the example of this embodiment, it is assumed that one user can set a desired value for each correction item at a time, and that all users are set sequentially.

  In the second and subsequent shootings, particularly when the terminal user setting has already been registered, the user is automatically specified immediately after shooting the image, and the screen of FIG. 18 is displayed. If another user has already been authenticated by identification information such as a password, the user is determined without re-entering the identification information, and the correction processing of FIGS. 14 to 16 is executed for each person. At this time, since the default values of the respective corrections can be read for the terminal user and other users, various correction processes are automatically executed without the user inputting or changing the correction setting values again. It may be. Or it is good also as a specification which does not automatically perform to a correction process, but performs to the setting of a default value automatically. In this case, the correction screen is selected from the beginning based on the default value. For example, in FIG. 14, an emphasis display is made such that the “cute” button, which is the default value for the correction of the terminal user “Mr. A”, is initially selected and the color changes with a thick frame. And the appearance is shown to be selected. In addition, the face image displayed in the image frame 610 has already been corrected based on a setting value registered in advance as the user's correction default value. When the user touches another button, for example, “Beautiful” or “Natural”, the highlighted object is switched to the touched button, and the correction applied to the face image in the image frame 610 is also applied to the taste selected by the user. Switch to the correction based on the corresponding correction setting value. In FIG. 15, “Eye 3”, “Outline 1”, and “Nose 1”, which are default values of “Mr. A”, are initially selected and highlighted. The face image displayed in the image frame 610 has also been corrected based on the default values “Eye 3”, “Outline 1”, and “Nose 1”. When the user touches another button, the highlighted object is switched to the touched button, and the correction applied to the face image in the image frame 610 is also corrected based on the correction setting value corresponding to the item selected by the user. Switch. In FIG. 16, “whitening”, which is the default value of “Mr. A”, is initially selected and highlighted. The face image displayed in the image frame 610 has already been corrected based on the default value “whitening”. When the user touches another button, the highlighted object is switched to the touched button, and the correction applied to the face image in the image frame 610 is also corrected based on the correction setting value corresponding to the item selected by the user. Switch.

(Second Embodiment)
This embodiment is different from the first embodiment in that information such as user face feature information, correction default values, and identification information is transmitted and received between the terminal devices 500 without using the network 200. That is, in the first embodiment, facial feature information, correction default values, identification information, and the like are acquired from the management server 300 via the network 200 for users other than the terminal user. Obtained from another terminal device 500 using the distance wireless technology. Hereinafter, the description will focus on differences from the first embodiment, and description of common points will be omitted.

  FIG. 19 is a diagram schematically illustrating an example of a user specifying screen in the second embodiment. First, the terminal user is automatically identified by collating with the feature information, and the point of displaying “Mr. A?” As the first user name 613 is the same as in the first embodiment. At the first time, feature information is not acquired and user identification is not performed. When the user performs a touch operation on the second user frame 612 in which the user is not specified, a first search button 642 written “search near” and a second search button 643 written “search by shaking” Is displayed.

  When the user selects “Find Near”, a terminal installed with the same application is detected in the short-range wireless communication range, and the user name is displayed as a selection candidate. “Can I send user information?” Is displayed. If the terminal user agrees, the user's characteristic information, correction default value, and identification information stored in the application are transmitted and can be received on the original application side.

  When the user selects “Search by shaking”, “Please shake your device” is displayed. When the user performs a gesture such as shaking the terminal to apply vibration, the terminal that has detected the vibration by the acceleration sensor acquires position information by the position acquisition unit 583 and transmits the position information to the management server 300 together with the identification information. At this time, if vibration is applied to another user's terminal at almost the same time or just before and after, the application on the terminal side similarly acquires the position information by the position acquisition unit 583 and the management server 300 together with the identification information. Send to. In the management server 300, mutual identification information is transmitted to each of a plurality of terminals that have transmitted location information that approximates the approximate time, and both terminals establish a direct connection using short-range wireless communication. For example, the identification information of terminal B is transmitted to terminal A, the identification information of terminal A is transmitted to terminal B, and terminal A and terminal B are connected by short-range wireless communication. The identification information in this case may include the MAC address of the terminal. Upon receiving the identification information of “Mr. B”, the terminal of “Mr. A” shown in the figure displays the second user name 614 of “Mr. B” below the second user frame 612, and features from the terminal of “Mr. B”. Information such as information and default values for correction is received by short-range wireless communication. As a result, the person in the second user frame 612 that is first touch-operated is identified as “Mr. B”, and correction based on the correction default value of “Mr. B” is performed on the subsequent screens in the face image range. Is done.

  As described above, when a user is specified using “search near” and “search by shaking”, since the user's terminal is close and the specific content is likely to be correct, the first embodiment and Differently, input of identification information such as a password is not required. However, in a modified example, it may be configured to request further input of identification information such as a password in order to further increase the certainty. In another modified example, the correction default value may be acquired from the management server 300 by simply using “search near” and “waving search” in order to recognize nearby terminals. It is good also as a structure which connects in the ad hoc mode of wireless LAN, and receives directly. In the second embodiment, for users other than the terminal user, the user is specified by “search near” and “search by shaking”, and collation based on facial feature information is not used. The feature information is acquired and stored in the data storage unit 550. In addition, in specifying the user in the second and subsequent shot images, other users are specified by collation based on feature information as in the first embodiment, without using “search near” and “search by shaking”. Can do. However, in the modification, the configuration is such that the feature information of other users is not acquired, and the user is identified by “search near” and “search by shaking” in the second and subsequent captured images as in the first image. It is good.

  In another modification, a user specifying method using a two-dimensional code may be further provided. An application can be provided with a function for displaying its identification information in a two-dimensional code, and the identification information can be transmitted by scanning the displayed two-dimensional code with a camera of another terminal.

  The image processing system 600 according to the embodiment of the present invention has been described above. The present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention.

  In each of the above embodiments, the configuration in which the feature information of the terminal user, the correction default value, and the identification information are stored in the data storage unit in the terminal has been described. In the modified example, the feature information of the terminal user, the default value of correction, and the identification information are not stored in the terminal, but are acquired from the management server 300 in the same manner as other users and authenticated by the identification information in the same manner as other users. It is good also as a system which specifies a user through.

  In each of the above-described embodiments, the configuration for correcting the user's face has been described. However, in practice, the user can take not only a face photo but also a whole body photo using this application. Therefore, in the modified example, not only the face correction but also the whole body style correction, for example, the length of the leg, the narrowness of the second arm, the narrowness of the waist, and the like may be used. In this case, the whole body captured image of the user is displayed in the image frame 610 of FIGS. 14 to 19, the user selects and sets based on the image, and correction is performed on the image.

  200 network, 310 data storage unit, 320 communication processing unit, 330 identification processing unit, 510 imaging unit, 520 identification processing unit, 530 image acquisition unit, 540 correction processing unit, 550 data storage unit, 571 display control unit, 580 communication processing Department.

Claims (1)

An imaging unit that captures an image including a user in a subject;
An image acquisition unit for acquiring a captured image of a user imaged by the imaging unit;
An identification processing unit that identifies a user included in the subject by person detection and person authentication in the captured image;
A correction processing unit for performing correction on the captured image;
A display control unit for displaying the captured image on a screen;
With
When the correction processing unit executes correction on a captured image including a plurality of users as subjects, the correction processing unit reads a preset correction value stored in advance for each user for each user specified by the identification processing unit, An image processing apparatus, wherein correction according to the read default value can be performed on each image range of the plurality of users.

Images