JP2016019092A - Information processor, information processing method, information processing system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of allowing a user to perform satisfactory self-photographing close to a camera eye line.SOLUTION: The information processor includes a control section that generates an eye line correction image in which an image of a part of an eye in one image based on an image of a part of an eye in the other image in two images which include a face of a person taken from different positions. The control section replaces, in the two images, the partial image of the eye in the one image which is taken from the relatively lower position with the partial image of the eye in the other image which is taken from the relatively upper position to generate an image with corrected eye line.SELECTED DRAWING: Figure 1

Description

  The present technology relates to an information processing apparatus, an information processing method, an information processing system, and a program having a function of processing a user's face image captured by a camera.

  Self-portrait is one way to enjoy mobile terminals such as mobile phones and smartphones. That is, many mobile terminals are equipped with a camera, and an image photographed by the camera can be displayed on a display screen in real time. Therefore, the user can easily photograph his / her face with a desired expression while checking his / her face on the display screen.

  Usually, the camera is arranged at a position deviating from the position of the display screen. For this reason, the image of the user's face photographed while looking at the display screen is slightly shifted in line of sight.

  A technique for correcting a user's gaze shift of a subject by correction is already known (see, for example, Patent Document 2). In the technique of Patent Document 2, in the camera-equipped mobile phone, the face display position is calculated from the center of the user's black eye calculated from the registered camera line-of-sight image and the user's face image captured by the upper camera and the lower camera, respectively. The amount of eyeline deviation is calculated from the amount of deviation from the center of the black eye in the corrected image. Based on the amount of line-of-sight deviation, an image in which the line of sight is corrected so as to be equivalent to the line of sight of the camera is generated.

JP 2012-114557 A (paragraph [0051])

  When a person's face is photographed using a camera built in a portable terminal such as a smartphone, only faces whose eyes are shifted in the same direction in the same direction tend to be photographed. Some users try to shoot with various expressions to take their best shot. However, due to the above-mentioned limitations, there are still insufficient points for good self-portrait close to the camera line of view, such as restrictions on variations in facial expression of the camera line of view that can be taken.

  In view of the circumstances as described above, it is an object of the present technology to provide an information processing apparatus, an information processing method, an information processing system, and a program that can satisfactorily take a self-portrait close to the camera line of sight.

  In order to solve the above-described problem, the information processing apparatus according to the first embodiment of the present technology applies an image of an eye portion of one of the two images including a human face photographed from different positions. And a control unit that generates a line-of-sight correction image in which the image of the eye portion in the other image is changed.

  In the information processing apparatus according to the first aspect, the control unit is configured to capture the person's face photographed by the first imaging unit and the second imaging unit from different positions in the vertical direction of the face of the person who is the subject. May be configured to acquire two images.

  In the information processing apparatus according to the first aspect, the control unit may extract a partial image of the eye of one of the two images acquired from a relatively lower position as the relative image. It may be configured to generate the line-of-sight correction image in which the partial image of the other image captured from the upper position is replaced.

  In the information processing apparatus according to the first aspect, the first imaging unit is an upper imaging unit in the vertical direction of the face of the person who is the subject, and the second imaging unit is the subject. The image capturing unit may be a lower image capturing unit in the vertical direction of the human face, and the first image capturing unit may have a higher quality of the captured image than the second image capturing unit.

  In the information processing apparatus according to the first aspect, the control unit is configured to perform signal processing on at least one of the images so as to adjust the tone of the two acquired images. It may be.

  In the information processing apparatus according to the first aspect, the control unit is configured to cause the display unit to simultaneously display at least one of the line-of-sight correction image and the acquired two images. It's okay.

  In the information processing apparatus according to the first aspect, the first imaging unit is incorporated in the information processing apparatus, and the second imaging unit is a separate unit that is detachably attached to the information processing apparatus. It may be an imaging unit.

  An information processing apparatus according to a second embodiment of the present technology includes an imaging unit that captures a person's face from different positions, and two images including the person's face captured by the imaging unit. And a control unit that generates a line-of-sight correction image in which the image of the eye part in the other image is changed based on the image of the eye part.

  In the information processing method according to the third aspect of the present technology, the control unit performs processing based on the image of the eye portion of one of the two images including the face of the person photographed from different positions. A line-of-sight correction image is generated by changing the image of the eye portion in the image.

  The information processing system according to the fourth aspect of the present technology is based on an image of an eye portion of one of the two images including the face of a person photographed from different positions. A control unit that generates a line-of-sight correction image obtained by changing the image of the part.

  The program according to the fifth aspect of the present technology is based on an image of an eye part in one image of two images including a human face photographed from different positions, and the eye part in the other image Is a program that causes a computer to function as a control unit that generates a line-of-sight correction image obtained by changing the image.

As described above, according to the present technology, it is possible to satisfactorily take a self-portrait close to the camera line of sight.
In addition, this effect is not limited as an effect of this technique, Any effect described in this indication may be an effect of this technique.

It is a figure showing the whole face image processing system composition of a 1st embodiment concerning this art. It is a figure which shows operation | movement until a synthetic | combination image by which eye line correction | amendment was performed from the face image of the user U each imaged by the two camera apparatuses 10 and 20 in the face image processing system of FIG. FIG. 7 is a diagram showing another operation on an image basis until a composite image in which eye line correction is performed is obtained from the face images of the user U captured by the two camera devices 10 and 20 in the face image processing system of FIG. 1. is there. It is a perspective view showing composition of face image processing system 1A of a 2nd embodiment concerning this art. FIG. 5 is a perspective view showing a state in which an external camera 10A is mounted on a portable terminal 40A in the face image processing system 1A of FIG. It is a block diagram which shows the hardware constitutions of 40 A of portable terminals. It is a perspective view which shows the external appearance of 10 A of external cameras. It is a perspective view which shows the imaging | photography state of 10 A of external cameras of FIG. It is a block diagram which shows the hardware constitutions of 10 A of external cameras of FIG. It is a figure which shows the 1st form which the user U image | photographs his / her face using the face image processing system 1A of FIG. It is a figure which shows the 2nd form which the user U image | photographs his / her face using the face image processing system 1A of FIG. It is a figure explaining the modification using four cameras.

Hereinafter, embodiments according to the present technology will be described with reference to the drawings.
<First Embodiment>

[Overall configuration of face image processing system]
FIG. 1 is a diagram illustrating an overall configuration of a face image processing system according to the first embodiment of the present technology.
The face image processing system 1 includes two camera apparatuses 10 and 20 (imaging units), a line-of-sight object 30 that is a target for the user U to match the line of sight E during shooting, and an information processing apparatus 40.

  More specifically, the line-of-sight object 30 may be a screen of a display that displays an image of the face of the user U taken by at least one of the two camera apparatuses 10 and 20. However, the line-of-sight object 30 is not necessarily a display screen. The line-of-sight object 30 may be a mirror, for example. Alternatively, an object with some mark that guides the user's U's line of sight E may be used. In the configuration of the present technology, the line-of-sight object 30 is not essential.

  The two camera apparatuses 10 and 20 are arranged so that the user U's face can be photographed from two different positions in the vertical direction of the user U's face. That is, the camera device 10 captures the face of the user U from a relatively upper position in the vertical direction, and the camera device 20 captures the face of the user U from a relatively lower position in the vertical direction. Placed in. More specifically, for example, the two camera devices 10 and 20 are configured such that the face of the user U who is aligned with the line-of-sight object 30 is slightly shifted upward from the height position of the line of sight E of the user U. It arrange | positions so that it can image from the height position which shifted | deviated a little from the position and the height position of the user's U eyes | visual_axis E, respectively. When the user E's line of sight E is facing directly in front, when the user U's face is photographed from a position slightly higher than the line of sight E by one camera device 10, the entire face of the user U in the photographed image includes the line of sight E somewhat. It looks down. When the other camera device 20 captures the face of the user U from a height slightly lower than the line of sight E, the entire face of the user U in the captured image is slightly upward including the line of sight E.

  In the following description, the “upper camera device 10” is the camera device 10 that captures the face of the user U from a position slightly higher than the line of sight E, and the “downward” is the camera device 20 that captures the face of the user U from a position slightly lower than the line of sight E. It will be referred to herein as “camera device 20”.

Each of the camera devices 10 and 20 includes a photographing lens, an image sensor, a development processing unit that develops an output of the image sensor, and a controller. Each of the camera devices 10 and 20 may be a camera mounted as an imaging unit on the housing 41 (see FIG. 4) of the information processing device 40. Alternatively, one camera may be a camera built in the information processing apparatus 40, and the other camera may be a camera that can be attached to and detached from the information processing apparatus 40, or a camera that is disposed independently of the information processing apparatus. . Alternatively, each of the camera devices 10 and 20 may be a camera installed independently with respect to the information processing device 40.
A more specific configuration of each of the camera devices 10 and 20 will be described in the second embodiment.

  The information processing apparatus 40 includes a communication interface for communicating with the two camera apparatuses 10 and 20 in a wired or wireless manner, a CPU (Central Processing Unit), a memory, a storage, an input unit for a user, and the like. . The CPU of the information processing device 40 controls the two camera devices connected through the communication interface, and performs a process of combining two images captured by the two camera devices 10 and 20, respectively.

Specifically, the information processing device 40 may be, for example, a smartphone, a personal computer, or a server device connected to the two camera devices 10 and 20 through a network.
A more specific configuration of the information processing apparatus 40 will be described in a second embodiment.

[Gaze correction by image composition]
In the face image processing system 1 described above, the information processing apparatus 40 can perform eye line correction as one of the image synthesis processes.
The line-of-sight correction is as follows.

  Here, eye line correction when a smartphone is used as the information processing apparatus 40 will be described. For example, when a user shoots his / her face with the camera built in the smartphone, the user operates the shutter button at any time while checking his / her face displayed in real time on the display screen, which is the object of eyesight. To do. Then, the image shot (displayed) at that time is cut out as a still image and stored in a storage or the like. In many smartphones, the camera is arranged at a position off the position where the display screen is arranged on the front surface of the housing, for example, a position above the display screen when viewed from the user. For this reason, when the user captures his / her face, the captured face image is such that the entire face including the direction of the line of sight faces slightly downward.

  In general, it is said that if you look slightly upward, you will have a positive feeling and an impressive eye. If you place the camera at a position below the display screen as seen by the user, you can shoot a face with an upward line of sight, but in this case, the entire face will be slightly upward and the upper line of sight will not stand out. .

  Note that there are various user needs, and there are cases where it is desired to photograph a face with a lower eyeline than an upper eyeline. In this case as well, if the user's face is photographed with a camera positioned slightly above the display screen as seen from the user, the face image is simply the face facing slightly downward, and the lower eye line is The face image does not stand out.

  As described above, in an environment in which a user looks at the target object 30 and shoots a user's face image with a camera disposed above or below the target object 30, the target line is in the same direction in the same direction. Because there is a tendency to take off-faced flare, there are limitations on the facial expression variations that can be taken, such as limitations on the facial expression of the face looking at the camera. .

  The face image processing system 1 according to the first embodiment of the present technology is intended to solve such a problem. However, the problem to be solved by the present technology is not limited to this.

[Gaze correction by facial image processing system]
Next, an outline of the eye line correction operation by the face image processing system 1 of the present embodiment will be described.

  (1) The CPU (control unit) of the information processing apparatus 40 performs control as follows according to the application program stored in the memory.

  In FIG. 1, first, the CPU of the information processing apparatus 40 causes the two camera apparatuses 10 and 20 to start imaging. In the two camera devices 10 and 20, the upper camera device 10 is installed so as to photograph the face of the user U from a position slightly higher than the line of sight E, and the lower camera device 20 is a position slightly lower than the line of sight E. It is installed to shoot from.

  (2) It is assumed that the line of sight E of the user U at the time of shooting is on a line-of-sight object 30, for example, a display screen, disposed between the two camera devices 10, 20. On the display screen, for example, the face image of the subject being photographed by the upper camera device 10 or the lower camera device 20 is displayed as a moving image. Or the moving image image | photographed by each of the two camera apparatuses 10 and 20 may be displayed simultaneously.

  (3) The user U operates the input unit of the information processing apparatus 40 to input a still image capturing instruction at an arbitrary timing while confirming the moving image of his / her face through the display screen.

  (4) When the CPU of the information processing apparatus 40 receives a still image capturing instruction from the user U, it supplies a control command for instructing the two camera apparatuses 10 and 20 to capture a still image.

  (5) Each of the two camera devices cuts out a still image from the moving image of the face of the user U and transmits it to the information processing device 40 in accordance with this control command.

  FIG. 2 is a diagram illustrating an operation based on an image base until a composite image in which eye line correction is performed is obtained from the face image of the user U captured by the two camera apparatuses 10 and 20.

  101 is a face image (still image) photographed by the upper camera device 10, and 201 is a face image (still image) photographed by the lower camera device 20. The face of the user U in the face image 101 photographed by the upper camera device 10 is slightly downward including the line of sight, and the face of the user U in the face image 201 photographed by the lower camera device 20 is slightly upward including the line of sight. .

  (6) The CPU of the information processing device 40 detects the eye images 102 and 202 from the face images 101 and 201 respectively captured by the two camera devices 10 and 20. Here, for the detection of the eye images 102 and 202 in the face image, for example, the feature data of the eye image obtained by analyzing the face images of a large number of people by a statistical learning method is used. And so on.

  Note that the CPU of the information processing device 40 notifies the user U of an error, for example, by display or sound when the detection of the image of the eye portion fails due to a focus shift or the like.

  (7) The CPU of the information processing apparatus 40 cuts out the image 202 of the eye portion detected in the face image 201 photographed by the lower camera apparatus 20 as the image 203 for eye line correction. Here, the line-of-sight correction image 203 is, for example, an image in a rectangular range including both eyes.

  (8) The CPU of the information processing apparatus 40 generates a line-of-sight correction image in which the line-of-sight correction image 203 is replaced with the eye part image 102 detected in the face image 101 photographed by the upper camera apparatus 10. . That is, the eye-line correction image 203 cut out from the face image 201 photographed by the lower camera device 20, and the image 103 other than the eye portion image 102 detected in the face image photographed by the upper camera device 10. For each pixel. At this time, the CPU of the information processing apparatus 40 may perform synthesis processing such as alpha blending on the joint portion in order to make the joint of the image inconspicuous.

  (9) As described above, the image 203 of the eye portion of the face image of the slightly upward user U photographed by the lower camera device 20 is used for the eyes of the face image of the slightly downward user U photographed by the upper camera device 10. A composite image 104 replaced with the partial image 102 is obtained. In this specification, such image composition is referred to as “upper eye line correction” as necessary.

Further, the following eye line correction is also possible.
As shown in FIG.
(7A) The CPU of the information processing apparatus 40 cuts out the eye part image 102 detected in the face image 101 photographed by the upper camera apparatus 10 as the eye-correction image 105.

  (8A) The CPU of the information processing device 40 performs image composition in which the eye-correction image 105 is replaced with the eye portion image 202 detected in the face image 201 photographed by the lower camera device 20. That is, an eyeline correction image 105 cut out from the face image 101 photographed by the upper camera device 10 and an image 205 other than the eye portion image 202 detected in the face image photographed by the lower camera device 20 For each pixel.

  (9A) As described above, the image 105 of the eye portion of the face image of the slightly downward user U photographed by the upper camera device 10 is converted into the eye of the face image of the slightly upward user U photographed by the lower camera device 20. A composite image 206 replaced with the partial image 102 is obtained. In this specification, such image composition is referred to as “lower eyeline correction” as necessary.

[Specifications of the two camera devices 10 and 20]
In the case where there is a difference in specifications that affect the image quality of the two camera apparatuses 10 and 20, it is desirable that the specifications of the camera apparatus that captures an image that occupies a wider area in the composite image are higher. Here, examples of specifications that influence the level of image quality include the size of the image sensor and lens characteristics. Of course, the specifications that determine the image quality are not limited to these.

  For example, in the above-described upper eyeline correction, the image 103 other than the eye portion of the face image 101 photographed by the upper camera device 10 is used in the synthesis. Therefore, it is desirable that the upper camera device 10 has higher specifications than the lower camera device 20 in order to improve the overall image quality.

  Further, in the lower eye line correction, the image 205 other than the eye portion of the face image 201 photographed by the lower camera device 20 is used in the synthesis, so that the lower camera device 20 is higher than the upper camera device 10. A spec is desirable to improve the overall image quality.

[Alignment of the tone of each face image taken by the two camera devices 10, 20]
For example, when the values of shooting parameters that affect the image quality such as exposure, white balance, hue, contrast, sharpness, etc. used at the time of photographing with the two camera devices 10 and 20 are different, the image tone is different in the eye area and other areas May be unnaturally different, and it may be easy to see at a glance that the image is a composite.

  In the face image processing system 1 of the present embodiment, the CPU of the information processing device 40 compares the values of the shooting parameters of the two camera devices 10 and 20 before shooting, and there is a difference that causes a difference in image tone. If there is, the tone of the image captured by at least one camera device is matched with the tone of the image captured by the other camera device by signal processing.

  For example, if there is a difference in the exposure value, the CPU of the information processing device 40, for example, determines the luminance value of the image captured by one camera device as the difference between the exposure values set in the respective camera devices. Signal processing is performed so as to increase or decrease only. Similarly, when there is a difference in white balance, hue, contrast, sharpness, etc., the difference in image tone is reduced by signal processing for one image.

  This makes it possible to match the image tone of the images taken by the two camera devices 10 and 20 as much as possible, and in the synthesized face image, the image tone of the eye region and other regions can be adjusted. Unnatural differences can be suppressed.

  Whether to align the image tone of the image captured by the upper or lower camera device may be uniquely determined, for example, or may be appropriately designated by the user.

  In order to reduce the total load of signal processing, it is desirable not to perform signal processing on the entire face image obtained from the camera device but to perform signal processing only on the image to be synthesized.

  In addition, here, a method for matching the tones of two images by signal processing has been described, but the two camera devices 10 and 20 are photographed by matching the photographing parameter values set in the camera devices. The image tone of each face image may be matched as much as possible.

  That is, in this method, the CPU of the information processing device 40 compares the values of the shooting parameters of the two camera devices 10 and 20 before shooting, and if there is a difference that causes a difference in image quality, This is realized by changing the value of the shooting parameter of at least one camera device to the value of the shooting parameter of the other camera device. Or you may match the value of the imaging | photography parameter of the two camera apparatuses 10 and 20 with each intermediate value.

  As described above, in the face image processing system 1 of the present embodiment, the image 203 of the eye portion of the face image of the user U slightly upward taken by the lower camera device 20 is taken somewhat by the upper camera device 10. A composite image 104 can be obtained by replacing the eye portion image 102 of the face image of the user U facing downward. In such a face image, since the whole face is slightly downward, only the eye part is a slightly upward face image, so the upper eyeline is more conspicuous than the image with the whole face slightly upward, An impressive face image can be obtained.

  Further, in the face image processing system 1 of the present embodiment, the slightly upward facing user 105 photographed by the lower camera device 20 is an image 105 of the eyes of the slightly facing downward user U face image photographed by the upper camera device 10. A composite image 206 in which the image 102 of the eye portion of the U face image is replaced can be obtained. In such a face image, the face of the whole face is slightly upward, and only the eye part is a slightly downward face image. A unique face image can be obtained.

<Second Embodiment>
Next, the configuration of the face image processing system 1A according to the second embodiment of the present technology will be described.
4 and 5 show a face image processing system 1A including a portable terminal 40A, which is a portable information processing apparatus provided with a built-in camera 20A, and a camera 10A that is detachably attached to the portable terminal 40A. It is a perspective view. The camera 10A that is detachably attached to the portable terminal 40A is referred to as an “external camera 10A” in the present specification.
4 shows a state before the external camera 10A is attached to the portable terminal 40A, and FIG. 5 shows a state where the external camera 10A is attached.

  Here, the external camera 10 </ b> A is used as the upper camera device 10, and the built-in camera 20 </ b> A is used as the lower camera device 20.

[Configuration of Mobile Terminal 40A]
As illustrated in FIGS. 4 and 5, the mobile terminal 40 </ b> A includes a housing 41 having a size and a shape that can be held by a user with one hand. The housing 41 has a plate-like rectangular parallelepiped shape constituted by a front surface, a back surface, and four side surfaces. On the front surface 42 of the housing 41, a display screen 43 composed of, for example, a liquid crystal panel, an organic EL panel, or the like is disposed. A transparent touch sensor panel (not shown) is disposed on the screen 43 of the display. The touch sensor panel is composed of, for example, a capacitive sensor type touch sensor panel. The touch sensor panel generates a signal corresponding to the position touched by the user, generates coordinate information from the signal, and supplies the coordinate information to the CPU.

  The portable terminal 40A incorporates the camera 20A as described above. Hereinafter, this camera 20A is referred to as “built-in camera 20A” in this specification.

  The front surface 42 of the housing 41 is provided with a light transmission portion 44 for taking light into an imaging lens (not shown) of the built-in camera 20A. Light captured through the imaging lens of the built-in camera 20A is converted into an electrical signal by an imaging element such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). This electrical signal is converted into digital data for each pixel by an A / D converter and becomes RAW data. The RAW data is adjusted by a signal processing circuit according to parameters in which, for example, exposure and white balance are set, and image data is generated through development processing.

  Thus, the image data obtained by the built-in camera 20A is displayed as data for display by the CPU and display control circuit in the portable terminal 40A, and is displayed on the screen 43 of the display as a moving image or a still image by being supplied to the display. The The CPU stores image data at a timing instructed by the user as a still image in a storage or a medium. At that time, it may be converted into a specific image format such as JPEG (Joint Photographic Experts Group).

Next, a specific hardware configuration of the mobile terminal 40A will be described.
FIG. 6 is a block diagram showing a hardware configuration of the portable terminal 40A. The hardware configuration of the portable terminal 40A is the same as that of the information processing apparatus 40 of the first embodiment.

  As shown in the figure, the mobile terminal 40A includes a CPU (control unit) 401, a main memory 402, a storage 403, a built-in camera (lower camera device) 20A, a display 404, a touch sensor panel 405, a display control unit 406, and a touch sensor. A control unit 407, an attitude sensor 408, a wireless communication unit 409, and a system bus 410 are provided.

  The CPU 401 is a controller that controls each unit of the mobile terminal 40A and executes arithmetic processing for executing an application program using the main memory 402 as a work space.

  The main memory 402 is composed of a RAM (Random Access Memory) or the like. The main memory 402 stores an operating system, an application program executed on the operating system, and various data necessary for the operation of the application program.

  The storage 403 is composed of a nonvolatile memory such as a flash memory, for example. The storage 403 stores files of various data such as an operating system called up to the main memory 402, various application programs, images, sounds, and documents.

  The built-in camera 20A is a camera built in the mobile terminal 40A as described above. The built-in camera 20A is controlled by the CPU 401 in accordance with a camera application program. The CPU 401 supplies, for example, activation of the built-in camera 20A, setting of various shooting conditions such as shooting parameters and resolution, control of development processing, and developed image data to the display control unit 406 in accordance with the camera application program. Control is performed to display a moving image or a still image on the display 404. Further, the CPU 401 performs image composition for eye-gaze correction using an image photographed by the built-in camera 20A and an image photographed by the external camera 10A according to the camera application program.

  The display 404 is composed of, for example, a liquid crystal panel, an organic EL (electroluminescence) panel, and the like, and displays various operation screens of the mobile terminal 40A, displays images such as still images and moving images, and contents of various files such as other documents. Or display.

  The display control unit 406 generates display data corresponding to the specifications of the display 404 for the various types of information described above.

  The touch sensor panel 405 is a device that detects the coordinates of the position selected by the user on the screen of the display 404 as described above.

  The touch sensor control unit 407 controls the touch sensor panel 405, generates coordinate data corresponding to the detected position, and supplies the coordinate data to the CPU 401.

  The attitude sensor 408 is a sensor for detecting the attitude of the mobile terminal 40A as described above. The posture sensor 408 includes, for example, an acceleration sensor that detects acceleration in three axes, a gyro sensor that detects angular velocity, and the like.

  The wireless communication unit 409 communicates with an external device (for example, the lower camera device 20), for example, a wireless local area network (WiFi) such as WiFi (registered trademark), NFC (Near field communication), or Blue Tooth (registered). This is an interface for performing near field communication such as a trademark.

  The system bus 410 is a transmission path for exchanging various signals between the blocks.

  Although the main hardware configuration of the mobile terminal 40A has been described above, other configurations include, for example, a WWAN (Wireless Wide Area Network) communication unit, a speaker, a microphone, a memory card reader / driver, a battery and its control circuit, GPS A (Global Positioning System) receiving unit, a terrestrial digital television receiving unit, and the like may be mounted.

[Configuration of External Camera 10A]
7 and 8 are perspective views showing the external appearance of the external camera 10A.

  This external camera 10 </ b> A includes a camera body 11 and an adapter 12. The adapter 12 can be attached to and detached from each of the camera body 11 and an external device (for example, the portable terminal 40A). However, the camera body 11 and the adapter 12 may be integrated.

  The camera body 11 has, for example, a cylindrical outer tube portion 13. An imaging lens, an image sensor, a circuit for signal processing and image processing, a control circuit such as a controller and a memory, a power supply unit, a storage for storing image data, a wireless communication circuit, and the like are accommodated in the outer cylinder unit 13. Has been.

  An inner cylinder portion 14 that is movable in the front-rear direction is supported at a front end portion of the outer cylinder portion 13. As shown in FIG. 8, in this external camera 10A, the inner tube portion 14 is drawn forward with respect to the outer tube portion 13 to be in a photographing state, and the inner tube portion 14 that has been drawn forward is moved backward. The retracted state (FIG. 7) is achieved by being inserted and housed inside the outer cylinder portion 13.

  A substantially rectangular light transmission hole 15 is formed in the front surface portion of the inner cylinder portion 14. Lens barriers 9 and 9 are supported on the rear surface side of the front surface portion of the inner cylinder portion 8 so as to be freely opened and closed. The lens barriers 9 and 9 are closed in the retracted state and opened in the photographing state by an opening / closing mechanism (not shown). Accordingly, in the retracted state, the light transmission holes 15 of the inner cylinder portion 8 are blocked by the lens barriers 9 and 9 (see FIG. 7), and in the photographing state, the light transmission holes 15 are opened by the lens barriers 9 and 9 (see FIG. 7). 8).

  A plurality of photographing lens groups are arranged in the camera body 11 so as to be separated from each other in the optical axis direction. When the light transmitting hole 15 is opened by the lens barriers 9 and 9, the photographing lens group is externally provided. Light is incident. The light that has passed through the photographing lens group enters the image sensor and is converted into an electrical signal for each pixel. An electrical signal for each pixel is converted into a digital signal by an A / D converter, and is converted into image data after development processing.

  An operation ring 16 is rotatably supported on the peripheral surface of the front end portion of the outer cylinder portion 7. When the operation ring 16 is rotated, manual focusing and zooming are performed, and the lens group is moved in the optical axis direction.

  A shutter button 17, a zoom switch 18, and other operation buttons / switches are provided on the outer peripheral surface of the outer cylinder portion 7.

  The adapter 12 has a pair of latching portions 19 and 19 that are latched by an external device (for example, the mobile terminal 40A). A pair of latching parts 19 and 19 can be expanded and contracted with respect to the adapter 12, and can be locked in arbitrary expansion and contraction positions.

  The pair of latching portions 19, 19 are extendable with respect to the adapter 12, so that the external camera 10 </ b> A can be attached to various parts of external devices having various shapes and sizes.

[Hardware configuration of external camera 10A]
FIG. 9 is a block diagram showing a hardware configuration of the external camera 10A.

  As shown in the figure, the external camera 10A includes a controller 501, an operation input unit 502, an imaging unit 503, a signal processing unit 504, a mechanical drive unit 505, a power supply unit 506, a storage 507, a wireless communication unit 508, and a system bus. 509.

  The controller 501 controls each part of the external camera 10A. The controller 501 has a CPU and a memory. The memory stores a RAM used as the main memory 402, and a program for executing ON / OFF of power, start of imaging, capturing of still images, communication control with an external device (for example, the portable terminal 40A), and the like. Has ROM.

  For example, when the controller 501 detects an external device (for example, the portable terminal 40A) existing within a short distance using the wireless communication unit 508, the controller 501 outputs a power control signal to the power supply unit 506, and the internal camera 10A The power supply to each part of is started. In addition, when the power supply to each unit of the external camera 10A is on, the controller 501 turns off the power supply to each unit when an external device (for example, the portable terminal 40A) is not detected within a short distance. A power supply control signal is supplied to the power supply unit 506. As a result, the controller 501 returns to the device detection state.

  The controller 501 receives various commands from an external device (for example, the mobile terminal 40A) connected using the wireless communication unit 508, and performs processing according to the commands. As an instruction from the external device (for example, the mobile terminal 40A), for example, an instruction to start shooting, extraction of a still image from the moving image being shot, wireless transmission to the (for example, the mobile terminal 40A), zoom-up / zoom-in, etc. There is.

  The operation input unit 502 receives an operation input from the user and outputs an operation signal to the controller 501.

  The imaging unit 503 includes an imaging lens and an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

  The signal processing unit 504 performs an A / D converter that converts the output of the imaging unit 503 into a digital signal, and adjustment of exposure, white balance, and development processing on RAW data that is the output of the A / D converter. Do. The signal processing unit 504 performs, for example, noise removal, color tone correction, and edge detection as necessary.

  The storage 507 is used to store a captured image data file. The storage 507 is composed of, for example, a flash memory.

  The mechanical drive unit 505 includes various mechanisms such as a mechanism that drives the lens group in the optical axis direction for focus adjustment and zoom change, a mechanism that moves the inner cylinder part 14 back and forth, and a mechanism that opens and closes the lens barriers 9 and 9. It is a circuit to drive.

  The power supply unit 506 generates power necessary for operating each part of the external camera 10A from the battery and the battery, and supplies the power to each part.

  The wireless communication unit 508 communicates with an external device (for example, the mobile terminal 40A), for example, a wireless local area network (WiFi) such as WiFi (registered trademark), NFC (Near field communication), or Blue Tooth (registered trademark). This is an interface for performing short-range wireless communication.

  In addition, the camera device may include a display, a speaker, a GPS receiver, a posture sensor, a media driter, and the like.

Next, an attitude when using the mobile terminal 40A will be described.
The casing 41 of the mobile terminal 40A has a rectangular parallelepiped shape, and is a thin rectangular parallelepiped because the size in the direction of one axis (Z axis) is particularly small. The two-dimensional space constituted by the directions of the other two axes (XY axes) is relatively wide in order to arrange the display screen 43. The relationship between the size Lx in the X-axis direction of the housing 41 and the size Ly in the Y-axis direction is Lx <Ly. The relationship between the size Dx of the display screen 43 in the X-axis direction and the size Dy in the Y-axis direction is also Dx <Dy.

  In the following description, the posture of the mobile terminal 40A with the Y-axis direction viewed from the user is called “vertical posture”, and the posture of the mobile terminal 40A with the X-axis direction viewed from the user is called “horizontal posture”. I will decide. The vertical posture includes a posture (first vertical posture) in which the position of the built-in camera 20 </ b> A is above the display screen 43 as viewed from the user, and a posture (second vertical posture) in the lower side. . Similarly, the horizontal posture includes a posture (first horizontal posture) where the position of the built-in camera 20A is the right side of the display screen 43 as viewed from the user and a posture (second horizontal posture) where the position is the left side.

  Each of the above postures of the mobile terminal 40A is detected by a built-in acceleration sensor, gyro sensor, or the like. The direction of the display data is rotated so as to match.

[Operation of Face Image Processing System 1A]
By attaching the external camera 10A to the portable terminal 40A via the adapter 12, the devices are detected by, for example, short-range wireless communication between the portable terminal 40A and the external camera 10A.

  When the controller 501 of the external camera 10A detects the portable terminal 40A, the controller 501 instructs the power supply unit 506 to supply power to each unit of the external camera 10A. As a result, the camera unit is ready for shooting and waits for a command from the mobile terminal 40A.

  On the other hand, when detecting the external camera 10A, the CPU 401 of the mobile terminal 40A activates an application program for the camera. The CPU 401 of the portable terminal 40A performs control such that the built-in camera 20A is activated to start photographing in accordance with the camera application program. Further, the CPU 401 of the mobile terminal 40A instructs the external camera 10A to start photographing by wireless communication. Thereby, shooting is started in each of the built-in camera 20A and the external camera 10A.

  When the built-in camera 20A is activated, an image taken by the built-in camera 20A is displayed as a moving image on the screen 43 of the display 404 of the portable terminal 40A.

  In order to photograph the user's face, the user determines the position and orientation of the mobile terminal 40A so that the user's face enters the field of view of the built-in camera 20A of the mobile terminal 40A while looking at the screen 43 of the display 404. When the shutter button of the built-in camera 20A is operated by the user, the CPU 401 of the mobile terminal 40A holds the image (still image) captured at that time in the main memory 402 as a first face image used for eye line correction.

  In addition, the CPU 401 of the mobile terminal 40A requests the external camera 10A to cut out a still image from the moving image being shot and transmit the still image by wireless communication. Upon receiving this request, the controller 501 of the camera unit cuts out a still image from the moving image being shot, and transmits this still image to the portable terminal 40A by wireless communication. The CPU 401 of the portable terminal 40A holds the image (still image) transmitted from the external camera 10A in the main memory 402 as a second face image used for eye line correction.

Thereafter, the CPU 401 of the mobile terminal 40 </ b> A executes the above-described combining process for eye line correction from the two face images held in the main memory 402, and displays the result on the screen 43 of the display 404.
Note that the matching of the tone of the two face images is performed in this embodiment as in the first embodiment.

  FIG. 10 is a diagram showing a first mode in which a user photographs his / her face using the face image processing system 1A of the present embodiment.

  In this first imaging mode, the external camera 10A is used as the upper camera device 10 (see FIG. 1) that captures the face of the user U from a position slightly higher than the line of sight E, and the built-in camera 20A is used for the line of sight E of the user U. A case is assumed where the camera is used as the lower camera device 20 (see FIG. 1) for photographing from a slightly lower position.

  This is a form optimal for the case where the external camera 10A has higher specifications than the built-in camera 20A of the portable terminal 40A, and the upper eyeline image is synthesized. It is. That is, a relatively high-quality image photographed by the external camera 10A is used for a wide range portion other than the eye portion.

  FIG. 11 is a diagram illustrating a second mode in which the user U captures his / her face using the face image processing system 1A of the present embodiment.

  In the second imaging mode, contrary to the first imaging mode, the external camera 10A is used as a lower camera device 20 (see FIG. 1) that captures the face of the user U from a position slightly lower than the line of sight E. It is assumed that the camera 20A is used as the upper camera device 10 (see FIG. 1) that captures the face of the user U from a position slightly higher than the line of sight E.

  Since the orientation sensor 408 detects the vertical orientation of the housing 41, the portable terminal 40A rotates the vertical orientation of the images captured by the built-in camera 20A and the external camera 10A of the portable terminal 40A. A composite image that has undergone eye line correction can be obtained.

  This is the case where the spec that determines the image quality of the camera is higher in the built-in camera 20A of the portable terminal 40A than in the external camera 10A, and is optimal for the case where the upper eye image is synthesized. It is.

<Modification 1>
After synthesizing the image subjected to the line-of-sight correction, the CPU 401 of the mobile terminal 40A displays the synthesized image and two face images before being synthesized or one of the two face images before being synthesized. It may be displayed on the screen simultaneously.

  The CPU 401 of the portable terminal 40A may cause the user to select a favorite face image among the face images displayed at the same time, and store only the selected face image.

<Modification 2>
In the eye line correction described above, processing that is instructed by the user may be simultaneously performed on the synthesized face image. For example, image editing contents such as processing for enlarging eyes and pupils, skin beautification processing, and the like may be selected by the user as options for eye line correction. The CPU 401 of the portable terminal 40A executes the above-described image editing on the synthesized face image according to the selected eye correction option.

<Modification 3>
The line of sight correction may be performed in the CPU of the server device on the network. That is, two face images photographed by two cameras are transmitted from the information processing apparatus to the server apparatus via the network. The CPU of the server device is configured to perform the above-described line-of-sight correction on the received two face images, and send the composite image back to the information processing device via the network.

<Modification 4>
The image transmitted from the external camera 10A to the portable terminal 40A may be RAW data. By performing development processing in the portable terminal 40A, it is possible to use an external camera that does not include a development processing unit.

<Modification 5>
In the above embodiment, two camera devices are used. However, one camera device is moved in the vertical direction of the user's face, and the user's face is photographed twice at each of the upper and lower positions. Also good. Therefore, in the present technology, the number of cameras is 1 or 2.

<Modification 6>
In the above embodiment, image synthesis for eye line correction is performed by the CPU of the information processing apparatus 40 or the portable terminal 40A. However, the processing is distributed and performed by an information processing system including a plurality of information processing apparatuses. You may do it. Alternatively, the processing may be performed in a distributed manner in an information processing system including one or more information processing devices and a server device connected thereto through a network. Alternatively, the processing may be performed in a distributed manner in an information processing system including a plurality of server devices connected through a network.

<Modification 7>
In the above, the line-of-sight correction for a still image has been described. However, the present technology can be similarly applied to line-of-sight correction for a moving image. That is, for example, image synthesis for eye line correction for a still image may be performed in units of frames or in units of 1 / N frames.

  In this case, the user holds the portable terminal 40 </ b> A so that his / her face falls within the angle of view of the two cameras. For this reason, due to camera shake or the like, the face may sometimes come off from within the angle of view of the two cameras or may not partially fit.

  In the modified example 7, if the image composition for eye-gaze correction fails because the face is out of the angle of view of the two cameras or does not fit in part, the CPU 401 of the information processing device 40 or the portable terminal 40A Finally, a frame or 1 / N frame that has failed to be combined is interpolated in the eye-corrected image that has been successfully combined. As a result, it is possible to display the moving image corrected for the line of sight without interruption.

<Modification 8>
The present technology can also be applied when three or more cameras are connected.
For example, as shown in FIG. 12, two cameras 10A and 20A having a field of view in front of the mobile terminal 40A (the side of the user U1) are arranged vertically and symmetrically with the rear of the mobile terminal 40A. It is assumed that two cameras 10B and 20B having a field of view are arranged vertically in the vertical direction.

  In this case, as shown in FIG. 12, the two cameras 10A and 20A having a visual field in front of the portable terminal 40A (the user U1 side) are used for photographing for generating their own eye-gaze corrected images. The two cameras 10B and 20B having a visual field behind the mobile terminal 40A are used for photographing for generating a line-of-sight correction image of the face of the person U2 on the opposite side across the position of the mobile terminal 40A from the user U1. .

  The CPU 401 of the portable terminal 40A detects a face from images captured by the four cameras 10A, 20A, 10B, and 20B, and performs image composition for eye-gaze correction based on an image pair including the face of the same person. Do. As a result, not only the face image corrected by the user U1 itself but also the face image corrected by the eye of another person U2 can be obtained. Further, when the face of the same person (person U2) is detected only in a pair of images taken by the two cameras 10B and 20B with the visual field behind the portable terminal 40A, the line of sight of the person U2 is corrected. Can be obtained by combining only the face images.

In addition, this technique can also take the following structures.
(1) A line-of-sight correction image obtained by changing an image of an eye part in the other image based on an image of an eye part in one of the two images including a human face photographed from different positions. An information processing apparatus including a control unit for generation.

(2) The information processing apparatus according to (1),
The control unit is configured to acquire two images including the face of the person photographed by the first imaging unit and the second imaging unit from different positions in the vertical direction of the face of the person who is the subject. Information processing device.

(3) The information processing apparatus according to (2),
The control unit selects a partial image of the eye of one of the two images acquired from a relatively lower position of the acquired two images, of the other image captured from the relatively upper position. An information processing apparatus configured to generate the line-of-sight correction image replaced with a partial image of an eye.

(4) The information processing apparatus according to (2) or (3),
The first imaging unit is an upper imaging unit in the vertical direction of the face of the person that is the subject, and the second imaging unit is an imaging unit that is lower in the vertical direction of the face of the person that is the subject. The first imaging unit has an image quality of a captured image higher than that of the second imaging unit.

(5) The information processing apparatus according to any one of (1) to (4),
The information processing apparatus configured to cause the control unit to perform signal processing on at least one of the images so as to match the tone of the two acquired images.

(6) The information processing apparatus according to any one of (1) to (5),
The information processing apparatus configured to cause the display unit to simultaneously display at least one of the line-of-sight correction image and the acquired two images on the display unit.

(7) The information processing apparatus according to any one of (2) to (6),
The information processing apparatus, wherein the control unit is configured to match shooting parameters of the first imaging unit and the second imaging unit so as to match the tones of the two acquired images.

(8) The information processing apparatus according to any one of (2) to (7),
The first imaging unit is built in the information processing apparatus;
The information processing apparatus, wherein the second imaging unit is a separate imaging unit that is detachably attached to the information processing apparatus.

(9) The information processing apparatus according to any one of (1) to (8),
The information processing apparatus configured to edit the generated line-of-sight correction image.

DESCRIPTION OF SYMBOLS 1 ... Face image processing system 1A ... Face image processing system 10 ... Upper camera apparatus 10A ... External camera 20 ... Lower camera apparatus 20A ... Built-in camera 40 ... Information processing apparatus (information processing system)
40A ... mobile terminal 401 ... CPU
404 ... Display 409 ... Wireless communication unit 501 ... Controller 503 ... Imaging unit 508 ... Wireless communication unit

Claims (13)

Control for generating a line-of-sight correction image in which the image of the eye part in one image is changed based on the image of the eye part in one of the two images including the face of a person photographed from different positions An information processing apparatus comprising a unit. The information processing apparatus according to claim 1,
The control unit is configured to acquire two images including the face of the person photographed by the first imaging unit and the second imaging unit from different positions in the vertical direction of the face of the person who is the subject. Information processing device. An information processing apparatus according to claim 2,
The control unit selects a partial image of the eye of one of the two images acquired from a relatively lower position of the acquired two images, of the other image captured from the relatively upper position. An information processing apparatus configured to generate the line-of-sight correction image replaced with a partial image of an eye. The information processing apparatus according to claim 3,
The first imaging unit is an upper imaging unit in the vertical direction of the face of the person that is the subject, and the second imaging unit is an imaging unit that is lower in the vertical direction of the face of the person that is the subject. The first imaging unit has an image quality of a captured image higher than that of the second imaging unit. The information processing apparatus according to claim 3,
The information processing apparatus configured to cause the control unit to perform signal processing on at least one of the images so as to match the tone of the two acquired images. The information processing apparatus according to claim 3,
The information processing apparatus configured to cause the display unit to simultaneously display at least one of the line-of-sight correction image and the acquired two images on the display unit. The information processing apparatus according to claim 3,
The information processing apparatus, wherein the control unit is configured to match shooting parameters of the first imaging unit and the second imaging unit so as to match the tones of the two acquired images. An information processing apparatus according to claim 2,
The first imaging unit is built in the information processing apparatus;
The information processing apparatus, wherein the second imaging unit is a separate imaging unit that is detachably attached to the information processing apparatus. The information processing apparatus according to claim 1,
The information processing apparatus configured to edit the generated line-of-sight correction image. An imaging unit for photographing a person's face from different positions;
Based on the image of the eye part in one of the two images including the face of the person photographed by the imaging unit, a line-of-sight correction image is generated by changing the image of the eye part in the other image And an information processing apparatus. The control unit
Based on the image of the eye part in one of the two images including the face of a person photographed from different positions, a line-of-sight correction image is generated by changing the image of the eye part in the other image Processing method. Control for generating a line-of-sight correction image in which the image of the eye part in one image is changed based on the image of the eye part in one of the two images including the face of a person photographed from different positions An information processing system comprising a section. Control for generating a line-of-sight correction image in which the image of the eye part in one image is changed based on the image of the eye part in one of the two images including the face of a person photographed from different positions A program that allows a computer to function as a part.

Images