JP4995935B2 - Image generating apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image generation apparatus, an image processing method, and a program that can generate an image in which a user's line-of-sight direction is appropriately corrected.

In recent years, products including a camera (an imaging unit such as a CCD) are sold in notebook personal computers (personal computers) and the like. For example, this camera is incorporated in a peripheral frame (an upper frame, a lower frame, or the like) that supports a display unit (display such as a liquid crystal), and can photograph a user (a face or the like).
Such a camera is used, for example, when video chatting is performed. That is, since images captured by the camera are sent to each other in each communicably connected personal computer, each user can chat while looking at the other party's face.

  Recently, products in which such a camera is incorporated are also sold in portable game devices and the like. In addition, a game device is known in which a player can attach and use a camera of a peripheral device even if the camera is not incorporated from the beginning.

  In a game device or the like capable of such video chatting, there is also disclosed a technique for outputting a voice or the like uttered in synchronization with video at one user terminal in synchronization with the video at the other user terminal (for example, a patent) Reference 1).

JP 2007-222411 A (pages 9-19, FIG. 3)

In the above-described game device capable of video chatting, the user proceeds with the conversation while viewing the image of the other user displayed on the display unit. As described above, since the camera is incorporated in the peripheral frame that supports the display unit, the optical axis direction of the camera is not aligned with the line-of-sight direction of the user, and a shift occurs. That is, an image of a user who has diverted his / her line of sight (out of line of sight) is taken.
For example, when the camera is incorporated in the upper frame of the display unit, the user is photographed from above, and an image of the user with the line of sight directed downward is photographed. When the photographed image is sent to the device on the other side, the other user often appears to be depressed (the user who is the transmission source) and often cannot enjoy the conversation.

  In the same way, the image of the user (another user) who has diverted his / her line of sight is also captured and sent to the other device. For this reason, in the conventional video chat, the users are talking with each other with their eyes on each other, which is extremely unnatural.

  Therefore, there has been a demand for development of a game device (image generation device) that can perform video chat without causing such an unnatural situation.

  SUMMARY An advantage of some aspects of the invention is that it provides an image generation apparatus, an image processing method, and a program that can generate an image in which a user's line-of-sight direction is appropriately corrected.

  An image generation apparatus according to a first aspect of the present invention includes a face detection unit, an image specification unit, and an image processing unit.

  First, the face detection unit detects a human face image from a photographed image photographed by a camera or the like, for example. Further, the image specifying unit specifies an eye part image (for example, a substantially circular eye part image including a part hidden behind the skin) from the detected face image. Then, the image processing unit processes the position of the pupil in the specified eye image so as to be at the center of the eye image, for example.

That is, in the present invention, a human face image (user face image) is detected from the photographed image, and an eye part image is specified from the face image. And the position of the pupil of the specified eye part image is processed so as to be the center of the eye part image, for example. That is, the eye image is processed so that the user looks at the camera (camera).
As a result, an image in which the user's line-of-sight direction is appropriately corrected can be generated.

  An image generation apparatus according to a second aspect of the present invention includes an imaging unit, a face detection unit, an image specification unit, an image processing unit, and an image transmission unit.

  First, the imaging unit captures a captured image including a person. The face detection unit detects a face image of the person by face recognition from the photographed photographed image. Further, the image specifying unit specifies an eye part image (for example, a substantially circular eye part image including a part hidden behind the skin) from the detected face image. Further, the image processing unit processes the pupil position in the identified eye part image so as to be in the center of the eye part image. Then, the image transmission unit transmits a captured image including a face image obtained by processing the eye image to another device.

That is, in the present invention, the eye part image is specified from the face image of the person (user) detected from the captured image. Then, the position of the pupil of the identified eye part image is processed so as to be the center of the eye part image, and a captured image including a face image obtained by processing the eye part is transmitted to another device. That is, the captured image obtained by processing the eye image is transmitted to another device so that the user looks at the camera (camera). For this reason, the other party (another apparatus) can obtain an image of a user who is looking at the user (the other party user), so that the video chat can be naturally performed without a sense of incongruity.
As a result, an image in which the user's line-of-sight direction is appropriately corrected can be generated.

The image processing unit may further combine a bright spot with a part of the pupil of the processed eye image. That is, the user's pupil is shined by further performing catch light processing.
In this case, the user's facial expression can be made richer by catch light processing.

The image generation apparatus further includes a calculation unit that calculates the gaze direction of the person from the detected face image,
The image processing unit may process the position of the pupil in the eye image when the calculated line-of-sight direction is within a predetermined range.
That is, when the user's line of sight is greatly deviated, an unnatural face image is obtained even if processing is performed with the pupil position in the eye image centered. Therefore, in the case where the line of sight is greatly deviated, the processing is not performed, and the position of the pupil of the eye image is processed only within a reasonable line of sight range.
In this case, an image in which the user's line-of-sight direction is corrected can be generated within a range that is not unnatural.

  An image processing method according to a third aspect of the present invention is an image processing method in an image generation apparatus having a face detection unit, an image specifying unit, and an image processing unit, the face detecting step, the image specifying step, and The image processing step is configured.

  First, in the face detection step, for example, a human face image is detected from a photographed image photographed by a camera or the like. In the image specifying step, an eye image (for example, a substantially circular eye image including a portion hidden by the skin) is specified from the detected face image. In the image processing step, the position of the pupil in the specified eye image is processed so as to be at the center of the eye image, for example.

That is, in the present invention, a human face image (user face image) is detected from the photographed image, and an eye part image is specified from the face image. And the position of the pupil of the specified eye part image is processed so as to be the center of the eye part image, for example. That is, the eye image is processed so that the user looks at the camera (camera).
As a result, an image in which the user's line-of-sight direction is appropriately corrected can be generated.

  A program according to a fourth aspect of the present invention is configured to cause a computer (including an electronic device) to function as the above-described image generation device.

  This program can be recorded on a computer-readable information recording medium such as a compact disk, flexible disk, hard disk, magneto-optical disk, digital video disk, magnetic tape, and semiconductor memory.

  The above program can be distributed and sold via a computer communication network independently of the computer on which the program is executed. The information recording medium can be distributed and sold independently of the computer.

  ADVANTAGE OF THE INVENTION According to this invention, the image which corrected the user's gaze direction appropriately can be produced | generated.

It is a mimetic diagram showing appearance of a game device concerning this embodiment. It is a block diagram which shows the schematic structure of the game device which concerns on this embodiment. It is a block diagram which shows the structure of the image generation apparatus which concerns on this embodiment. (A), (b) is a schematic diagram for demonstrating a mode that a person (user) is detected. (A)-(e) It is a schematic diagram for demonstrating a mode that all specify the eye part image which looks like an upper eye. (A)-(d) is a schematic diagram for demonstrating a mode that all specify the eye part image which has a tendency to bind down. (A)-(f) is a schematic diagram for demonstrating a mode that the position of the pupil in an eye part image is processed. (A), (b) is a schematic diagram which shows an example of catchlight processing, (c) is a schematic diagram which shows an example of the face image (eye part image) after a process. (A), (b) is a schematic diagram which shows an example of the video chat screen displayed on a display part. It is a flowchart for demonstrating the image processing process which concerns on this embodiment. It is a block diagram which shows the structure of the image generation apparatus which concerns on other embodiment. (A)-(e) is a schematic diagram for demonstrating a mode that all calculate a user's gaze direction.

  Embodiments of the present invention will be described below. In the following, for ease of understanding, an embodiment in which the present invention is applied to a portable game machine (game device) will be described. However, the present invention is similarly applied to information processing devices such as various computers, PDAs, and mobile phones. The invention can be applied. In other words, the embodiments described below are for illustrative purposes and do not limit the scope of the present invention. Therefore, those skilled in the art can employ embodiments in which each or all of these elements are replaced with equivalent ones, and these embodiments are also included in the scope of the present invention.

(Embodiment 1)
FIG. 1 is a schematic diagram showing an appearance of a typical game device in which the image generation device according to the embodiment of the present invention is realized. As an example, the game apparatus 1 is a portable game device, and includes a first display unit 18 and a second display unit 19, and a camera 21 is disposed between them. Yes. A touch panel 20 is disposed on the front surface of the second display unit 19.

  FIG. 2 is a schematic diagram showing a schematic configuration of the game apparatus 1. Hereinafter, the game apparatus 1 will be described with reference to FIG.

  The game apparatus 1 includes a processing control unit 10, a connector 11, a cartridge 12, a wireless communication unit 13, a communication controller 14, a sound amplifier 15, a speaker 16, an operation key 17, and a first display unit 18. A second display unit 19, a touch panel 20, and a camera 21.

  The processing control unit 10 includes a CPU (Central Processing Unit) core 10a, an image processing unit 10b, a DMA (Direct Memory Access) controller 10c, a VRAM (Video Random Access Memory) 10d, a WRAM (Work RAM) 10e, An LCD (Liquid Crystal Display) controller 10f and a touch panel controller 10g are provided.

  The CPU core 10a controls the overall operation of the game apparatus 1 and is connected to each component to exchange control signals and data. Specifically, a program and data stored in a ROM (Read Only Memory) 12a in the cartridge 12 are read out with the cartridge 12 mounted in the connector 11, and predetermined processing is executed.

  The image processing unit 10b processes the data read from the ROM 12a in the cartridge 12 and the data processed by the CPU core 10a, and stores the processed data in the VRAM 10d. For example, the image stored in the WRAM 10e is processed and stored in the VRAM 10d.

  The DMA controller 10c transfers the captured image captured by the camera 21 to the WRAM 10e or the like.

The VRAM 10d is a memory for storing information for display, and stores image information processed by the image processing unit 10b and the like.
The WRAM 10e stores work data and the like necessary when the CPU core 10a executes various processes according to the program.
The WRAM 10e temporarily stores the image transferred from the DMA controller 10c.

  The LCD controller 10f controls the first display unit 18 and the second display unit 19 to display a predetermined display image. For example, the LCD controller 10 f converts the image information stored in the VRAM 10 d into a display signal at a predetermined synchronization timing, and outputs it to the first display unit 18. Similarly, the LCD controller 10 f displays a predetermined display image on the second display unit 19.

  When the touch panel 20 is pressed with a touch pen or a player's finger, the touch panel controller 10g acquires the coordinates (input coordinates).

  The connector 11 is a terminal that can be detachably connected to the cartridge 12, and transmits / receives predetermined data to / from the cartridge 12 when the cartridge 12 is connected.

The cartridge 12 includes a ROM 12a and a RAM (Random Access Memory) 12b.
The ROM 12a stores a program for realizing the game and image data, sound data, and the like accompanying the game.
The RAM 12b stores various data indicating the progress of the game.

The wireless communication unit 13 is a unit that performs wireless communication with the wireless communication unit 13 of another game apparatus 1 and transmits / receives predetermined data via an antenna (not shown) (built-in antenna or the like).
The wireless communication unit 13 can also perform wireless communication with a predetermined wireless access point. The wireless communication unit 13 is assigned a unique MAC (Media Access Control) address.

The communication controller 14 controls the wireless communication unit 13 and mediates wireless communication performed between the process control unit 10 and the process control unit 10 of another game device 1 in accordance with a predetermined protocol.
Further, when the game apparatus 1 connects to the Internet via a nearby wireless access point or the like, wireless communication performed between the processing control unit 10 and the wireless access point or the like is performed according to a protocol compliant with the wireless LAN. Intermediary communication.

The sound amplifier 15 amplifies the audio signal generated by the processing control unit 10 and supplies it to the speaker 16.
The speaker 16 is composed of, for example, a stereo speaker, and outputs predetermined music sound, sound effect, and the like according to the audio signal amplified by the sound amplifier 15.

  The operation key 17 includes a plurality of key switches and the like appropriately arranged on the game apparatus 1 and receives a predetermined instruction input according to a user operation.

  The 1st display part 18 and the 2nd display part 19 consist of LCD etc., are controlled by LCD controller 10f, and display a game image etc. suitably.

The touch panel 20 is arranged so as to be superimposed on the front surface of the second display unit 19 and detects an input by a touch pen or a player's finger.
For example, the touch panel 20 includes a resistive film type touch sensor panel and the like, detects a press (pressing) with a touch pen or the like, and outputs information (signal or the like) according to the coordinates.

  As an example, the camera 21 includes a single-focus lens and an image sensor (CMOS, CCD, etc.) having a predetermined number of pixels. Then, the camera 21 captures an image in a predetermined imaging range with the front direction of the first display unit 18 as an optical axis. For example, a subject such as a user who performs a video chat is photographed.

In such a configuration, when the cartridge 12 in which the program and data for video chat are recorded in the ROM 12a is mounted and the game apparatus 1 is turned on, the program is executed, and the image generation according to the present embodiment is performed. A device is realized.
Hereinafter, an image generation apparatus realized by such a game apparatus 1 will be described.

(Outline of image generation device)
FIG. 3 is a block diagram showing a schematic configuration of the image generation apparatus 100 according to the present embodiment. As an example, the image generation apparatus 100 is an apparatus used for video chatting with another image generation apparatus 100 connected via a predetermined network, and the first display unit 18 described above. For example, an image of the user (user himself) captured by the camera 21 can be transmitted while displaying an image of the other user or the like. Hereinafter, the image generation apparatus 100 will be described with reference to the drawing, taking as an example the case where such a video chat is performed.

  As illustrated in FIG. 3, the image generation apparatus 100 includes an imaging unit 110, a face detection unit 120, an image specifying unit 130, an image processing unit 140, a communication unit 150, a display control unit 160, and a display unit 170. With.

First, the photographing unit 110 photographs a photographed image obtained by photographing a predetermined subject. For example, a captured image including a user (person) who performs video chat is captured.
Specifically, when performing video chat, the photographing unit 110 sequentially captures images including the user, and supplies the captured images (for example, moving images) to the face detection unit 120 and the like.
Note that the above-described camera 21 can function as such a photographing unit 110.

The face detection unit 120 detects a user's face image from the photographed image photographed by the photographing unit 110.
For example, the face detection unit 120 detects the face image of the user shown in the captured image by face recognition. Specifically, the face detection unit 120 searches the skin image areas A1 to A4 from the photographed image SG as shown in FIG. 4A, and determines whether or not a human facial feature part can be extracted from these areas. to decide. In this case, as shown in FIG. 4B, at least a feature part including both eyes and mouth can be extracted from the area A1, and can be recognized as a human face. K1 is detected.
Note that such a method of detecting the face image of the user (person) is an example, and other methods may be used. For example, the contour of the skin color region may be extracted by edge processing to determine whether or not the skin color region has a predetermined shape (such as a substantially elliptical shape), and the user's image image may be detected.
The CPU core 10a and the image processing unit 10b of the processing control unit 10 described above can function as such a face detection unit 120.

Returning to FIG. 3, the image specifying unit 130 specifies an eye part image from the face image of the user detected by the face detecting unit 120.
For example, the image specifying unit 130 specifies the eye part image in the face image from the positional relationship of the feature points in the detected face image.
Specifically, the image specifying unit 130 specifies the eye image G1, G2 of both eyes from the face image KG as shown in FIG. Note that the image in FIG. 5A is an example of the case where the user is paying attention to the first display unit 18 (looking at his / her line of sight). That is, as shown in FIG. 5B, the user Y has the line of sight L <b> 1 facing the first display unit 18, and the optical axis L <b> 2 from the camera 21 embedded in the lower frame of the first display unit 18. This is an example of shooting.
In this case, assuming that the user's eye is a circle (more specifically, a sphere), as shown in FIG. 5C, the line of sight L1 and the optical axis L2 are deviated and above the center CP of the eye. An image is taken so that the pupil H is positioned at Therefore, the image specifying unit 130 specifies eye images G1 and G2 that are slightly overlooked as shown in FIG. In more detail, the image specifying unit 130 specifies a substantially circular eye image including a portion hidden behind the skin. As an example, in the eye part images G1 and G2 as shown in FIG. 5D, up to the dotted line part CR hidden behind the skin is specified on the assumption that the eye part is substantially circular. That is, the image specifying unit 130 specifies a substantially circular eye image GB including the white eye SR and the pupil (that is, the black eye: iris and pupil) H as shown in FIG. . When a part of the pupil H is hidden behind the skin (eg, eyelid), the pupil H is assumed to be substantially circular, and an eye image GB that complements the hidden portion is specified.
On the other hand, as shown in FIG. 6 (a), the user Y is looking at the line of sight L3 toward the second display unit 19, and the upper frame of the second display unit 19 (the lower frame of the first display unit 18). When the image is taken from the embedded camera 21 with the optical axis L2, as shown in FIG. 6B, the line of sight L3 and the optical axis L2 are shifted, and the pupil H is located below the center CP of the eye part. Is taken so that is located. For this reason, the image specifying unit 130 specifies eye part images G1 and G2 that tend to be slightly bound as shown in FIG. Also in this case, up to the dotted line CR hidden behind the skin is specified assuming that the eye is substantially circular, the image specifying unit 130 for both eyes, as shown in FIG. A substantially circular eye image GB including the white eye SR and the pupil H (a complemented pupil H when part of the pupil is hidden by the skin) is specified.
As long as the user looks at the display unit 170 (the first display unit 18 or the second display unit 19), the pupil (pupil) is photographed regardless of the face orientation. In other words, if the user raises his face (sticks out his chin) or crawls his face (pulls his chin), some of his eyes may be hidden on the skin (upper and lower eyelids, etc.) The relationship between the line of sight and the optical axis is as shown in FIG. 5C and FIG. 6B. Therefore, the image specifying unit 130 specifies an eye part image GB as shown in FIGS. 5 (e) and 6 (d).
The CPU core 10a and the image processing unit 10b of the processing control unit 10 described above can function as such an image specifying unit 130.

Returning to FIG. 3, the image processing unit 140 processes the position of the pupil in the eye image specified by the image specifying unit 130.
For example, the image processing unit 140 processes the position of the pupil in each specified eye part image so as to be the center of the eye part image. Specifically, when the user is paying attention to the first display unit 18, the image processing unit 140 displays the pupil H of the eye part image GB that looks slightly upward as shown in FIG. As shown in FIG. 7B, a process of moving to the center of the eye image GB, that is, the center of a substantially circular shape is performed. In addition, when the user is paying attention to the second display unit 19, the image processing unit 140, for both eyes, displays the pupil H of the eye part image GB that tends to be slightly bowed as shown in FIG. As shown in FIG. 7D, a process of moving to the center of the eye image GB is performed.
Note that when moving the position of the pupil to the center of the eye image, the image processing unit 140 may appropriately move the position of the pupil in the pupil (iris). That is, as shown in FIG. 7E, not only the pupil H but also the pupil D is moved to the center of the eye image GB. Then, the image processing unit 140 cuts out (removes) the portion hidden in the skin, and returns the eye image G1 and G2 as shown in FIG.
The image processing unit 140 synthesizes a bright spot (bright spot: highlight) with a part of the processed pupil. That is, catch light processing is performed. Specifically, as shown in FIG. 8A, a bright spot KT1 reflecting light from a predetermined virtual light source is generated, and a predetermined location in the pupil H in the eye images G1 and G2 (as an example, the pupil ) And the like. Note that the shape and size of the bright spot to be combined, the combining position, and the like are arbitrary. For example, a substantially crescent-shaped bright spot KT2 as shown in FIG.
By performing such processing for both eyes, the image processing unit 140 finally has a user as shown in FIG. 8C turning his or her line of sight toward the photographing unit 110 (camera 21) (camera line of sight). A) face image KG.
Note that the CPU core 10a, the image processing unit 10b, and the like of the processing control unit 10 described above can function as such an image processing unit 140.

Returning to FIG. 3, the communication unit 150 controls communication with another image generation apparatus 100 connected via a predetermined network.
Specifically, the communication unit 150 includes an image transmission unit 151 and an image reception unit 152.
The image transmission unit 151 transmits a captured image including the face image (eye image) processed by the image processing unit 140 to another image generation apparatus 100. That is, a captured image obtained by processing the eye image in the user's face image is transmitted to the video chat partner image generation apparatus 100.
On the other hand, the image reception unit 152 receives an image transmitted from another image generation apparatus 100. That is, the image generation apparatus 100 of the video chat partner also receives a captured image including such a processed face image (eye part image).
Note that the above-described wireless communication unit 13, communication controller 14, and the like can function as such a communication unit 150.

The display control unit 160 displays a video chat screen in which the image received by the image receiving unit 152 and the captured image obtained by processing the face image (eye image) of the user (self) by the image processing unit 140 are displayed. 170.
For example, the display control unit 160 uses the image received by the image receiving unit 152 as a main, and synthesizes the captured image obtained by processing the face image by the image processing unit 140 as appropriate. Specifically, as shown in FIG. 9A, the display control unit 160 generates a video chat screen in which the reduced captured image KSG1 is combined with the upper right of the image ATG1 from the video chat partner.
The combining method of the display control unit 160 is an example, and both images may be combined by another method. For example, as shown in FIG. 9B, an image ATG2 from a video chatting partner and a processed photographed image KSG2 trimmed at the center may be combined and combined.
The CPU core 10a, the image processing unit 10b, and the like of the processing control unit 10 described above can function as such a display control unit 160.

The display unit 170 displays the video chat screen synthesized by the display control unit 160. That is, a video chat screen as shown in FIGS. 9A and 9B is displayed.
The first display unit 18 and the second display unit 19 described above can function as such a display unit 170.

(Outline of operation of image generation device)
Hereinafter, the operation of the image generating apparatus 100 having such a configuration will be described with reference to the drawings. As an example, the processing operation during video chat will be described with reference to FIG. FIG. 10 is a flowchart showing a flow of image processing performed by the image generation apparatus 100.

First, the image generation apparatus 100 captures a captured image including the user (step S201).
That is, the imaging unit 110 sequentially captures images including a user who performs a video chat, and supplies the captured images (for example, moving images) to the face detection unit 120.

The image generating apparatus 100 detects the face image of the user (person) from the captured image (step S202).
That is, the face detection unit 120 detects the face image of the user shown in the captured image by face recognition. That is, as shown in FIGS. 4A and 4B, the user's face image is detected.

The image generation device 100 identifies an eye image from the detected face image of the user (step S203).
That is, the image specifying unit 130 specifies the eye part image in the face image from the positional relationship of the feature points in the detected face image. That is, as shown in FIGS. 5D and 5E and FIGS. 6C and 6D, an eye image is specified from the detected face image.

The image generating apparatus 100 processes the position of the pupil in the identified eye image (step S204).
For example, the image processing unit 140 shows the position of the pupil H (pupil D) deviating from the center of the eye image GB as shown in FIGS. 7A and 7C described above, as shown in FIG. As shown in FIG. Further, as shown in FIGS. 8A and 8B described above, the image processing unit 140 synthesizes a bright spot with a part of the processed pupil.
That is, the image processing unit 140 performs such processing on both eyes, and finally the face image KG in which the user as shown in FIG. Is generated.

The image generation apparatus 100 transmits the captured image obtained by processing the face image (eye part image) to the other image generation apparatus 100, and receives an image sent from the other image generation apparatus 100 (step S205).
That is, the image transmission unit 151 transmits a captured image including the face image (eye image) processed by the image processing unit 140 to the image generation apparatus 100 of the video chat partner. On the other hand, the image receiving unit 152 receives an image transmitted from the image generation apparatus 100 of the video chat partner.

The image generating apparatus 100 appropriately synthesizes and displays the received image and a captured image obtained by processing the user's (self) face image (eye part image) (step S206).
That is, the display control unit 160 synthesizes the image received by the image receiving unit 152 and the captured image including its own face image processed by the image processing unit 140, and described above with reference to FIGS. The video chat screen as shown in FIG.

The image generating apparatus 100 determines whether the video chat has ended (step S207).
If it is determined that the video chat has not ended (step S207; No), the image generating apparatus 100 returns the process to step S201 and repeatedly executes the processes of steps S201 to S207 described above.
On the other hand, when it is determined that the video chat has ended (step S207; Yes), the image generation apparatus 100 ends the image processing.

In such an image processing process, an eye part image is specified from a face image of a person (user) detected from a captured image. That is, as shown in FIG. 7A and FIG. 7C described above, an eye image GB whose line of sight is shifted (not the camera line of sight) is specified.
Then, the image generation apparatus 100 processes the pupil position and the like of the identified eye image so as to be the center of the eye image, and further performs catch light processing. That is, the pupil H or the like is moved to the center of the eye image GB as shown in FIG. 7E, and the bright spot KT1 or the like is placed at a predetermined position of the pupil H as shown in FIG. Is synthesized.
In this way, by identifying a substantially circular eye image and moving the pupil in the eye image to the center of the eye image, the user's line of sight is taken as the camera line of sight regardless of the user's face orientation and line of sight. be able to.
Then, the image generation apparatus 100 transmits a captured image including the eye part image (face image) processed in this way to another apparatus. That is, a captured image including a face image processed into a face image in which the user turns his / her line of sight toward the photographing unit 110 (at the camera's line of sight) is transmitted to another device. For this reason, the other party (another apparatus) can obtain an image of a user who is looking at the user (the other party user), so that the video chat can be naturally performed without a sense of incongruity.
As a result, an image in which the user's line-of-sight direction is appropriately corrected can be generated.

(Other embodiments)
In the above embodiment, the position of the pupil in the user's face image (eye image) is unconditionally processed so as to be the center of the eye image, but inconvenience may occur depending on the user's line-of-sight direction. . For example, when the user's line of sight is greatly deviated from the display unit 170 (when looking away), an unnatural face image is obtained by performing processing with the pupil position of the eye image centered. End up. Therefore, the user's line-of-sight direction may be calculated, and the position of the pupil of the eye image may be processed only within a reasonable line-of-sight range.
Hereinafter, another embodiment, in which the position of the pupil of the eye image is processed after the user's line-of-sight direction is confirmed, will be described with reference to the drawings.

FIG. 11 is a block diagram showing a schematic configuration of an image generation apparatus 300 according to another embodiment of the present invention. The display generation apparatus 300 has a configuration in which a calculation unit 310 is added to the image generation apparatus 100 of FIG. 3 described above.
That is, the configuration of the photographing unit 110 to the display unit 170 is the same as that of the image generation device 100 of FIG.

The calculation unit 310 calculates the user's line-of-sight direction from the face image detected by the face detection unit 120. For example, the calculation unit 310 calculates the user's line-of-sight direction by the following calculation method.
The calculation unit 310 stores in advance a shape that indicates the positional relationship of the characteristic parts when a person faces the front. Specifically, as shown in FIG. 12A, for example, the calculation unit 310 connects a binocular and a mouth with a straight line in an image of a standard human face taken from the front at a certain distance. A reference plane SF of an elliptical plane (specifically, a triangle) is stored in advance. The reference shape SF may be obtained from an image obtained by photographing an actual user at a certain distance from the front.
In a state where the reference shape SF is stored in advance, as shown in FIG. 12B, when a shape F of a plane connecting the eyes and the mouth with a straight line in the detected face image KG of the user is obtained, this shape is obtained. By comparing F with the reference shape SF, the calculation unit 310 calculates the viewing direction of the observer.
For example, as shown in FIG. 12C, when the obtained shape F1 is similar to the reference shape SF, the calculation unit 310 calculates the line-of-sight direction GL1 that is directly in front. On the other hand, as shown in FIG. 12D, when the obtained shape F2 is not similar to the reference shape SF, the calculation unit 310 calculates the shape from the positional relationship of each point and the length of each side in the shape F2. A distortion of F2 (a distortion based on the reference shape SF) is obtained. Then, based on the distortion, a visual line direction GL2 that is an oblique direction is calculated. That is, it is possible to calculate the line-of-sight direction of the user who has largely missed the line of sight as shown in FIG.
Note that such a method of calculating the user's line-of-sight direction is an example, and other methods may be used. For example, the calculation unit 310 may more simply calculate the user's line-of-sight direction based on the positional relationship between both eyes.
The calculation unit 310 supplies the user's line-of-sight direction calculated in this way to the image processing unit 140.
The CPU core 10a and the image processing unit 10b of the processing control unit 10 described above can function as such a calculation unit 310.

Returning to FIG. 11, when the gaze direction of the user is calculated by such a calculation unit 310, the image processing unit 140 determines that the calculated gaze direction is within a predetermined range (for example, the optical axis of the imaging unit 110. The pupil position of the eye image is processed only when the deviation is within a relatively small range).
That is, when the calculated line-of-sight direction exceeds a predetermined range and the user's line-of-sight is greatly deviated, the image processing unit 140 does not perform processing centering on the position of the pupil of the eye image. Only the process of combining the bright spot with a part of the pupil.
That is, after confirming the user's line-of-sight direction, the image processing unit 140 processes the position of the pupil of the eye part image only within a reasonable line-of-sight range.

  For this reason, the image generation apparatus 300 illustrated in FIG. 11 can generate an image in which the user's line-of-sight direction is corrected within a range that does not become unnatural.

  In the above embodiment, the case where the image generation devices 100 and 300 of the present application are applied to the portable game device 1 has been described. However, the present invention is not limited to such a portable game device 1, and for example, a notebook type A personal computer etc. may be sufficient.

  As described above, according to the present invention, it is possible to provide an image generation apparatus, an image processing method, and a program that can generate an image in which a user's line-of-sight direction is appropriately corrected.

DESCRIPTION OF SYMBOLS 1 Game device 10 Processing control part 11 Connector 12 Cartridge 13 Wireless communication part 14 Communication controller 15 Sound amplifier 16 Speaker 17 Operation key 18 1st display part 19 2nd display part 20 Touch panel 21 Camera 100,300 Image generation apparatus 110 Photographing Unit 120 face detection unit 130 image specifying unit 140 image processing unit 150 communication unit 160 display control unit 170 display unit 310 calculation unit

Claims (6)

A face detection unit for detecting a human face image from a captured image;
An image specifying unit for specifying an eye part image from the detected face image;
A reference shape obtained from the pre-stored face of the person by comparing the obtained from the detected the face image shape, and a calculation unit for calculating the gaze direction of the person,
Calculation has been the viewing direction only if it is within a predetermined range, e Bei and an image processing unit for processing the position of the pupil in the identified the eye image,
The calculation unit stores a reference shape of an inverted triangle in which both eyes and a mouth are connected with a straight line when the person faces the front, and both the detected face images in the detected face image with the reference shape as a reference. Based on the distortion of the inverted triangle that connects the eyes and mouth with a straight line, the gaze direction of the person is calculated,
The image specifying unit specifies an eye image that complements a portion hidden in the skin from the detected face image,
The image processing unit performs processing to remove a portion hidden in the skin after moving the position of the pupil in the identified eye image.
An image generation apparatus characterized by that. A shooting section for shooting a shot image including a person;
A face detection unit that detects a face image of the person by face recognition from the captured image;
An image specifying unit for specifying an eye part image from the detected face image;
A reference shape obtained from the pre-stored face of the person by comparing the obtained from the detected the face image shape, and a calculation unit for calculating the gaze direction of the person,
An image processing unit that processes the position of the pupil in the specified eye image only in the center of the eye image only when the calculated line-of-sight direction is within a predetermined range;
E Bei captured image containing a face image which the eye image is processed, an image transmitting unit that transmits to the other device, and
The calculation unit stores a reference shape of an inverted triangle in which both eyes and a mouth are connected with a straight line when the person faces the front, and both the detected face images in the detected face image with the reference shape as a reference. Based on the distortion of the inverted triangle that connects the eyes and mouth with a straight line, the gaze direction of the person is calculated,
The image specifying unit specifies an eye image complemented with a portion hidden in the skin, assuming that the eye portion is substantially circular, from the detected face image,
The image processing unit performs processing for removing a portion hidden in the skin after moving the pupil position in the identified eye image to the center.
An image generation apparatus characterized by that. The image generation apparatus according to claim 1 or 2,
The image processing unit further synthesizes bright spots on a part of the pupil of the processed eye image.
An image generation apparatus characterized by that. The image generation apparatus according to any one of claims 1 to 3,
The image specifying unit specifies, from the detected face image, an eye part image including a white eye and a pupil, and an eye part image that complements the white eye and pupil hidden in the skin, assuming that the pupil is substantially circular.
An image generation apparatus characterized by that. An image processing method in an image generation apparatus having a face detection unit, an image specifying unit, a calculation unit, and an image processing unit,
A face detection step in which the face detection unit detects a face image of a person from a captured image;
An image specifying step in which the image specifying unit specifies an eye image from the detected face image;
The calculation unit, by comparing the reference shape obtained from the pre-stored face of the person, and were obtained from the detected the face image shape, a calculation step to calculate the gaze direction of the person,
Wherein the image processing unit, calculated by the said viewing direction only if it is within a predetermined range, e Bei and an image processing step for processing the position of the pupil in the identified the eye image,
In the calculation step, a reference shape of an inverted triangle obtained by connecting the eyes and the mouth with a straight line when the person is facing the front is stored, and both the detected face images in the detected face image are based on the reference shape. Based on the distortion of the inverted triangle that connects the eyes and mouth with a straight line, the gaze direction of the person is calculated,
In the image specifying step, from the detected face image, an eye image that complements a portion hidden in the skin is specified,
In the image processing step, after moving the position of the pupil in the specified eye image, processing to remove the portion hidden in the skin,
An image processing method characterized by the above. Computer
A face detection unit for detecting a human face image from a photographed image;
An image specifying unit for specifying an eye part image from the detected face image;
Previously stored and the reference shape obtained from the face of the person was, by comparing the obtained from the detected the face image shape, calculation unit for calculating the gaze direction of the person,
Only when the calculated gaze direction is within a predetermined range, function as an image processing unit that processes the position of the pupil in the specified eye image ,
The calculation unit stores a reference shape of an inverted triangle in which both eyes and a mouth are connected with a straight line when the person faces the front, and both the detected face images in the detected face image with the reference shape as a reference. Based on the distortion of the inverted triangle that connects the eyes and mouth with a straight line, the gaze direction of the person is calculated,
The image specifying unit specifies an eye image that complements a portion hidden in the skin from the detected face image,
The image processing unit functions to perform processing to remove a portion hidden in the skin after moving the position of the pupil in the identified eye image.
Program which is characterized a call.

Images