WO2021210340A1 - Image processing device, image processing method, and imaging device - Google Patents

Abstract

The present technology relates to an imaging device comprising an inclination calculation unit for calculating inclination of the face of a subject in a pitch direction, and a corresponding process unit for performing a corresponding process using the calculated inclination.

Description

Image processing equipment, image processing method and imaging equipment

This technology relates to an image processing device, an image processing method, and an imaging device for face detection in an image.

There are many opportunities to take a picture of a person with a camera, and in that case, the face of the person is detected and the person is focused on the face.
In focusing control on a person's face, for example, as shown in Patent Document 1, the focus may be on the center of the face frame or the portion closest to the image pickup device.

Japanese Unexamined Patent Publication No. 2007-328215

By the way, in recent years, the number of pixels of an image pickup device has been increased and the size of an image pickup device has been increased, and the number of image pickup devices having a shallow depth of field is increasing.
In the case of an image pickup device having a shallow depth of field, if the focus is on the center of the face frame or the portion closest to the image pickup device, there is a problem that the image pickup device is focused on an unintended position. In particular, depending on the upward or downward adjustment of the subject's face, the part closest to the imaging device becomes the chin or head.
If you focus on your chin or head, your complexion may become inappropriate as a result of the white balance adjustment process.

In view of such problems, the purpose of this technology is to perform appropriate response processing according to the upward adjustment and downward adjustment of the subject's face.

The image processing apparatus according to the present technology includes a tilt calculation unit that calculates the tilt of the face of the subject in the pitch direction, and a corresponding processing unit that performs corresponding processing using the calculated tilt.
By calculating the tilt of the subject's face, it is possible to perform optimal focusing control and white balance adjustment.

The image processing device described above includes a site specifying unit that detects the position of the nose and the position of the pupil in the subject, and the inclination calculating unit determines the face based on the detected position of the nose and the position of the pupil. The angle of the inclination in the pitch direction may be calculated.
As a result, the angle of the face in the pitch direction can be calculated with high accuracy.

The inclination calculation unit in the image processing apparatus described above may use face average data which is average distance information about a face in calculating the angle.
As a result, the angle of the face in the pitch direction can be calculated with higher accuracy.

The image processing apparatus described above includes an average data selection unit that selects one from the face average data provided at least for each race or age, and the inclination calculation unit is one selected by the average data selection unit. The angle may be calculated based on the two face average data.
As a result, the angle of the face is calculated with high accuracy based on the average face data according to the age.

The average data selection unit in the image processing apparatus described above clusters the distance information for each of a plurality of ranging areas provided on the image, and from the distribution of the distance information of the face of the subject obtained as a result of the clustering. The size of the subject's face in the depth direction may be calculated, and the face average data may be selected based on the calculated size of the face in the depth direction.
As a result, the face average data is selected based on the fact that the child's face is small and the adult's face is large.

The inclination calculation unit in the image processing device may calculate the inclination in the yaw direction and the inclination in the roll direction of the face of the subject by using the face average data.
This makes it possible to calculate the angle of inclination of the face in all directions.

In the above-mentioned image processing apparatus, when the face of the subject is determined to be a tilted face facing upward based on the inclination calculated by the inclination calculation unit, the corresponding processing unit performs the corresponding processing. , Focus control may be performed at a position other than the chin on the subject.
This prevents the chin from being in focus.

In the above-mentioned image processing apparatus, when the face of the subject is determined to be a downward-facing face based on the inclination calculated by the inclination calculation unit, the corresponding processing unit performs the corresponding processing as the corresponding processing. Focus control may be performed at a position other than the head of the subject.
This prevents the head from being in focus.

The corresponding processing unit in the image processing apparatus described above may perform focusing control at the position of the pupil of the subject as the corresponding process.
For example, when the depth of field is extremely shallow, such as in a high-resolution imaging device, the pupil may become unclear.

The corresponding processing unit in the image processing apparatus described above may perform a process of outputting the tilt information as the corresponding process.
As a result, for example, the external device can perform processing using the tilt information of the face.

In the image processing apparatus described above, when the position of the pupil can be detected by the site identification unit but the amount of defocus for the position of the pupil cannot be calculated, the corresponding processing unit performs the pupil as the corresponding process. Focus control may be performed using the defocus amount estimated for the position of.
For example, when an object such as a branch is located between the pupil and the image pickup device, the defocus amount may not be calculated appropriately.

The image processing method according to the present technology calculates the inclination of the face of the subject in the pitch direction, and performs the corresponding processing using the calculated inclination.

The imaging device according to the present technology includes an imaging unit that images a subject, a tilt calculation unit that calculates the tilt of the subject's face in the pitch direction, and a control unit that performs focusing control according to the calculated tilt. It is equipped with.

It is a block diagram of the image pickup apparatus of embodiment of this technique. It is a schematic explanatory drawing which shows the arrangement example of the PD division pixel as an image plane phase difference pixel. It is a block diagram which shows the functional configuration example of an image pickup apparatus. It is a flowchart of 1st example of correspondence processing. It is a flowchart which shows an example of the focusing area selection process. It is a schematic explanatory drawing which shows the relationship between a target area and a face frame. It is a histogram about the distance of the target area in a face frame. It is a figure which shows the target area classified into a face cluster by a clustering process. It is explanatory drawing for showing the head length and the distance between an ear and a nose. It is explanatory drawing which shows the relationship between a subject, a lens, an image sensor, and a focal length. It is the schematic which shows the pupil position and the nose position of the face facing the image pickup apparatus. It is the schematic which shows the pupil position and the nose position in the tilted face. It is the schematic which shows the pupil position and the nose position in the face down. It is a graph which shows the relationship between the distance and the inclination of the pupil position and the nose position. It is a flowchart which shows an example of the process flow for calculating the inclination of a face in a pitch direction using the information of a pupil position and a nose position. It is a figure which shows the 2nd example of the correspondence processing, and is the figure which shows the example of the processing which highlights the tip of the line of sight of a subject. FIG. 16 is a diagram showing a second example of the corresponding process together with FIG. 16, and is a diagram showing an example of a process of highlighting the tip of the line of sight of the subject.

Hereinafter, embodiments will be described in the following order with reference to the accompanying drawings.
<1. Imaging device configuration>
<2. Control function configuration>
<3. Correspondence processing using tilt information>
<3-1. First example of corresponding processing>
<3-2. Second example of corresponding processing>
<4. Summary>
<5. This technology>

<1. Imaging device configuration>
As one of the embodiments of the image processing apparatus of the present technology, an image pickup apparatus provided with an image processing unit will be described as an example.
However, the image processing device is not limited to the image processing device provided with the image processing unit, and may be an information processing device that performs image processing on the received captured image.

The image pickup device 1 can be considered in various forms as a video camera or a still camera.

The image pickup device 1 is provided with an image pickup element, various lenses, an operator, and the like inside and outside. Further, the image pickup apparatus 1 may be provided with a display monitor or an EVF (Electric Viewfinder).

FIG. 1 shows a block diagram of the image pickup apparatus 1.
The imaging device 1 includes a lens system 2, an imaging unit 3, a signal processing unit 4, a recording unit 5, a display unit 6, an output unit 7, an operation unit 8, a control unit 9, a memory unit 10, and a driver unit 11. ..

The lens system 2 is equipped with various optical lenses, an aperture mechanism, and the like.

The image pickup unit 3 is configured to include, for example, an image pickup element 12 of a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal-Oxide Semiconductor) type. The sensor surface of the image pickup device 12 is configured to include a sensing element in which a plurality of pixels are two-dimensionally arranged.

The image pickup unit 3 executes, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and the like on the electric signal obtained by photoelectric conversion of the light received by the image sensor 12, and further performs A / D (Analog) processing. / Digital) Performs conversion processing. The imaging unit 3 outputs the captured image signal as digital data to the signal processing unit 4 and the control unit 9.

The image sensor 12 includes, for example, an image plane phase difference pixel 12a that outputs a signal for calculating phase difference information for calculating the defocus amount.
All the pixels included in the image pickup device 12 may be the image plane phase difference pixel 12a, or some pixels may be the image plane phase difference pixel 12a.

FIG. 2 shows an example of arranging the image plane retardation pixels 12a.
Among the pixels provided with the Bayer-arranged color filters (R, G, B), a part of the pixels having the color filter having the spectral sensitivity of green (G) is defined as the image plane phase difference pixel 12a.

The image plane phase difference pixel 12a may be, for example, a PD divided pixel by a PD (Photodiode) division method or a light-shielding pixel by a light-shielding method. The example shown in FIG. 2 is an example in which the PD division pixel is adopted as the image plane retardation pixel 12a.

In the present embodiment, the defocus amount is calculated based on the signal output from the image plane phase difference pixel 12a included in the pixel region of a predetermined size (hereinafter referred to as “target region”), and the defocus amount is calculated from the defocus amount. The distance information between the reference position (for example, the image sensor 12) and the subject is calculated for each target area. Thereby, for example, the distance information for each part constituting the face of the subject is calculated and used in the processing described later.

The signal processing unit 4 is composed of, for example, a microcomputer specialized in digital signal processing such as a DSP (Digital Signal Processor), a microcomputer, or the like.

The signal processing unit 4 includes each unit for performing various signal processing on the digital signal (image captured image signal) sent from the imaging unit 3.

Specifically, processing such as correction processing between R, G, and B color channels, white balance correction, aberration correction, and shading correction is performed.
Further, the signal processing unit 4 generates (separates) a luminance (Y) signal and a color (C) signal from the image data of R, G, and B, a YC generation process, a process of adjusting the luminance and the color, and a knee correction. And each process such as gamma correction.
Further, the signal processing unit 4 performs conversion to the final output format by performing resolution conversion processing, codec processing for encoding for recording and communication, and the like. The image data converted into the final output format is stored in the memory unit 10. Further, by outputting the image data to the display unit 6, the image is displayed on the back side of the main body or on the monitor provided in the EVF. Further, by outputting from the output unit 7 provided with the external output terminal, it is displayed on a device such as a monitor provided outside the image pickup apparatus 1.

The recording unit 5 is composed of, for example, a non-volatile memory, and functions as a storage means for storing an image file (content file) such as still image data or moving image data, attribute information of the image file, a thumbnail image, or the like.
The image file is stored in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), or the like.
The actual form of the recording unit 5 can be considered in various ways. For example, the recording unit 5 may be configured as a flash memory built in the image pickup device 1, or a memory card (for example, a portable flash memory) that can be attached to and detached from the image pickup device 1 and storage / reading from the memory card. It may be composed of an access unit that performs access for. Further, it may be realized as an HDD (Hard Disk Drive) or the like as a form built in the image pickup apparatus 1.

The display unit 6 executes processing for performing various displays on the imager. The display unit 6 is, for example, a rear monitor or a finder monitor. The display unit 6 performs a process of displaying image data converted to an appropriate resolution input from the signal processing unit 4. As a result, a so-called through image, which is a captured image during the standby of the release, is displayed.
Further, the display unit 6 realizes on the screen the display of various operation menus, icons, messages, etc. as a GUI (Graphical User Interface) based on the instruction from the control unit 9.
Further, the display unit 6 can display a reproduced image of the image data read from the recording medium by the recording unit 5.

The output unit 7 performs data communication and network communication with an external device by wire or wireless. For example, captured image data (still image file or moving image file) is transmitted to an external display device, recording device, playback device, or the like.
Further, the output unit 7 may function as a network communication unit. For example, communication may be performed by various networks such as the Internet, a home network, and a LAN (Local Area Network), and various data may be transmitted and received to and from a server, a terminal, or the like on the network.

The operation unit 8 provided in the housing of the camera includes not only mechanical controls such as buttons and switches, but also a monitor that employs a touch panel method, and various operations such as tap operation and swipe operation of the imager. The operation information corresponding to the operation is output to the control unit 9.

The control unit 9 is composed of a microcomputer equipped with a CPU (Central Processing Unit) and performs overall control of the image pickup apparatus 1. For example, the shutter speed is controlled according to the operation of the imager, the signal processing unit 4 instructs various signal processing, the imaging operation and the recording operation are performed according to the user's operation, and the recorded image file is reproduced.

Further, the control unit 9 gives an instruction to the driver unit 11 in order to control various lenses included in the optical system 51.
For example, a process of designating an aperture value in order to secure a necessary amount of light for AF control, an operation instruction of an aperture mechanism according to the aperture value, and the like are performed.
The control unit 9 has a configuration capable of realizing various functions described later. Specifically, it will be described later.

The memory unit 10 stores information and the like used for processing executed by the control unit 9. The illustrated memory unit 10 comprehensively indicates, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, and the like.
The memory unit 10 may be a memory area built in the microcomputer chip as the control unit 9, or may be configured by a separate memory chip.

Programs and the like used by the control unit 9 are stored in the ROM, flash memory, and the like of the memory unit 10. In the ROM, flash memory, etc., in addition to content files such as an OS (Operating System) for the CPU to control each part and an image file, application programs and firmware for various operations are stored.
The control unit 9 controls the entire image pickup apparatus 1 by executing the program.

The RAM of the memory unit 10 is used as a work area of the control unit 9 by temporarily storing data, programs, and the like used in various data processing executed by the CPU of the control unit 9.

The driver unit 11 is provided with, for example, a motor driver for the zoom lens drive motor, a motor driver for the focus lens drive motor, an aperture mechanism driver for the motor that drives the aperture mechanism, and the like.
Each driver supplies a drive current to the corresponding drive motor in response to an instruction from the control unit 9.

<2. Control function configuration>
In the control unit 9 of the image pickup apparatus 1, the functional configuration as shown in FIG. 3 is constructed by executing the program stored in the ROM or RAM as the memory unit 10.
The functional configuration shown in FIG. 3 is not only constructed only in the control unit 9, but may be constructed so that the control unit 9 and the signal processing unit 4 cooperate to perform a function. Further, the functional configuration shown in FIG. 3 may be constructed in the signal processing unit 4.

The control unit 9 includes a subject identification unit 20, a site identification unit 21, a defocus amount calculation unit 22, an average data selection unit 23, an inclination calculation unit 24, an area selection unit 25, a focusing control unit 26, and an output control unit 27. I have.

The subject identification unit 20 detects and identifies a subject such as a person or an object in the image to be processed by a known technique for detecting the subject. As a method for detecting a subject, for example, a face / object recognition technique by template matching, a matching method based on the brightness distribution information of the subject, a method based on a skin-colored part included in an image, a feature amount of a human face, or the like is used. be able to. Further, these methods may be combined to improve the detection accuracy.
Further, the subject may be specified by executing image recognition processing using machine learning technology such as CNN (Convolutional Neural Network), or may be specified by using the information input by the user about the subject. good.

The site specifying unit 21 executes a process of identifying the position of the face portion of the subject on the captured image. The parts of the face to be identified are, for example, eyes, nose, chin, head, cheeks, ears, mouth and the like.
The identification of the facial part is performed by machine learning such as CNN. In CNN, the feature amount of the image is compressed by performing the convolution process and the pooling process. This makes it possible to identify the facial part of the subject.

The defocus amount calculation unit 22 calculates the distance between the face of the subject and the image sensor 12 by calculating the defocus amount for each target area. By combining the defocus amount for each target area and the position information of the facial part of the subject detected by the part identification unit 21, the distance from the image sensor 12 can be calculated for each part.

The average data selection unit 23 performs a process of selecting face average data (hereinafter referred to as “face average data”) based on subject attribute information.

Attribute information is, for example, gender, age, race, etc. In addition, the age may be not only a specific age but also a rough outline such as the age (teens, 20s, etc.), or more roughly, such as "adult" or "child". You may.

Face average data includes, for example, occipital / nasal tip distance, total head height, inter-pupil width, head circumference, head length, head width, buccal arch width, mandibular angle width, ears width, mastoid width, and internal eye angle. Distance information between parts such as width, external eye angle width, mouth fissure width, morphological face height, nose height, and subnasal / chin distance.
In the following description, the face average data consisting of these various distance information will be described based on an example in which two types, "for adults" and "for children", are provided.

In order to select face average data, first, as a result of image recognition using machine learning, position information about facial parts such as the position of the nose, the position of the eyes, and the position of the chin of the subject is obtained. Next, distance information between facial parts (for example, inter-pupil width, nose height, etc.) is calculated, and the information is compared with a plurality of face average data (children's face average data and adult face average data). Select the most suitable face average data.

The tilt calculation unit 24 calculates tilt information for the subject specified by the subject identification unit 20. Specifically, for example, the inclination of the pitch direction, the yaw direction, and the roll direction of the face of the person as the subject specified by the subject identification unit 20 is calculated. It should be noted that only a part of the inclinations in the pitch direction, the yaw direction and the roll direction may be calculated.

Here, the optical axis direction of the image pickup apparatus 1 is referred to as "front-back direction", the vertical direction is referred to as "vertical direction", and the direction orthogonal to both the front-back direction and the up-down direction is referred to as "left-right direction".
The inclination of the face in the pitch direction refers to the inclination of the face with the left-right direction as the rotation axis. The inclination of the face in the yaw direction refers to the inclination of the face with the vertical direction as the rotation axis. Further, the inclination in the roll direction refers to the inclination of the face with the rotation axis in the front-back direction.
In the following description, the inclinations in the pitch direction, the yaw direction, and the roll direction are collectively referred to as "face inclination".

There are several possible ways to calculate the inclination of the face in each direction. Specifically, it will be described later.

The area selection unit 25 performs a process of selecting an in-focus area, which is an area to be in-focus control, from the target area.
The focus area is selected based on which part of the subject is in focus. For example, when it is desired to focus on the pupil, the target area in which the pupil is imaged is selected as the focusing area.
When it is desired to select the target area in which the pupil of the subject is imaged as the focusing area, the target area may be unknown.

The focusing control unit 26 performs focusing control based on the focusing area selected by the area selection unit 25. There are several possible patterns for focusing control. An example will be described in which the focusing area is the area where the pupil of the subject is imaged.

First, if the focusing area is selected without any problem and the defocus amount of the focusing area can be calculated appropriately, the focusing control is performed based on the defocus amount of the focusing area.

Next, the focusing area is selected without any problem, but an obstacle (hereinafter referred to as "forward object") located between the pupil and the image sensor 1 is located, and the defocus amount in the focusing area is appropriate. If it cannot be calculated, the distance between the pupil and the image sensor 12 is estimated from the distance between the other part of the subject (for example, the nose) and the image sensor 12, and the difference information between the other part and the position of the pupil in the anteroposterior direction. Then focus control is performed.
Even if the focusing area cannot be specified, it is possible to estimate the distance between the pupil and the image sensor 12 from the relative positional relationship between the other portion and the pupil and focus on the focus.

The output control unit 27 outputs information such as the position of the subject and the position of the face in the captured image, or various inclinations (rotation angles around each axis) calculated by the inclination calculation unit 24 via the output unit 7. ..

<3. Correspondence processing using tilt information>
It is possible to execute various corresponding processes using the tilt information of the subject's face. Here are some examples.

<3-1. First example of corresponding processing>
In the first example of the correspondence process, the focus is controlled on the pupil of the subject. Specifically, it will be described with reference to the flowchart of FIG.

The control unit 9 performs face detection processing in step S101. This process is performed using machine learning such as CNN, for example, as described above.

Subsequently, in step S102, the control unit 9 determines whether or not the face of the subject can be detected and performs branch processing.

When it is determined that the face of the subject cannot be determined, the control unit 9 performs a process of causing the user to select one target area, for example, in step S103, and selects the selected target area as the focusing area. After that, the control unit 9 proceeds to the process of step S105.

On the other hand, if it is determined that the face of the subject can be detected, the control unit 9 proceeds to the focusing area selection process in step S104.

An example of the focusing area selection process is shown in FIG.
In the focusing area selection process, the control unit 9 determines in step S201 whether or not the pupil can be detected. When the pupil of the subject can be detected, the control unit 9 attempts to calculate the defocus amount for the target area corresponding to the pupil position detected in step S202, and determines whether or not the defocus amount can be calculated.

When it is determined that the defocus amount can be calculated, for example, when there is no obstacle (forward object) between the pupil of the subject and the imaging device 1, the control unit 9 is the target corresponding to the pupil position in step S203. The area is selected as the focusing area, and the focusing area selection process shown in FIG. 5 is completed.

On the other hand, if it is determined in step S201 that the pupil could not be detected, or if it is determined in step S202 that the pupil could be detected but the defocus amount could not be calculated, the control unit 9 performs each process from step S204 to step S211. By executing this, the focusing area for focusing on the pupil of the subject is selected.
For example, when a front object such as a branch is located between the pupil and the image pickup apparatus 1, the defocus amount may not be calculated normally even if the pupil can be detected. In such a case, the process proceeds to step S204.

In step S204, the control unit 9 calculates the defocus amount for each target area 31A included in the face frame 30.
For example, as shown in FIG. 6, the defocus amount is calculated for each target area 31A located inside the face frame 30 among the plurality of target areas 31 included in the captured image.

Subsequently, in step S205, the control unit 9 creates a histogram of the defocus amount calculated in step S204 and performs a clustering process.
FIG. 7 is an example of a histogram of the target area 31A. The horizontal axis in the histogram represents the distance to the image sensor 12, and the vertical axis represents the number of the corresponding target regions 31A.

Clustering is performed from the histogram shown in FIG. 7 using a method such as the k-means method. As a result of clustering, each target area 31A is classified into a "face cluster" in which the area in which the face is imaged is classified, a "front object cluster" in which the area in which the front object is imaged is classified, and an area in which the background object is imaged. Classified as "background object cluster".

Figure 8 shows an example of the classification results. As shown in the figure, each target area 31A is classified into one of a face cluster, a front object cluster, and a background cluster according to the distance from the image sensor 12.

When classifying into each cluster, the length of the face in the anterior-posterior direction (head length D1) and the distance D2 between the positions of the ears and the nose in the anterior-posterior direction may be taken into consideration (FIG. 9).

A specific example is given. For example, when the number of image pickup pixels of the image pickup apparatus 1 is 50 million, the focal length f is 85 mm, the aperture F value is 1.4, and the permissible circle of confusion diameter is 8.3 μm, the image sensor When focusing on a subject located at a distance of 12 to 2 m, the front depth of field is 6.41 mm and the rear depth of field is 6.45 mm. Let this be one depth of field.

Also, in the face average data, the distance from the center position of the face to the position of the ears is 96 mm. At this time, when the profile of the subject is imaged, the depth of 96 mm is about 15 depth of field (= 96 / 6.45). 15 The depth of field is 134.4 μm when converted to the defocus amount. That is, the range in which the defocus amount can be taken is 134.4 μm. Based on this value, the target area 31A classified into the face cluster may be determined.

In step S206 of FIG. 5, the control unit 9 performs a process of excluding the target area 31A classified into the front object cluster and the target area 31A classified into the background object cluster.

Next, in step S207, the control unit 9 estimates the distance between the image sensor 12 and the face from the data of the lens position and the defocus amount. In this process, for example, the median defocus amount of the defocus amount for each target area 31A classified into the face cluster is specified, and the distance between the imaging element 12 and the face is calculated from the median defocus amount. presume. The amount of defocus for the other target areas 31A classified into the face cluster may be calculated.

To calculate the distance between the image sensor 12 and the face based on the lens position and the amount of defocus, for example, the following formula (1) is used.

1 / f = 1 / L1 + 1 / L2 ... Equation (1)

Here, the variable f represents the focal length f. The variable L1 represents the distance L1 between the subject and the lens. The variable L2 represents the distance L2 between the lens and the image sensor 12.

Since the variable f and the variable L2 are known, the distance L1 between the subject and the lens can be calculated (see FIG. 10).

Further, by using the following formula (2), the distance between facial parts (for example, interpupillary width, inner canthus width, outer canthus width, etc.) can be calculated.

Y2 / Y1 = L2 / L1 ... Equation (2)

Here, the variable Y2 represents the distance Y2 on the image sensor 12. The variable Y1 represents the actual distance Y1.
Since the distance Y2, the distance L1 and the distance L2 are known, the distance Y1 can be calculated (see FIG. 10).

In step S208 of FIG. 5, the control unit 9 selects face average data based on the distance information. Specifically, one is selected from a plurality of prepared face average data (for example, adult face average data and child face average data) based on the ratio and size of each distance information.
As an example, the difference between the minimum defocus amount and the maximum defocus amount is calculated in the target area 31A classified into the face cluster, and the face average data for adults is selected or the face average for children is selected according to the magnitude of the difference. Decide whether to select the data.

In step S209, the control unit 9 calculates the position of the front pupil from the inclination of the face and the selected face average data. The front pupil is the pupil of the two pupils of the subject that is closer to the image pickup apparatus 1. That is, the distance between the front pupil and the image sensor 12 is calculated.
The information on the inclination of the face is the information acquired in the face detection process in step S101 of FIG.

The control unit 9 performs a three-axis correction process of the face angle in step S210. In the 3-axis correction process, the angle of the face, that is, the inclination of the face in the pitch direction, the yaw direction, and the roll direction is calculated, and the distance between the front pupil and the image sensor 12 is corrected.

In step S211 the control unit 9 selects the target area 31A having a close defocus amount as the focusing area. In selecting the focusing area, the target area 31A close to the pupil position may be selected from the target areas 31A having a close defocus amount.

Returning to the description of FIG.
The control unit 9 that has completed the focusing area selection process acquires the defocus amount of the selected focusing area in step S105.
Subsequently, in step S106, the control unit 9 drives the lens according to the acquired defocus amount to perform focusing control. As a result, the focus is on the front pupil.

The control unit 9 determines in step S107 whether or not the face can be detected. If the face cannot be detected, the process returns to step S101 again.

On the other hand, if face detection is successful as a result of focusing control, the series of processes shown in FIG. 4 is terminated.

Note that the inclination of the face in the pitch direction, yaw direction, and roll direction does not have to be output by machine learning such as CNN. For example, the inclination in each direction may be calculated according to the pupil position and the nose position.

Specifically, an example of calculating the inclination of the face in the pitch direction will be described with reference to FIG.
Let the distance X be the distance between the pupil position and the nose position of the face facing the image pickup device 1 in the vertical direction (see FIG. 11).
Further, the distance between the pupil position and the nose position in the vertical direction is defined as the distance Y in the tilted face with the subject's face facing upward (see FIG. 12).
Further, the distance between the pupil position and the nose position in the vertical direction is defined as the distance Z in the face with the subject's face facing downward (see FIG. 13).

It can be seen that the distance Y and the distance Z are shorter than the distance X (see FIG. 14). Therefore, it is possible to calculate how many times the face of the subject is tilted in the pitch direction according to the value of the distance Y or the distance Z.
The distance X may differ depending on the attributes of the subject (gender, age, race, etc.). In this case, the values of the distance Y and the distance Z also differ depending on the attributes of the subject. Specifically, if it is a child, the shape is such that the graph of FIG. 14 is crushed in the vertical direction.
Therefore, an appropriate graph may be selected using the face average data according to the attributes of the subject, and the inclination of the face may be calculated from the values of the distance Y and the distance Z.

Further, the inclination angle in the pitch direction may be calculated according to the ratio of the distance Y (distance Z) to the distance X.

The tilted face and the depressed face can be determined by the position of the eyes and the position of the nose with respect to the face frame 30. Specifically, if the pupil position and the nose position are above the face frame 30, it can be determined to be a tilted face, and if it is below, it can be determined to be a depressed face.

The relationship between the change in the distance X and the inclination angle in the pitch direction can be stored in advance in the memory unit 10 of the image pickup apparatus 1 in the form of a mathematical formula or a look-up table.

It is possible to calculate the tilt angle of the face not only in the pitch direction but also in the yaw direction and the roll direction by performing the same processing.
For example, in the case of inclination in the yaw direction, the distance between the nose position and the pupil position (or ear position) in the left-right direction is calculated. Then, the face average data is selected according to the attribute information of the subject. Further, based on the selected face average data, one graph showing the relationship between the horizontal distance between the nose position and the pupil position (or the ear position) and the tilt angle is selected to calculate the tilt angle in the yaw direction.

In this way, the inclination in the yaw direction can be calculated by using the distance information that changes depending on the degree of inclination in the yaw direction. The inclination in the roll direction can also be calculated by using the distance information that changes depending on the inclination in the roll direction.

The flow of processing when calculating the inclination of the face in the pitch direction based on the position of the pupil and the position of the nose of the subject will be described with reference to FIG.

The control unit 9 performs face detection processing in step S301. When calculating the inclination of the face to execute the process of step S209 of FIG. 5, the process of step S301 may be omitted because the face detection process has already been performed in step S101 of FIG. ..

In step S302, the control unit 9 calculates a distance Y (or distance Z) in the vertical direction from the pupil position and the nose position of the subject.

Next, in step S303, the control unit 9 determines whether the face is a tilted face or a depressed face according to the pupil position and the nose position with respect to the face frame 30.

Finally, in step S304, the control unit 9 calculates the tilt angle of the face using a mathematical formula, a look-up table, or the like.

After calculating the tilt angle in the pitch direction in step S304, the processing after step S209 in FIG. 5 is performed, and finally step S106 in FIG. 4 is executed to focus on the pupil position. That is, it is possible to prevent the chin from being in focus when the face is tilted and the head from being focused when the face is depressed.

<3-2. Second example of corresponding processing>
In the second example of the corresponding processing, processing other than focusing control is performed by using the tilt information of the subject's face in the pitch direction, the yaw direction, and the roll direction.
This process may be executed by the signal processing unit 4 or the control unit 9 of the image pickup apparatus 1. Alternatively, the tilt information of the face may be output from the output unit 7 and used in an information processing device provided outside the image pickup device 1 as an image processing device.

For example, it is conceivable to use the tilt information of the subject's face in the captured image for white balance adjustment. Specifically, when the subject of the captured image is a face down, the face may be adjusted to a bluish color by performing normal white balance adjustment. In such a case, by strengthening orange and red, which have a complementary color relationship with blue, it is possible to perform white balance adjustment that reproduces a natural complexion.

Further, as an example of using the tilt information of the face in the information processing device external to the image pickup device 1, post production can be mentioned.
What the person is looking at by automatically adjusting the blowout position of the person (subject) based on the captured image and the tilt information of the face output as metadata, or by moving the image in the direction of the person's line of sight. (See FIGS. 16 and 17) and the like can be considered.
The metadata output together with the captured image may include coordinate information of the pupil position and coordinate information of the nose position in addition to the tilt information. Further, for example, flag information for identifying a tilted face or a depressed face may be output as rougher information than the angle information.
Of course, not all of this information needs to be output as metadata, and some information may be output as metadata.

<4. Summary>
As described in each of the above examples, the image processing device (for example, the image pickup device 1) in the present technology corresponds to the inclination calculation unit 24 that calculates the inclination of the subject's face in the pitch direction by using the calculated inclination. It is provided with a corresponding processing unit (focusing control unit 26, output control unit 27) for performing processing.
By calculating the inclination of the subject's face and using it for various processes, it is possible to perform optimum focusing control and white balance adjustment.

As described with reference to FIGS. 3, 11 to 15, a site specifying unit 21 for detecting the position of the nose and the position of the pupil in the subject is provided, and the inclination calculation unit 24 includes the detected position of the nose and the pupil. The angle of inclination of the face in the pitch direction may be calculated based on the position of.
As a result, the angle of the face in the pitch direction can be calculated with high accuracy.
Therefore, focusing control and the like can be performed with higher accuracy. For example, when the hair color of the subject is black, if the head is in focus and the white balance is adjusted based on the area where the head is reflected, the entire image becomes bright. In addition, as a result of adjusting to prevent the face from becoming too bright in the case of a tilted face under outdoor shooting conditions, the entire image may become too dark. In such a case, by calculating the face angle with high accuracy, it is possible to perform appropriate focusing control and white balance adjustment.
Further, since the inclination information is calculated with high accuracy, it is possible to easily create an effective effect in the post-production described in the second example of the corresponding processing.

As described in the first example (FIG. 15) of the corresponding processing, the inclination calculation unit 24 may use the face average data which is the average distance information about the face in the calculation of the angle.
As a result, the angle of the face in the pitch direction can be calculated with higher accuracy.
Therefore, it is possible to perform focusing control and the like with higher accuracy and to create an effective effect in post-production.

As described in the first example of the corresponding processing (FIGS. 3 and 5), an average data selection unit 23 for selecting one from the face average data provided for at least race or age is provided, and a slope calculation unit is provided. 24 may calculate the angle based on one face average data selected by the average data selection unit 23.
As a result, the angle of the face is calculated with high accuracy based on the face average data according to the gender, age, race, and the like.
Therefore, it is possible to perform focusing control and the like with higher accuracy and to create a more effective effect in post-production.

As described in the first example of the corresponding processing (FIGS. 5, 7, 8), the average data selection unit 23 is the distance information for each of the plurality of ranging areas (target areas 31) provided on the image. The size of the subject's face in the depth direction is calculated from the distribution of the distance information of the subject's face obtained as a result of the clustering, and the face average data is selected based on the calculated size of the face in the depth direction. You may go.
As a result, the face average data is selected based on the fact that the child's face is small and the adult's face is large.
Therefore, regardless of whether the subject is a child or an adult, focusing control, post-production processing, and the like can be performed with high accuracy.

As described in the first example of the corresponding processing, the inclination calculation unit 24 may calculate the inclination in the yaw direction and the inclination in the roll direction with respect to the face of the subject by using the face average data. This makes it possible to calculate the angle of inclination of the face in all directions.
Therefore, for example, when performing focusing control, the distance to the pupil can be appropriately calculated, and highly accurate focusing control can be performed.

As described in the first example of the corresponding processing (FIGS. 15, FIG. 4, FIG. 5, etc.), when the subject's face is determined to be a tilted face facing upward based on the inclination calculated by the inclination calculation unit 24. In addition, the corresponding processing unit (focusing control unit 26) may perform focusing control at a position other than the chin on the subject as the corresponding processing.
This prevents the chin from being in focus.
Therefore, appropriate focusing control can be performed.

As described in the first example of the corresponding processing (FIGS. 15, FIG. 4, FIG. 5, etc.), when the subject's face is determined to be a downward-facing face based on the inclination calculated by the inclination calculation unit 24. In addition, the corresponding processing unit (focusing control unit 26) may perform focusing control at a position other than the head on the subject as the corresponding processing.
This prevents the head from being in focus.
Therefore, appropriate focusing control can be performed.

As described in the first example of the corresponding processing (FIGS. 4 to 15), the corresponding processing unit (focusing control unit 26) may perform focusing control at the position of the pupil of the subject as the corresponding processing. ..
For example, when the depth of field is extremely shallow, such as in the high-resolution image pickup apparatus 1, the pupil may become unclear.
Even in such a case, the distance to the pupil can be appropriately calculated and appropriate focusing control can be executed.

As described in the second example of the corresponding processing (FIGS. 16 and 17), the corresponding processing unit (output control unit 27) may perform a process of outputting tilt information as the corresponding process.
As a result, for example, the external device can perform processing using the tilt information of the face.
For example, it is possible to perform a process of arranging the image in front of the line of sight according to the direction of the face. Such processing is performed, for example, in post production.

As described with reference to FIG. 3 and the like, when the position of the pupil can be detected by the site identification unit 21 but the amount of defocus for the position of the pupil cannot be calculated, the corresponding processing unit (focus control unit 26). ) May perform focusing control using the defocus amount estimated for the position of the pupil as the corresponding process.
For example, when a front object such as a branch is located between the pupil and the image pickup apparatus 1, the defocus amount may not be calculated appropriately.
In such a case, appropriate focusing control can be performed by appropriately estimating the amount of defocus according to the distance to the pupil.

The image processing method of the present technology calculates the inclination of the face of the subject in the pitch direction, and performs the corresponding processing using the calculated inclination.

The imaging device 1 of the present technology includes an imaging unit 3 that images a subject, an inclination calculation unit 24 that calculates the inclination of the subject's face in the pitch direction, and a control unit 9 that performs focusing control according to the calculated inclination. , Is provided.

A program for realizing such an image processing device or an image pickup device 1 may be recorded in advance in an HDD as a recording medium built in a device such as the image pickup device 1 or a ROM in a microcomputer having a CPU. can.
Alternatively, flexible discs, CD-ROMs (Compact Disc Read Only Memory), MO (Magneto optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Discs (registered trademarks)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card. Such a removable recording medium can be provided as so-called package software.
In addition to installing such a program from a removable recording medium on a personal computer or the like, it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

Further, according to such a program, it is suitable for a wide range of provision of the image processing device (imaging device 1 and the like) of the embodiment. For example, by downloading a program to a mobile terminal device such as a smartphone or tablet equipped with a camera function, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), etc., these devices are disclosed. Can function as the image pickup device 1 of the above.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

<5. This technology>
The present technology can also adopt the following configurations.
(1)
A tilt calculation unit that calculates the tilt of the subject's face in the pitch direction,
An image processing device including a corresponding processing unit that performs corresponding processing using the calculated inclination.
(2)
A part specifying part for detecting the position of the nose and the position of the pupil in the subject is provided.
The image processing device according to (1) above, wherein the tilt calculation unit calculates the angle of the tilt in the pitch direction with respect to the face based on the detected position of the nose and the position of the pupil.
(3)
The image processing apparatus according to (2) above, wherein the inclination calculation unit uses face average data obtained as average distance information about a face in calculating the angle.
(4)
It is provided with an average data selection unit that selects one from the face average data provided at least for each race or age.
The image processing apparatus according to (3) above, wherein the inclination calculation unit calculates the angle based on one face average data selected by the average data selection unit.
(5)
The average data selection unit clusters distance information for each of a plurality of ranging areas provided on the image, and the depth of the subject's face is obtained from the distribution of the distance information of the subject's face obtained as a result of the clustering. The image processing apparatus according to (4) above, which calculates the size in the direction and selects the face average data based on the calculated size in the depth direction of the face.
(6)
The image processing apparatus according to (3) above, wherein the inclination calculation unit calculates the inclination in the yaw direction and the inclination in the roll direction with respect to the face of the subject by using the face average data.
(7)
When the face of the subject is determined to be an upwardly tilted face based on the inclination calculated by the inclination calculation unit,
The image processing apparatus according to any one of (1) to (6) above, wherein the corresponding processing unit performs focusing control at a position other than the chin on the subject as the corresponding processing.
(8)
When the face of the subject is determined to be a downward-facing face based on the inclination calculated by the inclination calculation unit,
The image processing apparatus according to any one of (1) to (7) above, wherein the corresponding processing unit performs focusing control at a position other than the head of the subject as the corresponding processing.
(9)
The image processing apparatus according to any one of (1) to (8) above, wherein the corresponding processing unit controls focusing at the position of the pupil of the subject as the corresponding processing.
(10)
The image processing apparatus according to any one of (1) to (9) above, wherein the corresponding processing unit performs a process of outputting information on the inclination as the corresponding process.
(11)
When the pupil position can be detected by the site identification unit but the defocus amount for the pupil position cannot be calculated, the corresponding processing unit estimates the defocus amount for the pupil position as the corresponding process. The image processing apparatus according to any one of (2) to (6) above, which controls focusing using the above.
(12)
Calculate the inclination of the subject's face in the pitch direction,
An image processing method that performs corresponding processing using the calculated inclination.
(13)
An imaging unit that captures the subject and
A tilt calculation unit that calculates the tilt of the subject's face in the pitch direction,
An imaging device including a control unit that performs focusing control according to the calculated inclination.

1 Imaging device (image processing device)
3 Imaging unit 21 Site identification unit 23 Average data selection unit 24 Tilt calculation unit 26 Focus control unit (corresponding processing unit)
27 Output control unit (corresponding processing unit)

Claims (13)

1. A tilt calculation unit that calculates the tilt of the subject's face in the pitch direction,
   An image processing device including a corresponding processing unit that performs corresponding processing using the calculated inclination.
2. A part specifying part for detecting the position of the nose and the position of the pupil in the subject is provided.
   The image processing device according to claim 1, wherein the tilt calculation unit calculates the angle of the tilt in the pitch direction with respect to the face based on the detected position of the nose and the position of the pupil.
3. The image processing apparatus according to claim 2, wherein the inclination calculation unit uses face average data as average distance information about a face in calculating the angle.
4. It is provided with an average data selection unit that selects one from the face average data provided at least for each race or age.
   The image processing apparatus according to claim 3, wherein the inclination calculation unit calculates the angle based on one face average data selected by the average data selection unit.
5. The average data selection unit clusters distance information for each of a plurality of ranging areas provided on the image, and the depth of the subject's face is obtained from the distribution of the distance information of the subject's face obtained as a result of the clustering. The image processing apparatus according to claim 4, wherein the size in the direction is calculated, and the face average data is selected based on the calculated size in the depth direction of the face.
6. The image processing apparatus according to claim 3, wherein the inclination calculation unit calculates an inclination in the yaw direction and an inclination in the roll direction with respect to the face of the subject by using the face average data.
7. When the face of the subject is determined to be an upwardly tilted face based on the inclination calculated by the inclination calculation unit,
   The image processing apparatus according to claim 1, wherein the corresponding processing unit performs focusing control at a position other than the chin on the subject as the corresponding processing.
8. When the face of the subject is determined to be a downward-facing face based on the inclination calculated by the inclination calculation unit,
   The image processing apparatus according to claim 1, wherein the corresponding processing unit performs focusing control at a position other than the head of the subject as the corresponding process.
9. The image processing apparatus according to claim 1, wherein the corresponding processing unit controls focusing at the position of the pupil of the subject as the corresponding process.
10. The image processing apparatus according to claim 1, wherein the corresponding processing unit performs a process of outputting the tilt information as the corresponding process.
11. When the pupil position can be detected by the site identification unit but the defocus amount for the pupil position cannot be calculated, the corresponding processing unit estimates the defocus amount for the pupil position as the corresponding process. The image processing apparatus according to claim 2, wherein focusing control is performed using the image processing apparatus according to claim 2.
12. Calculate the inclination of the subject's face in the pitch direction,
    An image processing method that performs corresponding processing using the calculated inclination.
13. An imaging unit that captures the subject and
    A tilt calculation unit that calculates the tilt of the subject's face in the pitch direction,
    An imaging device including a control unit that performs focusing control according to the calculated inclination.

Images