CN108289170B

CN108289170B - Photographing apparatus, method and computer readable medium capable of detecting measurement area

Info

Publication number: CN108289170B
Application number: CN201810030737.6A
Authority: CN
Inventors: 钟亮洪; 黄源浩; 许星
Original assignee: Shenzhen Orbbec Co Ltd
Current assignee: Shenzhen Orbbec Co Ltd
Priority date: 2018-01-12
Filing date: 2018-01-12
Publication date: 2020-02-14
Anticipated expiration: 2038-01-12
Also published as: CN108289170A

Abstract

The invention provides a photographing device capable of detecting a metering area, which comprises: a processor and a memory, the memory including instructions executable by the processor, the instructions comprising operations of: collecting an image containing depth information; the depth information is analyzed to determine a metrology area. The invention further provides a photographing method capable of detecting the metering area. According to the photographing device and method capable of detecting the metering area, the metering area is automatically determined by using the depth information, so that the face area can be automatically identified when the backlight light intensity is not appropriate or only part of faces exist in the camera view field, the definition, the exposure and the color saturation of the photographed picture can meet the general requirements of a user, and the photographing experience of the user is greatly improved.

Description

Photographing apparatus, method and computer readable medium capable of detecting measurement area

Technical Field

The invention belongs to the field of camera equipment, and particularly relates to a photographing device and method capable of detecting a metering area.

Background

The self-shooting is one of the more common activities in people's life, a camera is equipped on a mobile phone, a computer, a tablet and other computing equipment, the shooting is controlled by configuring software and hardware for controlling the camera in the equipment so as to complete the self-shooting, and most of the equipment is also provided with a display screen for displaying the state of a shot object in real time. The quality of the picture depends on a number of factors, such as the focus, exposure, etc. of the camera, and an experienced photographer may adjust the parameters of these factors to take a high quality picture, but not all users have this capability.

In this regard, more and more cameras or related photographing devices have automatic adjustment capabilities, such as Auto Focus (AF) technology, Auto Exposure (AE) technology, and the like. With the development of the face detection technology, the automatic face detection technology can also realize full-automatic rapid face automatic focusing and automatic exposure so as to realize high-quality self-shooting.

However, under some conditions, these techniques are difficult to apply. For example, when the backlight intensity is too strong, the intensity of the face region is too weak, so that the face cannot be detected; in addition, the human face cannot be automatically detected even when only a part of the human face exists in the field of view of the camera or a part of the human face area is blocked. These conditions degrade the user's photographing experience.

Disclosure of Invention

The invention aims to solve the problem that in the prior art, the photographing experience of a user is poor when the backlight light intensity is not appropriate or only partial faces exist in the field of view of a camera, and provides a photographing device and method capable of detecting a metering area.

The photographing apparatus capable of detecting a measurement area of the present invention includes: a processor and a memory, the memory including instructions executable by the processor, the instructions comprising operations of: collecting an image containing depth information; the depth information is analyzed to determine a metrology area.

In a preferred embodiment, the instructions further comprise the following operations: and calculating a metering value of the metering area, comparing the metering value with a preset threshold value, and adjusting the parameters of the camera based on the comparison result. In a more preferred embodiment, the metrology values comprise pixel values reflecting light intensity, and the parameters comprise one or more of shutter speed, aperture size, sensitivity; or in a more preferred embodiment, the metric comprises a distance and the parameter comprises a focal length of a lens of the camera.

In a preferred embodiment, the image containing depth information includes a depth image and a two-dimensional image, and the two-dimensional image includes one or more of a color image and a grayscale image.

In a preferred embodiment, said analyzing said depth information to determine a gauge region further comprises the sub-steps of: obtaining a closest point in the depth information; and determining the metering area according to the closest point. Or in a preferred embodiment, said analyzing said depth information to determine a gauge region further comprises the sub-steps of: analyzing the depth information to determine a target profile; and calculating a metering area according to the target profile.

The invention also proposes a computer-readable medium containing a computer program operable to cause a computer to perform the following operations: acquiring an image containing depth information by using a camera; analyzing the depth information to determine a metering region; calculating a metrology value for the metrology area to adjust a parameter of the camera; and acquiring an image according to the adjusted parameters.

The invention also provides a photographing method capable of detecting the metering area, which comprises the following steps: acquiring an image containing depth information by using a camera; analyzing the depth information to determine a metering region; calculating a metrology value for the metrology area to adjust a parameter of the camera; and acquiring an image according to the adjusted parameters.

In a preferred embodiment, before the capturing an image containing depth information with a camera, the method further comprises the following steps: acquiring an image by using a camera; and analyzing the image to judge whether a target exists, and if no target exists, carrying out the photographing method capable of detecting the metering area.

Compared with the prior art, the invention has the beneficial effects that:

according to the photographing device and method capable of detecting the metering area, the metering area is automatically determined by using the depth information, and then the camera parameters are automatically adjusted, so that the face area can be automatically identified when the backlight light intensity is not appropriate or only part of faces exist in the camera view field, the definition, the exposure and the color saturation of the photographed picture can meet the general requirements of a user, and the photographing experience of the user is greatly improved.

Drawings

FIG. 1 is a diagram illustrating a photograph taken by a user in one embodiment of the invention.

Fig. 2 is a schematic structural diagram of a photographing apparatus capable of detecting a measurement area according to an embodiment of the present invention.

FIG. 3 is a flow chart of automatic adjustment of camera parameters according to an embodiment of the present invention.

FIG. 4 is a flow chart of determining a gauge region using a depth image in one embodiment of the invention.

FIG. 5 is a flow chart of metering region determination in one embodiment of the present invention.

FIG. 6 is a diagram of a photographing apparatus and a photographing method according to an embodiment of the invention.

Fig. 7 is a schematic diagram of an incomplete face measurement area according to an embodiment of the present invention.

FIG. 8 is a schematic diagram of a metering area with a portion of a face occluded in accordance with an embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the following detailed description and accompanying drawings. Wherein like reference numerals refer to like parts unless otherwise specified. It should be emphasized that the following description is merely exemplary in nature and is not intended to limit the scope of the invention or its application.

FIG. 1 is a diagram illustrating a photograph taken by a user according to one embodiment of the invention. The user 10 takes a picture of himself with the photographing apparatus 11 in his hand to acquire an image, and aligns the camera 111 of the photographing apparatus 11 with the face 101 of the user to capture a face image at the time of photographing. This mode of taking pictures is commonly referred to as self-timer shooting, and in some embodiments, the picture taking device 11 may also be controlled by other users or automatically take pictures of the users. The photographing device 11 is provided with a camera 111 for photographing, and the cameras can be distributed at different positions of the device, for example, the front and back surfaces of the device are both provided with photographing cameras; the camera 111 may be any suitable camera type, such as a color camera, a grayscale camera, a wide-angle camera, a telephoto camera, a light field camera, a depth camera, a dual-camera, etc., and the captured image may correspondingly include two-dimensional images, such as color (e.g., RGB images), grayscale, etc., and may also include three-dimensional images, such as depth images, etc.

When a person, especially a human face, is photographed, focusing on a human face area is often needed to realize clear imaging of the human face. Besides the focal length, the imaging quality is affected by a shutter, an aperture, sensitivity and the like, which are hereinafter collectively referred to as exposure parameters, and properly setting the exposure parameters will make the face region appear with reasonable brightness to avoid a case of a small exposure or an excessive exposure.

The present invention will provide a photographing apparatus capable of detecting a measurement area, which can automatically adjust camera parameters during photographing, and will be described in the following description taking a color camera and a photographed image as an RGB image as an example. It is understood that aspects of the present invention are applicable to other cameras as well.

Fig. 2 is a schematic structural diagram of a photographing device according to an embodiment of the present invention. The photographing apparatus includes a camera processor 20, a camera 21, a memory 24, and in some embodiments, a display 22, a depth camera 23.

The camera 21 includes a lens 210, an aperture 211, an image sensor 212, a shutter (not shown), a lens adjustment system 213, and the like. The lens 210 is used for focusing the target light on the image sensor 212, and the image sensor 212 receives the light and converts the optical signal into an electrical signal, which is processed by an image processor (not shown) to output an image of the target. During the photographing process, the lens adjustment system 213 is used to control the lens to be adjusted, and includes a mechanical drive, such as a shutter motor, a voice coil motor, etc., for implementing the adjustment.

The camera 21 is connected to the processor 20, and the processor 20 is configured to receive the image data transmitted by the camera 21, and the image data may be further transmitted to the display 22 for displaying. Meanwhile, the processor 20 may also be used to control the adjustment of the camera, such as sending a control signal to the lens adjustment system to adjust the focal length and the exposure parameters.

For a photographing apparatus with auto-adjustment capability, it contains the relevant programs for auto-adjustment, such as the auto-focus program 240, the auto-exposure program 241, etc., which are stored in the memory 24, and when the corresponding auto-adjustment function is activated, the processor 20 will call the corresponding auto-adjustment program from the memory 24 and execute it. The memory 24 may be any type of storage medium such as ROM, RAM, FLASH, etc. The memory 24 may also include a gauge region detection program 242, an RGBD registration program 243, and may also include temporary data generated during program execution, and the like.

Processor 20 may be comprised of a single or multiple processor units. The display 22 may also include touch functionality to enable interaction with a user.

The depth camera 23 is used for acquiring depth images, and in the present invention will be a depth camera illuminated with active light, such as an infrared structured light depth camera, a TOF depth camera, or the like. In capturing the depth image, active light illumination, such as flood light, structured light illumination, or the like, is first provided using an active light illuminator. The depth camera 23 and the camera 21 may be an integrated single camera or may be separate cameras. When they are separate cameras, there is parallax due to the difference in position in space, and in some applications, it is necessary to register the images of the two. In one embodiment, the camera 21 is an RGB camera and the registration procedure is an RGBD registration procedure 243. After the camera 21 and the depth camera 23 acquire the RGB image and the depth image, the RGBD registration program 243 is executed by the processor 20 to achieve the alignment of the RGB image and the depth image. When an integrally integrated camera is used, it is understood that the camera is used to acquire an image containing depth information, such as an RGBD image or the like.

Fig. 3 is a flow diagram of automatic adjustment of camera parameters according to one embodiment of the invention. Taking the automatic exposure function as an example, after the program starts, step S301 is performed, the processor controls the camera to image the target and acquire a target image, and then step S302 is performed to determine a metering area in the image, where the metering area is an area in the image that needs to be clearly imaged, such as a human face area. The determination of the photometric metering area may also be determined by calling a metering area detection program during the execution of the automatic adjustment program of the present invention, for example, a metering area detection program based on human face detection, by which a human face area is automatically detected and the human face area is used as the photometric metering area. In some embodiments, the photometric metering area can also be determined manually by a user, such as by touch interaction to designate the photometric metering area. Then, in step S303, a measure value in the measure region is calculated, where the measure value may be a pixel value (an average value, a weighted average value, or the like) reflecting the light intensity in the region, or may be a value reflecting the image definition or brightness defined in other manners. Next, step S304 is performed, i.e. adjusting the camera parameters according to the metric value, and more specifically, in one embodiment, comparing the metric value with a preset threshold value, for example, in case the metric value is an average pixel value, setting the threshold value to 18% of the maximum value of the pixel value, i.e. in case of an 8-bit pixel format, the pixel value is between 0-255, and then selecting the threshold value to be 46. When the measurement value is less than 46, the exposure parameters are adjusted, such as reducing the shutter speed, increasing the aperture value, increasing the sensitivity (ISO) of the image sensor, and the like, and vice versa. Finally, the processor adjusts the camera 21 through the lens adjustment system 213 according to the adjusted parameters and captures an image based on the adjusted parameters (305). It should be noted that, in step S301, the target image captured by the processor controlled camera is a preview image, and the display effect of the general image is not good, and in step S305, the image is captured based on the adjusted parameters, and the exposure and the definition have been adjusted to a better state, so as to generally meet the photographing requirement of the user.

For the auto-focus procedure, the principle is similar to that of the auto-exposure procedure, except that the measurement area may be a surface area or a single point, the measurement value of the measurement area is the distance information of the focus area/point, and the adjusted camera parameter is the focal length of the lens.

The core link in the automatic adjustment program is the determination of the metering area, and the metering area detection mode based on the traditional face recognition has defects, namely when the backlight is too strong or the face area is not complete, the face area is difficult to detect, so that the automatic adjustment cannot be realized.

FIG. 4 is a flow diagram for determining a metrology area using a depth image in accordance with one embodiment of the present invention. The program of instructions reflected in the flowchart may be part of an auto-adjustment program, such as auto-focus program 240, auto-exposure program 241, or may be part of another program, such as metrology area detection program 242 or metrology area detection program 242. After the program starts, step S401 is performed, the processor calls the depth camera to acquire a depth image, step S402 is performed, the depth image is analyzed, whether a face exists is judged based on the analysis result, and when the face exists, step S403 is performed, namely, a metering area is determined according to the face; and when the human face is not detected, performing step S404 to obtain the closest point in the depth image, and then performing step S405 to determine a metering area according to the area where the closest point is located. In step S402, the method for detecting a human face by using a depth image may adopt any suitable method, such as a template matching method, feature extraction, and the like. Because the acquisition of the depth image is usually based on active light illumination, such as infrared light, the backlight influence on the face detection based on the depth image is small, and the problem that the face detection fails due to the fact that the backlight intensity is not appropriate in the traditional method is solved.

In some embodiments, when only a portion of the face is in the depth image, face detection algorithms based on template matching, feature extraction, etc., will have difficulty detecting the face region. Considering that the face is the closest region to the camera when taking a picture, the face region may be determined in one embodiment by searching for the closest point in the depth image. In step S404, first obtaining a closest point in the depth image, where the closest point may refer to a region composed of multiple pixels instead of a single pixel point, and then determining a metering region according to the closest point in step S405, for example, growing a face region from the closest point by using a region growing method; in some embodiments, each part of the image can be segmented by an image segmentation algorithm, and the part containing the closest point is a face region; in some embodiments, the face region may be segmented using geodesic distance calculations based on the closest points.

In the embodiment shown in fig. 4, after the metering region is acquired by using the depth image in step S403 or step S405, the metering region may be applied to step S302 in the automatic adjustment flow in the embodiment shown in fig. 3. When the depth camera 23 and the camera 21 are independent cameras, since there is a parallax between them, this step also needs to invoke the RGBD registration program 243 to align the depth image with the RGB image, so that the metering region in the RGB image can be acquired according to the metering region in the depth image. In some embodiments, the depth camera 23 is integrated with the camera 21, whereby a depth image without parallax and an RGB image can be acquired at the same time, in other words, an RGB image containing depth information is acquired.

It is understood that, in some embodiments, steps S402 and S403 may also be omitted, that is, after the depth image is acquired in step S401, steps S404 and S405 are directly performed, that is, the closest point in the depth image is acquired, and the metering region is acquired according to the closest point, so that the metering region can be quickly located, and thus the complicated and time-consuming processes of image analysis and face detection are skipped.

In some embodiments, the metrology area may also be determined based on the RGB image in combination with the depth image. FIG. 5 is a flow diagram of metering region determination, according to one embodiment of the present invention. After the program starts, firstly, step S501 is performed, that is, an RGB image is collected, then step S502 is performed, the RGB image is analyzed to determine whether a face exists, when a face exists, step S503 is performed, that is, a metering area is determined according to the face, if a face is not detected (that is, when conditions such as too strong backlight and incomplete face occur), a metering area determination flow based on a depth image as shown in fig. 4 is adopted, in the embodiment shown in fig. 5, step S504 and step S505 are specifically included, that is, a depth image in a field of view of a depth camera is collected and analyzed, whether a face exists is determined based on an analysis result, and when a face exists, step S506 is performed, that is, a metering area is determined according to the face; and when the human face is not detected, performing step S507 to obtain the closest point in the depth image, and then performing step S508 to determine the metering area according to the area where the closest point is located. It will also be appreciated that, as described above, in some embodiments, steps S505 and S506 may be eliminated, that is, after the depth image is acquired in step S504, steps S507 and S508 are performed directly, that is, the closest point in the depth image is acquired, and the metering region is acquired according to the closest point, so that the metering region is located quickly, and thus the complicated and time-consuming processes of image analysis and face detection are skipped.

Fig. 6 is a schematic diagram of a photographing apparatus and photographing according to an embodiment of the invention. The apparatus 60 includes a front camera 602 for capturing RGB images, a structured light depth camera (including an infrared structured light projection module 604, an infrared camera 601) for capturing depth images, a display 605, a processor (not shown), and other sensors 603. When the user uses the device 60 to perform self-timer shooting, the processor turns on the front camera 602 to acquire the RGB image of the current face and displays the face image 61 on the display 605 in real time, and then the processor will call and execute an automatic adjustment program (such as an auto-focusing program, an auto-exposure program, etc.) to achieve auto-focusing and exposure. The specific method can be seen in the embodiments described above in conjunction with fig. 3 to 5.

In the embodiments shown in fig. 4 and 5, an algorithm for determining the measurement region based on the human face is involved in each of step S403, step S405, step S506, and step S508. In general, the conventional two-dimensional face detection algorithm outputs a square region including a face. Considering that the intensity of the edge of the face is obviously changed relative to the intensity of the middle area in some abnormal exposure situations, if the measurement area is set to be too large and includes the edge area of the face, the measurement value will not be accurate in calculating the measurement value (the measurement value needs to accurately reflect the intensity of the face area, and if the measurement value includes the edge or the outer area of the face, the measurement value will not accurately reflect the intensity of the face area), therefore, the square area is generally set to be smaller than the face area and only includes the area of the main features of the face, such as the eyes, nose, mouth, etc., such as the measurement area 62 in fig. 6.

When the measurement area is smaller than the face, the situation that the measurement value cannot completely reflect the whole face intensity occurs. In the present invention, when a face is detected using a depth image, for example, in the above-described face detection method based on the closest point, an edge contour of a face (for example, an image segmentation method, a region growing method, or the like) may be detected, and a near-elliptical region including a main part of the face (for example, a geodesic distance method, or the like) may also be detected as the face contour 63 shown in fig. 6. Subsequent computation of the metric value for this metric region will accurately reflect the face intensity value (the intensity value reflects the sharpness, exposure, etc. of the image). In some embodiments, the measurement area may also be reduced to some extent, for example, the outline area is reduced by several pixel values, so as to avoid the problem of significant intensity variation in the edge area. Here, the near-elliptical region may be detected based on a geodesic distance method or the like, and the detection may be performed based on the closest point specified in the depth image, or another method may be employed, such as detecting the near-elliptical region including the main part of the human face by analyzing dot matrix information in the depth image.

In some embodiments, the front camera 602 may also include multiple cameras, such as dual cameras. At this time, the depth image can be acquired by the depth camera and then RGBD registration can be performed with the images of the two cameras, so that the metering area acquired by the depth image can be applied to the two cameras for further automatic focusing, automatic exposure and the like.

Fig. 7 is a schematic view of an incomplete face measurement area according to an embodiment of the present invention. FIG. 8 is a schematic diagram of an occluded face measurement area according to an embodiment of the present invention. Fig. 7 and 8 are different from the embodiment shown in fig. 6 in that the current face partially appears in the camera field of view or is blocked, and the captured image only includes a part of the face, and the face detection algorithm in the prior art is difficult to identify the face position due to incomplete face information. In the present invention, the depth image is used to detect the measurement region, for example, based on the above-mentioned face detection method based on the closest point, that is, the contour 70 of the face region is obtained in fig. 7, the contour 80 of the face region is obtained in fig. 8, and the measurement region is further determined based on the contour.

It is understood that the method or apparatus shown in the above embodiments can be used for photographing other objects besides face photographing, such as whole body photographing of human body, animal photographing, etc. The photographed objects such as a human face contour, a whole body contour, an animal contour, etc. may be collectively referred to as a target contour, and when the target contour partially appears in the camera field of view or is occluded, the target contour may be determined by analyzing the depth image by the metering region detecting apparatus of the present invention, and then the metering region may be calculated based on the target contour. When the target contour is determined by analyzing the depth image, the target contour may be determined by detecting a region including a main part of the subject by analyzing the dot matrix information in the depth image based on the closest point in the depth information, such as an image segmentation method or a region growing method.

It should be noted that, according to the implementation requirement, each step/component described in the present invention can be divided into more steps/components, or two or more steps/components or partial operations of the steps/components can be combined into new steps/components to achieve the purpose of the present invention, and all of the steps/components or the steps/components can be simply modified or changed, and fall into the protection scope of the present invention.

The above-described flow method of the present invention for acquiring a metering area by analyzing a depth image may be implemented in hardware, firmware, or as software or computer code storable in a recording medium such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk, or as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware such as an ASIC or FPGA. It will be appreciated that the computer, processor, microprocessor controller or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the processing methods described herein. Further, when a general-purpose computer accesses code for implementing the processes shown herein, execution of the code transforms the general-purpose computer into a special-purpose computer for performing the processes shown herein.

The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.

Claims

1. A photographing device capable of detecting a measurement area, comprising:

a processor and a memory, the memory including instructions executable by the processor, the instructions comprising the following operations;

when a user is photographed, an image containing depth information is collected;

analyzing the depth information to determine a metering area, and determining a face area by searching a closest point in a depth image, wherein the closest point in the depth information is obtained, and the metering area is determined according to the closest point;

and calculating a metering value of the metering area and adjusting camera parameters according to the metering value.

2. The photographing apparatus capable of detecting a metering area according to claim 1, wherein the instructions further comprise operations of:

and calculating a metering value of the metering area, comparing the metering value with a preset threshold value, and adjusting the parameters of the camera based on the comparison result.

3. The photographing apparatus capable of detecting a measurement region according to claim 2, wherein the measurement value includes a pixel value reflecting light intensity, and the parameter includes one or more of shutter speed, aperture size, and sensitivity.

4. The photographing apparatus capable of detecting a measurement region according to claim 2, wherein the measurement value includes a distance, and the parameter includes a focal length of a lens of the camera.

5. The photographing apparatus capable of detecting a measurement region according to claim 1, wherein the image including the depth information includes a depth image and a two-dimensional image, and the two-dimensional image includes one or more of a color image and a grayscale image.

6. The photographing apparatus capable of detecting a measurement region according to any one of claims 1 to 5, wherein the analyzing the depth information to determine the measurement region further comprises the steps of:

analyzing the depth information to determine a target profile;

and calculating a metering area according to the target profile.

7. A computer readable medium embodying a computer program operable to cause a computer to:

acquiring an image containing depth information when a user is photographed by using a camera; analyzing the depth information to determine a metering area, and determining a face area by searching a closest point in a depth image, wherein the closest point in the depth information is obtained, and the metering area is determined according to the closest point; calculating a metrology value for the metrology area to adjust a parameter of the camera;

and acquiring an image according to the adjusted parameters.

8. A photographing method capable of detecting a metering area is characterized by comprising the following steps:

acquiring an image containing depth information when a user is photographed by using a camera;

calculating a metrology value for the metrology area to adjust a parameter of the camera;

and acquiring an image according to the adjusted parameters.

9. The photographing method capable of detecting a metering area according to claim 8, further comprising the following steps before the capturing an image containing depth information with a camera:

acquiring an image by using a camera;

analyzing the image to determine whether a target is present, and if no target is present, performing the photographing method capable of detecting a metering area according to claim 8.