CN113673285B - Depth reconstruction method, system, equipment and medium during capturing of depth camera - Google Patents
Depth reconstruction method, system, equipment and medium during capturing of depth camera Download PDFInfo
- Publication number
- CN113673285B CN113673285B CN202010411183.1A CN202010411183A CN113673285B CN 113673285 B CN113673285 B CN 113673285B CN 202010411183 A CN202010411183 A CN 202010411183A CN 113673285 B CN113673285 B CN 113673285B
- Authority
- CN
- China
- Prior art keywords
- image
- face
- depth
- light spot
- rgb
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Abstract
The application provides a depth reconstruction method, a depth reconstruction system, depth reconstruction equipment and a depth reconstruction medium during snapshot of a depth camera, which comprise the following steps: acquiring an RGB image, an IR image and an infrared light spot image of a target face; performing face detection on the RGB image and the IR image, and preprocessing the infrared light spot image to extract a plurality of light spot areas in the infrared light spot image; when a face area is detected in an RGB image and an IR image, carrying out expression detection on the face area, and determining the expression type of the face area; and when the expression type is any one of the preset expression type sets, performing depth reconstruction on the target face according to the plurality of light spot areas of the infrared light spot image and the face area in the RGB image to generate a depth face image. According to the application, the infrared light spot image is preprocessed during face detection, and the infrared light spot image is deeply reconstructed only according to the face region in the RGB image after expression optimization, so that the efficiency of deep reconstruction is improved.
Description
Technical Field
The application relates to the field of 3D imaging, in particular to a depth reconstruction method, a depth reconstruction system, depth reconstruction equipment and a depth reconstruction medium during capturing by a depth camera.
Background
In recent years, with the development of consumer electronics industry, depth cameras with depth sensing function are receiving attention from consumer electronics industry. The current well-established depth measurement methods are structured light schemes and ToF techniques.
ToF (time of flight) is a 3D imaging technique that emits measurement light from a projector and reflects the measurement light back to a receiver through a target face so that the spatial distance of the object to the sensor can be obtained from the propagation time of the measurement light in this propagation path. Common ToF techniques include single point scanning projection methods and face light projection methods.
The structured light three scheme is based on the principle of optical triangulation. The optical projector projects a certain mode of structured light on the surface of the object, and a light bar three-dimensional image modulated by the surface shape of the measured object is formed on the surface. The three-dimensional image is detected by a camera at another location, thereby obtaining a two-dimensional distorted image of the light bar. The degree of distortion of the light bar depends on the relative position between the optical projector and the camera and the object surface profile (height). Intuitively, the displacement (or offset) displayed along the light bar is proportional to the object surface height, kinks represent changes in plane, and discontinuities represent physical gaps in the surface. When the relative position between the optical projector and the camera is fixed, the three-dimensional shape outline of the object surface can be reproduced by the distorted two-dimensional light bar image coordinates.
The depth camera module widens the dimension of front end perception, can well solve the problems of false body attack resistance and recognition accuracy reduction under extreme conditions encountered by 2D face recognition, has the effect of being accepted by the market, has strong demand, and can be applied to the scenes of door locks, access control, payment and the like based on 3D face recognition. However, when the method is applied to scenes such as payment, the snapshot efficiency of the face image is improved, the quick payment is realized, and no corresponding solution exists in the prior art.
Disclosure of Invention
Aiming at the defects in the prior art, the application aims to provide a depth reconstruction method, a system, equipment and a medium during capturing by a depth camera.
The depth reconstruction method for depth reconstruction in the process of capturing by the depth camera provided by the application comprises the following steps:
step S1: acquiring an RGB image, an IR image and an infrared light spot image of a target face;
step S2: performing face detection on the RGB image and the IR image, and preprocessing the infrared light spot image at the same time to extract a plurality of light spot areas in the infrared light spot image;
step S3: when the face area is detected in the RGB image and the IR image, carrying out expression detection on the face area, and determining the expression type of the face area;
step S4: and when the expression type is any expression type in a preset expression type set, performing depth reconstruction on the target face according to a plurality of light spot areas of the infrared light spot image and the face area in the RGB image to generate a depth face image.
Preferably, when the target face is subjected to depth reconstruction to generate a depth face image, the following steps are simultaneously performed:
-quality detecting said RGB image, said IR image;
-performing a live detection of the IR image when the IR image meets a preset quality criterion.
Preferably, the step S1 includes the steps of:
step S101: alternately acquiring an IR image and an infrared spot image of the target face through an infrared camera;
step S102: collecting RGB images of the target face through an RGB camera;
step S103: and acquiring an RGB image, an IR image and an infrared light spot image of the target face, and previewing the RGB image in real time.
Preferably, the step S3 includes the steps of:
step S301: triggering step S303 when the face area is detected in the RGB image and the IR image;
step S302: when no face area is detected in the RGB image and the IR image, returning to the step S1 to acquire the RGB image, the IR image and the infrared light spot image of the target face again;
step S303: and carrying out expression detection on the face area, and determining the expression type of the face area.
Preferably, the step S4 includes the steps of:
step S401: acquiring a preset expression type set;
step S402: judging whether the expression type is an expression type in a preset expression type set, triggering step S403 when the expression type is an expression type in the preset expression type set, sending out first prompt information when the expression type is not an expression type in the preset expression type set, and returning to step S1;
step S403: and carrying out depth reconstruction on the target face according to the plurality of light spot areas of the infrared light spot image and the RGB image to generate a depth face image.
Preferably, when quality detection is performed on the RGB image and the IR image, the method includes the steps of:
step M1: acquiring a preset image quality standard;
step M2: judging whether the RGB image and the IR image accord with preset image quality standards, triggering step M3 when the RGB image and the IR image accord with preset image quality standards, sending out second prompt information when the RGB image and the IR image do not accord with preset image quality standards, and returning to step S1;
step M3: and performing living body detection on the target face according to the IR image.
Preferably, the step S403 includes the steps of:
step S4031: mapping the face area of the RGB image into the infrared light spot image, and determining a plurality of light spots of the face area in the infrared light spot image;
step S4032: extracting a plurality of light spot areas of the face area;
step S4033: and generating a depth face image of the face region according to the deformation or displacement of the spot area corresponding to the face region.
The depth reconstruction system for capturing the depth camera provided by the application comprises the following modules:
the image acquisition module is used for acquiring an RGB image, an IR image and an infrared light spot image of the target face;
the face detection module is used for carrying out face detection on the RGB image and the IR image;
the depth data preprocessing module is used for preprocessing the infrared light spot image while detecting the human face so as to extract a plurality of light spot areas in the infrared light spot image;
the expression detection module is used for carrying out expression detection on the human face region when the RGB image and the IR image detect the human face region, and determining the expression type of the human face region;
and the depth reconstruction module is used for performing depth reconstruction on the target face according to the infrared light spot image and the RGB image to generate a depth face image when the expression type is any expression type in a preset expression type set.
The depth reconstruction device for capturing by the depth camera provided by the application comprises the following components:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the depth reconstruction method when the depth camera is snapped via execution of the executable instructions.
According to the computer readable storage medium provided by the application, the computer readable storage medium is used for storing a program, and the program is executed to realize the steps of the depth reconstruction method when the depth camera is used for capturing.
Compared with the prior art, the application has the following beneficial effects:
according to the application, the RGB images are subjected to face detection and expression optimization in sequence, the infrared light spot images are preprocessed during face detection, and the infrared light spot images are subjected to depth reconstruction only according to the face areas in the RGB images after the expression optimization, so that the efficiency of depth reconstruction is improved;
according to the application, the RGB image and the IR image are deeply reconstructed after the expression is optimized, so that failure caused by too poor image quality in the deep reconstruction is avoided, and the depth face image can be generated after the IR image is subjected to living detection, so that the snapshot process can be compactly executed, the time of the whole snapshot process is shortened, and the face brushing payment efficiency is improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art. Other features, objects and advantages of the present application will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
fig. 1 is a flow chart of steps of a depth reconstruction method during capturing by a depth camera according to an embodiment of the present application;
FIG. 2 is a flowchart showing steps for capturing RGB images, IR images, and IR spot images of a target face according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating steps of face detection according to an embodiment of the present application;
FIG. 4 is a flowchart illustrating a method for detecting a table condition according to an embodiment of the present application;
FIG. 5 is a flowchart illustrating steps for quality detection of RGB images and IR images according to an embodiment of the present application;
FIG. 6 is a flowchart illustrating steps for performing depth reconstruction on a target face according to an embodiment of the present application;
FIG. 7 is a schematic block diagram of a depth reconstruction system when capturing images with a depth camera according to an embodiment of the present application;
fig. 8 is a schematic structural diagram of a depth reconstruction device when a depth camera captures images in an embodiment of the present application; and
fig. 9 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application.
Detailed Description
The present application will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present application, but are not intended to limit the application in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present application.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme of the application is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
The application provides a depth reconstruction method in capturing by a depth camera, which aims to solve the problems in the prior art.
The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 1 is a step flowchart of a depth reconstruction method in capturing with a depth camera according to an embodiment of the present application, where, as shown in fig. 1, the depth reconstruction method in capturing with a depth camera provided in the present application includes the following steps:
step S1: acquiring an RGB image, an IR image and an infrared light spot image of a target face;
fig. 2 is a flowchart of steps for acquiring an RGB image, an IR image, and an infrared spot image of a target face in an embodiment of the present application, as shown in fig. 2, the step S1 includes the following steps:
step S101: alternately acquiring an IR image and an infrared spot image of the target face through an infrared camera;
step S102: collecting RGB images of the target face through an RGB camera;
step S103: and acquiring an RGB image, an IR image and an infrared light spot image of the target face, and previewing the RGB image in real time.
In the embodiment of the application, the RGB image, the IR image and the infrared light spot image are carried out by a depth camera;
the depth camera comprises a discrete light beam projector, a surface light source projector, an RGB camera and an infrared camera
The discrete light beam projector is used for projecting a plurality of discrete collimated light beams to a target face;
the surface light source projector is used for projecting floodlight to the target face body;
the infrared camera is used for receiving the scattered collimated light beam reflected by the target face, obtaining an infrared light spot image of the surface of the target face according to the scattered collimated light beam reflected by the target face, receiving floodlight reflected by the target face, and obtaining IR image data of the surface of the target face according to the floodlight reflected by the target face.
The RGB camera is used for collecting RGB images of the target face.
Step S2: performing face detection on the RGB image and the IR image, and preprocessing the infrared light spot image at the same time to extract a plurality of light spot areas in the infrared light spot image;
in the embodiment of the present application, the detected face area may be a pixel range of the face area, where the face area in the RGB image and the IR image is framed, and the face area is actually detected. The expression detection can be performed by an expression detection model based on a neural network.
Step S3: when the face area is detected in the RGB image and the IR image, carrying out expression detection on the face area, and determining the expression type of the face area;
fig. 3 is a flowchart of steps of face detection in the embodiment of the present application, as shown in fig. 3, the step S3 includes the following steps:
step S301: triggering step S303 when the face area is detected in the RGB image and the IR image;
step S302: when no face area is detected in the RGB image and the IR image, returning to the step S1 to acquire the RGB image, the IR image and the infrared light spot image of the target face again;
step S303: and carrying out expression detection on the face area, and determining the expression type of the face area.
In the embodiment of the application, the face area in the RGB image is determined by a face detection model based on a neural network.
Step S4: and when the expression type is any expression type in a preset expression type set, performing depth reconstruction on the target face according to a plurality of light spot areas of the infrared light spot image and the face area in the RGB image to generate a depth face image.
Fig. 4 is a flowchart of a step of table condition detection in the embodiment of the present application, as shown in fig. 4, the step S4 includes the following steps:
step S401: acquiring a preset expression type set;
step S402: judging whether the expression type is an expression type in a preset expression type set, triggering step S403 when the expression type is an expression type in the preset expression type set, sending out first prompt information when the expression type is not an expression type in the preset expression type set, and returning to step S1;
step S403: and carrying out depth reconstruction on the target face according to the infrared light spot image and the RGB image to generate a depth face image.
In an embodiment of the present application, the expression type set includes expression blurriness and smiling. The first prompt message may be head setting, keeping the expression flat, please look forward, etc.
In the embodiment of the application, when the target face is subjected to depth reconstruction to generate a depth face image, the following steps are simultaneously carried out:
-quality detecting said RGB image, said IR image;
-performing a live detection of the IR image when the IR image meets a preset quality criterion.
Fig. 5 is a flowchart illustrating steps for quality detection of RGB images and IR images according to an embodiment of the present application, where, as shown in fig. 5, the quality detection of RGB images and IR images includes the following steps:
step M1: acquiring a preset image quality standard;
step M2: judging whether the RGB image and the IR image accord with preset image quality standards, triggering step M3 when the RGB image and the IR image accord with preset image quality standards, sending out second prompt information when the RGB image and the IR image do not accord with preset image quality standards, and returning to step S1;
step M3: and performing living body detection on the target face according to the IR image.
In an embodiment of the present application, the image quality standard may be a contrast threshold, and the contrast threshold may be set to 150: and 1, when the contrast ratio of the RGB image and the IR image is larger than the contrast ratio threshold value, recognizing that the RGB image and the IR image accord with a preset image quality standard.
In the embodiment of the application, the image quality standard can also adopt a PSNR (Peak Signal to Noise Ratio ) threshold value; the PSNR threshold may be set to 30dB, and when the contrast of the RGB image and the IR image is greater than the PSNR threshold, the RGB image and the IR image are recognized to meet a preset image quality standard.
The second prompt information can improve the exposure time, reduce the exposure time, perform backlight compensation and the like.
In the embodiment of the application, when the IR image passes through living body detection, living body detection is carried out on the depth face image, and after the depth face image passes through living body detection, a living body face recognition result is output.
In the embodiment of the application, the face recognition result of the living body can be a depth face image, an IR image and an RGB image which are detected by the living body, and can also be a successful result of the living body detection.
In the embodiment of the application, when the IR image passes through living body detection, specifically, whether the light spot image is a living body face light spot image or not is judged according to the light spot definition of the IR image, whether the light spot definition is in a preset light spot definition threshold interval is judged, when the light spot definition of the pixel area is in the preset light spot definition threshold interval, the light spot image is judged to be the living body face light spot image, and the light spot definition threshold interval is 10-30. The light spot definition has the following numerical value:c is the total number of pixels in the pixel area, D (f) is the value of the spot definition, and G (x, y) is the value of the center pixel after convolution.
In the embodiment of the application, the IR image can be detected in vivo by a neural network-based living body detection model, and the depth face image can be detected in vivo by another neural network-based living body detection model.
Fig. 6 is a flowchart of a step of performing depth reconstruction on a target face according to an embodiment of the present application, where, as shown in fig. 6, the step of performing depth reconstruction on the target face according to the infrared speckle image and the RGB image includes the following steps:
the step S403 includes the steps of:
step S4031: mapping the face area of the RGB image into the infrared light spot image, and determining a plurality of light spots of the face area in the infrared light spot image;
step S4032: extracting a plurality of spot areas of the face area;
step S4033: and generating a depth face image of the face region according to the deformation or displacement of the spot area corresponding to the face region.
In the embodiment of the application, the depth face image is produced according to a structured light technology, specifically, the depth face image of the face area is obtained according to deformation or displacement of a light spot area, and the rugged depth information of the face area is obtained.
In the embodiment of the application, the depth image of the face area can also be obtained through the time delay or the phase difference of a plurality of infrared light spot images, namely, the depth image is calculated through a TOF technology.
When the depth reconstruction method for the snap shooting of the depth camera is realized, the method is realized through a Hai Si chip with the model of Hi3516DV300, when the RGB image and the IR image are subjected to face detection through a neural network reasoning engine (NNIE, neural Network Inference Engine), and simultaneously, the infrared light spot image is preprocessed through an intelligent video engine (IVW, intelligent Video Engine) so as to extract a plurality of light spot areas in the infrared light spot image; when the CPU and the intelligent video engine are used for carrying out depth reconstruction on the target face to generate a depth face image, the neural network reasoning engine is used for carrying out quality detection on the RGB image and the IR image and carrying out living body detection on the IR image in sequence.
Fig. 7 is a schematic block diagram of a depth reconstruction system in capturing with a depth camera according to an embodiment of the present application, where, as shown in fig. 7, the depth reconstruction system in capturing with a depth camera provided by the present application includes the following modules:
the image acquisition module is used for acquiring an RGB image, an IR image and an infrared light spot image of the target face;
the face detection module is used for carrying out face detection on the RGB image and the IR image;
the depth data preprocessing module is used for preprocessing the infrared light spot image while detecting the human face so as to extract a plurality of light spot areas in the infrared light spot image;
the expression detection module is used for carrying out expression detection on the human face region when the RGB image and the IR image detect the human face region, and determining the expression type of the human face region;
and the depth reconstruction module is used for performing depth reconstruction on the target face according to the infrared light spot image and the RGB image to generate a depth face image when the expression type is any expression type in a preset expression type set.
The embodiment of the application also provides depth reconstruction equipment for capturing by the depth camera, which comprises a processor. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to execute the steps of the depth reconstruction method upon a snapshot of the depth camera via execution of the executable instructions.
As described above, in this embodiment, the RGB images are sequentially subjected to face detection and expression optimization, and the infrared spot image is preprocessed during face detection, and the infrared spot image is deeply reconstructed only according to the face region in the RGB images after expression optimization, so that the efficiency of deep reconstruction is improved.
Those skilled in the art will appreciate that the various aspects of the application may be implemented as a system, method, or program product. Accordingly, aspects of the application may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 8 is a schematic structural diagram of a depth reconstruction device when a depth camera captures images in an embodiment of the present application. An electronic device 600 according to this embodiment of the application is described below with reference to fig. 8. The electronic device 600 shown in fig. 8 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present application.
As shown in fig. 8, the electronic device 600 is in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code that can be executed by the processing unit 610, such that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the above-described depth reconstruction method section when a depth camera is snap shot. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.
The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in fig. 8, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the application also provides a computer readable storage medium for storing a program, and the method is used for realizing the steps of the depth reconstruction method when the program is executed. In some possible embodiments, the aspects of the application may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the application as described in the above-mentioned depth camera snapshot depth reconstruction method section of the specification when the program product is run on the terminal device.
As described above, when the program of the computer readable storage medium of this embodiment is executed, the RGB images sequentially perform face detection and expression optimization, the infrared spot image is preprocessed during face detection, and the infrared spot image is deeply reconstructed only according to the face region in the RGB images after the expression optimization, thereby improving the efficiency of deep reconstruction.
Fig. 9 is a schematic structural view of a computer-readable storage medium in an embodiment of the present application. Referring to fig. 9, a program product 800 for implementing the above-described method according to an embodiment of the present application is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In the embodiment of the application, the RGB images are subjected to face detection and expression optimization in sequence, the infrared speckle images are preprocessed during face detection, and the infrared speckle images are subjected to depth reconstruction only according to the face areas in the RGB images after expression optimization, so that the efficiency of depth reconstruction is improved;
according to the application, the RGB image and the IR image are deeply reconstructed after the expression is optimized, so that failure caused by too poor image quality in the deep reconstruction is avoided, and the depth face image can be generated after the IR image is subjected to living detection, so that the snapshot process can be compactly executed, the time of the whole snapshot process is shortened, and the face brushing payment efficiency is improved.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing describes specific embodiments of the present application. It is to be understood that the application is not limited to the particular embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the claims without affecting the spirit of the application.
Claims (8)
1. The depth reconstruction method during capturing by the depth camera is characterized by comprising the following steps of:
step S1: acquiring an RGB image, an IR image and an infrared light spot image of a target face;
step S2: performing face detection on the RGB image and the IR image, and preprocessing the infrared light spot image at the same time to extract a plurality of light spot areas in the infrared light spot image;
step S3: when the face area is detected in the RGB image and the IR image, carrying out expression detection on the face area, and determining the expression type of the face area;
step S4: when the expression type is any expression type in a preset expression type set, performing depth reconstruction on the target face according to a plurality of light spot areas of the infrared light spot image and a face area in the RGB image to generate a depth face image;
when the target face is subjected to depth reconstruction to generate a depth face image, the following steps are carried out simultaneously:
performing quality detection on the RGB image and the IR image;
when the IR image accords with a preset quality standard, performing living body detection on the IR image;
the step S1 includes the steps of:
step S101: alternately acquiring an IR image and an infrared spot image of the target face through an infrared camera;
step S102: collecting RGB images of the target face through an RGB camera;
step S103: and acquiring an RGB image, an IR image and an infrared light spot image of the target face, and previewing the RGB image in real time.
2. The depth reconstruction method according to claim 1, wherein the step S3 includes the steps of:
step S301: triggering step S303 when the face area is detected in the RGB image and the IR image;
step S302: when no face area is detected in the RGB image and the IR image, returning to the step S1 to acquire the RGB image, the IR image and the infrared light spot image of the target face again;
step S303: and carrying out expression detection on the face area, and determining the expression type of the face area.
3. The depth reconstruction method according to claim 1, wherein the step S4 includes the steps of:
step S401: acquiring a preset expression type set;
step S402: judging whether the expression type is an expression type in a preset expression type set, triggering step S403 when the expression type is an expression type in the preset expression type set, sending out first prompt information when the expression type is not an expression type in the preset expression type set, and returning to step S1;
step S403: and carrying out depth reconstruction on the target face according to the plurality of light spot areas of the infrared light spot image and the RGB image to generate a depth face image.
4. The depth reconstruction method when capturing images with a depth camera according to claim 1, wherein when quality detection is performed on the RGB image and the IR image, comprising the steps of:
step M1: acquiring a preset image quality standard;
step M2: judging whether the RGB image and the IR image accord with preset image quality standards, triggering step M3 when the RGB image and the IR image accord with preset image quality standards, sending out second prompt information when the RGB image and the IR image do not accord with preset image quality standards, and returning to step S1;
step M3: and performing living body detection on the target face according to the IR image.
5. A depth reconstruction method when a depth camera is snap shot according to claim 3, wherein said step S403 comprises the steps of:
step S4031: mapping the face area of the RGB image into the infrared light spot image, and determining a plurality of light spots of the face area in the infrared light spot image;
step S4032: extracting a plurality of light spot areas of the face area;
step S4033: and generating a depth face image of the face region according to the deformation or displacement of the spot area corresponding to the face region.
6. The depth reconstruction system for the snapshot of the depth camera is characterized by comprising the following modules:
the image acquisition module is used for acquiring an RGB image, an IR image and an infrared light spot image of the target face;
the face detection module is used for carrying out face detection on the RGB image and the IR image;
the depth data preprocessing module is used for preprocessing the infrared light spot image while detecting the human face so as to extract a plurality of light spot areas in the infrared light spot image;
the expression detection module is used for carrying out expression detection on the human face region when the RGB image and the IR image detect the human face region, and determining the expression type of the human face region;
and the depth reconstruction module is used for performing depth reconstruction on the target face according to the infrared light spot image and the RGB image to generate a depth face image when the expression type is any expression type in a preset expression type set.
7. A depth reconstruction device when a depth camera captures, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the depth reconstruction method when the depth camera of any one of claims 1 to 5 is snap shot via execution of the executable instructions.
8. A computer-readable storage medium storing a program, characterized in that the program when executed implements the steps of the depth reconstruction method at the time of capturing with a depth camera according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411183.1A CN113673285B (en) | 2020-05-15 | 2020-05-15 | Depth reconstruction method, system, equipment and medium during capturing of depth camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411183.1A CN113673285B (en) | 2020-05-15 | 2020-05-15 | Depth reconstruction method, system, equipment and medium during capturing of depth camera |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113673285A CN113673285A (en) | 2021-11-19 |
| CN113673285B true CN113673285B (en) | 2023-09-12 |
Family
ID=78537508
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010411183.1A Active CN113673285B (en) | 2020-05-15 | 2020-05-15 | Depth reconstruction method, system, equipment and medium during capturing of depth camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113673285B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1811771A1 (en) * | 2006-01-20 | 2007-07-25 | Infrared Solutions, Inc. | Camera with visible light and infrared image blending |
| CN108629278A (en) * | 2018-03-26 | 2018-10-09 | 深圳奥比中光科技有限公司 | The system and method that information security is shown is realized based on depth camera |
| CN108650447A (en) * | 2018-07-06 | 2018-10-12 | 上海图漾信息科技有限公司 | Imaging sensor, depth data measurement head and measuring system |
| CN108764052A (en) * | 2018-04-28 | 2018-11-06 | Oppo广东移动通信有限公司 | Image processing method, device, computer readable storage medium and electronic equipment |
| CN109583304A (en) * | 2018-10-23 | 2019-04-05 | 宁波盈芯信息科技有限公司 | A kind of quick 3D face point cloud generation method and device based on structure optical mode group |
| CN109635539A (en) * | 2018-10-30 | 2019-04-16 | 华为技术有限公司 | A kind of face identification method and electronic equipment |
| CN109840475A (en) * | 2018-12-28 | 2019-06-04 | 深圳奥比中光科技有限公司 | Face identification method and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9959455B2 (en) * | 2016-06-30 | 2018-05-01 | The United States Of America As Represented By The Secretary Of The Army | System and method for face recognition using three dimensions |
-
2020
- 2020-05-15 CN CN202010411183.1A patent/CN113673285B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1811771A1 (en) * | 2006-01-20 | 2007-07-25 | Infrared Solutions, Inc. | Camera with visible light and infrared image blending |
| CN108629278A (en) * | 2018-03-26 | 2018-10-09 | 深圳奥比中光科技有限公司 | The system and method that information security is shown is realized based on depth camera |
| CN108764052A (en) * | 2018-04-28 | 2018-11-06 | Oppo广东移动通信有限公司 | Image processing method, device, computer readable storage medium and electronic equipment |
| CN108650447A (en) * | 2018-07-06 | 2018-10-12 | 上海图漾信息科技有限公司 | Imaging sensor, depth data measurement head and measuring system |
| CN109583304A (en) * | 2018-10-23 | 2019-04-05 | 宁波盈芯信息科技有限公司 | A kind of quick 3D face point cloud generation method and device based on structure optical mode group |
| CN109635539A (en) * | 2018-10-30 | 2019-04-16 | 华为技术有限公司 | A kind of face identification method and electronic equipment |
| CN109840475A (en) * | 2018-12-28 | 2019-06-04 | 深圳奥比中光科技有限公司 | Face identification method and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| Good match exploration for infrared face recognition;Changcai Yang;《ResearchGate》;第111-115页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113673285A (en) | 2021-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6239594B2 (en) | 3D information processing apparatus and method | |
| US8619122B2 (en) | Depth camera compatibility | |
| US9460337B2 (en) | Information processor and information processing method | |
| US20130335535A1 (en) | Digital 3d camera using periodic illumination | |
| EP3135033B1 (en) | Structured stereo | |
| JP6476831B2 (en) | Parallax calculation system, information processing apparatus, information processing method, and program | |
| JP2020042772A (en) | Depth data processing system capable of optimizing depth data by image positioning with respect to depth map | |
| US20110187819A1 (en) | Depth camera compatibility | |
| KR20130099735A (en) | Method and fusion system of time-of-flight camera and stereo camera for reliable wide range depth acquisition | |
| CN108513121B (en) | Method and apparatus for depth map evaluation of a scene | |
| US20160245641A1 (en) | Projection transformations for depth estimation | |
| CN108495113B (en) | Control method and device for binocular vision system | |
| CN113673285B (en) | Depth reconstruction method, system, equipment and medium during capturing of depth camera | |
| CN112749610A (en) | Depth image, reference structured light image generation method and device and electronic equipment | |
| CN113673287B (en) | Depth reconstruction method, system, equipment and medium based on target time node | |
| CN113673286B (en) | Depth reconstruction method, system, equipment and medium based on target area | |
| CN112150529B (en) | Depth information determination method and device for image feature points | |
| CN113673284B (en) | Depth camera snapshot method, system, equipment and medium | |
| US11195290B2 (en) | Apparatus and method for encoding in structured depth camera system | |
| CN111260781B (en) | Method and device for generating image information and electronic equipment | |
| CN107589834B (en) | Terminal device operation method and device and terminal device | |
| CN116704001A (en) | Multi-frame depth reconstruction method, system, equipment and storage medium | |
| YUNFAN et al. | Indepth: Real-time depth inpainting for mobile augmented reality | |
| CN115201834A (en) | Distance detection method, system, equipment and storage medium based on light spot image | |
| CN117557628A (en) | Structured light rapid depth reconstruction method, system, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |