WO2020078243A1

WO2020078243A1 - Image processing and face image identification method, apparatus and device

Info

Publication number: WO2020078243A1
Application number: PCT/CN2019/110266
Authority: WO
Inventors: 郎平; 姚迪狄
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2018-10-19
Filing date: 2019-10-10
Publication date: 2020-04-23
Also published as: CN111161205A; CN111161205B

Abstract

Disclosed are an image processing and face image identification method, apparatus and device. The method comprises: acquiring a first image and a second image of a target object under different lighting conditions; performing quality evaluation processing on one of the images; if there is a first region with a quality evaluation value less than a pre-set quality evaluation threshold value, determining a second region, in the other image, corresponding to the first region; and compensating the first region based on image data of the second region.

Description

Image processing and face image recognition method, device and equipment

This application requires the priority of the Chinese patent application filed on October 19, 2018 with the application number 201811222689.7 and the invention titled "An Image Processing and Face Image Recognition Method, Device, and Equipment" In this application.

Technical field

The present application relates to the field of computer technology, and in particular, to an image processing and face image recognition method, device, and equipment.

Background technique

Image quality evaluation is one of the basic techniques in image processing. It mainly analyzes and studies the characteristics of the image, and then evaluates the quality of the image.

Currently, when the quality evaluation value of an image of an object does not meet the standard, the image is generally fused with other types of images of the object to improve the quality evaluation value.

Therefore, a more reliable solution is needed.

Summary of the invention

The embodiments of the present specification provide an image processing method for optimizing image quality.

An embodiment of this specification also provides an image processing method, including:

Acquiring a first image and a second image of the target object, the first image and the second image being images collected under different lighting conditions;

Performing quality evaluation processing on the first image;

If there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second image corresponding to the first area;

Based on the image data of the second area, the first area is compensated.

An embodiment of the present specification also provides a face image recognition method, including:

Acquiring a first face image and a second face image of the target object, where the first face image and the second face image are images collected under different lighting conditions;

Performing quality evaluation processing on the first face image;

If there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second face image corresponding to the first area;

Based on the image data of the second area, the first area is compensated, and the compensated first face image is identified.

An embodiment of this specification also provides an image processing device, including:

An obtaining module, configured to obtain a first image and a second image of the target object, where the first image and the second image are images collected under different lighting conditions;

An evaluation module, configured to perform quality evaluation processing on the first image;

A determining module, configured to determine a second area in the second image corresponding to the first area if there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold;

The compensation module is used to compensate the first area based on the image data of the second area.

An embodiment of this specification also provides a face image recognition device, including:

An obtaining module, configured to obtain a first face image and a second face image of the target object, where the first face image and the second face image are images collected under different lighting conditions;

An evaluation module, configured to perform quality evaluation processing on the first face image;

A determining module, configured to determine a second area in the second face image corresponding to the first area if there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold ;

The compensation module is configured to compensate the first area based on the image data of the second area and identify the compensated first face image.

An embodiment of the present specification also provides a face image recognition device, including: an RGB camera, a black and white camera, an infrared fill light, and a processing chip; wherein:

The RGB camera is used to collect RGB face images of the target object;

The infrared fill light is used to emit infrared light to the face of the target object;

The black-and-white camera is used to collect infrared face images of the target object under infrared lighting conditions;

The processing chip is configured to use one of the RGB face image and the infrared face image as a first face image and the other as a second face image, and perform any of claims 15 to 17 A method step.

An embodiment of this specification also provides an electronic device, including:

Processor; and

A memory arranged to store computer-executable instructions, which when executed, causes the processor to perform the steps of the image processing method or face image recognition method as described above.

Embodiments of this specification also provide a computer-readable storage medium that stores a computer program on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the image processing method or face image recognition method described above are implemented .

The above at least one technical solution adopted by the embodiments of the present specification can achieve the following beneficial effects:

By acquiring the first image and the second image collected under different lighting conditions of the same target object, and performing regional quality evaluation on it, to detect whether there is a quality evaluation in any one of the first image and the second image When the value of the first area is less than the quality evaluation threshold, the image data of another second area corresponding to the first area is used to compensate the first area to improve the overall image quality. Compared with the prior art solution for fusing images to improve image quality, this solution can locate areas that require quality compensation and perform image quality compensation regionally, thereby simplifying image quality compensation.

BRIEF DESCRIPTION

The drawings described here are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an undue limitation on the present application. In the drawings:

FIG. 1a is a schematic diagram of an application scenario provided by this specification;

FIG. 1b is a schematic diagram of another application scenario provided by this specification;

2 is a schematic flowchart of an image processing method according to an embodiment of the present specification;

FIG. 3 is a schematic flowchart of the quality evaluation steps provided by this specification;

4 is a schematic diagram of brightness distribution data of an image provided by the specification;

FIG. 5 is a schematic flowchart of a face image recognition method according to an embodiment of the present specification;

6 is a schematic flowchart of a face image recognition method according to another embodiment of the present specification;

7 is a schematic structural diagram of hardware and a system provided by an embodiment of the present specification;

8 is a schematic diagram of different working modes of a black and white camera provided by an embodiment of the present specification;

9a is a schematic diagram of the data format of each frame of the 2D infrared output provided by an embodiment of the present specification;

9b is a schematic diagram of the data format of each frame of 3D depth output provided by an embodiment of the present specification;

10 is a schematic diagram of the data format of each frame of RGB and infrared output provided by an embodiment of the present specification;

11 is a schematic structural diagram of an image processing device according to an embodiment of the present specification;

12 is a schematic structural diagram of a face image recognition device according to an embodiment of the present specification;

13 is a schematic structural diagram of an electronic device according to an embodiment of the present specification;

14 is a schematic structural diagram of an electronic device according to another embodiment of the present specification.

detailed description

In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be described clearly and completely in conjunction with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of this application.

As stated in the background section, the prior art generally combines the advantages of image acquisition under different lighting conditions through the steps of preprocessing-registration-image fusion to prevent the impact of dark light and occlusion on the image quality, and then Provide image processing support for business scenarios with high security requirements. However, because the preprocessing, registration, image fusion and other steps are all based on the post-algorithm, the algorithm complexity is very large.

Based on this, the present invention provides an image processing method by collecting the first image and the second image under different lighting conditions of the same target object, and performing a regional quality evaluation value to detect the first image and When one of the second images has a first area whose quality evaluation value is less than the quality evaluation threshold, the image data of the second area corresponding to the first area in the other is used to compensate the first area to improve the overall image quality. Compared with the prior art, this solution can locate the area where quality compensation is required, and then perform quality compensation in a targeted manner, thereby achieving the purpose of simplifying image quality compensation.

Among them, image quality evaluation is one of the basic techniques in image processing, mainly by analyzing the characteristics of the image, and then assessing the quality of the image (the degree of image distortion).

For ease of description, the first image is specifically an RGB image, the second image is an infrared image, or the first image is an infrared image, and the second image is an RGB image. .

Referring to FIG. 1a, one application scenario may be:

The image acquisition device collects the RGB image and infrared image of the object and inputs it to the image processing terminal;

The image processing terminal performs image quality evaluation on the RGB image and the infrared image to determine whether one of the RGB image and the infrared image has an area whose quality evaluation value is less than the quality evaluation threshold, and if so, the image data of the corresponding area of the other image is used Perform compensation to obtain the compensated RGB image and infrared image whose quality evaluation value is greater than or equal to the quality evaluation threshold.

Referring to FIG. 1b, another application scenario may be:

The image collection device collects the user's RGB face image and infrared face image, and inputs it to the image processing terminal;

The image processing terminal performs image quality evaluation on the RGB face image and the infrared face image to determine whether the RGB face image and the infrared face image have areas where the quality evaluation value is less than the quality evaluation threshold, and if so, use the image from another image Compensate the image data of the corresponding area, get the compensated RGB face image and infrared face image whose quality evaluation value is greater than or equal to the quality evaluation threshold, and input it to the business processing terminal, otherwise directly convert the RGB face image and infrared face The image is input to the business processing terminal;

The service processing terminal verifies the user's identity information and authorization information based on the compensated RGB face image and / or infrared face image; after the verification is passed, the service requested by the user is performed, and the service processing result is output.

Among them, the image acquisition device can be an integrated machine with the image processing terminal; the image processing terminal and the business processing terminal can be an integrated machine, which can be exemplified by image quality optimization software, banking counters, offline stores, hotel room access control, etc. The image processing terminal and business processing terminal can be a PC terminal, a mobile terminal, or a mobile communication terminal refers to a computer device that can be used on the move. Broadly speaking, it includes mobile phones, notebooks, tablet computers, POS machines and even Including car computer. But in most cases it refers to mobile phones or smartphones and tablets with multiple application functions.

The technical solutions provided by the embodiments of the present application will be described in detail below in conjunction with the drawings.

FIG. 2 is a schematic flowchart of an image processing method according to an embodiment of the present specification. The method may be executed by the image processing terminal in FIG. 1a. Referring to FIG. 2, the method may specifically include the following steps:

Step 220: Acquire a first image and a second image of the target object, where the first image and the second image are images collected under different lighting conditions;

The first image and the second image are preferably images acquired synchronously.

For ease of description, one example of the first image and the second image in this embodiment is an RGB image, and the other example is an infrared image.

Accordingly, an implementation manner of step 220 may be:

Use multi-eye camera to collect RGB and infrared images of the target object simultaneously.

The multi-eye camera may be a binocular camera or a trinocular camera, including a color camera part and a black-and-white camera part. RGB image refers to the image collected by the color camera part under visible light conditions. It is an image displayed in the RGB color mode. The RGB color mode is a color standard in the industry. It is through the red (R), green (G ), The blue (B) three color channels change and superimpose each other to get a variety of colors, RGB is the color representing the three channels of red, green, and blue; infrared image refers to infrared The image collected by the black-and-white camera part under lighting conditions is an image obtained by acquiring the intensity of infrared light of an object.

Step 240: Perform quality evaluation processing on the first image; referring to FIG. 3, an implementation manner thereof may be:

Step 320: Determine the quality-related parameter distribution data of the first image;

Among them, the quality-related parameters include: one or more of sharpness, resolution, and brightness; sharpness refers to the sharpness of each detail shadow and its boundary on the image; resolution refers to the precision of the screen image; Brightness refers to the brightness of the picture.

Referring to FIG. 4, the RGB image is taken as an example to illustrate the brightness distribution data in the quality-related parameter distribution data in the first image. In this RGB image, due to the problem of light source deployment, the brightness of the image changes from left to right (401-406 The corresponding column) gradually decreases, and the uniform brightness value is lower.

The area 401 may be a pixel-level area, and the brightness value may be the brightness of the pixel; it may also be a pre-divided area size, and the brightness value may be a uniform brightness value of the area.

Similarly, the distribution data of sharpness and resolution can be constructed.

Step 340: Based on the quality-related parameter distribution data, determine the quality evaluation value distribution data of the first image; with reference to FIG. 4, specific examples may be:

Based on the distribution data of each quality-related parameter constructed in step 320, the brightness, sharpness, resolution, etc. corresponding to any area can be obtained, and then compared with the preset threshold corresponding to each quality-related parameter to obtain a difference, and Based on the difference and preset weight corresponding to each quality-related parameter, the quality evaluation value of the area is calculated, and the quality evaluation values of other areas are obtained by analogy, and then the distribution data of the quality evaluation value of the image is obtained.

In addition, because the infrared image can ensure uniform light, the brightness parameter can be ignored when the first image is an infrared image and the quality evaluation value distribution data is calculated; when the first image is the RGB image and the quality evaluation value distribution data is calculated, the The uniform brightness value of the infrared image is used as the preset threshold of the brightness of the RGB image.

Step 360: Based on the quality evaluation value distribution data, determine an area in the first image whose quality evaluation value is less than a preset quality evaluation threshold. With reference to Figure 4, specific examples can be:

The quality evaluation value of each area in the image is compared with a preset quality evaluation threshold to determine an area where the quality evaluation value is less than the preset quality evaluation threshold, and this area is used as the first area.

Step 260: If there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second image corresponding to the first area; an implementation manner thereof Can be:

S1. Perform image registration processing on infrared images and RGB images; specific examples can be:

First, perform feature extraction processing on the two images to obtain feature points, and find matching feature point pairs by performing similarity measurement processing; then, obtain image space coordinate transformation parameters from the matched feature point pairs; and finally, coordinate transformation Parameters for image registration.

S2. Based on the spatial coordinate transformation relationship between the RGB image and the infrared image obtained by the registration, determine a second area with coordinates corresponding to the coordinates of the first area.

That is, if there is a first area in the infrared image, the second area in the RGB image is determined; if there is a first area in the RGB image, the second area in the infrared image is determined.

In another implementation manner of step 260, the steps of image registration processing may be completed by a multi-eye camera, and specific examples may be:

Obtain the RGB image and infrared image collected by the multi-eye camera simultaneously; based on the parallax between the RGB camera part and the infrared camera part in the multi-eye camera, internally calibrate the RGB image and the infrared image to ensure that the RGB and infrared images have complete screen content Consistent to complete the registration.

Step 280: Compensate the first area based on the image data of the second area.

If the first image is an RGB image and the second image is an infrared image, one implementation manner may be:

If the brightness of the first area of the RGB image is less than a preset brightness threshold, based on the image data of the second area of the infrared image, compensate the brightness of the first area of the RGB image; or,

If the sharpness of the first area of the RGB image is less than a preset sharpness threshold, the sharpness of the first area of the RGB image is compensated based on the image data of the second area of the infrared image.

If the first image is an infrared image and the second image is an RGB image, another implementation manner may be:

If the first area of the infrared image is a blocked area, the image data of the target object in the first area of the infrared image is compensated based on the image data of the second area of the RGB image; or,

If the resolution of the first region of the infrared image is less than a preset resolution threshold, the first region resolution of the infrared image is compensated based on the image data of the second region of the RGB image.

Another implementation of step 280 may be:

Fusing the image data of the first area and the second area, and using the fused image data as the compensated image data of the first area.

It can be seen that in this embodiment, by acquiring the first image and the second image collected under different lighting conditions of the same target object, and performing a regional quality evaluation value thereof, one of the first image and the second image is detected When there is a first area whose quality evaluation value is less than the quality evaluation threshold, the image data of another second area corresponding to the first area is used to compensate the first area to improve the overall image quality. Compared with the scheme of fusing images in the prior art to improve image quality, it can accurately locate areas requiring quality compensation, and perform targeted image data compensation, thereby simplifying image quality compensation.

FIG. 5 is a schematic flowchart of a face image recognition method provided by an embodiment of the present specification. The method may be jointly executed by the image processing terminal and the business processing terminal in FIG.

Step 502: Acquire a first face image and a second face image of a target object, where the first face image and the second face image are images collected under different lighting conditions;

Wherein, the first face image and the second face image are preferably images collected synchronously.

For ease of description, one example of the first face image and the second face image in this embodiment is an RGB image, and the other example is an infrared image. Accordingly, it should be noted that an implementation manner of step 502 may be:

Simultaneously collect RGB face images and infrared face images of the target object through the multi-eye camera;

Among them, the color camera and the black and white camera of the binocular camera are in the normally open state.

Another implementation manner of step 502 may be:

Collect the RGB image of the target object through a binocular camera, and when it is determined that there is a human face in the RGB image, simultaneously collect the RGB facial image and the infrared facial image of the target object;

Among them, the color camera is in a normally open state, and the black and white camera is in a normally closed state.

Step 504: Perform quality evaluation processing on the first face image;

Step 506: If there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second face image corresponding to the first area;

Wherein, the preset quality evaluation threshold may be an image quality evaluation threshold required by the target user to handle the business.

Step 508: Compensate the first area based on the image data of the second area;

Steps 504 to 508 correspond to steps 240 to 280 in the embodiment shown in FIG. 2 respectively, and their implementations are also similar. Therefore, they will not be repeated here.

Further, before step 510 is performed, it also includes the step of detecting whether the compensated first face image can meet the requirements of face recognition, and one implementation manner thereof may be:

Performing quality evaluation processing on the compensated first face image;

If the quality evaluation value of the compensated first face image meets the face recognition quality evaluation requirements, it is allowed to identify the compensated first face image; otherwise, return to step 502 until the quality evaluation value is up to standard The first image after compensation or the second image after compensation.

Step 510: Identify the compensated first face image. One of its implementation methods can be:

Performing face feature extraction processing on the compensated first face image;

Based on the extracted face features, face recognition processing is performed to determine the identity information of the target object.

It is not difficult to understand that, after determining the identity information of the target object (target user), related services can be handled based on the identity information of the target object.

Further, when it is determined that the security level of the business requirements of the target object reaches a preset security threshold, the target object may also be subjected to a living body detection before handling related business. For example, the business requested by the target object is payment. Because the security required by the payment business is relatively high, it is necessary to further determine whether the target object is a living body. One of its implementation methods can be:

Determining the depth face image of the target object, and performing live detection on the target object based on the depth face image. Specific examples can be:

First, select the most distinguishing feature to train the classifier; then, use the 3D face data corresponding to the deep face image as the input of the training classifier, and obtain the result that the target object output by the training classifier is a living or non-living body.

In addition, in order to further provide reduction in living body detection, the compensated first face image and the depth map face image can also be fused, and the target object can be subjected to living body detection based on the fused image.

Another way to implement the living body detection step can be:

Determine the living body motion of the target object, and determine whether the target object is a living body based on the living body motion.

Among them, living body movements include blinking, shaking head, opening mouth, etc.

It can be seen that in this embodiment, by acquiring the first face image and the second face image under different lighting conditions of the object, and determining the quality evaluation distribution data, the first face image and the second face image are detected If any one of the first areas has a quality evaluation value less than the quality evaluation threshold, use the image data of the second area corresponding to the first area to compensate the first area to improve the overall quality of the face image To further improve the accuracy of face recognition; moreover, in this embodiment, a living body detection is performed on the user based on the deep face image to improve the security of business handling.

FIG. 6 is a schematic flowchart of a face image recognition method according to another embodiment of the present specification. Referring to FIG. 6, the method may specifically include the following steps:

Step 602: Initialize the hardware and system configuration;

With reference to Fig. 7, the hardware and system design part includes: RGB part, black and white part, IR structured light projector, special ISP and SOC processing chip, and infrared fill light, in which:

The RGB part, which is the color camera Color, includes: a dedicated lens + RGB image sensor, used to collect normal color images, taking into account the actual business needs, using the resolution of the corresponding specifications, for example: for the business needs of new retail scenarios, use The resolution of 1080P specification (1920 * 1080), the FOV of the special lens is larger (such as the diagonal of 110 °), and it is customized according to the security specifications to ensure no distortion (required for face recognition).

The black and white part, the infrared camera Infrared Camera, includes: a dedicated lens + black and white image sensor, used to collect grayscale images, considering the business needs of new retail scenarios and ensuring alignment with RGB pixels, the same resolution of 1080P (1920 * 1080), the FOV of the special lens is large (for example, 110 ° diagonal), customized according to the security specifications to ensure no distortion (required for face recognition).

With reference to FIG. 8, it should be noted that the infrared camera collects infrared grayscale images, 2D pure infrared image collection (filling through a narrow-band infrared fill light), and 3D depth Depth image collection (through a specially coded structure Light) uses Infrared Camera for image collection, so the two parts need to be processed in time-sharing, that is, the two processes are independent, and it is not possible to collect 2D pure infrared images and 3D depth Depth images at the same time.

Among them, the 2D pure infrared image acquisition condition is triggered, confirm to turn off the IR structured light projector (IR Laser Projector), turn on the custom band (850nm / 940nm) infrared fill light, and start collecting 2D infrared face images; 3D depth information The image acquisition condition is triggered, confirm to turn off the infrared fill light of the custom band (850nm / 940nm), turn on the IR structured light projector (IR Laser Projector), and start to collect the face image of 3D depth information.

Moreover, in view of the fact that it is not possible to collect 2D pure infrared images and 3D depth Depth images at the same time, it must be processed accordingly when the final output is output, that is, the grayscale of each frame is 2D infrared data when collecting the 2D pure infrared image process The specific output format is shown in FIG. 9a; when the image flow of collecting 3D depth information is output, the degree of each frame is 3D depth data, and the specific output format is shown in FIG. 9b. In FIGS. 9a and 9b, "Y" indicates brightness, "U" indicates chroma, "V" indicates density, and h is 1080P.

The IR structured light projector is a light source with a special structure, such as discrete light spots, fringe light, coded structured light, etc., used to project onto the object or plane to be detected, when the structured light after coding is projected on a plane of different depths The texture of light will change, so you can calculate different depths by collecting these texture changes.

Dedicated image signal processing (Image) Processing (ISP) and SOC processing chip (System-on-a-Chip), in addition to calculating depth information, digital image processing of RGB and grayscale images, such as lens distortion Adjust LSC, automatic white balance AWB, automatic exposure AE, wide dynamic WDR, color matrix adjustment CCM, 2D noise reduction, etc., optimize and format RGB and grayscale (RAW format is converted to the final YUV format).

The infrared fill light mainly fills the infrared infrared face (such as 850nm and 940nm) to ensure the uniform brightness of the face under different light source scenes.

The initial configuration parameters include: RGB sensor is turned on by default, MONO sensor, infrared fill light and IR structure light emitter are turned off to collect only RGB image data in each frame.

Step 604: Detect whether there is a human face in each frame of RGB images;

If yes, go to step 606; if no, continue to collect RGB image data.

Among them, the face recognition technology is a relatively mature technology, therefore, the face recognition based on RGB images will not be repeated here.

Step 606: Turn on the infrared fill light and the black-and-white image sensor MONO Sensor to collect infrared image data while collecting RGB image data in each frame;

Preferably, in order to ensure the flexibility of the scene and the adjusted image after triggering, this embodiment further integrates the output formats of RGB image data and infrared image data, that is, each frame of data includes both RGB image data and infrared image data. Taking h as 1080P for example, the specific number of single frames is shown in Figure 10.

Step 608: Perform face feature extraction on the infrared image and the RGB image to determine whether the face quality meets the recognition feature extraction requirements;

If yes, perform step 610; if not, perform compensation processing on the face image, and perform step 608 again. If the recognition feature extraction requirements are not met after multiple compensation processing, return to step 606 to reacquire infrared images and RGB image;

Among them, different services have different identification feature extraction requirements.

Step 610: Perform face recognition processing on the face image to determine the user's identity, and then conduct related business processing; turn off the infrared fill light to return to step 604 in the next round;

Step 612: Based on the security required by the service requested by the user, determine whether a living body test is required;

If yes, go to step 614; if no, the process ends.

Among them, for services with high security requirements such as payment and hotel access control, it can be configured to perform live detection; for mobile phone package processing, banking business opening and other security requirements, it can be configured to not require live detection.

Step 614: Turn on the IR structure light coding transmitter, and the MONO Sensor collects the depth image and performs 3D calculation;

Step 616: Cyclically collect the grayscale image of each frame of RGB + Depth and output a real-time depth map;

Step 618: Based on the depth map, determine whether to pass the living body detection;

If yes, go to step 620; if no, go back to step 616;

Step 620, transact related business; synchronize, turn off the IR structure optical coding transmitter to return to step 604 in the next round;

It can be seen that this embodiment compensates the RGB image with an infrared image to ensure the uniform brightness of the image, and solves the problem of uneven exposure of the face when RGB is exposed to various complex lights, such as a yin and yang face; by compensating the infrared image with the RGB image, In order to solve the problem of face occlusion caused by reflective objects such as glasses, RGB + IR redundancy strategy is ensured; when performing live detection in business scenarios with high security requirements for payment, the IR-encoded structured light emitter is also collected through MONOSensor Project the returned depth information (point cloud image) to ensure safety.

In addition, the above method embodiments are described as a series of action combinations for the sake of simple description, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the described sequence of actions, because according to this In embodiments of the invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

FIG. 11 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present specification. Referring to FIG. 11, it may specifically include: an acquisition module 111, an evaluation module 112, a determination module 113, and a compensation module 114, where:

The obtaining module 111 is used to obtain a first image and a second image of the target object, where the first image and the second image are images collected under different lighting conditions;

The evaluation module 112 is configured to perform quality evaluation processing on the first image;

The determining module 113 is configured to determine a second area in the second image corresponding to the first area if there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold;

The compensation module 114 is configured to compensate the first area based on the image data of the second area.

Optionally, the evaluation module 112 is specifically used for:

Determining quality-related parameter distribution data of the first image; determining quality evaluation value distribution data of the first image based on the quality related parameter distribution data; determining the first image based on the quality evaluation value distribution data The area where the medium quality evaluation value is less than the preset quality evaluation threshold.

Wherein, the quality-related parameters include one or more of clarity, resolution, and brightness.

Optionally, the first image is an RGB image, and the second image is an infrared image; the compensation module 114 is specifically used to:

Optionally, the first image is an infrared image, and the second image is an RGB image; the compensation module 114 is specifically used to:

Optionally, the compensation module 114 is specifically used for:

Optionally, the first image and the second image are collected and registered by a binocular camera.

Optionally, the target object includes a human face, the first image is a first human face image, and the second image is a second human face image.

Optionally, the obtaining module 111 is specifically used for:

Collect the first image of the target object; when it is determined that there is a human face in the first image, synchronously acquire the first human face image and the second human face image of the target object.

Optionally, the device further includes:

The face recognition module is used to recognize the compensated first face image and determine the identity information of the target object.

Optionally, the device further includes:

The living body detection module is configured to determine that the safety of the business requirements of the target object reaches a preset safety threshold, and perform living body detection on the target object.

Optionally, the living body detection module is specifically used for:

Determining the depth face image of the target object, and performing live detection on the target object based on the depth face image.

Optionally, the living body detection module is specifically used for:

Fusing the deep face image and the compensated first face image, and performing live detection on the target object based on the fused image.

Optionally, the living body detection module is specifically used for:

12 is a schematic structural diagram of a face image recognition device according to an embodiment of the present specification. Referring to FIG. 12, the device may specifically include: an acquisition module 121, an evaluation module 122, a determination module 123, a compensation module 124, and a recognition module 125 ,among them:

The obtaining module 121 is used to obtain a first face image and a second face image of the target object, where the first face image and the second face image are images collected under different lighting conditions;

The evaluation module 122 is configured to perform quality evaluation processing on the first face image;

The determining module 123 is configured to determine a second area corresponding to the first area in the second face image if there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold region;

The compensation module 124 is configured to compensate the first area based on the image data of the second area;

The recognition module 125 is used to recognize the compensated first face image.

Optionally, the evaluation module 122 is also used to:

Performing quality evaluation processing on the compensated first face image; if the quality evaluation value of the compensated first face image meets the face recognition quality evaluation requirement, it is allowed to recognize the compensated first person Face image.

Optionally, the identification module 125 is specifically used for:

Perform face feature extraction processing on the compensated first face image; perform face recognition processing based on the extracted face features to determine the identity information of the target object.

In addition, for the above-mentioned device implementations, since they are basically similar to the method implementations, the description is relatively simple, and the relevant parts can be referred to the description of the method implementations. Moreover, it should be noted that among the various components of the device of the present invention, the components therein are logically divided according to the functions to be realized, but the present invention is not limited to this, and each component can be Re-divide or combine.

FIG. 7 is a schematic structural diagram of hardware and a system provided by an embodiment of the present specification. The face image recognition device provided by this embodiment will be described in detail below with reference to FIG. Light lamps and processing chips; including:

The RGB camera (Color Camera) is used to collect RGB face images of the target object;

The black and white camera (Inrared Camera) is used to collect infrared face images of the target object under infrared illumination conditions;

The processing chip (for example: an image signal processing unit ISP + integrated circuit SOC) is configured to use one of the RGB face image and the infrared face image as a first face image and the other as a second face Image and execute the steps of the method described in any one of Embodiment 7 corresponding to FIG. 2, FIG. 5, and FIG. 6 above.

Optionally, the RGB camera and the black-and-white camera belong to the same binocular camera.

Optionally, the RGB camera is in a normally-on state, and the infrared camera and the infrared fill light process a normally-off state;

The processing chip is also used to determine that there is a face in the RGB face image collected by the RGB camera, and wake up the infrared camera and the infrared fill light to synchronize the infrared camera with the RGB camera Collect a face image of the target object.

Optionally, the processing chip is also used to turn off the infrared fill light when identifying the identity information of the target object.

Optionally, the device further includes: a structured light emitter in a normally closed state;

The processing chip is also used to determine that the structured light emitter needs to be awakened when it is necessary to perform a live detection on the target object;

The structured light emitter is configured to emit structured light to the face of the target object for the black-and-white camera to collect the depth face image of the target object under structured light illumination conditions;

The processing chip is also used to turn off the structured light emitter when it is determined that the target object passes live detection based on the depth face image.

It can be seen that in this embodiment, by customizing the hardware, the startup / shutdown timing of each device of the hardware is reconstructed in the processes of face detection, quality judgment, feature extraction, and comparison recognition to ensure the optimal face recognition strategy in various environments .

FIG. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present specification. Referring to FIG. 13, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may include other business offices. Required hardware. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to form an image processing device at a logical level. Of course, in addition to the software implementation, this application does not exclude other implementations, such as a logic device or a combination of software and hardware, etc., that is to say, the execution body of the following processing flow is not limited to each logical unit, it can also be Hardware or logic device.

The network interface, processor and memory can be connected to each other via a bus system. The bus may be an ISA (Industry Standard Architecture, Industry Standard Architecture) bus, a PCI (Peripheral Components Interconnect, Peripheral Component Interconnect Standard) bus, or an EISA (Extended Industry Standard, Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, and a control bus. For ease of representation, only one bidirectional arrow is used in FIG. 13, but it does not mean that there is only one bus or one type of bus.

The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include read-only memory and random access memory, and provide instructions and data to the processor. The memory may include a high-speed random access memory (Random-Access Memory, RAM), or may also include a non-volatile memory (non-volatile memory), for example, at least one disk memory.

The processor is used to execute the program stored in the memory and specifically execute:

Performing quality evaluation processing on the first image;

Based on the image data of the second area, the first area is compensated.

The method executed by the image processing apparatus or the master node disclosed in the embodiment shown in FIG. 11 of the present application may be applied to a processor, or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The aforementioned processor may be a general-purpose processor, including a central processor (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc .; it may also be a digital signal processor (Digital Signal Processor, DSP), dedicated integration Circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied and executed by a hardware decoding processor, or may be executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, and register. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

The image processing apparatus can also execute the methods of FIGS. 2 to 3 and 6 and implement the method executed by the manager node.

Based on the same invention, the embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs include multiple application programs. When the electronic device executes, the electronic device is caused to execute the image processing method provided in the embodiments corresponding to FIG. 2 to FIG. 3, and FIG. 6.

FIG. 14 is a schematic structural diagram of an electronic device according to an embodiment of the present specification. Referring to FIG. 14, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may include other business offices. Required hardware. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to form a face image recognition device at a logical level. Of course, in addition to the software implementation, this application does not exclude other implementations, such as a logic device or a combination of software and hardware, etc., that is to say, the execution body of the following processing flow is not limited to each logical unit, it can also be Hardware or logic device.

The network interface, processor and memory can be connected to each other via a bus system. The bus may be an ISA (Industry Standard Architecture, Industry Standard Architecture) bus, a PCI (Peripheral Components Interconnect, Peripheral Component Interconnect Standard) bus, or an EISA (Extended Industry Standard, Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, and a control bus. For ease of representation, only one bidirectional arrow is used in FIG. 14, but it does not mean that there is only one bus or one type of bus.

Performing quality evaluation processing on the first face image;

The method performed by the face image recognition device or the master node disclosed in the embodiment shown in FIG. 12 of the present application may be applied to a processor, or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method may be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The aforementioned processor may be a general-purpose processor, including a central processor (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc .; it may also be a digital signal processor (Digital Signal Processor, DSP), dedicated integration Circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied and executed by a hardware decoding processor, or may be executed and completed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, and register. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

The face image recognition device can also execute the methods of FIGS. 5 to 6 and implement the method executed by the manager node.

Based on the same invention, the embodiments of the present application also provide a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs include multiple application programs. When the electronic device executes, the electronic device is caused to execute the face image recognition method provided in the embodiments corresponding to FIG. 5 to FIG. 6.

The embodiments in this specification are described in a progressive manner. The same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method embodiment.

The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram and a combination of the flow and / or block in the flowchart and / or block diagram may be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processing machine, or other programmable data processing device to produce a machine that enables the generation of instructions executed by the processor of the computer or other programmable data processing device A device for realizing the functions specified in one block or multiple blocks of one flow or multiple blocks of a flowchart.

These computer program instructions may also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instructions The device implements the functions specified in one block or multiple blocks of the flowchart one flow or multiple flows and / or block diagrams.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are performed on the computer or other programmable device to produce computer-implemented processing, which is executed on the computer or other programmable device The instructions provide steps for implementing the functions specified in one block or multiple blocks of the flowchart one flow or multiple flows and / or block diagrams.

In a typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

The memory may include non-permanent memory, random access memory (RAM) and / or non-volatile memory in computer-readable media, such as read only memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer-readable media, including permanent and non-permanent, removable and non-removable media, can store information by any method or technology. The information may be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, Magnetic tape cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media can be used to store information that can be accessed by computing devices. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

It should also be noted that the terms "include", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device that includes a series of elements not only includes those elements, but also includes Other elements not explicitly listed, or include elements inherent to such processes, methods, goods, or equipment. In the absence of more restrictions, the elements defined by the sentence "include one ..." do not exclude that there are other identical elements in the process, method, commodity or equipment that includes the elements.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

The above is only an embodiment of the present application, and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included in the scope of the claims of this application.

Claims

An image processing method, characterized in that it includes:

Acquiring a first image and a second image of the target object, the first image and the second image being images collected under different lighting conditions;

Performing quality evaluation processing on the first image;

If there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second image corresponding to the first area;

Based on the image data of the second area, the first area is compensated.
The method according to claim 1, wherein performing quality evaluation processing on the first image includes:

Determining the quality-related parameter distribution data of the first image;

Based on the quality-related parameter distribution data, determining the quality evaluation value distribution data of the first image;

Based on the quality evaluation value distribution data, an area in the first image whose quality evaluation value is less than a preset quality evaluation threshold is determined.
The method according to claim 2, wherein the quality-related parameters include one or more of sharpness, resolution, and brightness.
The method according to claim 1, wherein the first image is an RGB image and the second image is an infrared image;

Wherein, based on the image data of the second area, compensating the image data of the first area includes:

If the brightness of the first area of the RGB image is less than a preset brightness threshold, based on the image data of the second area of the infrared image, compensate the brightness of the first area of the RGB image; or,

If the sharpness of the first area of the RGB image is less than a preset sharpness threshold, the sharpness of the first area of the RGB image is compensated based on the image data of the second area of the infrared image.
The method according to claim 1, wherein the first image is an infrared image and the second image is an RGB image;

Wherein, based on the image data of the second area, compensating the image data of the first area includes:

If the first area of the infrared image is a blocked area, the image data of the target object in the first area of the infrared image is compensated based on the image data of the second area of the RGB image; or,

If the resolution of the first region of the infrared image is less than a preset resolution threshold, the first region resolution of the infrared image is compensated based on the image data of the second region of the RGB image.
The method according to claim 1, wherein, based on the image data of the second area, compensating the first area includes:

Fusing the image data of the first area and the second area, and using the fused image data as the compensated image data of the first area.
The method according to claim 1, wherein the first image and the second image are collected and registered by a binocular camera.
The method of claim 1, wherein the target object includes a human face, the first image is a first human face image, and the second image is a second human face image.
The method according to claim 8, wherein acquiring the first image and the second image of the target object comprises:

Collect the first image of the target object;

When it is determined that there is a face in the first image, the first face image and the second face image of the target object are collected synchronously.
The method of claim 8, further comprising:

Identify the compensated first face image and determine the identity information of the target object.
The method according to claim 10, further comprising:

When it is determined that the security level of the business requirements of the target object reaches a preset security level threshold, live detection is performed on the target object.
The method according to claim 11, wherein the biopsy of the target object comprises:

Determining the depth face image of the target object, and performing live detection on the target object based on the depth face image.
The method according to claim 12, wherein the living body detection of the target object based on the depth image comprises:

Fusing the deep face image and the compensated first face image, and performing live detection on the target object based on the fused image.
The method according to claim 11, wherein the biopsy of the target object comprises:

Determine the living body motion of the target object, and determine whether the target object is a living body based on the living body motion.
A face image recognition method, characterized in that it includes:

Acquiring a first face image and a second face image of the target object, where the first face image and the second face image are images collected under different lighting conditions;

Performing quality evaluation processing on the first face image;

If there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold, determine a second area in the second face image corresponding to the first area;

Based on the image data of the second area, the first area is compensated, and the compensated first face image is identified.
The method according to claim 15, wherein before identifying the compensated first face image, further comprising:

Performing quality evaluation processing on the compensated first face image;

If the quality evaluation value of the compensated first face image meets the face recognition quality evaluation requirement, the first face image after compensation is allowed to be recognized.
The method of claim 15, wherein identifying the compensated first face image comprises:

Performing face feature extraction processing on the compensated first face image;

Based on the extracted face features, face recognition processing is performed to determine the identity information of the target object.
An image processing device, characterized in that it includes:

An obtaining module, configured to obtain a first image and a second image of the target object, where the first image and the second image are images collected under different lighting conditions;

An evaluation module, configured to perform quality evaluation processing on the first image;

A determining module, configured to determine a second area in the second image corresponding to the first area if there is a first area in the first image whose quality evaluation value is less than a preset quality evaluation threshold;

The compensation module is used to compensate the first area based on the image data of the second area.
A face image recognition device, characterized in that it includes:

An obtaining module, configured to obtain a first face image and a second face image of the target object, where the first face image and the second face image are images collected under different lighting conditions;

An evaluation module, configured to perform quality evaluation processing on the first face image;

A determining module, configured to determine a second area in the second face image corresponding to the first area if there is a first area in the first face image whose quality evaluation value is less than a preset quality evaluation threshold ;

A compensation module for compensating the first area based on the image data of the second area;

A recognition module is used to recognize the compensated first face image.
A face image recognition device, characterized by comprising: an RGB camera, a black and white camera, an infrared fill light and a processing chip; wherein:

The RGB camera is used to collect RGB face images of the target object;

The infrared fill light is used to emit infrared light to the face of the target object;

The black-and-white camera is used to collect infrared face images of the target object under infrared lighting conditions;

The processing chip is configured to use one of the RGB face image and the infrared face image as a first face image and the other as a second face image, and perform any of claims 15 to 17 A method step.
The device according to claim 20, wherein the RGB camera and the monochrome camera belong to the same binocular camera.
The device according to claim 20, wherein the RGB camera is in a normally open state, and the infrared camera and the infrared fill light process a normally closed state;

The processing chip is also used to determine that there is a face in the RGB face image collected by the RGB camera, and wake up the infrared camera and the infrared fill light to synchronize the infrared camera with the RGB camera Collect a face image of the target object.
The device according to claim 21, characterized in that

The processing chip is also used to turn off the infrared fill light when identifying the identity information of the target object.
The device according to claim 21, further comprising: a structured light emitter in a normally closed state;

The processing chip is also used to determine that the structured light emitter needs to be awakened when it is necessary to perform a live detection on the target object;

The structured light emitter is configured to emit structured light to the face of the target object for the black-and-white camera to collect the depth face image of the target object under structured light illumination conditions;

The processing chip is also used to turn off the structured light emitter when it is determined that the target object passes live detection based on the depth face image.
An electronic device, characterized in that it includes:

Processor; and

A memory arranged to store computer executable instructions which, when executed, causes the processor to perform the steps of the method according to any one of claims 1 to 17.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented .