WO2023160075A1

WO2023160075A1 - Image inpainting method and apparatus, and device and medium

Info

Publication number: WO2023160075A1
Application number: PCT/CN2022/134873
Authority: WO
Inventors: 邵昌旭; 许亮; 李轲
Original assignee: 上海商汤智能科技有限公司
Priority date: 2022-02-28
Filing date: 2022-11-29
Publication date: 2023-08-31
Also published as: CN114565531A

Abstract

Provided in the embodiments of the present disclosure are an image inpainting method and apparatus, and a device and a medium. The method comprises: acquiring a facial image of a target object and an environment image, which comprises the surrounding environment of the target object, wherein the facial image includes a spectacle lens area image of glasses worn by the target object; according to a matching result of the spectacle lens area image and the environment image, determining a light reflection area in the spectacle lens area image; and according to the light reflection area, inpainting the spectacle lens area image, so as to obtain an inpainted target image. By means of the present method, a spectacle lens area image can be inpainted on the basis of a light reflection area, thereby achieving a better light reflection cancellation effect.

Description

Image restoration method, device, equipment and medium

Cross References to Related Applications

This patent application claims priority to Chinese Patent Application No. 202210188238.6 filed on February 28, 2022, which is incorporated herein by reference.

technical field

The present disclosure relates to the technical field of image processing, and in particular to an image restoration method, device, equipment and medium.

Background technique

In many fields that rely on visual algorithms, relevant information can be obtained through face recognition or face detection and analysis. However, if a person wears glasses, it is likely that the face information will be blocked due to the reflection of the lens, which will affect the recognition accuracy and make the algorithm invalid.

Exemplarily, in the smart cabin technology that brings great convenience to people's transportation, the face image in the vehicle can be analyzed to obtain the attributes and states of objects such as drivers or passengers.

For example, by judging whether the driver's eyes are open or closed, it is possible to identify whether the driver is tired.

When the driver wears glasses, the fatigue detection algorithm is likely to output wrong state detection results due to the reflection of the glasses. Missing or false positives about the driver's state will increase driving risks and affect user experience.

Contents of the invention

In view of this, the embodiments of the present disclosure provide at least one image restoration method, device, device and medium.

Specifically, the embodiments of the present disclosure are achieved through the following technical solutions:

In a first aspect, an image restoration method is provided, the method comprising:

Acquiring a face image of the target object and an environment image including the surrounding environment of the target object, the face image including a lens area image of glasses worn by the target object;

determining a reflective area in the lens area image according to a matching result between the lens area image and the environment image;

According to the reflective area, the image of the lens area is repaired to obtain a repaired target image.

In some optional embodiments, the determining the reflective area in the lens area image according to the matching result of the lens area image and the environment image includes: combining the lens area image with the environment image Matching is performed to determine the marked area in the environment image that matches the lens area image; extracting the feature profile of the marked area; using the feature profile of the marked area to segment the lens area image to obtain the reflective areas.

In some optional embodiments, after the facial image of the target object is acquired, the method includes: performing glasses recognition on the facial image, and determining glasses worn by the target object in the facial image Image of the lens area in glasses.

In some optional embodiments, the determining the reflective area in the lens area image according to the matching result of the lens area image and the environment image includes: responding to determining that there is reflection in the lens area image According to the matching result of the lens area image and the environment image, the reflective area in the lens area image is determined.

In some optional embodiments, the determining that there is a reflection phenomenon in the lens area image includes: in response to the existence of an image area that successfully matches the lens area image and the environment image, determining There are reflections.

In some optional embodiments, the determining that there is a reflection phenomenon in the lens area image includes: determining a first area in the lens area image whose pixel brightness value is greater than or equal to a preset brightness threshold; in response to the The area ratio of the first area to the lens area image satisfies a preset area condition, and it is determined that there is a reflection phenomenon in the lens area image.

In some optional embodiments, the determining that there is a reflection phenomenon in the lens area image includes: determining a first area in the lens area image whose pixel brightness value is greater than or equal to a preset brightness threshold; in response to the The eye area in the lens area is blocked by the first area, and it is determined that there is reflection phenomenon in the image of the lens area.

In some optional embodiments, the acquiring the facial image of the target object and the environment image including the surrounding environment of the target object includes: acquiring the facial image of the target object in the vehicle captured by the first camera; acquiring The environment image collected by the second camera, the environment image includes an external environment image of the vehicle.

In some optional embodiments, after the image of the lens region is repaired according to the reflective region to obtain the repaired target image, the method further includes: based on the target image, identifying the The state of the target object.

In a second aspect, an image restoration device is provided, the device comprising:

An image acquisition module, configured to acquire a face image of the target object and an environment image including the surrounding environment of the target object, the face image including a lens area image of glasses worn by the target object;

A reflective area determining module, configured to determine the reflective area in the lens area image according to the matching result of the lens area image and the environment image;

The image processing module is configured to repair the image of the lens area according to the reflective area to obtain a repaired target image.

In some optional embodiments, the reflective area determining module is specifically configured to: match the lens area image with the environment image, and determine a mark in the environment image that matches the lens area image region; extracting the characteristic contour of the marked region; performing region segmentation on the lens region image by using the characteristic contour of the marked region to obtain the reflective region.

In some optional embodiments, after the facial image of the target object is acquired, the image acquisition module is further configured to: perform glasses recognition on the facial image, and determine the An image of the lens area in the glasses worn by the target subject.

In some optional embodiments, the reflective area determining module is specifically configured to: in response to determining that there is a reflective phenomenon in the lens area image, according to the matching result of the lens area image and the environment image, determine the reflective areas in the image of the lens area described above.

In some optional embodiments, when the reflective area determination module is used to determine that there is a reflective phenomenon in the lens area image, it is specifically configured to: respond to the successful matching between the lens area image and the environment image In the image area of the lens, it is determined that there is a reflection phenomenon in the image of the lens area.

In some optional embodiments, when the reflective area determination module is used to determine that there is a reflective phenomenon in the image of the lens area, it is specifically used to: determine that the pixel brightness value in the image of the lens area is greater than or equal to a preset The first area of the brightness threshold; in response to the area ratio of the first area to the lens area image meeting a preset area condition, it is determined that there is a reflection phenomenon in the lens area image.

In some optional embodiments, when the reflective area determination module is used to determine that there is a reflective phenomenon in the image of the lens area, it is specifically used to: determine that the pixel brightness value in the image of the lens area is greater than or equal to a preset A first area of the brightness threshold; in response to the eye area in the lens area being blocked by the first area, it is determined that there is a reflection phenomenon in the image of the lens area.

In some optional embodiments, the image acquisition module is specifically configured to: acquire the face image of the target object in the vehicle captured by the first camera; acquire the environment image captured by the second camera, and the environment image An image of the external environment of the vehicle is included.

In some optional embodiments, the device further includes a state recognition module, configured to, after repairing the lens area image and obtaining the repaired target image: identify the target object based on the target image state.

In a third aspect, an electronic device is provided, the device includes a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to implement the present disclosure when executing the computer instructions The image restoration method described in any one of the embodiments.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the image restoration method described in any embodiment of the present disclosure is implemented.

The image repair method provided by the technical solution of the embodiment of the present disclosure can accurately find the reflective area in the glasses by matching the lens area image in the glasses worn by the target object with the environment image of the surrounding environment, so that the image can be detected based on the position and shape of the reflective area. and other information to repair the image of the lens area, so as to achieve a better effect of weakening or eliminating the reflection in the image of the lens area. The occluded face information of the target object is restored in the repaired target image, which reduces the impact of the reflection on the lens on the algorithm in the subsequent application and improves the accuracy of the subsequent algorithm.

Description of drawings

In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings that need to be used in the descriptions of the embodiments or related technologies. Obviously, the accompanying drawings in the following description The drawings are only some embodiments described in one or more embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

Fig. 1 is a flowchart of an image restoration method shown in at least one embodiment of the present disclosure;

Fig. 2 is a flowchart of another image restoration method shown in at least one embodiment of the present disclosure;

Fig. 3 is a flowchart of another image restoration method shown in at least one embodiment of the present disclosure;

Fig. 4 is a block diagram of an image restoration device shown in at least one embodiment of the present disclosure;

Fig. 5 is a block diagram of another image restoration device shown in at least one embodiment of the present disclosure;

Fig. 6 is a schematic diagram of a hardware structure of an electronic device according to at least one embodiment of the present disclosure.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present specification as recited in the appended claims.

The terms used in this specification are for the purpose of describing particular embodiments only, and are not intended to limit the specification. As used in this specification and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

As shown in FIG. 1, FIG. 1 is a flowchart of an image restoration method shown in at least one embodiment of the present disclosure, and the method may include the following steps:

In step 102, a face image of the target object and an environment image including the surrounding environment of the target object are acquired.

Wherein, the face image includes a lens area image of glasses worn by the target object. In this embodiment, the target object wears glasses, and the lens area image is an image of the area where the lens in the glasses is located.

In this step, the environment image captured by at least one camera can be obtained. When there are multiple environment images, the multiple environment images can be images of environments in different orientations around the target object.

This embodiment does not limit the manner of acquiring the face image of the target object and the environment image of the surrounding environment of the target object. When the target object is in different application scenarios, different acquisition methods may be used. The following are examples of several acquisition methods:

For example, when the target object uses a mobile phone to take a selfie, the face image of the target object can be based on the image collected by the front camera of the mobile phone on the face of the target object, and the environment image can be based on the image captured by the rear camera of the mobile phone in front of the target object. Images collected from the environment. For example, a face image and an environment image of a target object may be acquired based on an image captured by a camera. For another example, when the target object is inside the vehicle, the face image of the target object can be collected based on the camera inside the vehicle, and the environment image outside the vehicle can be collected based on the camera outside the vehicle or facing outside the vehicle.

It should be noted that the collection time of the face image and the environment image may be at the same time or different time. When the target object is an object moving in the environment, the face image and the environment image with the same acquisition time or adjacent acquisition time may be used.

In step 104, the reflective area in the lens area image is determined according to the matching result of the lens area image and the environment image.

When the lens in the glasses worn by the target object has reflective phenomenon, the image of the scene in the surrounding environment reflected by the lens will appear in the lens area image, and the area where the image is located in the lens area image is the reflective area.

In this step, the lens region image can be matched with the environment image to find a region with high similarity in the two images, and then determine the reflective region in the lens region image according to the region with high similarity.

For example, match the lens area image with the environment image. If the similarity between a certain part in the lens area image and a certain area in the environment image meets certain conditions, directly determine this part in the lens area image as reflection area.

It should be noted that when the lens has reflections, when reflecting the surrounding scenery, due to various factors such as the reflection angle, the curvature of the lens, and the material of the lens, generally speaking, the image of the scene in the reflective area is compared with the environment image. The scene image in the image will change in shape, color, etc. When matching the lens area image with the environment image, and looking for an area with high similarity, those skilled in the art can set the similarity condition in consideration of the influence of the above factors. Exemplarily, the similarity condition may be a similarity requirement on the shape of the region, the pixel value, and the like.

In one embodiment, in response to determining that there is a reflection phenomenon in the lens area image, the reflection area in the lens area image is determined according to a matching result between the lens area image and the environment image. In this implementation manner, it may first be determined whether there is a reflection phenomenon in the image of the lens area. When there is a reflection phenomenon, then determine the reflection area in the image of the lens area, so as to reduce unnecessary consumption of computing resources. The following examples illustrate several methods for determining whether there is a reflection phenomenon in the image of the lens area, but it can be understood that the specific implementation is not limited to the following examples:

In an example, it may be determined that there is a reflection phenomenon in the lens area image in response to an image area that successfully matches the lens area image and the environment image. In this example, the image of the lens area is still first matched with the image of the environment. If the similarity between the image of a certain part or all of the image of the lens area and the image of a certain local area of the environment image satisfies the set condition, then it is determined that there is an image area that successfully matches the image of the lens area and the environment image, and the lens area There are reflections in the image. Then the reflective area can be determined according to the successfully matched image area. If the similarity between the image of a certain part or all of the area in the lens area image and the image of a certain local area of the environment image does not meet the set condition, then it is determined that there is no image area that matches successfully between the lens area image and the environment image, There is no reflective phenomenon in the image of the lens area, and subsequent repair processing can no longer be performed.

When the surrounding environment is bright, the image area of the lens due to reflection is likely to have a higher brightness value than other parts without reflection. In another example, it may be determined whether there is a reflection phenomenon in the lens area image according to the pixel brightness value in the lens area image.

For example, according to the brightness values of pixels in the image of the lens area, the first area in the image of the lens area whose pixel brightness value is greater than or equal to a preset brightness threshold may be determined. In response to the area ratio of the first area in the lens area image meeting a preset area condition, it is determined that there is a reflection phenomenon in the lens area image. Combining pixels in the image of the lens area with pixel luminance values greater than or equal to a preset luminance threshold into a first area. When the area ratio of the first area to the lens area image satisfies the preset area condition, for example, when the area ratio of the first area to the lens area image is greater than or equal to 10%, it indicates that there is reflection phenomenon in the lens area image. When the area ratio of the first area to the image of the lens area does not meet the preset area condition, it means that there may be no reflection phenomenon on the lens or the reflection area is too small to ignore the reflection phenomenon. In other examples, it may also be determined directly according to the area size of the first area whether there is a reflection phenomenon in the image of the lens area.

For another example, according to the pixel brightness values in the lens area image, the first area in the lens area image whose pixel brightness values are greater than or equal to a preset brightness threshold may be determined. In response to the eye area in the lens area being blocked by the first area, it is determined that there is a glare phenomenon in the image of the lens area. For the subsequent algorithm of the eye area, if the reflection on the lens does not affect the image of the eye behind the lens, the reflection can be ignored. And if the reflection on the lens blocks the eye area, the reflection cannot be ignored. Combining pixels with pixel brightness values greater than or equal to the preset brightness threshold in the lens area image into the first area, when the first area in the lens area image blocks the eye area, for example, the outline of the eye may be incomplete, the lens area image If the complete eyes of the target object are not detected in the area other than the first area, it is considered that there is a reflection phenomenon in the image of the lens area and the reflection has an impact on the eye area of the target object, and image restoration needs to be continued. When the eye area in the lens area image is not blocked by the first area, it is considered that there is no reflection phenomenon in the lens area image or the reflection phenomenon can be ignored, and no subsequent repair processing is performed.

In step 106, the lens area image is repaired according to the reflective area to obtain a repaired target image.

In this step, according to the information provided by the reflective area, the image restoration technology can be used to repair the reflective area in the image of the lens area, eliminate or weaken the impact caused by the reflection, and obtain the image of the non-reflective lens area, that is, the target after repair image.

In the reflective area, the image reflected by the lens may be superimposed and mixed with the image of the face area of the target object behind the lens, making it difficult to identify the face information of the target object. When repairing the image of the lens area, on the one hand, information such as the structural shape of the reflective area and the edge color of the reflective area can be used to infer the information content of the reflective area, and then the reflective area can be filled. On the other hand, the image information in the area where the reflective area matches the environment image can be used as a reference to repair the reflective area and restore the image behind the lens.

This embodiment does not limit the restoration algorithm specifically used in the above restoration process, for example, an image quality enhancement algorithm, a picture completion algorithm, a super-resolution technology, and the like. In one example, a neural network model for image restoration can be pre-trained, and the reflective area, lens area image, and environment image are input into the neural network model, and an inpainted target image is output. The neural network can predict the face information of the target object in the lens area image by learning the image information of the non-reflective area around the reflective area and learning the batch image samples.

The image repair method provided by the technical solution of the embodiment of the present disclosure can accurately find the reflective area in the glasses by matching the lens area image in the glasses worn by the target object with the environment image of the surrounding environment, so that the image can be detected based on the position and shape of the reflective area. and other information to repair the image of the lens area. In the traditional reflection repair method, since the specific position of the reflection area cannot be determined, the image of the lens area can only be roughly repaired as a whole. Compared with the traditional reflection restoration method, this method better realizes the effect of weakening or eliminating the reflection in the image of the lens area. The occluded face information of the target object is restored in the repaired target image, especially the information of the eye area, which reduces the impact of the reflection on the lens on the algorithm in the subsequent application and improves the accuracy of the subsequent algorithm.

Fig. 2 is another image restoration method provided by at least one embodiment of the present disclosure, and the method may include the following processing, wherein the same steps as the flow in Fig. 1 will not be described in detail again.

In step 202, a face image of the target object and an environment image including the surrounding environment of the target object are acquired.

Wherein, the face image includes a lens area image of glasses worn by the target object. The image restoration method in this embodiment can be used as a preprocessing step of various image recognition algorithms, for example, it can be applied to various cabin vision algorithms.

In step 204, the lens area image is matched with the environment image, and a marked area in the environment image that matches the lens area image is determined.

In this step, the lens area image can be matched with the environment image. If the similarity between a certain part in the lens area image and a certain area in the environment image satisfies certain conditions, then the area in the environment image is determined is the marked area that matches the image of the lens area. The marked area contains scenes in the known environment, such as houses, trees, and so on.

In step 206, the feature contour of the marked area is extracted.

The feature contour is a group or several groups of interconnected curves that outline the scene in the marked area, and these curves are composed of a series of edge points. For example, the marked area includes a tree and an electric pole, and the extracted feature contours include the outline of the above-mentioned tree and the contour of an electric pole.

This embodiment is not limited to the specific manner of extracting the feature contour of the marked area, for example, image segmentation, edge detection and other manners may be used for extraction.

In step 208, the image of the lens area is segmented using the characteristic contour of the marked area to obtain the reflective area.

This embodiment does not limit the specific manner used for region segmentation.

For example, the extracted feature contour can be used as a region mask, and the region mask covers the region within the range of the feature contour. Continuing the above example, the feature contour of the combination of the above-mentioned outline of a tree and the outline of a utility pole can be used as an area mask, or it can be used as two area masks respectively, that is, an area mask corresponding to a tree and an electric pole corresponding to An area mask of . According to the shape of the area covered by the area mask, an area similar to the shape is fitted in the image of the lens area, and the reflective area is obtained by segmentation. When there are multiple area masks, multiple reflective areas can be divided.

For another example, the feature contour can be used as the detection target, and the target detection is performed in the lens region image to obtain the region contour with the highest confidence, and the reflective region is segmented from the lens region image according to the region contour.

It should be noted that, in the method for matching the lens area image and the environment image shown in the previous embodiment to determine the reflective area, if there is a similarity between a certain part in the lens area image and a certain area in the environment image that satisfies Under certain conditions, the part in the image of the lens area is directly determined as the reflective area. Such a method is faster than the method of this embodiment in speed, but the accuracy of the determined reflective area is not as high as that of the method of this embodiment. The reason is that the image presented in the lens area image is a mixture of the image of the environment reflected by the lens and the image of the target subject's face area behind the lens, so that it is difficult to separate the real image from the lens area image when matching. reflective area.

In step 210, the lens area image is repaired according to the reflective area to obtain a repaired target image.

The image restoration method provided by the technical solution of the embodiment of the present disclosure finds the marked area in the environment image that matches the image in the lens area image by matching the lens area image in the glasses worn by the target object with the environment image including the surrounding environment. Since the marked area is an area in the exact and clear environment image, the feature outline extracted based on the marked area will be clearer and more reliable, and closer to the shape of the actual scene in the environment, and the reflection of the lens reflection is exactly the scene in the environment. In this way, the reflective area obtained through feature contour segmentation is closer to the actual reflective area on the lens, so that the image of the lens area can be repaired based on information such as the position and shape of the reflective area. The higher the accuracy of the reflective area, the more reliable the information provided to the restoration algorithm, and the better the restoration and completion effect will be. The occluded face information of the target object is restored in the repaired target image, especially the information of the eye area, which reduces the impact of the reflection on the lens on the algorithm in the subsequent application and improves the accuracy of the subsequent algorithm.

Fig. 3 provides the image restoration method of another embodiment of the present disclosure, and this method can be applied to the field of intelligent cabin, for example, can be by DMS (Driver Monitoring System, driver monitoring system), OMS (Occupancy Monitoring System, passenger monitoring system ), executed by the intelligent driving system or the cloud, etc., including the following processing, wherein the same steps as those shown in Fig. 1 and Fig. 2 will not be described in detail again.

In step 302, the face image of the target object in the vehicle captured by the first camera is acquired.

For example, the first camera may be a camera facing the interior of the vehicle, and the first camera captures images inside the vehicle to obtain an image inside the vehicle, and performs image analysis on the image inside the vehicle to obtain a face image of the target object. Target objects can be drivers, passengers, safety officers, etc. in the vehicle.

In step 304, glasses recognition is performed on the face image, and an image of a lens area in glasses worn by the target object in the face image is determined.

For example, glasses detection may be performed on the face image first, and if glasses are detected, glasses recognition may be further performed on the face image to obtain an image of the lens area in the glasses worn by the target object.

Another example is to directly perform glasses recognition on the facial image, determine whether the target object wears glasses, and obtain an image of the lens area in the glasses worn by the target object when it is determined that the target object wears glasses.

In step 306, the environment image captured by the second camera is acquired, and the environment image includes an external environment image of the vehicle.

For example, the second camera may be a camera facing outside the vehicle, and the second camera captures an environmental image outside the vehicle to obtain an environmental image. The environment image contains data of at least one camera, or data of multiple cameras.

In an example, the method in this embodiment can be used to restore the driver's face image collected by the DMS. Considering that the reflection of the driver's glasses is usually formed by the imaging of the scene in front of or on the side of the vehicle, the second camera can use the vehicle's front-facing camera and/or side-view camera, and the captured environmental images include the front and/or side of the vehicle external environment. Alternatively, a surround-view camera facing the outside of the vehicle can be used, and the captured environment image includes a panoramic image outside the vehicle.

In step 308, the lens area image is matched with the environment image, and a marked area in the environment image that matches the lens area image is determined.

For example, if the image of the image of the lens area matches a certain area of the image outside the vehicle of the environment image, mark the area outside the vehicle as a marked area.

In step 310, the feature contour of the marked region is extracted to obtain a region mask.

The feature contour of the marked region is extracted, and the feature contour is used as a region mask.

In step 312, the region image of the lens region is segmented by using the region mask to obtain the reflective region.

In step 314, the lens area image is repaired according to the reflective area to obtain a repaired target image.

In step 316, the state of the target object is identified based on the target image.

Wherein, the state of the target object may represent the emotional or physical state of the target object, specifically, may include at least one of the following: normal state, fatigue state, and distraction state.

For example, the target image, that is, the image of the repaired lens area, can be input into the state recognition model. The state recognition model can be a pre-trained neural network model, which can recognize the state of the target object based on closed eyes, distance between eyelids, fast blinking speed, gaze direction, and jumping movement.

For another example, the target image may be filled into the facial image to obtain a repaired facial image. The face image is input to the state recognition model, and the state recognition model can combine the features of the eyes and other features of the face, such as yawning of the mouth, changes in facial expressions, etc., to recognize the state of the target object.

In some embodiments, the eye-related state of the target object can be identified based on the repaired target image. Specifically, eye features can be extracted from the target image to identify the direction of sight of the target object or the state of eye opening and closing, and the length of time for the sight of the target object to maintain one direction or the duration of eye closure can be detected according to the video stream to determine whether the target object is Being distracted or fatigued, or determining the target object's level of distraction or fatigue.

In the image repair method in this embodiment, the image can be pre-processed through the lens reflection elimination technology, and then poured into the algorithm module to improve the accuracy and availability of the recognition algorithm. Taking the fatigue detection algorithm as an example, in the case of reflective lenses, the detection accuracy will be significantly reduced due to the reduction of feature information on the image, that is, false positives or missed negatives. The method of this embodiment can segment and restore the reflection of the lens according to the outline of the scene outside the vehicle, restore the key information required by the fatigue monitoring algorithm, and improve the accuracy of the vision algorithm inside the vehicle.

As shown in FIG. 4, the figure is a block diagram of an image restoration device shown in at least one embodiment of the present disclosure, and the device includes:

The image acquisition module 41 is configured to acquire a face image of the target object and an environment image including the surrounding environment of the target object, the face image including a lens area image of glasses worn by the target object.

The reflective area determination module 42 is configured to determine the reflective area in the lens area image according to the matching result of the lens area image and the environment image.

The image processing module 43 is configured to repair the image of the lens area according to the reflective area to obtain a repaired target image.

In one example, the reflective area determination module 42 is specifically configured to: match the lens area image with the environment image, determine a marked area in the environment image that matches the lens area image; extract The characteristic contour of the marked area; using the characteristic contour of the marked area to segment the image of the lens area to obtain the reflective area.

In one example, the image acquisition module 41 is further configured to: perform glasses recognition on the facial image after acquiring the facial image of the target object, and determine the location of the target object in the facial image. Image of the lens area in the worn glasses.

In an example, the reflective area determination module 42 is specifically configured to: determine the lens area according to the matching result between the lens area image and the environment image in response to determining that there is a reflective phenomenon in the lens area image Reflective areas in the image.

In an example, when the reflective area determination module 42 is used to determine that there is a reflective phenomenon in the lens area image, it is specifically configured to: respond to the existence of an image area that successfully matches the lens area image and the environment image , to determine that there is a reflection phenomenon in the image of the lens area.

In one example, when the reflective area determination module 42 is used to determine that there is a reflective phenomenon in the image of the lens area, it is specifically used to: determine that the pixel brightness value in the image of the lens area is greater than or equal to a preset brightness threshold The first area: in response to the area ratio of the first area in the lens area image meeting a preset area condition, determine that there is a reflection phenomenon in the lens area image.

In one example, when the reflective area determination module 42 is used to determine that there is a reflective phenomenon in the image of the lens area, it is specifically used to: determine that the pixel brightness value in the image of the lens area is greater than or equal to a preset brightness threshold First area: in response to the eye area in the lens area being blocked by the first area, determine that there is a reflection phenomenon in the image of the lens area.

In one example, the image acquisition module 41 is specifically configured to: acquire the face image of the target object in the vehicle captured by the first camera; acquire the environment image captured by the second camera, the environment image includes the An image of the vehicle's external environment.

In one example, as shown in FIG. 5 , the device further includes a state recognition module 44, configured to, after repairing the image of the lens area and obtaining the repaired target image: based on the target image, identify the The state of the target object.

For the implementation process of the functions and effects of each module in the above-mentioned device, please refer to the implementation process of the corresponding steps in the above-mentioned method for details, and details will not be repeated here.

An embodiment of the present disclosure also provides an electronic device. As shown in FIG. The device 62 is configured to implement the image restoration method described in any embodiment of the present disclosure when executing the computer instructions.

An embodiment of the present disclosure further provides a computer program product, the product includes a computer program/instruction, and when the computer program/instruction is executed by a processor, the image restoration method described in any embodiment of the present disclosure is implemented.

An embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image restoration method described in any embodiment of the present disclosure is implemented.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in One place, or it can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution in this specification. It can be understood and implemented by those skilled in the art without creative effort.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Other embodiments of the description will readily occur to those skilled in the art from consideration of the specification and practice of the invention claimed herein. This description is intended to cover any modification, use or adaptation of this description. These modifications, uses or adaptations follow the general principles of this description and include common knowledge or conventional technical means in this technical field for which this description does not apply . The specification and examples are to be considered exemplary only, with a true scope and spirit of the specification being indicated by the following claims.

It should be understood that this specification is not limited to the precise constructions which have been described above and shown in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the specification is limited only by the appended claims.

The above descriptions are only preferred embodiments of this specification, and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included in this specification. within the scope of protection.

Claims

A method for image restoration, comprising:

Acquiring a face image of the target object and an environment image including the surrounding environment of the target object, the face image including a lens area image of glasses worn by the target object;

determining a reflective area in the lens area image according to a matching result between the lens area image and the environment image;

According to the reflective area, the image of the lens area is repaired to obtain a repaired target image.
The method according to claim 1, wherein the determining the reflective area in the lens area image according to the matching result of the lens area image and the environment image comprises:

matching the lens area image with the environment image, and determining a marked area in the environment image that matches the lens area image;

extracting a feature profile of the marked region;

The reflective area is obtained by performing area segmentation on the lens area image by using the characteristic contour of the marked area.
The method according to claim 1, characterized in that, after the acquisition of the face image of the target object, the method comprises:

Glasses recognition is performed on the facial image, and an image of a lens area in the glasses worn by the target object in the facial image is determined.
The method according to any one of claims 1-3, wherein the determining the reflective area in the lens area image according to the matching result of the lens area image and the environment image comprises:

In response to determining that there is a reflection phenomenon in the lens area image, a reflection area in the lens area image is determined according to a matching result between the lens area image and the environment image.
The method according to claim 4, wherein the determining that there is a reflection phenomenon in the image of the lens area comprises:

In response to an image area that successfully matches the lens area image and the environment image, it is determined that there is a reflection phenomenon in the lens area image.
The method according to claim 4, wherein the determining that there is a reflection phenomenon in the image of the lens area comprises:

determining a first region in the lens region image whose pixel brightness value is greater than or equal to a preset brightness threshold;

In response to the area ratio of the first area in the lens area image meeting a preset area condition, it is determined that there is a reflection phenomenon in the lens area image.
The method according to claim 4, wherein the determining that there is a reflection phenomenon in the image of the lens area comprises:

determining a first region in the lens region image whose pixel brightness value is greater than or equal to a preset brightness threshold;

In response to the eye area in the lens area being blocked by the first area, it is determined that there is a reflection phenomenon in the image of the lens area.
The method according to any one of claims 1-7, wherein the acquiring the face image of the target object and the environment image including the surrounding environment of the target object includes:

Obtain the face image of the target object in the vehicle captured by the first camera;

Acquire the environment image captured by the second camera, where the environment image includes an external environment image of the vehicle.
The method according to any one of claims 1-8, characterized in that, after repairing the lens region image according to the reflective region to obtain the repaired target image, the method further includes:

A state of the target object is identified based on the target image.
An image restoration device, comprising:

An image acquisition module, configured to acquire a face image of the target object and an environment image including the surrounding environment of the target object, the face image including a lens area image of glasses worn by the target object;

A reflective area determining module, configured to determine the reflective area in the lens area image according to the matching result of the lens area image and the environment image;

The image processing module is configured to repair the image of the lens area according to the reflective area to obtain a repaired target image.
An electronic device, characterized in that the device comprises a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to implement claim 1 when executing the computer instructions to the method described in any one of 9.
A computer-readable storage medium, on which a computer program is stored, wherein the program implements the method according to any one of claims 1 to 9 when the program is executed by a processor.