WO2019080061A1

WO2019080061A1 - Camera device-based occlusion detection and repair device, and occlusion detection and repair method therefor

Info

Publication number: WO2019080061A1
Application number: PCT/CN2017/107875
Authority: WO
Inventors: 谢俊; 赵聪; 杨松龄; 陈爽新
Original assignee: 深圳市柔宇科技有限公司
Priority date: 2017-10-26
Filing date: 2017-10-26
Publication date: 2019-05-02
Also published as: CN110770786A

Abstract

Disclosed in the present application is a camera device-based occlusion detection and repair method, said method comprising the following steps: photographing a first image; detecting whether there is a target object, which is at a preset distance away from a camera unit, within a framing range of the camera unit; if so, calculating an occlusion area of the first image; extracting a first feature point of the first image; acquiring a plurality of second images photographed by the camera unit, extracting from each of the second images, second feature points corresponding to the first feature point; calculating the matching degree between the second feature point of each of the second images and the corresponding first feature point of the first image, selecting a second image, the matching degree of which satisfies a preset matching degree, as a repair source image; and calculating a corresponding position of the occlusion area in the repair source image, using an image of the repair source image corresponding to the occlusion area of the first image to repair the occlusion area of the first image. The present application can effectively perform occlusion detection and image repair, has a good repair effect and a high filming rate.

Description

Occlusion detection and repair device based on camera device and occlusion detection and repair method thereof

Technical field

The present invention relates to an image pickup apparatus, and more particularly to an occlusion detection and repair apparatus based on an image pickup apparatus and an occlusion detection and repair method thereof.

Background technique

The camera quality of electronic devices is one of the main considerations for consumers when they choose to purchase electronic devices. In other words, if an electronic device has excellent image quality, it will become a major selling point of the electronic device. However, when the existing electronic device is photographed, if there is a target object, such as a finger, a stain, or the like, the camera of the electronic device is blocked, the film taken will have a dark area, and the filming rate is low.

Summary of the invention

The embodiment of the invention discloses an occlusion detection and repairing device and an occlusion detection and repairing method thereof, which can detect occlusion and repair the occlusion area of the film, can effectively improve the splicing rate, and the repairing effect is good.

The occlusion detection and repair device based on the imaging device disclosed in the embodiment of the invention. The occlusion detection and repairing device includes: an image capturing unit, the image capturing unit captures a first image; and a detecting module, wherein the detecting module detects whether the camera unit is present in the framing range of the camera unit a target within a preset distance; a memory storing the framing range, the preset distance, and a preset matching degree, the memory further storing a plurality of second images, wherein the plurality of second images are An image captured by the camera unit; and a processor, the camera unit, the detection module, and the memory are respectively electrically connected to the processor, and the processor is configured to: detect in the detection module Calculating an occlusion region of the first image when the target object within the preset distance is within the framing range of the camera unit; and extracting a first feature of the first image Obtaining the plurality of second images, and extracting second feature points corresponding to the first feature points from each of the second images; calculating second feature points of each of the second images With the first image Matching degree between a feature point, and selecting one of the second images whose matching degree satisfies the preset matching degree as a repair source image; and calculating the occlusion region corresponding to the repair source image Location and use the repair source An image of the occlusion region corresponding to the first image in the image repairs an occlusion region of the first image.

The occlusion processing method based on the imaging device disclosed in the embodiment of the present invention. The occlusion processing method includes the steps of: capturing a first image; detecting whether a target object within a preset range of the image capturing unit is within a preset distance; and detecting the image capturing unit Calculating an occlusion region of the first image when a target object within the preset distance is within the framing range; extracting a first feature point of the first image; acquiring a plurality of second images, And extracting, from each of the second images, a second feature point corresponding to the first feature point; calculating a second feature point of each of the second images and a first feature point of the first image a degree of matching between the two, and selecting one of the second images whose matching degree satisfies the preset matching degree as a repair source image; and calculating a position of the occlusion region corresponding to the repair source image, and An occlusion region of the first image is repaired using an image of the occlusion region of the repair source image corresponding to the first image.

A computer readable storage medium, wherein the computer readable storage medium stores a plurality of program instructions, where the program instructions are executed by the processor, and the step of: capturing the first image; detecting the framing of the camera unit Whether there is a target within a predetermined distance from the camera unit in the range; when it is detected that the target object within the preset distance of the camera unit is within the view range of the camera unit, Calculating an occlusion region of the first image; extracting a first feature point of the first image; acquiring a plurality of second images, and extracting, corresponding to the first feature point, from each of the second images a second feature point; calculating a matching degree between the second feature point of each of the second images and the first feature point of the first image, and selecting one of the matching degrees to satisfy the preset matching degree The second image is used as a repair source image; and calculating a position of the occlusion region corresponding to the repair source image, and adopting an image of the occlusion region corresponding to the first image in the repair source image Multiplexing the occlusion region of the first image.

The occlusion detection and repair device and the occlusion detection and repair method of the present invention, when the detection unit detects that the camera unit is occluded, uses the image captured by the camera unit to perform occlusion repair, and can detect occlusion in time to avoid continuation. Occlusion, and even if there is occlusion, it can be repaired in time, improve the filming rate, and the repair effect is good.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings to be used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

FIG. 1 is a structural block diagram of an occlusion detection and repair apparatus according to an embodiment of the present invention.

2 is a schematic diagram of an imaging unit and a detection module of an occlusion detection and repair device according to an embodiment of the invention.

FIG. 3 is a schematic diagram of an imaging unit and a detection module of an occlusion detection and repair device according to another embodiment of the present invention.

4 is a schematic diagram of a minimum shooting distance, a common portion, and a non-common portion when an image is taken when the camera unit is a binocular camera according to an embodiment of the present invention.

FIG. 5 is a schematic diagram of image comparison of the detection module of the occlusion detection and repair device when the detection module is an image detection unit according to an embodiment of the invention.

FIG. 6 is a flowchart of an occlusion detection and repair method according to an embodiment of the present invention.

Figure 7 is a sub-flow diagram of step S602 of Figure 6 in an embodiment.

Figure 8 is a sub-flow diagram of step S602 of Figure 6 in another embodiment.

Detailed ways

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

Please refer to FIG. 1 , which is a structural block diagram of an occlusion detection and repair device 100 based on an imaging device according to an embodiment of the present invention. The occlusion detection repair device 100 is applied to an electronic device. The electronic device includes but is not limited to a camera, a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc. Think of smart helmets, smart glasses and other wearable devices. The occlusion detection and repair device 100 includes a processor 10, a memory 20, an imaging unit 30, and a detection module 40. The memory 20, the camera unit 30, and the detection module 40 are electrically connected to the processor 10, respectively.

The camera unit 30 is configured to take a photo or video for a shooting scene to obtain image or video information of the shooting scene. Specifically, the camera unit 30 is configured to capture a first image. Referring to FIG. 2 together, the camera unit 30 may include at least one camera 31. In some embodiments, the camera unit 30 includes a camera 31. In other embodiments, the camera unit 30 includes two cameras 31, a binocular camera. It can be understood that in other embodiments, the camera unit 30 can include three or more cameras 31, which can be specifically set according to actual needs.

The memory 20 is configured to store a framing range, a preset distance, and a preset matching degree. The framing range is a framing range of the imaging unit 30 at the time of shooting. The preset distance is a preset distance between the camera unit 30 and the target when the target object is photographed. The matching degree is the similarity between the two images. The higher the similarity, the higher the matching degree. Conversely, the lower the similarity, the lower the matching degree. The preset matching degree, that is, the similarity between the two images reaches a predetermined similarity. The memory also stores a plurality of second images. The plurality of second images are images captured by the imaging unit 30.

The detecting module 40 is configured to detect whether there is a target object in the framing range of the image capturing unit 30 or that the camera 31 of the camera unit 30 is to be blocked. The processor 10 is configured to calculate an occlusion region of the first image when the detection module 40 detects the target within the framing range of the camera unit 30.

In some embodiments, the detection module 40 includes an inductive detection unit 41. The sensing detection unit 41 is electrically connected to the processor 10 . The sensing detection unit 41 is configured to generate a sensing signal including occlusion position information when detecting that a target object is about to approach the imaging unit 30 or has partially blocked all or part of the camera 31 of the imaging unit 30. The processor 10 is configured to calculate an occlusion region of the first image according to occlusion position information in the sensing signal.

Referring to FIG. 2 together, in some embodiments, the sensing detection unit 41 includes at least one proximity sensor 411. The proximity sensor 411 can also be a distance sensor. The at least one proximity sensor 411 is disposed within a preset distance range around the camera 31. In another embodiment, the proximity sensor 411 is disposed between the two cameras 31. The processor 10 senses the The position information of the proximity sensor 411 of the target calculates an occlusion area of the first image. Specifically, when the imaging unit 30 includes one camera 31, the at least one proximity sensor 411 is disposed within a preset distance range around the camera 31. When the camera unit 30 includes two cameras 31, the at least one proximity sensor 411 may also be disposed at a position between two adjacent cameras 31.

In some embodiments, referring to FIG. 3 , the sensing detection unit 41 includes a touch module 413 mounted on the at least one camera 31 of the camera unit 30 . It can be understood that the touch module 413 can be a flexible touch unit. The touch module 413 generates an inductive signal including touch position coordinates when sensing that a target object is in contact therewith. The processor 10 calculates an occlusion region of the first image according to the touch position coordinates in the sensing signal.

In some embodiments, the touch module 413 is also disposed within a preset distance range around the camera 31 of the camera unit 30. The touch module 413 generates an inductive signal including touch position coordinates when sensing a target within a preset distance range around the camera 31. The processor 10 determines that the camera unit 30 is about to be blocked according to the touch position coordinates, and issues an occlusion reminder.

In some embodiments, the detection module 40 further includes an imaging detection unit 43. The imaging detecting unit 43 is the imaging unit 30 itself. The imaging unit 30 captures a third image. The third image is an image captured by the imaging unit 30 prior to the first image and stored in the memory 20. It can be understood that the memory 20 also stores a color threshold, a first difference threshold, and a connected number. Wherein, the color threshold is a maximum value of a color of an image captured when the camera 31 is blocked. The first difference threshold is a difference between color values of corresponding positions of the two images. The number of connections is the number of blocks in which the difference is greater than or equal to the first difference threshold and communicates with each other.

Referring to FIG. 4 and FIG. 5 together, the processor 10 is further configured to cut the first image into a plurality of first small blocks M according to a predetermined size, and cut the third image into the predetermined size as a plurality of second small blocks N, wherein each of the first small blocks M corresponds to one of the second small blocks N, that is, the first small block M and the corresponding second small block N are cameras 31 Images taken at the same shooting position. The processor 10 is further configured to calculate a color average of each of the first small block M and each of the second small blocks N. The processor 10 is further configured to determine whether a color average value of each of the first small blocks M is smaller than the color threshold, that is, whether the color of the first small block M is dark. The treatment The device 10 is further configured to determine whether a difference between a color average value of each of the first small blocks M and a color average value of the corresponding second small block N is smaller than the first difference threshold, that is, The first small block M is close to the color of the corresponding second small block N, and the color change is relatively small, and the area may be occluded. The processor 10 is further configured to: when the color average of the first small block M is smaller than the color threshold, and the difference corresponding to the first small block M is smaller than the first difference threshold, The first small piece M is marked. The processor 10 is further configured to determine, when the number of the first small blocks M that are marked and communicated with each other is greater than or equal to the number of connected pieces, determine an area where the first small blocks M are located. The occlusion area forms a dark area. It can be understood that when the two first small blocks M that have been marked are adjacent, the two first small blocks M are in communication. It can be understood that the first image and the third image described above are images taken by the same camera 31 of the imaging unit 30.

In some embodiments, the camera unit 30 is a binocular camera. The third image captured by the camera unit 30 and the first image are images taken by the binocular camera simultaneously for the same shooting scene, and the first image and the third image are followed by a minimum shooting distance. Divided into public and non-public parts. The calculation of the occlusion region of the non-common portion is the same as the above embodiment, that is, the calculation of the occlusion region by the image captured by the camera 31 that captures the first image is performed. However, for the calculation of the occlusion area of the common portion, the following manner can be adopted, which is described in detail below.

The memory 20 also stores a second difference threshold. The processor 10 is further configured to cut a common portion of the first image into a plurality of first small pieces M according to a predetermined size, and cut a common portion of the third image into the second small size according to the predetermined size. Block N, each of the first small blocks M corresponding to one of the second small blocks N, that is, each of the first small blocks M and the corresponding second small blocks N are the binocular cameras An image taken for the same target. The processor 10 is further configured to calculate a color average of each of the first small block M and each of the second small blocks N. The processor 10 is further configured to determine whether a color average value of each of the first small blocks M is smaller than the color threshold, that is, whether the color of the first small block M is dark. The processor 10 is further configured to determine whether a difference between a color average value of each of the first small blocks M and a color average value of the corresponding second small block N is greater than the second difference threshold. That is, the images taken by the binocular camera for the same target have a large difference, and therefore, there may be a case where one of the cameras 31 is blocked. The processor 10 is further configured to: the color average value of the first small block M is smaller than the color threshold, and the difference between the first small block M and the corresponding second small block N is greater than When the second difference threshold is used, the first small block M is marked. The processor 10 is further configured to determine, when the number of the first small blocks M that are marked and communicated with each other is greater than or equal to the number of connected pieces, determine an area where the first small blocks M are located. Occlusion area.

The processor 10 is further configured to repair an occlusion region in the first image.

Specifically, the processor 10 is configured to extract a first feature point of the first image. Preferably, the processor 10 is configured to extract a first feature point of the first image other than the occlusion region. The processor 10 is configured to acquire the plurality of second images and extract a second feature point of each of the second images. It can be understood that the first feature point and the second feature point may be a SIFT (Scale Invariant Feature Transform) feature, a FAST (Features from Accelerated Segment Test) feature, or an ORB (Oriented FAST and Rotated BRIEF) feature. Select at least one according to actual needs.

The processor 10 is configured to calculate a matching degree between a second feature point of each of the second images and a corresponding first feature point of the first image, and select one of the matching degrees to satisfy the pre- The second image of the matching degree is set as the repair source image.

The processor 10 is configured to calculate a position of the occlusion area corresponding to the image in the repair source, and repair an occlusion of the first image by using an image of the occlusion area corresponding to the first image in the repair source image. region. Specifically, the processor 10 is configured to perform matrix transformation on an image corresponding to the occlusion region in the repair source image. The matrix transformation describes a conversion relationship between an image corresponding to the occlusion region and a corresponding pixel of the occlusion region in the repair source image, which is equivalent to a perspective transformation.

In some embodiments, the processor 10 is further configured to perform color adjustment on the repair source image before calculating the position of the occlusion region in the repair source image, so that the repair source image and the The color of the first image is consistent, wherein the color adjustment includes color gamut adjustment, brightness adjustment, and the like. Since the white balance parameter between the first image and the repair source image may be inconsistent, there is a slight difference between the first image and the repair source image, and therefore, before the occlusion region is repaired, Fix the source image for color adjustment to better fix the occlusion area.

In some embodiments, the processor 10 is further configured to select, as the repair source image, the second image whose matching degree satisfies a preset matching degree and the matching degree is the highest.

In some embodiments, when the camera unit 30 is a binocular camera, the processor 10 is further configured to preferentially capture an image captured by the camera 31 of the first image from the binocular camera. Selecting the second image, when the camera 31 that captures the first image in the binocular camera does not satisfy the image of the preset matching degree in the image captured first, another from the binocular camera The second image is selected from images captured by the camera 31.

The processor 10 can be a microcontroller, a microprocessor, a single chip, a digital signal processor, or the like.

The memory 20 can be a computer readable storage medium such as a memory card, a solid state memory, a micro hard disk, an optical disk, or the like. In some embodiments, the memory 20 stores a number of program instructions that can be executed by the processor 10 to perform the aforementioned functions.

Please refer to FIG. 6 , which is a flowchart of a method for detecting occlusion detection in an embodiment of the present invention. The occlusion detection repairing method is applied to the occlusion detection repairing apparatus 100 described above, and the order of execution is not limited to the order shown in FIG. 6. The method includes the steps of:

In step S601, it is detected whether there is a target occlusion or a camera that will block the camera unit 30. If yes, the process proceeds to step S602, otherwise, the process ends. In some embodiments, the sensing detection unit 41 generates an inductive signal including occlusion position information when detecting that a target object is about to approach the imaging unit 30 or has partially blocked all or part of the camera 31 of the imaging unit 30. . The proximity detecting unit 41 may be a proximity sensor (or distance sensor) 411 disposed within a preset distance range of the at least one camera 31 and/or between the at least one camera 31. The sensing detection unit 41 can also be a touch module 413 mounted on at least one camera 31. In other embodiments, the camera unit 30 itself is used to detect whether there is a target that is approaching the camera unit 30 or has partially or completely blocked the camera 31 of the camera unit 30.

Specifically, when it is detected that the target object within the preset distance of the image capturing unit 30 exists in the viewing range of the image capturing unit 30, it is determined that the target object is occluding or about to block the image capturing unit. 30 cameras.

Step S602, calculating an occlusion area of the first image. In some embodiments, the processor 10 is configured to calculate an occlusion region of the first image according to occlusion position information in the sensing signal. In other embodiments, the processor 10 is configured to calculate an occlusion region according to the first image captured by the imaging unit 30 and the third image captured previously.

Step S603, the processor 10 repairs an occlusion area in the first image. Specifically, the processor 10 extracts a first feature point of the first image, acquires the plurality of second images, and And extracting, in each of the second images, a second feature point corresponding to the first feature point. The processor 10 calculates a matching degree between a second feature point of each of the second images and a corresponding first feature point of the first image, and selects one of the matching degrees to satisfy the preset matching The second image of degrees is used as a repair source image. The processor 10 calculates a position corresponding to the occlusion area in the repair source image, and repairs an occlusion area of the first image by using an image of the occlusion area corresponding to the first image in the repair source image.

Please refer to FIG. 7, which is a sub-flowchart of step S602 in some embodiments. As shown in FIG. 5, the step S602 includes:

In step S6021, the processor 10 cuts the first image into a plurality of first small blocks M according to a predetermined size.

Step S6022, the processor 10 acquires a third image, and cuts the third image into a plurality of second small blocks N according to the predetermined size, wherein each of the first small blocks M and one of the The second small block N corresponds. Specifically, the third image is an image captured by the imaging unit 30 prior to the first image and stored in the memory 20. The memory 20 also stores a color threshold, a first difference threshold, and a connected number.

In step S6023, the processor 10 calculates a color average value of each of the first small block M and each of the second small blocks N.

In step S6024, the processor 10 determines whether the average value of the color of each of the first small blocks M is smaller than the color threshold, and if yes, proceeds to step S6025, otherwise, ends.

Step S6025, the processor 10 determines whether the difference between the color average value of each of the first small blocks M and the corresponding color average value of the second small block N is smaller than the first difference threshold. If yes, go to step S6026, otherwise, end.

In step S6026, the processor 10 marks the first small block M.

In step S6027, when the number of the first small blocks M that are marked and connected to each other is greater than or equal to the number of connected, the processor 10 determines that the area where the first small blocks M are located is Occlusion area.

It can be understood that, when the camera unit 30 is a binocular camera, the first image and the third image are images simultaneously captured by the binocular camera for the same shooting scene, the first image and the third image. The image is divided into a common portion and a non-public portion according to a minimum shooting distance. Wherein said The calculation of the occlusion region of the non-common portion is the same as the above embodiment, that is, the calculation of the occlusion region is performed using the image previously captured by the camera 31 that captured the first image. However, for the occlusion area calculation of the common portion, the manner shown in FIG. 6 can be adopted, which is described in detail below.

Please refer to FIG. 8, which is a sub-flowchart in step S602 in other embodiments. As shown in FIG. 6, the step S602 includes:

In step S6021', the processor 10 cuts the common portion of the first image into a plurality of first patches M according to a predetermined size.

Step S6022', the processor 10 cuts the common portion of the third image into a plurality of second small blocks N according to the predetermined size, and each of the first small blocks M and one of the second small blocks N corresponds to each of the first small block M and the corresponding second small block N being an image taken by the binocular camera for the same target.

In step S6023', the processor 10 calculates a color average of each of the first small block M and each of the second small blocks N.

In step S6024', the processor 10 determines whether the average value of the color of each of the first small blocks M is smaller than the color threshold, and if so, proceeds to step S6025', otherwise, ends.

Step S6025', the processor 10 determines whether the difference between the color average value of each of the first small blocks M and the corresponding color average value of the second small block N is greater than the second difference value. The threshold, if yes, proceeds to step S6026', otherwise, ends.

In step S6026', the processor 10 marks the first small block M.

Step S6027', when the number of the first small blocks M that are marked and connected to each other is greater than or equal to the number of connected, the processor determines that the area where the first small blocks M are located is Occlusion area.

Wherein, when a plurality of program instructions are stored in the memory 40, the plurality of program instructions are used by the processor 10 to perform execution to perform the steps in any of the methods of FIGS. 6-8.

In some embodiments, the present invention further provides a computer readable storage medium having stored therein a plurality of program instructions, which are executed by the processor 10 for execution, and are executed in FIG. 6-8. Any method step of calculating an occlusion region in the first image and repairing an occlusion region in the first image. In some embodiments, the computer storage medium is the memory 20, and may be any storable information such as a memory card, a solid state memory, a micro hard disk, an optical disk, or the like. Storage device.

Therefore, the occlusion detection and repair device based on the imaging device of the present invention and the occlusion detection and repair method thereof can perform occlusion detection when capturing an image, repair the occlusion region of the captured image, improve the splicing rate, and have a good repair effect. .

The above is a preferred embodiment of the present invention, and it should be noted that those skilled in the art can also make several improvements and retouchings without departing from the principles of the present invention. It is the scope of protection of the present invention.

Claims

An occlusion detection and repair device based on an imaging device, wherein the occlusion detection and repair device comprises:

a camera unit that captures a first image;

a detecting module, the detecting module detecting whether there is a target object within a preset range of the camera unit within a preset range of the camera unit;

a memory that stores the framing range, the preset distance, and a preset matching degree, the memory further storing a plurality of second images, the plurality of second images being images captured by the camera unit;

The processor, the camera unit, the detection module, and the memory are respectively electrically connected to the processor, and the processor is configured to:

Calculating an occlusion region of the first image when the detecting module detects that a target object within the preset distance of the image capturing unit exists within the viewing range of the image capturing unit;

Extracting a first feature point of the first image;

Obtaining the plurality of second images, and extracting, from each of the second images, a second feature point corresponding to the first feature point;

Calculating a matching degree between the second feature point of each of the second images and the first feature point of the first image, and selecting one of the second matching points that satisfies the second matching degree Image as a repair source image; and

Calculating a position of the occlusion area corresponding to the repair source image, and repairing an occlusion area of the first image by using an image of the occlusion area corresponding to the first image in the repair source image.
The occlusion detection and repair device of claim 1 , wherein the detection module comprises an inductive detection unit, the inductive detection unit is electrically connected to the processor, and the inductive detection unit is When detecting that a target object is about to approach the camera unit or has partially or completely blocked the camera unit, generating a sensing signal including occlusion position information, and the processor according to the occlusion position information in the sensing signal An occlusion region of the first image is calculated.
The occlusion detection and repair device according to claim 2, wherein the imaging unit comprises at least one camera, the sensing detection unit comprises at least one proximity sensor or distance sensor, and the at least one proximity sensor or distance sensor is configured. The processor calculates an occlusion region of the first image according to position information of the proximity sensor or the distance sensor that senses the target within a preset distance range around the at least one camera.
The occlusion detection and repair device of claim 2, wherein the camera unit comprises at least one camera, and the sensor detection unit comprises a touch module mounted on the at least one camera, the touch module A sensing signal including touch position coordinates is generated when the target object is sensed, and the processor calculates an occlusion region of the first image according to the touch position coordinates in the sensing signal.
The occlusion detection and repair device of claim 4, wherein the touch module is further disposed within a preset distance range around the at least one camera, and the touch module senses the target in the The sensing signal including the coordinates of the touch position is generated when the preset distance is within the range of the camera. The processor determines that the camera unit is about to be blocked according to the touch position coordinate, and sends an occlusion reminder.
The occlusion detection and repair device according to claim 1, wherein the imaging unit captures a third image, and the third image is an image captured by the imaging unit prior to the first image, the memory The color threshold, the first difference threshold, and the connected number are also stored, and the processor is further configured to:

Cutting the first image into a plurality of first patches according to a predetermined size;

Cutting the third image into a plurality of second patches according to the predetermined size, each of the first patches corresponding to one of the second patches;

Calculating a color average of each of the first small block and each of the second small blocks;

Determining whether a color average value of each of the first small blocks is smaller than the color threshold, and a difference between a color average value of each of the first small blocks and a color average value of the corresponding second small block Whether the value is less than the first difference threshold;

When the color average of the first small block is smaller than the color threshold, and the difference corresponding to the first small block is smaller than the first difference threshold, marking the first small block; and

The number of the first small blocks that have been marked and communicated with each other is greater than or equal to the connectivity When the number is determined, it is determined that the area where the first small block is located is an occlusion area.
The occlusion detection and repair device according to claim 1, wherein the imaging unit is a binocular camera, the imaging unit captures a third image, and the first image and the third image are the binocular The two cameras of the camera respectively respectively capture images at the same time for the same shooting scene, the first image and the third image are divided into a common portion and a non-common portion according to a minimum shooting distance; the memory also stores a color threshold The second difference threshold and the number of connected, the processor is further configured to:

Cutting a common portion of the first image into a plurality of first patches according to a predetermined size;

Cutting a common portion of the third image into a plurality of second patches according to the predetermined size, each of the first patches corresponding to one of the second patches;

Calculating a color average of each of the first small block and each of the second small blocks;

Determining whether a color average value of each of the first small blocks is smaller than the color threshold, and a difference between a color average value of each of the first small blocks and a color average value of the corresponding second small block Whether the value is greater than the second difference threshold;

When the average value of the color of the first small block is smaller than the color threshold, and the difference corresponding to the first small block is greater than the second difference threshold, marking the first small block; and

When the number of the first small blocks that are marked and connected to each other is greater than or equal to the number of connected pieces, it is determined that the area where the first small blocks are located is an occlusion area.
The occlusion detection repairing apparatus according to claim 1, wherein the processor is further configured to: perform color adjustment on the repair source image before calculating the position of the occlusion region corresponding to the repair source image, so that The repair source image is consistent with the color of the first image, wherein the color adjustment includes color gamut adjustment and brightness adjustment.
The occlusion detection and repair apparatus according to claim 1, wherein the processor is further configured to select the second image whose matching degree satisfies a preset matching degree and the matching degree is the highest as the repair source. image.
The occlusion detection and repair device according to claim 1, wherein when the camera unit is a binocular camera, the processor is further configured to preferentially capture the camera of the first image from the binocular camera. Selecting the second image from the image captured first, and when the image captured by the camera of the first image in the binocular camera does not satisfy the preset matching degree, The second image is selected from an image captured by another camera of the binocular camera.
An occlusion processing method based on an imaging device, characterized in that the occlusion processing method comprises the steps of:

Taking the first image;

Detecting whether there is a target within a preset range of the camera unit within a preset distance from the camera unit;

Calculating an occlusion region of the first image when detecting that the target object within the preset distance is within the framing range of the camera unit;

Extracting a first feature point of the first image;

Acquiring a plurality of second images, and extracting, from each of the second images, a second feature point corresponding to the first feature point;

Calculating a matching degree between the second feature point of each of the second images and the first feature point of the first image, and selecting one of the second matching points that satisfies the second matching degree Image as a repair source image; and

Calculating a position of the occlusion area corresponding to the repair source image, and repairing an occlusion area of the first image by using an image of the occlusion area corresponding to the first image in the repair source image.
The occlusion processing method according to claim 11, wherein the occlusion processing method further comprises the steps of:

A sensing signal including occlusion position information is generated when detecting that a target object is approaching the camera unit or has partially blocked all or part of the camera unit; and

And calculating an occlusion region of the first image according to the occlusion position information in the sensing signal.
The occlusion processing method according to claim 11, wherein the image capturing unit comprises at least one camera, and the occlusion processing method further comprises the steps of:

At least one proximity sensor or distance sensor is disposed within a preset distance range around the at least one camera;

Generating an inductive signal including occlusion position information when the at least one proximity sensor or distance sensor detects that a target is approaching the at least one camera or has partially or completely blocked the at least one camera; and

And calculating an occlusion region of the first image according to the occlusion position information in the sensing signal.
The occlusion processing method according to claim 12, wherein the image capturing unit comprises at least one camera, and the occlusion processing method further comprises the steps of:

a touch module mounted on the at least one camera of the camera unit generates a sensing signal including coordinates of a touch position when sensing that a target object is in contact therewith; and

And calculating an occlusion region of the first image according to the touch position coordinates in the sensing signal.
The occlusion processing method according to claim 14, wherein the occlusion processing method further comprises the steps of:

a touch module installed within a preset distance range around the at least one camera of the camera unit generates a sensing signal including touch position coordinates when sensing a target object; and

The camera unit is determined to be occluded according to the touch position coordinates, and an occlusion reminder is issued.
The occlusion processing method according to claim 11, wherein the occlusion processing method further comprises the steps of:

Taking a third image, where the third image is an image captured by the imaging unit prior to the first image;

Cutting the first image into a plurality of first patches according to a predetermined size;

Cutting the third image into a plurality of second patches according to the predetermined size, each of the first patches corresponding to one of the second patches;

Calculating a color average of each of the first small block and each of the second small blocks;

Determining whether a color average value of each of the first small blocks is smaller than the color threshold, and a difference between a color average value of each of the first small blocks and a color average value of the corresponding second small block Whether the value is less than the first difference threshold;

When the color average of the first small block is smaller than the color threshold, and the difference corresponding to the first small block is smaller than the first difference threshold, marking the first small block; and

When the number of the first small blocks that are marked and connected to each other is greater than or equal to the number of connected pieces, it is determined that the area where the first small blocks are located is an occlusion area.
The occlusion processing method according to claim 11, wherein the image capturing unit is a binocular camera, and the occlusion processing method further comprises the steps of:

The image capturing unit captures a third image, where the first image and the third image are images respectively captured by two cameras of the binocular camera at the same time for the same shooting scene, the first image and the image The third image is divided into a common portion and a non-public portion according to a minimum shooting distance;

Cutting a common portion of the first image into a plurality of first patches according to a predetermined size;

Cutting a common portion of the third image into a plurality of second patches according to the predetermined size, each of the first patches corresponding to one of the second patches;

Calculating a color average of each of the first small block and each of the second small blocks;

Determining whether a color average value of each of the first small blocks is smaller than a color threshold, and whether a difference between a color average value of each of the first small blocks and a color average value of the corresponding second small block is Greater than the second difference threshold;

When the average value of the color of the first small block is smaller than the color threshold, and the difference corresponding to the first small block is greater than the second difference threshold, marking the first small block; and

When the number of the first small blocks that are marked and connected to each other is greater than or equal to the number of connected pieces, it is determined that the area where the first small blocks are located is an occlusion area.
The occlusion processing method according to claim 11, wherein before the calculating the position of the occlusion region in the restoration source image, the occlusion processing method further comprises the steps of:

The repair source image is color-adjusted such that the repair source image is consistent with the color of the first image, wherein the color adjustment includes color gamut adjustment and brightness adjustment.
The occlusion processing method according to claim 11, wherein the image capturing unit is a binocular camera, and the occlusion processing method further comprises the steps of:

Selecting the second image from the image captured by the camera that captures the first image from the binocular camera preferentially; and

Selecting from an image taken by another camera of the binocular camera when an image of the preset matching degree is not included in the image captured by the camera of the first image in the binocular camera The second image is described.
A computer readable storage medium storing a plurality of program instructions, wherein the program instructions are executed by a processor, and the steps are:

Taking the first image;

Detecting whether there is a target within a preset range of the camera unit within a preset distance from the camera unit;

Calculating an occlusion region of the first image when detecting that the target object within the preset distance is within the framing range of the camera unit;

Extracting a first feature point of the first image;

Acquiring a plurality of second images, and extracting, from each of the second images, a second feature point corresponding to the first feature point;

Calculating a matching degree between the second feature point of each of the second images and the first feature point of the first image, and selecting one of the second matching points that satisfies the second matching degree Image as a repair source image; and

Calculating a position of the occlusion area corresponding to the repair source image, and repairing an occlusion area of the first image by using an image of the occlusion area corresponding to the first image in the repair source image.