CN107705334B - Camera abnormity detection method and device - Google Patents
Camera abnormity detection method and device Download PDFInfo
- Publication number
- CN107705334B CN107705334B CN201710742191.2A CN201710742191A CN107705334B CN 107705334 B CN107705334 B CN 107705334B CN 201710742191 A CN201710742191 A CN 201710742191A CN 107705334 B CN107705334 B CN 107705334B
- Authority
- CN
- China
- Prior art keywords
- image
- abnormal
- semantic segmentation
- camera
- segmentation model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Abstract
The invention discloses a camera abnormity detection method and a device, which detect whether a camera is abnormal in real time through a computer vision method and give an alarm when the camera is detected to be abnormal. The method comprises the following steps: training a neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in an image, wherein the type comprises an abnormal type and a normal type; inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image; and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a camera abnormity detection method and device.
Background
In an application scene of a vision-based unmanned vehicle, a robot, or the like, a camera mounted on an autonomous vehicle, an advanced assistant driving vehicle, a robot, or the like generally captures an environment around the vehicle, and a decision control unit makes a decision based on an image captured by the camera and controls the autonomous vehicle, the advanced assistant driving vehicle, the robot, or the like. Therefore, the quality of the image shot by the camera plays an important role in the decision of the decision control unit.
However, in practical applications, the camera may have poor quality of the captured image due to some abnormal situations, which may affect the accuracy of the decision making by the decision control unit, for example: the camera lens has insect corpses, viscous liquid is stained, raindrops and abnormal conditions such as overexposure or underexposure of images caused by problems of an exposure device of the lens. As shown in fig. 1A, 1B, 1C, 1D, 1E, and 1F, the abnormal images captured by the camera are respectively an opaque stain image, a raindrop image, a transparent stain image, an insect liquid image, an overexposed image, and an underexposed image. Therefore, how to detect the camera abnormality in real time and alarm becomes a technical problem to be solved urgently.
Disclosure of Invention
In view of the above problems, the present invention provides a camera abnormality detection method and device, which detects whether an abnormality occurs in a camera in real time by a computer vision method and alarms when an abnormality occurs.
In one aspect, an embodiment of the present invention provides a method for detecting camera anomalies, where the method includes:
training a neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in an image, wherein the type comprises an abnormal type and a normal type;
inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;
and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.
The embodiment of the present invention further provides a camera anomaly detection device, including:
the training unit is used for training the neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in the image, wherein the type comprises an abnormal type and a normal type;
the information acquisition unit is used for inputting the image shot by the camera into the semantic segmentation model to obtain the characteristic information corresponding to the image;
and the abnormality detection unit is used for judging whether the camera is abnormal according to the characteristic information corresponding to the image and giving an alarm when the abnormality is determined.
According to the embodiment of the invention, on one hand, whether the camera is abnormal or not can be detected in real time, and an alarm can be given when the camera is abnormal, so that an operator can know the abnormality of the camera in time and process the camera in time; on the other hand, the neural network is trained through the abnormal image set and the normal image set, so that the robustness and the accuracy of the semantic segmentation model obtained through training can be improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention.
FIGS. 1A to 1F are abnormal images photographed by a camera;
FIG. 2 is a flowchart of a method for detecting camera shooting anomalies according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a semantic segmentation model obtained by training a neural network according to an embodiment of the present invention;
FIG. 4 is a second schematic diagram of a semantic segmentation model obtained by training a neural network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram illustrating an embodiment of determining whether a camera is abnormal according to feature information corresponding to consecutive multi-frame images;
FIGS. 6A-6C are process diagrams of an embodiment of the method of FIG. 5;
FIGS. 7A-7C are process diagrams of another embodiment of the method of FIG. 5;
FIG. 8 is a second schematic diagram illustrating a determination of whether a camera is abnormal according to feature information corresponding to consecutive multi-frame images according to an embodiment of the present invention;
FIG. 9 is a schematic diagram illustrating an embodiment of determining whether a camera is abnormal according to feature information corresponding to a frame of image;
FIG. 10 is a second schematic diagram illustrating a method for determining whether a camera is abnormal according to feature information corresponding to a frame of image according to an embodiment of the present invention;
fig. 11 is a schematic structural diagram of an imaging abnormality detection apparatus according to an embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The camera abnormity detection method provided by the embodiment of the invention is mainly suitable for automatic driving vehicles, advanced auxiliary driving vehicles, robots and other artificial intelligent equipment.
Example one
Referring to fig. 2, a flowchart of a camera anomaly detection method according to an embodiment of the present invention is shown, where the method includes:
In a first example, the set of abnormal images includes a transparent dirty image and an opaque dirty image, the normal type is no dirty, and the abnormal type is dirty.
In a second example, the set of abnormal images includes a transparent defacement image and an opaque defacement image, the normal type is no defacement, and the abnormal type includes a transparent defacement and an opaque defacement.
In a third example, the abnormal image set includes an overexposed image and an underexposed image, the abnormal type includes overexposure and underexposure, and the normal type is normal exposure.
As a fourth example, the set of abnormal images includes a transparent defacement image, an opaque defacement image, an exposed image, and an underexposed image, the normal types include normal exposure and no defacement, and the abnormal types include defacement, no defacement, overexposure, and underexposure.
As a fifth example, the set of abnormal images includes a clear defacement image, an opaque defacement image, an exposed image, and an underexposed image, the normal types include normal exposure and no defacement, and the abnormal types include clear defacement, opaque defacement, no defacement, overexposure, and underexposure.
In the embodiment of the present invention, the abnormal image set, the normal image set, the abnormal type, and the normal type are not limited to the first to fifth examples, and are not exhaustive.
In the embodiment of the invention, the abnormal image set can be obtained by respectively acquiring images through cameras of the same type (namely, cameras of the same type which are judged to be abnormal or not) with lenses containing different pollutants in advance. The normal image set can be obtained by acquiring images through cameras of the same type without defilements on the lens in advance. This application is not described in detail.
The neural network used for training the semantic segmentation model can be any neural network model with a semantic segmentation function, and the application is not strictly limited.
And 203, judging whether the camera is abnormal or not according to the characteristic information corresponding to the image, and alarming when the abnormality is determined.
In an embodiment, the foregoing step 201 may be specifically implemented by the process shown in fig. 3, and includes:
Compared with the abnormal image, the normal image has lower acquisition difficulty, and in order to improve the training efficiency of the semantic segmentation model, only the normal image which is output after passing through the primary semantic segmentation model and is detected wrongly is added into the abnormal image set.
In the embodiment of the invention, after an initial semantic segmentation model is obtained through training of an abnormal image set, the initial semantic segmentation model is tested through a normal image set, an image (namely a positive sample) which is detected in error is selected, a label of a normal type is marked on each pixel of the image, and the label is added into the abnormal image set; and training the initial semantic segmentation model based on the new abnormal image set to obtain the semantic segmentation model, wherein the aim of the training is to improve the robustness of the semantic segmentation model and reduce the error detection rate of the semantic segmentation model by adding a positive sample for error detection.
Preferably, to further improve the robustness and accuracy of the trained semantic segmentation model, one skilled in the art may also derive an alternative as shown in fig. 4 to implement step 201 based on the teaching of the flowchart shown in fig. 3, where the method includes:
Step 201C ', determining the number of normal images with wrong semantic segmentation, judging whether a preset iteration stopping condition is met or not according to the number, if not, executing step 201D ', and if so, executing step 201F '.
In this embodiment of the present invention, the condition for stopping iteration may be that the number is smaller than a preset number threshold, or that a ratio of the number to the total number of normal images input into the initial semantic segmentation model is smaller than a preset proportion threshold. The present application is not limited strictly, and those skilled in the art can flexibly set the setting according to actual requirements.
In the embodiment of the present invention, the specific implementation of the foregoing step 202 to step 203 can be implemented by, but not limited to, any of the following modes (mode 1 to mode 2):
and 203, judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame images, and alarming when the abnormality is determined.
In practical application, if the camera is abnormal, the images continuously shot within a period of time are all abnormal, so that the method 1 combines the characteristic information corresponding to the continuous multi-frame images to judge whether the camera is abnormal more accurately.
In one embodiment, the feature information is a type to which each pixel output by the semantic segmentation model softmax layer belongs; that is, after an image is input into a semantic segmentation model, the softmax layer of the semantic segmentation model outputs the type to which each pixel belongs as feature information. In mode 1, in step 203, the specific implementation of determining whether the camera is abnormal according to the feature information corresponding to the multiple frames of images may be as follows: simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain a semantic segmentation model; and when the semantic segmentation result graph contains pixels with abnormal types, the number of the pixels is judged to be larger than or equal to a preset number threshold value, and the camera is determined to be abnormal. The quantity threshold value is determined according to an empirical value, for example, according to an pollutant causing an abnormal type, for example, through experiments, pollutant areas with different sizes are caused on a camera lens by pollutants with different sizes, the influence degree of an image obtained by the camera under the pollutant areas with different sizes on a decision making control unit is judged, and the quantity threshold value is determined according to a lossless area with the minimum size, wherein the influence degree reaches a preset influence degree. As shown in fig. 5, after the N consecutive frame images P1, P2, …, PN are input to the semantic segmentation model, corresponding N semantic segmentation results G1, G2, …, GN are output from the softmax layer of the semantic segmentation model, and a semantic segmentation result map is obtained from the semantic segmentation results G1, G2, …, GN. Two specific examples are described below.
Example 1, assume that an abnormal image set includes a transparent dirty image and an opaque dirty image, the normal type is no dirty, the abnormal type is dirty, and the semantic segmentation model labels each pixel in the input image in a category, for example, if the type to which the pixel belongs is no dirty, the corresponding category label is 0, and if the type to which the pixel belongs is dirty, the corresponding category label is 1. Taking 4-frame images of 4 × 4 size continuously captured by a camera as an example, the 4-frame images are respectively represented by P1, P2, P3 and P4, and as shown in fig. 6A, the pixels in P1, P2, P3 and P4 are numbered as x1, x2, … and x16 sequentially from left to right and from top to bottom. Respectively inputting P1, P2, P3 and P4 into a semantic segmentation model to obtain a semantic segmentation result shown in FIG. 6B, and virtualizing pixels at the same positions (namely, the same numbers) in P1, P2, P3 and P4 into one pixel to obtain virtual pixels x1 ', x2 ', … and x16 '; and for each virtual pixel, determining the type of the virtual pixel according to the types of 4 pixels of the virtualized virtual pixel. For example, in the mode 1, the number N1 of the type 0 and the number N2 of the type 1 of the 4 pixels are counted, and the type corresponding to the number of values greater than a preset value (the value is generally set to be 1/2 of N, taking the continuous N-frame image as an example, and when the number of 0 of 10 x1 pixels in the 10-frame image exceeds 5, the type 0 of the corresponding virtual pixel is determined) in N1 and N2 is taken as the type to which the virtual pixel belongs, in the mode 2, the type with the larger values of N1 and N2 is taken as the type of the virtual pixel, taking the pixel x1 ' as an example, and the types to which the four pixels corresponding to the pixel x1 ' belong are 0001 respectively, so that the type to which the pixel x1 ' belongs is 0 (i.e., there is no contamination). Thus, a semantic segmentation result graph as shown in fig. 6C is obtained.
Example 2, assuming that the abnormal image set includes a transparent dirty image, an opaque dirty image, an underexposed image, and an overexposed image, where the normal types are no dirt and normal exposure, and the abnormal types are dirt, overexposure, and underexposed, the semantic segmentation model labels a category for each pixel in the input image, for example, if the type is no dirt, the category label is 0, if the type is dirt, the category label is 1, if the type is overexposed, the category label is 2, and if the type is underexposed, the category label is 4. Taking a 6-frame image with a size of 4 × 4 continuously captured by a camera as an example, the 6-frame image is represented by P1, P2, P3, P4, P5 and P6, and as shown in fig. 7A, the pixels in the P1, P2, P3, P4, P5 and P6 are numbered as x1, x2, … and x16 sequentially from left to right and from top to bottom. Respectively inputting P1, P2, P3, P4, P5 and P6 into a semantic segmentation model to obtain a semantic segmentation result shown in FIG. 7B, and virtualizing pixels at the same positions (namely, the same numbers) in P1, P2, P3, P4, P5 and P6 into one pixel to obtain virtual pixels x1 ', x2 ', … and x16 '; for each virtual pixel, determining the type to which the virtual pixel belongs according to the type to which 6 pixels of the virtual pixel belong, for example, counting the number N1 of the type 0, the number N2 of the type 1, the number N3 of the type 2, and the number N4 of the type 3 in the 6 pixels, taking the type with a large value in N1, N2, N3, and N4 as the type of the virtual pixel, taking the pixel x1 ' as an example, the types to which the 6 pixels corresponding to the pixel x1 ' belong are 0, 1, 2, and 3, respectively, so that the type to which the pixel x1 ' belongs is 1 (i.e., there is a stain), thereby obtaining the segmentation semantic result graph shown in fig. 7C.
In another embodiment, the feature information is a feature map output by a neural network convolution layer previous to the softmax layer in the semantic segmentation model; namely, after the image is input into the semantic segmentation model, the feature map output by the neural network convolution layer which is the layer before the softmax layer of the semantic segmentation model is used as feature information. In mode 1, the specific implementation of determining whether the camera is abnormal according to the feature information corresponding to the multiple frames of images in step 203 may be as follows: merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain the result of whether the camera is abnormal. In the embodiment of the present invention, after obtaining the semantic segmentation model through training of the abnormal image set and the normal image set, the neural network model is trained according to the sample image (the sample image may be an image in the abnormal image set, or may not be an image in the abnormal image set, which is not strictly limited in the present application) and the trained semantic segmentation model to obtain the first post-processing model, that is, the semantic segmentation model remains unchanged during the training process. Or, the neural network model for training the semantic segmentation model and the neural network model for training the first post-processing model may be jointly trained through the abnormal image set and the normal image set to obtain the semantic segmentation model and the first post-processing model. The training mode may be an existing neural network training mode, such as a gradient descent mode, and the present application is not limited thereto. The neural network model used for training the post-processing model may be a full-connection layer network or a convolution layer network, and the present application is not limited strictly. Specifically, as shown in fig. 8, after the multiple frames of images P1, P2, …, and PN are input into the semantic segmentation model, N feature maps T1, T2, …, and TN corresponding to the multiple frames of images are output from the neural network convolution layer that is one layer before the softmax layer of the semantic segmentation model, and feature maps T1, T2, …, and TN are combined into feature information of one three-dimensional space according to a preset order and input into the first post-processing model, so as to obtain result information of whether the camera is normal. Of course, a person skilled in the art may also use a feature map output by any neural network convolution layer before the softmax layer in the semantic segmentation model as the feature information in the embodiment of the present invention, and obtain the semantic segmentation result map according to the manner shown in fig. 8, which is not limited in this application.
The preset sequence can be the time sequence of the collected multiple frames of images. The preset sequence is the same as the sequence adopted by the pre-training post-processing model.
and step 203, judging whether the camera is normal or not according to the characteristic information corresponding to the image.
The mode 2 can realize that the camera abnormity judgment is carried out once when one frame of image is shot, the camera abnormity judgment is carried out again after continuous multi-frame images are shot, and compared with the mode 1, the mode 2 has stronger real-time performance. The skilled person can select the mode 1 or the mode 2 according to the actual requirement, and the application is not limited strictly.
In one embodiment, the feature information is a type to which each pixel output by the semantic segmentation model softmax layer belongs; that is, after an image is input into a semantic segmentation model, the softmax layer of the semantic segmentation model outputs the type to which each pixel belongs as feature information. In a mode 2, in the step 203, determining whether the camera is abnormal according to the feature information corresponding to the image includes: and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal. The quantity threshold is determined empirically, for example, based on the type of insult causing the anomaly. For example, through experiments, the stained areas with different sizes are caused to the camera lens by the stained objects with different sizes, the influence degree of the image obtained by the camera under the stained areas with different sizes on the decision making of the decision making control unit is judged, and the quantity threshold value is determined according to the smallest-sized lossless area with the influence degree reaching the preset influence degree. As shown in fig. 9, after the image P is input into the semantic segmentation model, the semantic segmentation result G corresponding to the image P is output from the softmax layer of the semantic segmentation model, and the result information of whether the camera is normal is obtained from the semantic segmentation result G.
In another embodiment, the feature information is a feature map output by a neural network convolution layer previous to the softmax layer in the semantic segmentation model; namely, after the image is input into the semantic segmentation model, the feature map output by the neural network convolution layer which is the layer before the softmax layer of the semantic segmentation model is used as feature information. In mode 2, the determining, in the step 203, whether the camera is abnormal according to the feature information corresponding to the image specifically includes: and inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain the result of whether the camera is abnormal or not. In the embodiment of the present invention, after the semantic segmentation model is obtained by training the abnormal image set and the normal image set, the neural network model is trained according to the sample image (the sample image may be an image in the abnormal image set, or may not be an image in the abnormal image set, which is not strictly limited in the present application) and the trained semantic segmentation model to obtain the second post-processing model, that is, the semantic segmentation model is kept unchanged during the training process. Or, the neural network model for training the semantic segmentation model and the neural network model for training the second post-processing model may be jointly trained through the abnormal image set and the normal image set to obtain the semantic segmentation model and the second post-processing model. The training mode may be an existing neural network training mode, such as a gradient descent mode, and the present application is not limited thereto. The neural network model used for training the post-processing model may be a full-connection layer network or a convolution layer network, and the present application is not limited strictly. Specifically, as shown in fig. 10, after the image P is input into the semantic segmentation model, the corresponding feature map T is output from the neural network convolution layer that is one layer before the softmax layer of the semantic segmentation model, and the feature map T is input into the second post-processing model, so as to obtain result information on whether the camera is normal. Of course, a person skilled in the art may also use a feature map output by any neural network convolution layer before the softmax layer in the semantic segmentation model as the feature information in the embodiment of the present invention, and the present application is not limited thereto.
On one hand, the camera abnormity detection method provided by the embodiment of the invention can realize the real-time detection of whether the camera is abnormal or not and alarm when the camera is abnormal, so that an operator can know the abnormity of the camera in time and process the camera in time; on the other hand, the neural network is trained through the abnormal image set and the normal image set, so that the robustness and the accuracy of the semantic segmentation model obtained through training can be improved.
Example two
Based on the same concept of the camera anomaly detection method provided by the first embodiment, the second embodiment of the present invention provides a camera anomaly detection device, the structure of which is shown in fig. 11, and the device comprises a training unit 11, an information acquisition unit 12, and an anomaly detection unit 13, wherein:
the training unit 11 is configured to train the neural network according to a preset abnormal image set and a preset normal image set, to obtain a semantic segmentation model for determining a type to which each pixel in the image belongs, where the type includes an abnormal type and a normal type.
The specific information of the abnormal image set, the normal image set, the abnormal type and the normal type may be as in the first to fifth examples of the first embodiment, and will not be described herein again.
An information obtaining unit 12, configured to input an image captured by a camera into the semantic segmentation model, so as to obtain feature information corresponding to the image.
And the abnormality detection unit 13 is used for judging whether the camera is abnormal according to the characteristic information corresponding to the image and giving an alarm when the abnormality is determined.
The foregoing information acquisition unit 12 and the abnormality detection unit 13 can be implemented in, but not limited to, the following two ways:
The information obtaining unit 12 is specifically configured to: inputting continuous multi-frame images shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to each frame of image;
the abnormality detection unit 13 is specifically configured to: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame images, and giving an alarm when the abnormality is determined.
In one example, the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs; the abnormality detecting unit 13 determines whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes: simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain the semantic segmentation result graph; and determining that the camera is abnormal when the semantic segmentation result graph contains pixels with abnormal types in a number greater than or equal to a preset number threshold.
In another example, the feature information is a feature map output by a neural network convolution layer previous to the softmax layer in the semantic segmentation model; the abnormality detecting unit 13 determines whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes: merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain a result of whether the camera is abnormal; the first post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
The information obtaining unit 12 is specifically configured to: inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;
the abnormality detection unit 13 is specifically configured to: and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.
In one example, the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs; the abnormality detecting unit 13 determines whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes: and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal.
In another example, the feature information is a feature map output by a neural network convolution layer previous to the softmax layer in the semantic segmentation model; the abnormality detecting unit 13 determines whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes: inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain a result of whether the camera is abnormal or not; and the second post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
The training unit 11 is specifically configured to: training the neural network by adopting an abnormal image set to obtain a primary semantic segmentation model; inputting normal images in a normal image set into the primary semantic segmentation model, outputting semantic segmentation results corresponding to the normal images, and adding normal images with wrong semantic segmentation results into the abnormal image set to obtain a new abnormal image set; and training the primary semantic segmentation model by adopting a new abnormal image set to obtain a semantic segmentation model.
The foregoing training unit 11 may also be implemented by the process shown in fig. 4, which is not described herein again.
The technical scheme of the invention brings the following technical effects:
the method has the technical effects that after the neural network model is trained through the preset abnormal image set to obtain the initial semantic segmentation model, the initial semantic segmentation model is tested through the normal image set, the normal image (namely, the positive sample) with the detection error is added into the abnormal image set to obtain a new abnormal image set, and then the initial semantic segmentation model is trained based on the new abnormal image set. According to the embodiment of the invention, the normal image with error detection in the normal image set is automatically added into the abnormal image set, and the normal image with error detection has typicality reflecting the detection error of the initial semantic segmentation model, so that the normal image with error detection can form a good training positive sample, the training positive sample is added into the abnormal image set, so that the abnormal image set can be automatically expanded, the labor cost is reduced, and the semantic segmentation model trained on the basis of the new abnormal image set has stronger robustness and lower error detection rate.
The foregoing is the core idea of the present invention, and in order to make the technical solutions in the embodiments of the present invention better understood and make the above objects, features and advantages of the embodiments of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention are further described in detail with reference to the accompanying drawings.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (18)
1. A camera abnormality detection method characterized by comprising:
training a neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in an image, wherein the type comprises an abnormal type and a normal type;
inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;
and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.
2. The method according to claim 1, wherein the image captured by the camera is input into the semantic segmentation model to obtain feature information corresponding to the image, specifically: inputting continuous multi-frame images shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to each frame of image;
judging whether the camera is abnormal according to the characteristic information corresponding to the image, specifically: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame image.
3. The method according to claim 2, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;
judging whether the camera is abnormal according to the characteristic information corresponding to the multi-frame images, specifically comprising:
simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain the semantic segmentation result graph;
and when the semantic segmentation result graph contains pixels with abnormal types, the number of the pixels is judged to be larger than or equal to a preset number threshold value, and the camera is determined to be abnormal.
4. The method according to claim 2, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;
judging whether the camera is abnormal according to the characteristic information corresponding to the multi-frame images, specifically comprising:
merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain a result of whether the camera is abnormal; the first post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
5. The method according to claim 1, wherein the image captured by the camera is input into the semantic segmentation model to obtain feature information corresponding to the image, specifically: inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;
judging whether the camera is abnormal according to the characteristic information corresponding to the image, specifically: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the image.
6. The method according to claim 5, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;
judging whether the camera is abnormal according to the characteristic information corresponding to the image, which specifically comprises the following steps:
and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal.
7. The method according to claim 5, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;
judging whether the camera is abnormal according to the characteristic information corresponding to the image, which specifically comprises the following steps:
inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain a result of whether the camera is abnormal or not; and the second post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
8. The method according to any one of claims 1 to 7, wherein the training of the neural network is performed according to a preset abnormal image set and a normal image set to obtain a semantic segmentation model for determining the abnormal type of each pixel in the image, and specifically comprises:
training the neural network by adopting an abnormal image set to obtain a primary semantic segmentation model;
inputting normal images in a normal image set into the primary semantic segmentation model, outputting semantic segmentation results corresponding to the normal images, and adding normal images with wrong semantic segmentation results into the abnormal image set to obtain a new abnormal image set;
and training the primary semantic segmentation model by adopting a new abnormal image set to obtain a semantic segmentation model.
9. The method of any of claims 1 to 7, wherein the set of abnormal images comprises transparent defacement images and opaque defacement images, the normal type is no defacement, the abnormal type is defacement, or the abnormal type comprises transparent defacement and opaque defacement;
or the abnormal image set comprises an overexposed image and an underexposed image, the abnormal type comprises overexposure and underexposure, and the normal type is normal exposure;
or the abnormal image set comprises a transparent stained image, an opaque stained image, an exposed image and an underexposed image, the normal type comprises normal exposure and no stain, and the abnormal type comprises stain, no stain, overexposure and underexposure; alternatively, the types of anomalies include clear smudge, opaque smudge, no smudge, overexposure, and underexposure.
10. A camera abnormality detection device characterized by comprising:
the training unit is used for training the neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in the image, wherein the type comprises an abnormal type and a normal type;
the information acquisition unit is used for inputting the image shot by the camera into the semantic segmentation model to obtain the characteristic information corresponding to the image;
and the abnormality detection unit is used for judging whether the camera is abnormal according to the characteristic information corresponding to the image and giving an alarm when the abnormality is determined.
11. The apparatus according to claim 10, wherein the information obtaining unit is specifically configured to: inputting continuous multi-frame images shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to each frame of image;
the abnormality detection unit is specifically configured to: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame images, and giving an alarm when the abnormality is determined.
12. The apparatus according to claim 11, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;
the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes:
simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain the semantic segmentation result graph;
and when the semantic segmentation result graph contains pixels with abnormal types, the number of the pixels is judged to be larger than or equal to a preset number threshold value, and the camera is determined to be abnormal.
13. The apparatus according to claim 11, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;
the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes:
merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain a result of whether the camera is abnormal; the first post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
14. The apparatus according to claim 10, wherein the information obtaining unit is specifically configured to: inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;
the abnormality detection unit is specifically configured to: and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.
15. The apparatus according to claim 14, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;
the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes:
and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal.
16. The apparatus according to claim 14, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;
the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes:
inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain a result of whether the camera is abnormal or not; and the second post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.
17. The device according to any one of claims 10 to 16, wherein the training unit is specifically configured to:
training the neural network by adopting an abnormal image set to obtain a primary semantic segmentation model;
inputting normal images in a normal image set into the primary semantic segmentation model, outputting semantic segmentation results corresponding to the normal images, and adding normal images with wrong semantic segmentation results into the abnormal image set to obtain a new abnormal image set;
and training the primary semantic segmentation model by adopting a new abnormal image set to obtain a semantic segmentation model.
18. The apparatus according to any one of claims 10 to 16, wherein the abnormal image set comprises a transparent defacement image and an opaque defacement image, the normal type is no defacement, the abnormal type is defacement, or the abnormal type comprises a transparent defacement and an opaque defacement;
or the abnormal image set comprises an overexposed image and an underexposed image, the abnormal type comprises overexposure and underexposure, and the normal type is normal exposure;
or the abnormal image set comprises a transparent stained image, an opaque stained image, an exposed image and an underexposed image, the normal type comprises normal exposure and no stain, and the abnormal type comprises stain, no stain, overexposure and underexposure; alternatively, the types of anomalies include clear smudge, opaque smudge, no smudge, overexposure, and underexposure.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710742191.2A CN107705334B (en) | 2017-08-25 | 2017-08-25 | Camera abnormity detection method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710742191.2A CN107705334B (en) | 2017-08-25 | 2017-08-25 | Camera abnormity detection method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107705334A CN107705334A (en) | 2018-02-16 |
| CN107705334B true CN107705334B (en) | 2020-08-25 |
Family
ID=61170393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710742191.2A Active CN107705334B (en) | 2017-08-25 | 2017-08-25 | Camera abnormity detection method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107705334B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110198471A (en) * | 2018-02-27 | 2019-09-03 | 北京猎户星空科技有限公司 | Abnormality recognition method, device, smart machine and storage medium |
| CN108629180B (en) * | 2018-03-29 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Abnormal operation determination method and device, storage medium and electronic device |
| CN109308449B (en) * | 2018-08-06 | 2021-06-18 | 瑞芯微电子股份有限公司 | Foreign matter filtering video coding chip and method based on deep learning |
| CN109167998A (en) * | 2018-11-19 | 2019-01-08 | 深兰科技（上海）有限公司 | Detect method and device, the electronic equipment, storage medium of camera status |
| CN111291778B (en) * | 2018-12-07 | 2021-07-06 | 马上消费金融股份有限公司 | Training method of depth classification model, exposure anomaly detection method and device |
| CN109685131A (en) * | 2018-12-20 | 2019-04-26 | 斑马网络技术有限公司 | Automobile vehicle device system exception recognition methods and device |
| CN110855976B (en) * | 2019-10-08 | 2022-03-11 | 南京云计趟信息技术有限公司 | Camera abnormity detection method and device and terminal equipment |
| CN111325715A (en) * | 2020-01-21 | 2020-06-23 | 上海悦易网络信息技术有限公司 | Camera color spot detection method and device |
| CN114757947B (en) * | 2022-06-14 | 2022-09-27 | 苏州魔视智能科技有限公司 | Method, device and system for detecting fouling of camera lens |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102176758A (en) * | 2011-03-07 | 2011-09-07 | 北京文安科技发展有限公司 | Video quality diagnosis system and realization method thereof |
| CN103152601A (en) * | 2013-03-15 | 2013-06-12 | 广州市澳视光电子技术有限公司 | Intelligent failure-reporting camera and network management client system thereof |
| CN104378629A (en) * | 2014-12-01 | 2015-02-25 | 广州市浩云安防科技股份有限公司 | Camera fault detection method |
| CN105975956A (en) * | 2016-05-30 | 2016-09-28 | 重庆大学 | Infrared-panorama-pick-up-head-based abnormal behavior identification method of elderly people living alone |
| CN106851263A (en) * | 2017-03-30 | 2017-06-13 | 安徽四创电子股份有限公司 | Video quality diagnosing method and system based on timing self-learning module |
| CN107025457A (en) * | 2017-03-29 | 2017-08-08 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10290106B2 (en) * | 2016-02-04 | 2019-05-14 | Nec Corporation | Video monitoring using semantic segmentation based on global optimization |
-
2017
- 2017-08-25 CN CN201710742191.2A patent/CN107705334B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102176758A (en) * | 2011-03-07 | 2011-09-07 | 北京文安科技发展有限公司 | Video quality diagnosis system and realization method thereof |
| CN103152601A (en) * | 2013-03-15 | 2013-06-12 | 广州市澳视光电子技术有限公司 | Intelligent failure-reporting camera and network management client system thereof |
| CN104378629A (en) * | 2014-12-01 | 2015-02-25 | 广州市浩云安防科技股份有限公司 | Camera fault detection method |
| CN105975956A (en) * | 2016-05-30 | 2016-09-28 | 重庆大学 | Infrared-panorama-pick-up-head-based abnormal behavior identification method of elderly people living alone |
| CN107025457A (en) * | 2017-03-29 | 2017-08-08 | 腾讯科技（深圳）有限公司 | A kind of image processing method and device |
| CN106851263A (en) * | 2017-03-30 | 2017-06-13 | 安徽四创电子股份有限公司 | Video quality diagnosing method and system based on timing self-learning module |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107705334A (en) | 2018-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107705334B (en) | Camera abnormity detection method and device | |
| CN108460362B (en) | System and method for detecting human body part | |
| CN100373423C (en) | Method and system for evaluating moving image quality of displays | |
| CN110650292B (en) | Method and device for assisting user in shooting vehicle video | |
| CN107491790B (en) | Neural network training method and device | |
| CN109284674A (en) | A kind of method and device of determining lane line | |
| US11748894B2 (en) | Video stabilization method and apparatus and non-transitory computer-readable medium | |
| CN104867128B (en) | Image blurring detection method and device | |
| Yang et al. | A robotic system towards concrete structure spalling and crack database | |
| CN109253722A (en) | Merge monocular range-measurement system, method, equipment and the storage medium of semantic segmentation | |
| CN114091620B (en) | Template matching detection method, computer equipment and storage medium | |
| CN103096117B (en) | Video noise detection method and device | |
| Nodado et al. | Intelligent traffic light system using computer vision with android monitoring and control | |
| CN112991374A (en) | Canny algorithm-based edge enhancement method, device, equipment and storage medium | |
| WO2024061194A1 (en) | Sample label acquisition method and lens failure detection model training method | |
| CN116403162B (en) | Airport scene target behavior recognition method and system and electronic equipment | |
| CN109960990B (en) | Method for evaluating reliability of obstacle detection | |
| CN114219758A (en) | Defect detection method, system, electronic device and computer readable storage medium | |
| CN112052883B (en) | Clothes detection method, device and storage medium | |
| CN113128499A (en) | Vibration testing method of visual imaging device, computer device and storage medium | |
| Pooyoi et al. | Snow scene segmentation using cnn-based approach with transfer learning | |
| CN111815705A (en) | Laser tracker light filtering protective lens pollution identification method and device and electronic equipment | |
| CN114757947B (en) | Method, device and system for detecting fouling of camera lens | |
| CN117079085B (en) | Training method of raindrop detection model, vehicle control method, device and medium | |
| CN117152707B (en) | Calculation method and device for offset distance of vehicle and processing equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20200325 Address after: 101300, No. two, 1 road, Shunyi Park, Zhongguancun science and Technology Park, Beijing, Shunyi District Applicant after: BEIJING TUSENZHITU TECHNOLOGY Co.,Ltd. Address before: 101300, No. two, 1 road, Shunyi Park, Zhongguancun science and Technology Park, Beijing, Shunyi District Applicant before: TuSimple |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |