TWI703511B - Screen crack detection system - Google Patents

Abstract

A screen crack detection system includes an input module, a feature extraction module and a target frame selection module. The input module is configured to receive a screen image. The feature extraction module includes a first convolutional neural network model that converts the screen image into a first feature. The target frame selection module includes a region suggestion network model, and the region suggestion network model receives the first feature transmitted by the feature extraction module, and identifies whether there is at least one cracked region in the screen image. If the cracked region is existed, the cracked region is framed to form at least one target frame.

Description

Screen crack detection system

The invention refers to a detection system, especially a screen crack detection system.

With the development of mobile phone technology, mobile phone functions include network, fingerprint recognition, face recognition, photography, video and audio, etc. besides calling. While the functions have been improved, the price of mobile phones has also increased. Nowadays, brand-new mobile phones cost tens of thousands of Taiwan dollars, and the maintenance costs of mobile phones have also increased significantly compared with the past. In response to the gradual increase in mobile phone prices, the industry has also introduced mobile phone insurance to reduce the burden of users in repairing mobile phones through collective charging. Generally, mobile phone insurance determines the amount of compensation based on the damage of the mobile phone. Among them, damage to the mobile phone screen accounts for the majority of mobile phone damage. At present, the judgment of the degree of damage to the mobile phone, especially the appearance damage, is usually visually judged by the claims adjuster or repairer, but the visual judgment is prone to vary from person to person. In addition, it is also possible that the repairer or the mobile phone owner colludes to misrepresent the damage of the mobile phone, thereby defrauding a higher insurance premium. Patent Publication No. TW201839666A invention patent discloses an image recognition method that recognizes the degree of damage to the exterior of a car and uses it as a basis for car damage insurance compensation. However, the car itself is large in size, and the damaged photos taken usually have a large area of the car body surface, and there is less interference elements such as background environment, so the identification is less error-free. On the contrary, the size of the mobile phone is small, and the image of the damaged mobile phone is usually mixed with other images such as the background environment, which greatly increases the difficulty of identification. Therefore, the technical features disclosed in Patent Publication No. TW201839666A cannot be directly applied to the damage identification of the mobile phone screen. In summary, how to effectively use artificial intelligence to identify the degree of damage to the screen of a mobile phone is worth considering for those with ordinary knowledge in the field.

The present invention provides a screen crack detection system, which includes an input module, a feature extraction module and a target frame selection module. The input module is suitable for receiving a screen image. The feature extraction module includes a first convolutional neural network model, and the first convolutional neural network model converts the screen image into a first feature. The target frame selection module includes a regional suggestion network model. The regional suggestion network model receives the first feature sent by the feature extraction module and identifies whether there is at least one cracked area in the screen image, If the cracked area exists, frame the cracked area to form at least one target frame. Among them, the screen crack detection system includes the following steps in the training phase: A10: Search and collect multiple first reference images from the Internet with multiple keywords; A20: Send the above-mentioned first reference image to a second convolutional neural network model; A30: According to the above-mentioned first reference image, the second convolutional neural network model outputs a plurality of second features; A40: Use an unsupervised model to group the above-mentioned first reference images according to the second feature; and A50: Input the first reference image after grouping and containing the cracked area into a training database; A90: Train the screen crack detection system according to the first reference image stored in the training database. In the above-mentioned screen crack detection system, the first convolutional neural network model and the second convolutional neural network model are VGG model, ResNet model, DenseNet model or Inception model. In the above-mentioned screen crack detection system, the suggested network model for this area is faster-RCNN model, YOLO model, CTPN model or EAST model. In the above-mentioned screen crack detection system, the target frame selection module further includes a binary classifier and detects whether there is the cracked area in the screen image through the binary classifier. In the above-mentioned screen crack detection system, the screen crack detection system further includes the following steps in the training phase: A60: Select multiple second reference images that do not contain cracked areas on the screen; and A70: Simulate at least one cracked area on each of the above-mentioned second reference images; A80: Input the simulated second reference image into the training database; Wherein, in step A90, the screen crack detection system is trained based on the first reference image and the second reference image stored in the training database. In the above-mentioned screen crack detection system, in step A10, the keyword includes broken or cracked. In the above-mentioned screen crack detection system, in step A10, the keyword also includes screen or mobile phone. In the above screen crack detection system, the first feature is a two-dimensional matrix, and the two-dimensional matrix is mainly composed of a part of the hidden layer of the first convolutional neural network model. In the above-mentioned screen crack detection system, the second feature is a vector. In the above-mentioned screen crack detection system, in step A40, the unsupervised model performs K-means, DBSCAN, or Expectation Maximization algorithm to group the above-mentioned first reference image. In order to make the above-mentioned features and advantages of the present invention more comprehensible and understandable, a detailed description is given below of preferred embodiments in conjunction with the accompanying drawings.

The present invention can be better understood with reference to the detailed content set forth herein and the description of the drawings. Various embodiments will be discussed below with reference to the drawings. However, those skilled in the art will easily understand that the detailed description given here with respect to the drawings is only for explanatory purposes, because these methods and systems may exceed the described embodiments. For example, the given teachings and specific application requirements may produce a variety of alternative and suitable methods to implement any detailed functions described herein. Therefore, any method can extend beyond the specific implementation options described and illustrated in the following embodiments. In the specification and subsequent patent applications, certain words are used to refer to specific elements. Those with ordinary knowledge in the field should understand that different manufacturers may use different terms to refer to the same components. The scope of this specification and subsequent patent applications does not use differences in names as a way to distinguish elements, but uses differences in functions of elements as a criterion for distinguishing. The "include" or "include" mentioned in the entire specification and the subsequent request items is an open term, so it should be interpreted as "include but not limited to". In addition, the term "coupled" or "connected" herein includes any direct and indirect electrical or communication connection means. Therefore, if it is described that a first device is coupled to a second device, it means that the first device can be directly and electrically connected to the second device, or indirectly, electrically or communicatively connected to the second device through other devices or connection means. The second device. Please refer to FIG. 1. FIG. 1 illustrates an embodiment of the screen crack detection system of the present invention. The screen crack detection system 100 includes an input module 110, a feature extraction module 120, a target frame selection module 130, and an output module 140. The input module 110 is, for example, electrically connected to an image input device 40. The image input device 40 is a smart phone with a camera function in this embodiment, but it can also be a digital camera. With the image input device 40 and the input module 110, a captured image 10 (for example, the photo shown in FIG. 2A) can be imported into the input module 110. After that, the input module 110 sends the image 10 to the feature extraction module 120. The feature extraction module 120 includes a first convolutional neural network model 122, and the first convolutional neural network model 122 The screen image 10 is converted into a first feature 12, and the first feature 12 is transmitted to the target frame selection module 130. The target frame selection module 130 includes a regional suggestion network model 132. The regional suggestion network model 132 receives the first feature 12 sent by the feature extraction module 120 and recognizes whether there is at least one crack in the screen image 10 Area, if there is a cracked area, frame the cracked area to form at least one target frame 14 (in this embodiment, there are four target frames 14 in total). After that, the output module 140 together with the screen image 10 and the target frame 14 are output and displayed, for example, on the display screen 60 of the user, so that the user can clearly recognize how many cracked areas exist in the screen image 10. In addition, the output module 140 can also be connected to other computers to perform further processing on the output data, such as registering the number of cracked areas in the screen image 10 in the database. In this embodiment, the input module 110, the feature extraction module 120, the target frame selection module 130, and the output module 140 are arranged on the server side, and the server side is, for example, composed of one or more servers. In the above embodiment, the first feature 12 is a two-dimensional matrix composed of multiple hidden layers of the first convolutional neural network model 122, which will be described in detail below. Generally speaking, a neural network model includes an input layer, multiple hidden layers, and an output layer. Therefore, as shown in FIG. 3, the first convolutional neural network model 122 also includes an input layer 122a, a plurality of hidden layers 122b, and an output layer 122c. Among them, each neuron (cell) in each hidden layer 122b is represented by a number, so the hidden layer 122b can be regarded as a vector, and several hidden layers 122b can form a two-dimensional matrix, which is The first feature 12. In this embodiment, the first feature 12 is composed of three hidden layers 122b closest to the output layer 122c, but those skilled in the art can also adjust it according to the situation. In the foregoing embodiment, the first convolutional neural network model 122 is, for example, a VGG model, a ResNet model, a DenseNet model, or an Inception model. Among them, the VGG model is a neural network model proposed by the Visual Geometry Group of Oxford University, the ResNet model is short for Deep residual network, and the DenseNet model is a dense convolutional neural network ( Short for Dense Convolutional Network). In addition, the regional suggestion network model 132 may be a faster-RCNN model, a YOLO model, a CTPN model or an EAST model. Among them, Yolo is the abbreviation of You only look once, which is a neural network model for object detection. CTPN is the abbreviation of Connectionist Text Proposal Network, and the EAST model is the neural network model disclosed in the paper An Efficient and Accurate Scene Text Detector. It should be noted that the neural network models mentioned above are all open source materials, and those with ordinary knowledge in the field can modify them appropriately to meet the required purpose. In the above embodiment, the target frame selection module 130 preferably further includes a binary classifier 134. The binary classifier 134 executes algorithms related to binary classification. The binary classifier 134 also It can be a neural network model, or algorithms such as support vector machines can also be executed. With the binary classifier 134, it is possible to more accurately determine and detect whether there is a cracked area in the screen image 10. As far as neural network models are concerned, the quality and quantity of data provided during the training phase are critical to the effectiveness of neural network models. In the training phase, with the data provided by each training, the neural network model will adjust the parameters (ie: weights) of the connections between the neurons. Therefore, even if the two neural network models have the same architecture, they will still be quite different after training. Therefore, as the quality and quantity of the data provided are different, even the same neural network model will have considerable differences in effectiveness. As far as this case is concerned, the quality and quantity of the data provided are also critical to the performance of the screen crack detection system 100. Therefore, the following will introduce how to collect relevant training data for the screen crack detection system 100 to perform training during the training phase. Please refer to FIGS. 4 and 5 at the same time. FIG. 4 shows the model used in data collection, and FIG. 5 shows the data collection process of the screen crack detection system of the present invention in the training phase. First, please refer to step S110 to input multiple keywords into the search engine 210 to search and collect multiple pictures from the Internet. Here, the keyword 20 is, for example, mobile phone screen, broken, and the search engine 210 used is, for example, Google's image search. However, those with general knowledge in the field can also use other keywords, such as: cracked, mobile device, etc. Here, the pictures collected through the above keywords are referred to as the first reference image 22. Next, referring to step S120, the first reference image is sent and input to a second convolutional neural network model 220. In this embodiment, the second convolutional neural network model 220 is, for example, a VGG model, a ResNet model, a DenseNet model, or an Inception model. In a preferred embodiment, the second convolutional neural network model 220 has been pre-trained, for example, it has been pre-trained using the imagenet database. This second convolutional neural network model 220 belongs to a convolutional neural network, which mainly includes a convolutional layer and a pooling layer (both convolutional layer and sampling layer are not shown in the figure) Drawing type), where the convolutional layer is mainly used for feature extraction, and the sampling layer is used to reduce the parameters required by the second convolutional neural network model 220 to avoid overfitting. Next, in step S130, the second convolutional neural network model 220 generates a plurality of second features 24 according to the input first reference image 22, and each second feature 24 corresponds to one of the first reference images 22. Here, the second feature 24 is, for example, a vector. After that, step S140 is performed to group the second features 24 with an unsupervised model 230. In this embodiment, the unsupervised model 230 performs K-means, DBSCAN (Density-based spatial clustering of applications with noise, density-based clustering method with noise), or Unsupervised classification such as Expectation Maximization. Algorithm. Since each second feature 24 corresponds to one of the first reference images 22, grouping the second feature 24 is grouping the first reference image 22. After that, step S150 is executed to input the first reference image 22 into a training database 240. Here, explain the reasons for grouping. Because the first reference images 22 collected by using the keywords 22 in the search engine 210 are not necessarily all the required images. For example, the pictures searched by the keywords of mobile phone screen and broken may include pictures of good phone screens or pictures of repair shops, so irrelevant pictures need to be removed. However, due to the large number of collected pictures, it is very inefficient to remove irrelevant pictures by manpower alone. Therefore, in this embodiment, the first reference image 22 is grouped by the unsupervised model 230, and the more similar pictures have been grouped into one category, thereby removing irrelevant pictures. In addition, in order to make the execution of the unsupervised model 230 faster and more efficient, it is necessary to use the second convolutional neural network model 220 for feature extraction first. In a preferred embodiment, manual methods can be supplemented to identify irrelevant pictures from the grouped pictures and shave them. After completing the construction of the training database 240, the training database 240 can be used to train the screen crack detection system 100. Since the images and pictures in the training database 240 are found through search engines, they contain a large number and styles. In other words, the training database 240 contains not only images of broken mobile phone screens, but also a variety of different images. Screen size, model, and cracked type of mobile phone screen damage image. Also due to the large number of pictures in the training database 240, the screen crack detection system 100 can have very good results after training based on the training database 240. It can accurately determine whether the mobile phone screen is damaged and recognize it. The number and scope of the cracked area. In order to increase the diversity of the pictures in the training database 240, the number of pictures in the training database 240 can also be increased through the process shown in FIG. 6. First, perform step S160 to select a plurality of pictures on the screen that do not contain cracked areas (as shown in FIG. 7A), and these pictures are referred to as second reference images 30 herein. After that, step S170 is executed to simulate at least one cracked area 32 on the second reference image 20' (as shown in FIG. 7B). Here, drawing software (such as Photoshop) or programming can be used to simulate various cracks on the second reference image 30. Then, step S180 is executed to input the simulated second reference image 30 into the training database 240. In summary, because the screen crack detection system 100 can accurately determine whether the mobile phone screen is broken and identify the number and range of the cracked area, it can reduce the cost of manual judgment and there is no need to worry about repairers who lie about the damage of the mobile phone. The problem. In addition, the screen crack detection system 100 of this case can not only be used to determine whether the mobile phone screen is damaged, but also can be used to determine whether the screen of other devices is damaged by changing the training data. The description of the present invention is as above, but it is not intended to limit the scope of patent rights claimed in this creation. The scope of its patent protection shall be determined by the attached scope of patent application and its equivalent fields. Anyone with ordinary knowledge in the field, without departing from the spirit or scope of this patent, makes changes or modifications that are equivalent changes or designs completed under the spirit of this creation, and should be included in the following patent scope Inside.

10: Image 12: The first feature 14: target box 40: Video input device 100: Screen crack detection system 110: Input module 120: Feature Extraction Module 122: The first convolutional neural network model 122a: Input layer 122b: hidden layer 122c: output layer 130: Target frame selection module 132: Regional Recommendation Network Model 134: Binary classifier 140: output module 210: search engine 220: The second convolutional neural network model 230: unsupervised model 240: Training Database 30: Second reference image 32: Cracked area 60: display screen S110~S180: Flow chart steps

Fig. 1 shows an embodiment of the screen crack detection system of the present invention. Figure 2A shows a schematic diagram of the image. FIG. 2B shows a schematic diagram of forming a target frame in an image. Fig. 3 shows a schematic diagram of the first convolutional neural network model. Figure 4 shows the model used in data collection. FIG. 5 illustrates the data collection process of the screen crack detection system of the present invention in the training phase. Figure 6 shows the process of increasing the number of pictures in the training database. FIG. 7A shows a schematic diagram of a picture with no cracked area on the screen. FIG. 7B is a schematic diagram of simulating the cracked area on the second reference image.

10: Image

12: The first feature

40: Video input device

100: Screen crack detection system

110: Input module

120: Feature Extraction Module

122: The first convolutional neural network model

130: Target frame selection module

132: Regional Recommendation Network Model

134: Binary classifier

140: output module

60: display screen

Claims (10)

A screen crack detection system, including: An input module suitable for receiving a screen image; A feature extraction module including a first convolutional neural network model that converts the screen image into a first feature; and A target frame selection module includes a regional suggestion network model, the regional suggestion network model receives the first feature sent by the feature extraction module, and recognizes whether there is at least one cracked area in the screen image , If the cracked area exists, frame the cracked area to form at least one target frame; Among them, the screen crack detection system includes the following steps in the training phase: A10: Search and collect multiple first reference images from the Internet with multiple keywords; A20: Send the above-mentioned first reference image to a second convolutional neural network model; A30: According to the above-mentioned first reference image, the second convolutional neural network model outputs a plurality of second features; A40: Use an unsupervised model to group the above-mentioned first reference images according to the second feature; and A50: Input the first reference image after grouping and containing the cracked area into a training database; A90: Train the screen crack detection system according to the first reference image stored in the training database. The screen crack detection system described in the first item of the scope of patent application, wherein the first convolutional neural network model and the second convolutional neural network model are VGG model, ResNet model, DenseNet model or Inception model. Such as the screen crack detection system described in item 1 of the scope of patent application, wherein the proposed network model for this area is the faster-RCNN model, the YOLO model, the CTPN model or the EAST model. For the screen crack detection system described in item 1 of the scope of patent application, the target frame selection module further includes a binary classifier and detects whether there is the cracked area in the screen image through the binary classifier . For the screen crack detection system described in item 1 of the scope of patent application, the screen crack detection system further includes the following steps in the training phase: A60: Select multiple second reference images that do not contain cracked areas on the screen; and A70: Simulate at least one cracked area on each of the above-mentioned second reference images; A80: Input the simulated second reference image into the training database; Wherein, in step A90, the screen crack detection system is trained based on the first reference image and the second reference image stored in the training database. For the screen crack detection system described in item 1 of the scope of patent application, in step A10, the keyword includes damage or crack. Such as the screen crack detection system described in item 6 of the scope of patent application, wherein in step A10, the keyword also includes screen or mobile phone. The screen crack detection system described in item 1 of the scope of patent application, wherein the first feature is a two-dimensional matrix, and the two-dimensional matrix is mainly composed of a part of the hidden layer of the first convolutional neural network model . In the screen crack detection system described in item 1 of the scope of patent application, the second feature is a vector. In the screen crack detection system described in the first item of the scope of patent application, in step A40, the unsupervised model executes K-means, DBSCAN, or Expectation Maximization algorithms to group the first reference image.

Images