US11144799B2 - Image classification method, computer device and medium - Google Patents

Image classification method, computer device and medium Download PDF

Info

Publication number
US11144799B2
US11144799B2 US16/556,697 US201916556697A US11144799B2 US 11144799 B2 US11144799 B2 US 11144799B2 US 201916556697 A US201916556697 A US 201916556697A US 11144799 B2 US11144799 B2 US 11144799B2
Authority
US
United States
Prior art keywords
classified
image
level
features
contained
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US16/556,697
Other versions
US20200285917A1 (en
Inventor
Xiangjun Peng
Yunqi Wang
Chenxi Zhao
Yachong Xue
Gang Li
Yaoyu Lv
Shuo Zhang
Huidong HE
Yadong DING
Minglei Chu
Lili Chen
Hao Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Original Assignee
BOE Technology Group Co Ltd
Beijing BOE Optoelectronics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BOE Technology Group Co Ltd, Beijing BOE Optoelectronics Technology Co Ltd filed Critical BOE Technology Group Co Ltd
Assigned to BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD., BOE TECHNOLOGY GROUP CO., LTD. reassignment BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: DING, Yadong, Wang, Yunqi, CHEN, LILI, CHU, MINGLEI, HE, HUIDONG, LI, GANG, LV, Yaoyu, PENG, Xiangjun, XUE, Yachong, ZHANG, HAO, ZHANG, Shuo, ZHAO, CHENXI
Publication of US20200285917A1 publication Critical patent/US20200285917A1/en
Application granted granted Critical
Publication of US11144799B2 publication Critical patent/US11144799B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G06K9/726—
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06K9/6268—
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration using histogram techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G06T7/0008—Industrial image inspection checking presence/absence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/174—Segmentation; Edge detection involving the use of two or more images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G06V30/191—Design or setup of recognition systems or techniques; Extraction of features in feature space; Clustering techniques; Blind source separation
    • G06V30/19173—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/26—Techniques for post-processing, e.g. correcting the recognition result
    • G06V30/262—Techniques for post-processing, e.g. correcting the recognition result using context analysis, e.g. lexical, syntactic or semantic context
    • G06V30/274—Syntactic or semantic context, e.g. balancing
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
    • G01N21/84—Systems specially adapted for particular applications
    • G01N21/88—Investigating the presence of flaws or contamination
    • G01N21/8851—Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges
    • G01N2021/8854—Grading and classifying of flaws
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning

Definitions

  • the present disclosure relates to the field of image processing technology, and more particularly, to an image classification method, a computer device, and a medium.
  • an image classification method comprising:
  • acquiring a middle-level semantic feature of an image to be classified through a visual dictionary comprises:
  • acquiring a target to be classified contained in the image to be classified comprises: acquiring the target to be classified contained in the image to be classified through image segmentation.
  • the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
  • the low-level feature is a histogram of oriented gradient feature.
  • acquiring the middle-level semantic feature of the image to be classified according to the low-level feature through the visual dictionary comprises: acquiring, according to the low-level feature, a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified.
  • the method before acquiring the target to be classified contained in the image to be classified, the method further comprises: performing image enhancement on the image to be classified.
  • performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
  • the method before acquiring a middle-level semantic feature of an image to be classified through a visual dictionary, the method further comprises:
  • the method before classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features, the method further comprises:
  • acquiring the targets to be classified contained in the plurality of training images comprises: acquiring the targets to be classified contained in the plurality of training images through image segmentation.
  • the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
  • the low-level features are histogram of oriented gradient features.
  • constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images comprises: clustering histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using a K-means algorithm to obtain visual words, and constructing the visual dictionary according to the visual words.
  • acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary comprises: acquiring, according to the low-level features, visual words which have the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images as the middle-level semantic features of the plurality of training images.
  • the method before acquiring the targets to be classified contained in the plurality of training images, the method further comprises: performing image enhancement on the plurality of training images.
  • performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
  • a computer device comprising a memory, a processor, and a computer program stored on the memory and operative on the processor, wherein the program, when executed by the processor, implements the image classification method according to the first aspect of the present disclosure.
  • a computer readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the image classification method according to the first aspect of the present disclosure.
  • FIG. 1 illustrates a flowchart of an image classification method according to an embodiment of the present disclosure.
  • FIG. 2 illustrates a flowchart of performing image enhancement.
  • FIG. 3 illustrates a schematic diagram of constructing a visual dictionary according to low-level features of targets to be classified contained in a plurality of training images.
  • FIG. 4 illustrates a flowchart of an image classification method for classifying display screen defects according to an embodiment of the present disclosure.
  • FIG. 5 illustrates a schematic diagram of an image classification system according to an embodiment of the present disclosure.
  • FIG. 6 illustrates a schematic structural diagram of a computer system for performing image classification according to an embodiment of the present disclosure.
  • Classification of display screen defects is taken as an example.
  • display screen defects which are generally classified into spot defects, line defects and mura defects.
  • the spot defects comprise dark spots, bright spots etc.
  • the line defects comprise film defects, scratches etc.
  • the mura defects comprise glass breakage, liquid leakage etc.
  • camera parameters, a shooting mode, and a shooting environment of a camera change while the camera collects a display image of a display screen under test
  • low-level features of the same type of defects in the image such as geometric shapes, textures, local description etc., may also change accordingly.
  • an embodiment of the present disclosure provides an image classification method.
  • step 101 a middle-level semantic feature of an image to be classified is acquired through a visual dictionary.
  • step 102 the image to be classified is classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
  • the middle-level semantic feature of the image to be classified is acquired and the image is classified based on the middle-level semantic feature of the image to be classified, which reduces a semantic gap between a low-level feature and a high-level feature, and has advantages such as high accuracy, high robustness, high efficiency etc. Furthermore computer storage space and a large amount of calculation can be reduced.
  • acquiring a middle-level semantic feature of an image to be classified through a visual dictionary further comprises:
  • An image has hierarchical features.
  • Low-level features are basic features of the image, and represent objective features of the image. During extraction, the low-level features are extracted based on an image level without any prior knowledge.
  • Low-level features of images which are widely used currently comprise geometric shapes, textures, local invariant features, transform domain features, etc.
  • Middle-level features are middle-level semantic features obtained by analyzing a statistical distribution of the low-level features, and comprise a bag of visual words and a semantic topic.
  • High-level features are more abstract semantics of the image, which correspond to behavioral semantics, emotional semantics, and scene semantics etc. The scene semantics describe content of the image, the behavioral semantics describe motion information in the image, and the emotional semantics describe emotions, such as happiness. sadness etc. which are transferred by the image to humans.
  • the middle-level semantic feature of the image to be classified is formed by analyzing a statistical distribution of the low-level feature of the target to be classified contained in the image to be classified, i.e., establishing a relationship with semantics by analyzing a statistical distribution of the low-level feature, which reduces a semantic gap between a low-level feature and a high-level feature, and has advantages such as high accuracy, high robustness, high efficiency etc. Furthermore computer storage space and a large amount of calculation can be reduced.
  • the method before acquiring the target to be classified contained in the image to be classified, the method further comprises: performing image enhancement on the image to be classified.
  • the accuracy of the image classification may further be improved.
  • performing image enhancement on the image to be classified further comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence on the image to be classified.
  • the graying may greatly reduce the amount of calculation for subsequent image processing based on the distribution characteristics of brightness and chromaticity of the image.
  • the wavelet de-noising may highlight high-frequency information such as an edge, a structure etc. of the target to be classified contained in the image, and improve a contrast between the target to be classified and a background, thereby highlighting the target to be classified contained in the image.
  • the otsu threshold segmentation is used to binarize the image.
  • the median filtering may perform edge smoothing on the target contained in the binarized image.
  • the binary expansion and the binary corrosion may remove holes and isolated false points from the binarized image.
  • acquiring a target to be classified contained in the image to be classified further comprises: acquiring the target to be classified contained in the image to be classified through image segmentation.
  • acquiring the target to be classified contained in the image to be classified through image segmentation further comprises: performing image segmentation on the image to be classified based on a minimum circumscribed rectangle algorithm, to obtain the target to be classified contained in the image to be classified.
  • a minimum circumscribed rectangle of the target contained in the image is obtained using a target rotation method. Specifically, the target contained in the image is rotated at equal intervals in a range of 90°, an area of a circumscribed rectangle parallel to a coordinate axis direction is calculated each time the target is rotated, a circumscribed rectangle with the smallest area is obtained as a minimum circumscribed rectangle of the target to be classified contained in the image, and then a region of the minimum circumscribed rectangle is segmented to obtain the target to be classified contained in the image.
  • the low-level feature of the target to be classified contained in the image to be classified is a Histogram of Oriented Gradient (HOG) feature.
  • the histogram of oriented gradient feature is composed of histograms of oriented gradient of local regions of the image. Gradients in an image or oriented densities of edges in the image may accurately represent shape attributes of local regions of the target to be classified. Therefore, the histogram of oriented gradient feature may be used as the low-level feature to ensure the accuracy of the image classification.
  • a flow of extracting an HOG feature is as follows.
  • a magnitude value and an orientation of the gradient at the pixel point (x, y) are as follows respectively:
  • a gradient histogram is established, and specifically, the image is firstly divided into a plurality of cells, for example, cells each composed of 2*2 pixels, an orientation value of the gradient histogram is set to 0-180°, and a bin is obtained every 20°, wherein a weight of oriented gradient within the gradient histogram is determined by a magnitude value of the gradient.
  • acquiring the middle-level semantic feature of the image to be classified according to the low-level feature of the target to be classified contained in the image to be classified through the visual dictionary further comprises: acquiring a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified.
  • the method before acquiring a middle-level semantic feature of an image to be classified through a visual dictionary, the method further comprises:
  • the method before acquiring the targets to be classified contained in the plurality of training images, the method further comprises: performing image enhancement on the plurality of training images.
  • performing image enhancement on the plurality of training images further comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence on the plurality of training images.
  • the method before classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features, the method further comprises: acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary; and
  • acquiring the targets to be classified contained in the plurality of training images further comprises: acquiring the targets to be classified contained in the plurality of training images through image segmentation.
  • acquiring the targets to be classified contained in the plurality of training images through image segmentation further comprises: performing image segmentation on the plurality of training images based on a minimum circumscribed rectangle algorithm, to obtain the targets to be classified contained in the plurality of training images.
  • the low-level features of the targets to be classified contained in the plurality of training images are histogram of oriented gradient features.
  • constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images further comprises: clustering histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using a K-means algorithm to obtain visual words, and constructing the visual dictionary according to the visual words.
  • acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary comprises: acquiring visual words which have the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images as the middle-level semantic features of the plurality of training images.
  • a Bag of Words (BoW) model is such a middle-level feature representation method.
  • the BoW model treats the image as a document composed of visual words, so that the BoW model and related theory in the field of text classification are applied to the understanding of the image, without analyzing and interpreting specific composition of targets in an object, and instead, a plurality of sample images are applied as training samples, low-level features of the sample images are quantized into visual words, and content of an unknown target is expressed using a distribution histogram of the visual words of the image.
  • the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images are clustered using a K-means algorithm to obtain visual words, and a flow of constructing a visual dictionary according to the visual words is as follows.
  • Visual words for example, t 1 , t 2 , t 3 , t 4 as shown in FIG. 3 , are formed from the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using the K-means algorithm.
  • t 1 , t 2 , t 3 , t 4 are centroids of clusters, and all the visual words form a visual dictionary.
  • the K-means algorithm is an unsupervised machine learning algorithm. When clustering is performed using the algorithm, N objects are divided into k classes based on a criterion that there is a high similarity within the classes and a low similarity between the classes.
  • a specific flow is as follows.
  • a specific flow of acquiring a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified is as follows.
  • V i [ v 1 +v 2 + . . . +v k ], v k ⁇ R k ⁇ 1
  • the histogram of the visual words is the middle-level semantic feature of the image, and is applied to sample images. Firstly, low-level features of targets to be classified contained in the images are quantized into visual words, and then content of the unknown targets is expressed by a distribution histogram of the visual words of the images. Since overall statistical information of the images is applied to the middle-level semantic feature without analyzing specific composition of the targets contained in the images, there are high accuracy and high robustness in modeling the features of the targets of the images.
  • visual words having the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images may be acquired as middle-level semantic features of the plurality of training images in a specific flow similar to that described above, and will not be described in detail here.
  • Image enhancement is performed on the training samples and the test sample respectively.
  • the middle-level semantic features of the training samples are trained using a machine learning algorithm to obtain a classification model, and the test sample is classified according to the middle-level semantic feature of the test sample using the classification model.
  • Feature extraction is a key step in the classification of display screen defects.
  • Low-level features such as geometry, textures, shapes, local descriptors etc. used in the existing display screen defect classification method based on image classification are extracted based on information at an image level. Since the feature extraction has not been sufficiently generalized and abstracted, it is far from a concept level, display screen defect classification based on low-level features is not highly accurate, and it is difficult to accurately classify an image having a defect outside a training set.
  • the specific flow of the present example comprises two phases, which are a training phase and a classification phase respectively.
  • image enhancement is performed on the training samples
  • a visual dictionary is constructed according to the low-level features of the display screen defect targets of the training samples, and middle-level semantic features of the training samples are acquired through the visual dictionary;
  • the middle-level semantic features of the training samples are trained using a machine learning algorithm to obtain a classification model.
  • At least one second display image of at least one second display screen to be classified under test is collected, and the second display image is used as a test sample;
  • a middle-level semantic feature of the test sample is acquired through the visual dictionary
  • FIG. 5 another embodiment of the present disclosure provides an image classification system which performs the image classification method described above, comprising:
  • a middle-level semantic feature acquisition module configured to acquire a middle-level semantic feature of a target to be classified in an image to be classified through a visual dictionary
  • a classification module configured to classify the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
  • a computer system suitable for implementing the image classification system according to the present embodiment, comprising a Central Processing Unit (CPU) which may perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) or a program loaded from a storage portion into a Random Access Memory (RAM).
  • ROM Read Only Memory
  • RAM Random Access Memory
  • Various programs and data required for operations of a computer system are also stored in the RAM.
  • the CPU, the ROM, and the RAM are connected to each other through a bus.
  • An Input/Output (I/O) interface is also connected to the bus.
  • the following components are connected to the I/O interface: an input part comprising a keyboard, a mouse, etc.; an output part comprising a Liquid Crystal Display (LCD), etc., and a speaker; a storage part comprising a hard disk etc.; and a communication part comprising a network interface card such as a LAN card, a modem, etc.
  • the communication part performs communication processing via a network such as the Internet.
  • the driver is also connected to the I/O interface as needed.
  • a removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory etc., is mounted on the driver as needed, so that a computer program read therefrom is installed into the storage part as needed.
  • the processes described above in the flowcharts may be implemented as a computer software program.
  • the present embodiment comprises a computer program product comprising a computer program included tangibly on a computer readable medium, the computer program comprising program codes for performing the methods illustrated in the above flowcharts.
  • the computer program may be downloaded and installed from the network via a communication portion, and/or installed from a removable medium.
  • each block of the flowcharts or diagrams may represent a module, a program segment, or a portion of codes, which comprises one or more executable instructions for implementing specified logic functions.
  • the functions illustrated in the blocks may also occur in a different order than that illustrated in the accompanying drawings. For example, two successively represented blocks may in fact be executed substantially in parallel, or they may sometimes be executed in a reverse order, depending upon functionality involved.
  • each block of the diagrams and/or flowcharts, and combinations of blocks in the diagrams and/or flowcharts may be implemented by a dedicated hardware-based system which performs specified functions or operations, or may be implemented by combination of dedicated hardware and computer instructions.
  • Modules described in the present embodiment may be implemented by software, or may be implemented by hardware, and the described modules may also be disposed in a processor.
  • a processor comprising a middle-level semantic feature acquisition module and a classification module.
  • names of these modules do not in any way constitute a limitation on the modules themselves.
  • the middle-level semantic feature acquisition module may also be described as “visual dictionary module”.
  • the present embodiment further provides a non-volatile computer storage medium, which may be a non-volatile computer storage medium included in the above apparatus in the above embodiments; or may exist separately but is not assembled into a non-volatile computer storage medium in a terminal.
  • the above non-volatile computer storage medium stores one or more programs which, when executed by a device, cause the device to:

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Pathology (AREA)
  • Immunology (AREA)
  • General Health & Medical Sciences (AREA)
  • Biochemistry (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

An image classification method, a computer device, and a medium are disclosed. The method includes: acquiring a middle-level semantic feature of an image to be classified through a visual dictionary; and classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)
This application claims priority to the Chinese Patent Application No. CN201910175377.3, filed on Mar. 8, 2019, which is incorporated herein by reference in its entirety.
TECHNICAL FIELD
The present disclosure relates to the field of image processing technology, and more particularly, to an image classification method, a computer device, and a medium.
BACKGROUND
In the production of display screens, not only display screen defects need to be detected, but also the display screen defects need to be classified and statistics need to be compiled on types of the display screen defects to analyze causes of the defects, thereby improving the industrial production methods and improving the yield of the production of the display screens. Since all the classification methods for the display screen defects are based on a low-level feature of a target to be classified contained in an image, this results in a large amount of calculation and low efficiency for image classification, and difficulty in ensuring accuracy and robustness.
SUMMARY
According to a first aspect of the embodiments of the present disclosure, there is provided an image classification method, comprising:
acquiring a middle-level semantic feature of an image to be classified through a visual dictionary; and
classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
In an embodiment, acquiring a middle-level semantic feature of an image to be classified through a visual dictionary comprises:
acquiring a target to be classified contained in the image to be classified;
extracting a low-level feature of the target to be classified contained in the image to be classified; and
acquiring the middle-level semantic feature of the image to be classified according to the low-level feature through the visual dictionary.
In an embodiment, acquiring a target to be classified contained in the image to be classified comprises: acquiring the target to be classified contained in the image to be classified through image segmentation.
In an embodiment, the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
In an embodiment, the low-level feature is a histogram of oriented gradient feature.
In an embodiment, acquiring the middle-level semantic feature of the image to be classified according to the low-level feature through the visual dictionary comprises: acquiring, according to the low-level feature, a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified.
In an embodiment, before acquiring the target to be classified contained in the image to be classified, the method further comprises: performing image enhancement on the image to be classified.
In an embodiment, performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
In an embodiment, before acquiring a middle-level semantic feature of an image to be classified through a visual dictionary, the method further comprises:
acquiring targets to be classified contained in a plurality of training images, extracting low-level features of the targets to be classified contained in the plurality of training images, and constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images.
In an embodiment, before classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features, the method further comprises:
acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary; and
training the middle-level semantic features of the plurality of training images as training samples using a machine learning algorithm to obtain the classification model based on the middle-level semantic features.
In an embodiment, acquiring the targets to be classified contained in the plurality of training images comprises: acquiring the targets to be classified contained in the plurality of training images through image segmentation.
In an embodiment, the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
In an embodiment, the low-level features are histogram of oriented gradient features.
In an embodiment, constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images comprises: clustering histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using a K-means algorithm to obtain visual words, and constructing the visual dictionary according to the visual words.
In an embodiment, acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary comprises: acquiring, according to the low-level features, visual words which have the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images as the middle-level semantic features of the plurality of training images.
In an embodiment, before acquiring the targets to be classified contained in the plurality of training images, the method further comprises: performing image enhancement on the plurality of training images.
In an embodiment, performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
According to a second aspect of the embodiments of the present disclosure, there is provided a computer device comprising a memory, a processor, and a computer program stored on the memory and operative on the processor, wherein the program, when executed by the processor, implements the image classification method according to the first aspect of the present disclosure.
According to a third aspect of the embodiments of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the image classification method according to the first aspect of the present disclosure.
BRIEF DESCRIPTION OF THE ACCOMPANYING DRAWINGS
Specific embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.
FIG. 1 illustrates a flowchart of an image classification method according to an embodiment of the present disclosure.
FIG. 2 illustrates a flowchart of performing image enhancement.
FIG. 3 illustrates a schematic diagram of constructing a visual dictionary according to low-level features of targets to be classified contained in a plurality of training images.
FIG. 4 illustrates a flowchart of an image classification method for classifying display screen defects according to an embodiment of the present disclosure.
FIG. 5 illustrates a schematic diagram of an image classification system according to an embodiment of the present disclosure.
FIG. 6 illustrates a schematic structural diagram of a computer system for performing image classification according to an embodiment of the present disclosure.
 DETAILED DESCRIPTION
In order to explain the present disclosure more clearly, the present disclosure will be further described in conjunction with preferred embodiments and the accompanying drawings. Similar components in the accompanying drawings are denoted by the same reference signs. It should be understood by those skilled in the art that the following detailed description is illustrative and is non-restrictive, and should not be used to limit the protection scope of the present disclosure.
Classification of display screen defects is taken as an example. There are many types of display screen defects, which are generally classified into spot defects, line defects and mura defects. The spot defects comprise dark spots, bright spots etc., the line defects comprise film defects, scratches etc., and the mura defects comprise glass breakage, liquid leakage etc. When camera parameters, a shooting mode, and a shooting environment of a camera change while the camera collects a display image of a display screen under test, low-level features of the same type of defects in the image, such as geometric shapes, textures, local description etc., may also change accordingly. Currently, in order to ensure the accuracy of the classification of the display screen defects, it is necessary to acquire features of the display screen defects in a case of various camera parameters, shooting modes and shooting environments, that is, the classification needs to be performed after the collection is performed under different factors respectively, which leads to the need to model, store and extract a large number of low-level features in the process of classification, thereby resulting in a complicated workflow, and a large amount of calculation and low efficiency for classification.
As shown in FIG. 1, an embodiment of the present disclosure provides an image classification method.
In step 101, a middle-level semantic feature of an image to be classified is acquired through a visual dictionary.
In step 102, the image to be classified is classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
With the image classification method according to the present disclosure, the middle-level semantic feature of the image to be classified is acquired and the image is classified based on the middle-level semantic feature of the image to be classified, which reduces a semantic gap between a low-level feature and a high-level feature, and has advantages such as high accuracy, high robustness, high efficiency etc. Furthermore computer storage space and a large amount of calculation can be reduced.
In an embodiment, acquiring a middle-level semantic feature of an image to be classified through a visual dictionary further comprises:
acquiring a target to be classified contained in the image to be classified;
extracting a low-level feature of the target to be classified contained in the image to be classified; and
acquiring the middle-level semantic feature of the image to be classified according to the low-level feature of the target to be classified contained in the image to be classified through the visual dictionary.
An image has hierarchical features. Low-level features are basic features of the image, and represent objective features of the image. During extraction, the low-level features are extracted based on an image level without any prior knowledge. Currently, low-level features of images which are widely used currently comprise geometric shapes, textures, local invariant features, transform domain features, etc. Middle-level features are middle-level semantic features obtained by analyzing a statistical distribution of the low-level features, and comprise a bag of visual words and a semantic topic. High-level features are more abstract semantics of the image, which correspond to behavioral semantics, emotional semantics, and scene semantics etc. The scene semantics describe content of the image, the behavioral semantics describe motion information in the image, and the emotional semantics describe emotions, such as happiness. sadness etc. which are transferred by the image to humans.
With the image classification method according to the present embodiment, the middle-level semantic feature of the image to be classified is formed by analyzing a statistical distribution of the low-level feature of the target to be classified contained in the image to be classified, i.e., establishing a relationship with semantics by analyzing a statistical distribution of the low-level feature, which reduces a semantic gap between a low-level feature and a high-level feature, and has advantages such as high accuracy, high robustness, high efficiency etc. Furthermore computer storage space and a large amount of calculation can be reduced.
In some implementations of the present embodiment, before acquiring the target to be classified contained in the image to be classified, the method further comprises: performing image enhancement on the image to be classified.
With this implementation, the accuracy of the image classification may further be improved.
In some implementations of the present embodiment, as shown in FIG. 2, performing image enhancement on the image to be classified further comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence on the image to be classified. Here, the graying may greatly reduce the amount of calculation for subsequent image processing based on the distribution characteristics of brightness and chromaticity of the image. In a specific example, the image is grayed using a weighted average method in the present implementation, that is, assuming that three primary color component values of a RGB image are R, G, and B respectively, and a converted grayscale value of the image is GRAY, the grayscale value of the image is calculated according to GRAY=0.3R+0.59G+0.11B. The wavelet de-noising may highlight high-frequency information such as an edge, a structure etc. of the target to be classified contained in the image, and improve a contrast between the target to be classified and a background, thereby highlighting the target to be classified contained in the image. The otsu threshold segmentation is used to binarize the image. The median filtering may perform edge smoothing on the target contained in the binarized image. The binary expansion and the binary corrosion may remove holes and isolated false points from the binarized image.
In some implementations of the present embodiment, acquiring a target to be classified contained in the image to be classified further comprises: acquiring the target to be classified contained in the image to be classified through image segmentation.
In some implementations of the present embodiment, acquiring the target to be classified contained in the image to be classified through image segmentation further comprises: performing image segmentation on the image to be classified based on a minimum circumscribed rectangle algorithm, to obtain the target to be classified contained in the image to be classified.
In a specific example, a minimum circumscribed rectangle of the target contained in the image is obtained using a target rotation method. Specifically, the target contained in the image is rotated at equal intervals in a range of 90°, an area of a circumscribed rectangle parallel to a coordinate axis direction is calculated each time the target is rotated, a circumscribed rectangle with the smallest area is obtained as a minimum circumscribed rectangle of the target to be classified contained in the image, and then a region of the minimum circumscribed rectangle is segmented to obtain the target to be classified contained in the image.
In some implementations of the present embodiment, the low-level feature of the target to be classified contained in the image to be classified is a Histogram of Oriented Gradient (HOG) feature. The histogram of oriented gradient feature is composed of histograms of oriented gradient of local regions of the image. Gradients in an image or oriented densities of edges in the image may accurately represent shape attributes of local regions of the target to be classified. Therefore, the histogram of oriented gradient feature may be used as the low-level feature to ensure the accuracy of the image classification.
In a specific example, a flow of extracting an HOG feature is as follows.
An input image is normalized using a Gamma correction method to suppress noise interference. A Gamma compression formula is as follows:
I(x,y)=I(x,y)gamma
where I(x, y) is the minimum circumscribed rectangle of the target to be classified contained in the image which is obtained in the above example, and gamma=½.
(2) Gradients of pixels in horizontal and vertical directions of the image are calculated respectively, to obtain a vector of oriented gradient at a position of each pixel point, which is used to capture edge information of the target to be classified contained in the image, wherein in an image I(x, y), a gradient at a pixel point (x, y) in the horizontal direction is Gx(x, y), and a gradient at the pixel point (x, y) in the vertical direction is Gy(x, y):
G x(x,y)=I(x+1,y)−I(x−1,y) and
G y(x,y)=I(x,y+1)−I(x,y−1)
A magnitude value and an orientation of the gradient at the pixel point (x, y) are as follows respectively:
G ⁡ ( x , ⁢ y ) = G x ⁡ ( x , y ) 2 + G y ⁡ ( x , y ) 2 ⁢ ⁢ and α ⁡ ( x , y ) = tan - 1 ⁢ G y ⁡ ( x , y ) G x ⁡ ( x , y ) .
(3) A gradient histogram is established, and specifically, the image is firstly divided into a plurality of cells, for example, cells each composed of 2*2 pixels, an orientation value of the gradient histogram is set to 0-180°, and a bin is obtained every 20°, wherein a weight of oriented gradient within the gradient histogram is determined by a magnitude value of the gradient.
(4) Several ones of the cells are grouped into one block, for example, 2*2 cells are grouped into one block, and gradient histogram features of all cells in one block are connected in series and normalized, wherein the cells are usually overlapped to some extent. Assuming that v denotes a gradient intensity distribution vector which is not normalized, a formula may be obtained by applying a normalization method as follows:
L ⁢ 2 - n ⁢ orm ⁢ : ⁢ ⁢ f = v v 2 2 + e 2
where e is a constant term with a small value.
(5) In the entire image, HOG features of all the blocks are combined to obtain the HOG feature describing the image.
In some implementations of the present embodiment, acquiring the middle-level semantic feature of the image to be classified according to the low-level feature of the target to be classified contained in the image to be classified through the visual dictionary further comprises: acquiring a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified.
In some implementations of the present embodiment, before acquiring a middle-level semantic feature of an image to be classified through a visual dictionary, the method further comprises:
acquiring targets to be classified contained in a plurality of training images, extracting low-level features of the targets to be classified contained in the plurality of training images, and constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images.
In some implementations of the present embodiment, before acquiring the targets to be classified contained in the plurality of training images, the method further comprises: performing image enhancement on the plurality of training images.
In some implementations of the present embodiment, performing image enhancement on the plurality of training images further comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence on the plurality of training images.
In some implementations of the present embodiment, before classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features, the method further comprises: acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary; and
training the middle-level semantic features of the plurality of training images as training samples using a machine learning algorithm to obtain the classification model based on the middle-level semantic features.
In some implementations of the present embodiment, acquiring the targets to be classified contained in the plurality of training images further comprises: acquiring the targets to be classified contained in the plurality of training images through image segmentation.
In some implementations of the present embodiment, acquiring the targets to be classified contained in the plurality of training images through image segmentation further comprises: performing image segmentation on the plurality of training images based on a minimum circumscribed rectangle algorithm, to obtain the targets to be classified contained in the plurality of training images.
In some implementations of the present embodiment, the low-level features of the targets to be classified contained in the plurality of training images are histogram of oriented gradient features.
In some implementations of the present embodiment, constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images further comprises: clustering histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using a K-means algorithm to obtain visual words, and constructing the visual dictionary according to the visual words.
In some implementations of the present embodiment, acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary comprises: acquiring visual words which have the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images as the middle-level semantic features of the plurality of training images.
It is beneficial to model the features of the target of the image by applying overall statistical information of the image in the image analysis without analyzing specific composition of the target to be classified contained in the image. A Bag of Words (BoW) model is such a middle-level feature representation method. The BoW model treats the image as a document composed of visual words, so that the BoW model and related theory in the field of text classification are applied to the understanding of the image, without analyzing and interpreting specific composition of targets in an object, and instead, a plurality of sample images are applied as training samples, low-level features of the sample images are quantized into visual words, and content of an unknown target is expressed using a distribution histogram of the visual words of the image.
In a specific example, as shown in FIG. 3, the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images are clustered using a K-means algorithm to obtain visual words, and a flow of constructing a visual dictionary according to the visual words is as follows.
Visual words, for example, t1, t2, t3, t4 as shown in FIG. 3, are formed from the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using the K-means algorithm. In fact, t1, t2, t3, t4 are centroids of clusters, and all the visual words form a visual dictionary. The K-means algorithm is an unsupervised machine learning algorithm. When clustering is performed using the algorithm, N objects are divided into k classes based on a criterion that there is a high similarity within the classes and a low similarity between the classes. A specific flow is as follows.
(1) Extracted features of various objects of interest form a feature space: X=[x1, x2, . . . , xN], where xn∈Rd is a low-level feature.
(2) k cluster centers u1, u2, . . . , uk are randomly selected;
(3) Distances from each low-level feature xi to the cluster centers are calculated, then each feature is re-clustered to a cluster center nearest thereto, and Rk records feature tags contained in class k:
R k ={i|∥x i −u k ∥≤∥x i −u i ∥,k≠l},k=1,2, . . . ,K
Thereby, the targets are re-clustered into k classes {R1, R2, . . . , Rk}.
(4) A center of each new class is re-calculated and the visual dictionary is updated:
u k = 1 R k ⁢ ∑ i ∈ R k ⁢ x i
(5) The above steps are repeated until the convergence requirements are met to obtain an optimal visual dictionary:
U={u 1 ,u 2 , . . . ,u k },u k ∈R d.
A specific flow of acquiring a visual word which has the closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified is as follows.
A histogram of visual words is extracted. According to the nearest neighbor criterion, each feature is quantized into a visual word having the closest Euclidean distance thereto. A formula is expressed as follows:
min V ⁢ X - UV 2 2 , ⁢ s . t . ⁢ v i 0 = v i 1 = 1
where X is a low-level feature space of each image, U is a visual dictionary obtained by K-means clustering, V represents a correspondence relationship between low-level features and visual words, a conditional constraint ∥vi∥0=1 indicates that there may only be one non-zero value in a vector vi, a conditional constraint ∥vi∥1=1 indicates that a cumulative sum of absolute values of numbers in the vector vi is 1, and thereby ∥vi∥0=∥vi∥1=1 defines that there is only one “1” in the vector vi, that is, a low-level feature may be quantized into a visual word having the closest Euclidean distance thereto.
Then, a histogram of visual words of each image is:
V i=[v 1 +v 2 + . . . +v k],v k ∈R k×1
The histogram of the visual words is the middle-level semantic feature of the image, and is applied to sample images. Firstly, low-level features of targets to be classified contained in the images are quantized into visual words, and then content of the unknown targets is expressed by a distribution histogram of the visual words of the images. Since overall statistical information of the images is applied to the middle-level semantic feature without analyzing specific composition of the targets contained in the images, there are high accuracy and high robustness in modeling the features of the targets of the images.
It may be understood that visual words having the closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images may be acquired as middle-level semantic features of the plurality of training images in a specific flow similar to that described above, and will not be described in detail here.
In some implementations of the present embodiment, a machine learning algorithm is an SVM algorithm. In a specific example, the middle-level semantic features of the plurality of training images are trained using a linear SVM classifier to obtain a classification model based on the middle-level semantic features, and an image to be classified is classified according to the classification model based on the middle-level semantic features, to obtain a result of the image classification.
In a specific example, as shown in FIG. 4, when the image classification method according to the present embodiment is applied to the classification of display screen defects, a specific flow of the method is as follows.
A plurality of first display images (i.e., a plurality of training images) of a plurality of first display screens under test (which usually display an image in a pure color, so that a target to be classified different from a background represents a display screen defect) and at least one second display image (i.e., an image to be classified) of at least one second display screen under test having a display screen defect to be classified are collected respectively, the first display images are used as training samples, and the second display image is used as a test sample, wherein, “under test” indicates that the display screen displays a test image, and a camera collects the test image displayed on the display screen to classify the display screen defect.
Image enhancement is performed on the training samples and the test sample respectively.
Display screen defect targets contained in the training samples and the test sample are obtained through image segmentation respectively.
Low-level features of the display screen defect targets of the training samples and the test sample are extracted respectively.
A visual dictionary is constructed according to the low-level features of the display screen defect targets of the training samples, and middle-level semantic features of the training samples and the test sample are acquired according to the visual dictionary respectively.
The middle-level semantic features of the training samples are trained using a machine learning algorithm to obtain a classification model, and the test sample is classified according to the middle-level semantic feature of the test sample using the classification model.
Feature extraction is a key step in the classification of display screen defects. Low-level features such as geometry, textures, shapes, local descriptors etc. used in the existing display screen defect classification method based on image classification are extracted based on information at an image level. Since the feature extraction has not been sufficiently generalized and abstracted, it is far from a concept level, display screen defect classification based on low-level features is not highly accurate, and it is difficult to accurately classify an image having a defect outside a training set.
With the flow in the present example, the middle-level semantic feature is formed by analyzing a statistical distribution of the low-level feature of the display screen defect target, i.e., establishing a relationship with semantics by analyzing a statistical distribution of the low-level feature, which reduces a semantic gap between a low-level feature and a high-level feature, avoids the influence on the classification of display screen defects by factors such as camera parameters, a shooting mode and a shooting environment etc. of a camera when the camera collects a display image of a display screen under test, solves the problem of large intra-class variation when the classification of display screen defects is performed based on the low-level feature, has advantages such as high accuracy, high robustness, high efficiency, etc., and thus has a good foundation for subsequent analysis of causes of the display screen defects, improvement of industrial production methods, and improvement of the yield of the production of display screens. Furthermore computer storage space and a large amount of calculation can be reduced
In another description, the specific flow of the present example comprises two phases, which are a training phase and a classification phase respectively.
In the training phase,
a plurality of first display images of a plurality of first display screens under test are collected, and the first display images are used as training samples;
image enhancement is performed on the training samples;
display screen defect targets of the training samples are obtained through image segmentation;
low-level features of the display screen defect targets of the training samples are extracted;
a visual dictionary is constructed according to the low-level features of the display screen defect targets of the training samples, and middle-level semantic features of the training samples are acquired through the visual dictionary; and
the middle-level semantic features of the training samples are trained using a machine learning algorithm to obtain a classification model.
In the classification phase,
at least one second display image of at least one second display screen to be classified under test is collected, and the second display image is used as a test sample;
image enhancement is performed on the test sample;
a display screen defect target of the test sample is obtained through image segmentation;
a low-level feature of the display screen defect target of the test sample is extracted;
a middle-level semantic feature of the test sample is acquired through the visual dictionary; and
the test sample is classified using the classification model according to the middle-level semantic feature of the test sample.
As shown in FIG. 5, another embodiment of the present disclosure provides an image classification system which performs the image classification method described above, comprising:
a middle-level semantic feature acquisition module configured to acquire a middle-level semantic feature of a target to be classified in an image to be classified through a visual dictionary; and
a classification module configured to classify the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
It should be illustrated that the principle and the working flow of the image classification system according to the present embodiment are similar to those of the image classification method described above, and related description may be known with reference to the above description, and will not be described in detail here.
As shown in FIG. 6, illustrated is a computer system suitable for implementing the image classification system according to the present embodiment, comprising a Central Processing Unit (CPU) which may perform various appropriate actions and processes based on a program stored in a Read Only Memory (ROM) or a program loaded from a storage portion into a Random Access Memory (RAM). Various programs and data required for operations of a computer system are also stored in the RAM. The CPU, the ROM, and the RAM are connected to each other through a bus. An Input/Output (I/O) interface is also connected to the bus.
The following components are connected to the I/O interface: an input part comprising a keyboard, a mouse, etc.; an output part comprising a Liquid Crystal Display (LCD), etc., and a speaker; a storage part comprising a hard disk etc.; and a communication part comprising a network interface card such as a LAN card, a modem, etc. The communication part performs communication processing via a network such as the Internet. The driver is also connected to the I/O interface as needed. A removable medium, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory etc., is mounted on the driver as needed, so that a computer program read therefrom is installed into the storage part as needed.
Particularly, according to the present embodiment, the processes described above in the flowcharts may be implemented as a computer software program. For example, the present embodiment comprises a computer program product comprising a computer program included tangibly on a computer readable medium, the computer program comprising program codes for performing the methods illustrated in the above flowcharts. In such an embodiment, the computer program may be downloaded and installed from the network via a communication portion, and/or installed from a removable medium.
The flowcharts and diagrams in the accompanying drawings illustrate architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to the present embodiment. In this regard, each block of the flowcharts or diagrams may represent a module, a program segment, or a portion of codes, which comprises one or more executable instructions for implementing specified logic functions. It should also be illustrated that in some alternative implementations, the functions illustrated in the blocks may also occur in a different order than that illustrated in the accompanying drawings. For example, two successively represented blocks may in fact be executed substantially in parallel, or they may sometimes be executed in a reverse order, depending upon functionality involved. It is also illustrated that each block of the diagrams and/or flowcharts, and combinations of blocks in the diagrams and/or flowcharts, may be implemented by a dedicated hardware-based system which performs specified functions or operations, or may be implemented by combination of dedicated hardware and computer instructions.
Modules described in the present embodiment may be implemented by software, or may be implemented by hardware, and the described modules may also be disposed in a processor. For example, there may be a description of a processor, comprising a middle-level semantic feature acquisition module and a classification module. Here, names of these modules do not in any way constitute a limitation on the modules themselves. For example, the middle-level semantic feature acquisition module may also be described as “visual dictionary module”.
In another aspect, the present embodiment further provides a non-volatile computer storage medium, which may be a non-volatile computer storage medium included in the above apparatus in the above embodiments; or may exist separately but is not assembled into a non-volatile computer storage medium in a terminal. The above non-volatile computer storage medium stores one or more programs which, when executed by a device, cause the device to:
acquire a middle-level semantic feature of an image to be classified through a visual dictionary; and
classify the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features.
It should also be illustrated that in the description of the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation without necessarily requiring or implying that there is any such actual relationship or order between these entities or operations. Moreover, the terms “comprises”, “includes” or any other variation thereof are intended to cover a non-exclusive inclusion, so that a process, method, article, or device which comprises a series of elements comprises not only these elements but also comprises other elements which are not explicitly listed, or elements inherent to such a process, method, article, or device. In a case of no more limitation, an element defined by a sentence “including one . . . ” does not exclude that there is another element which is the same as the element in the process, the method, the article, or the device including the element.
It should be understood that the above embodiments of the present disclosure are merely examples for clearly illustrating the present disclosure, and are not limitations on the embodiments of the present disclosure. For those of ordinary skill in the art, other changes or modifications in other forms can also be made based on the above description. All embodiments cannot be exhaustively described herein, and any obvious changes or modifications derived from the technical solutions of the present disclosure are still within the protection scope of the present disclosure.

Claims (18)

We claim:
1. An image classification method, comprising:
acquiring a target to be classified contained in the image to be classified;
extracting a low-level feature of the target to be classified contained in the image to be classified;
acquiring a middle-level semantic feature of an image to be classified according to the low-level feature through a visual dictionary; and
classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features
wherein the low-level feature comprises geometric shapes, textures, local invariant features, and transform domain features, and the middle-level semantic feature comprises a bag of visual words and a semantic topic.
2. The method according to claim 1, wherein acquiring a target to be classified contained in the image to be classified comprises: acquiring the target to be classified contained in the image to be classified through image segmentation.
3. The method according to claim 2, wherein the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
4. The method according to claim 3, wherein the low-level feature is a histogram of oriented gradient feature.
5. The method according to claim 4, wherein acquiring the middle-level semantic feature of the image to be classified according to the low-level feature through the visual dictionary comprises:
acquiring, according to the low-level feature, a visual word which has a closest Euclidean distance to the histogram of oriented gradient feature of the target to be classified contained in the image to be classified as the middle-level semantic feature of the image to be classified.
6. The method according to claim 1, wherein before acquiring the target to be classified contained in the image to be classified, the method further comprises:
performing image enhancement on the image to be classified.
7. The method according to claim 6, wherein performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
8. The method according to claim 1, wherein before acquiring a middle-level semantic feature of an image to be classified through a visual dictionary, the method further comprises:
acquiring targets to be classified contained in a plurality of training images, extracting low-level features of the targets to be classified contained in the plurality of training images, and constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images.
9. The method according to claim 8, wherein before classifying the image to be classified according to the middle-level semantic feature of the image to be classified using a classification model based on middle-level semantic features, the method further comprises:
acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary; and
training the middle-level semantic features of the plurality of training images as training samples using a machine learning algorithm to obtain the classification model based on the middle-level semantic features.
10. The method according to claim 9, wherein acquiring the targets to be classified contained in the plurality of training images comprises: acquiring the targets to be classified contained in the plurality of training images through image segmentation.
11. The method according to claim 10, wherein:
the image segmentation is implemented using a minimum circumscribed rectangle algorithm.
12. The method according to claim 11, wherein the low-level features are histogram of oriented gradient features.
13. The method according to claim 12, wherein constructing the visual dictionary according to the low-level features of the targets to be classified contained in the plurality of training images comprises:
clustering histogram of oriented gradient features of the targets to be classified contained in the plurality of training images using a K-means algorithm to obtain visual words; and
constructing the visual dictionary according to the visual words.
14. The method according to claim 13, wherein acquiring middle-level semantic features of the plurality of training images according to the low-level features of the targets to be classified contained in the plurality of training images through the visual dictionary comprises:
acquiring, according to the low-level features, visual words which have closest Euclidean distances to the histogram of oriented gradient features of the targets to be classified contained in the plurality of training images as the middle-level semantic features of the plurality of training images.
15. The method according to claim 8, wherein before acquiring the targets to be classified contained in the plurality of training images, the method further comprises: performing image enhancement on the plurality of training images.
16. The method according to claim 15, wherein performing image enhancement comprises: performing graying, wavelet de-noising, otsu threshold segmentation, binary expansion, median filtering, and binary corrosion in sequence.
17. A computer device comprising a memory, a processor, and a computer program stored on the memory and operative on the processor, wherein the program, when executed by the processor, implements the method according to claim 1.
18. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method according to claim 1.
US16/556,697 2019-03-08 2019-08-30 Image classification method, computer device and medium Active 2039-11-08 US11144799B2 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910175377.3 2019-03-08
CN201910175377.3A CN109858570A (en) 2019-03-08 2019-03-08 Image classification method and system, computer equipment and medium

Publications (2)

Publication Number Publication Date
US20200285917A1 US20200285917A1 (en) 2020-09-10
US11144799B2 true US11144799B2 (en) 2021-10-12

Family

ID=66900257

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/556,697 Active 2039-11-08 US11144799B2 (en) 2019-03-08 2019-08-30 Image classification method, computer device and medium

Country Status (2)

Country Link
US (1) US11144799B2 (en)
CN (1) CN109858570A (en)

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110414538B (en) * 2019-07-24 2022-05-27 京东方科技集团股份有限公司 Defect classification method, defect classification training method and device thereof
US12056880B2 (en) * 2020-08-03 2024-08-06 Korea Advanced Institute Of Science And Technology Method of classifying lesion of chest x-ray radiograph based on data normalization and local patch and apparatus thereof
CN112183559B (en) * 2020-10-27 2022-01-11 深圳市威富视界有限公司 Image recognition model training method, image recognition method and device
CN112347899B (en) * 2020-11-03 2023-09-19 广州杰赛科技股份有限公司 A moving target image extraction method, device, equipment and storage medium
CN112668567A (en) * 2020-12-25 2021-04-16 深圳太极云软技术有限公司 Image clipping algorithm based on deep learning
CN115205571B (en) * 2021-04-13 2026-01-02 武汉Tcl集团工业研究院有限公司 An image classification method, apparatus, terminal device, and storage medium
CN115619800A (en) * 2021-07-13 2023-01-17 电子科技大学 Segmentation method and device for COVID-19 CT images based on adaptive threshold segmentation
CN116863198A (en) * 2023-05-26 2023-10-10 中国银行股份有限公司 Image classification method, apparatus, computer device and storage medium
CN116721099B (en) * 2023-08-09 2023-11-21 山东奥洛瑞医疗科技有限公司 Image segmentation method of liver CT image based on clustering
CN116994109B (en) * 2023-08-10 2025-09-16 商汤集团有限公司 Image processing method, device, computer equipment and storage medium

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102208038A (en) 2011-06-27 2011-10-05 清华大学 Image classification method based on visual dictionary
US20120075638A1 (en) * 2010-08-02 2012-03-29 Case Western Reserve University Segmentation and quantification for intravascular optical coherence tomography images
CN102509110A (en) 2011-10-24 2012-06-20 中国科学院自动化研究所 Method for classifying images by performing pairwise-constraint-based online dictionary reweighting
US20120290577A1 (en) 2011-05-13 2012-11-15 Microsoft Corporation Identifying visual contextual synonyms
US20130080426A1 (en) * 2011-09-26 2013-03-28 Xue-wen Chen System and methods of integrating visual features and textual features for image searching
CN104850859A (en) 2015-05-25 2015-08-19 电子科技大学 Multi-scale analysis based image feature bag constructing method
US20160132754A1 (en) * 2012-05-25 2016-05-12 The Johns Hopkins University Integrated real-time tracking system for normal and anomaly tracking and the methods therefor
CN107515905A (en) 2017-08-02 2017-12-26 北京邮电大学 An Interactive Image Search and Fusion Method Based on Sketch
US20190331301A1 (en) * 2016-12-30 2019-10-31 Du Yuchuan Method for leakage detection of underground pipeline corridor based on dynamic infrared thermal image processing

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120075638A1 (en) * 2010-08-02 2012-03-29 Case Western Reserve University Segmentation and quantification for intravascular optical coherence tomography images
US20120290577A1 (en) 2011-05-13 2012-11-15 Microsoft Corporation Identifying visual contextual synonyms
CN102208038A (en) 2011-06-27 2011-10-05 清华大学 Image classification method based on visual dictionary
US20130080426A1 (en) * 2011-09-26 2013-03-28 Xue-wen Chen System and methods of integrating visual features and textual features for image searching
CN102509110A (en) 2011-10-24 2012-06-20 中国科学院自动化研究所 Method for classifying images by performing pairwise-constraint-based online dictionary reweighting
US20160132754A1 (en) * 2012-05-25 2016-05-12 The Johns Hopkins University Integrated real-time tracking system for normal and anomaly tracking and the methods therefor
CN104850859A (en) 2015-05-25 2015-08-19 电子科技大学 Multi-scale analysis based image feature bag constructing method
US20190331301A1 (en) * 2016-12-30 2019-10-31 Du Yuchuan Method for leakage detection of underground pipeline corridor based on dynamic infrared thermal image processing
CN107515905A (en) 2017-08-02 2017-12-26 北京邮电大学 An Interactive Image Search and Fusion Method Based on Sketch

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
First Chinese Office Action dated Jul. 23, 2020, for corresponding Chinese Application No. 201910175377.3.
Liu Yuhuan, "Product Defect Detection Based on Computer Vision", Mar. 2018.

Also Published As

Publication number Publication date
US20200285917A1 (en) 2020-09-10
CN109858570A (en) 2019-06-07

Similar Documents

Publication Publication Date Title
US11144799B2 (en) Image classification method, computer device and medium
US11334982B2 (en) Method for defect classification, method for training defect classifier, and apparatus thereof
CN110334706B (en) Image target identification method and device
Al-Shemarry et al. Ensemble of adaboost cascades of 3L-LBPs classifiers for license plates detection with low quality images
Chen et al. Text detection and recognition in images and video frames
Jia et al. Region-based license plate detection
Pan et al. A robust system to detect and localize texts in natural scene images
EP2701098B1 (en) Region refocusing for data-driven object localization
CN111126115B (en) Violent sorting behavior identification method and device
WO2019232853A1 (en) Chinese model training method, chinese image recognition method, device, apparatus and medium
Molina-Moreno et al. Efficient scale-adaptive license plate detection system
CN111507344A (en) Method and device for recognizing text from images
Liu et al. A novel SVM network using HOG feature for prohibition traffic sign recognition
Shi et al. License plate localization in complex environments based on improved GrabCut algorithm
Ismail et al. Statistical Binarization Techniques for Document Image Analysis.
CN104834891A (en) Method and system for filtering Chinese character image type spam
CN113344047A (en) Platen state identification method based on improved K-means algorithm
Gaceb et al. Adaptative smart-binarization method: For images of business documents
Gaddour et al. A new method for arabic text detection in natural scene image based on the color homogeneity
Qin et al. Video scene text frames categorization for text detection and recognition
Huang et al. Text extraction in natural scenes using region-based method
Tung et al. Efficient uneven-lighting image binarization by support vector machines
Pack et al. Augmentation-based pseudo-ground truth generation for deep learning in historical document segmentation for greater levels of archival description and access
Kaur et al. Text Extraction from Natural Scene using PCA.
Kosala et al. Robust License Plate Detection in Complex Scene using MSER-Dominant Vertical Sobel.

Legal Events

Date Code Title Description
AS Assignment

Owner name: BEIJING BOE OPTOELECTRONICS TECHNOLOGY CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PENG, XIANGJUN;WANG, YUNQI;ZHAO, CHENXI;AND OTHERS;SIGNING DATES FROM 20190603 TO 20190718;REEL/FRAME:050237/0281

Owner name: BOE TECHNOLOGY GROUP CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PENG, XIANGJUN;WANG, YUNQI;ZHAO, CHENXI;AND OTHERS;SIGNING DATES FROM 20190603 TO 20190718;REEL/FRAME:050237/0281

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4