KR102158967B1 - Image analysis apparatus, image analysis method and recording medium - Google Patents

Abstract

An image analysis method and an image analysis apparatus are provided. The image analysis method of the present invention includes obtaining an image including one or more objects, extracting a plurality of detailed regions from the image, analyzing a hierarchical structure between the plurality of detailed regions, and the hierarchical structure It may include classifying the object included in the image by using.

Description

Image analysis device, image analysis method and recording medium {IMAGE ANALYSIS APPARATUS, IMAGE ANALYSIS METHOD AND RECORDING MEDIUM}

The present disclosure relates to an image analysis apparatus and method. More specifically, the present disclosure relates to an apparatus and method for detecting and classifying an object included in an input image. The input image may be, for example, a 3D image. The 3D image may be, for example, a 3D security CT image.

In security facilities such as ports, airports, or research facilities, there is a need to search for the belongings of passers-by in order to strengthen security and prevent technology leakage. At this time, as a technology for more quickly and efficiently searching for belongings for a large number of passengers, an article search system using radiation (X-ray) or the like is often used.

This product retrieval system is particularly widely used in customs electronic customs clearance systems or security inspection systems. For example, the customs electronic customs clearance system is a computerized customs clearance service for import and export cargo, which can improve the efficiency of multilateral customs administration.

In addition, the security inspection system is a computerized security inspection task that determines whether there are items that may cause safety or security problems in the belongings of passers-by, and through this, it is possible to enhance the security of the security area.

On the other hand, deep learning is to learn a very large amount of data, and when new data is input, the highest probability is selected based on the learning result, and can be adaptively operated according to the image. In addition, since feature factors are automatically found in the process of learning models based on data, attempts to utilize them in the field of artificial intelligence are increasing.

In addition, security search using 3D security CT images is currently in progress, and research on a new method that considers the characteristics of 3D images differentiated from 2D images is required.

An object of the present disclosure is to provide an image analysis method and apparatus for classifying objects included in an image.

In addition, an object of the present disclosure is to provide an image analysis method and apparatus for classifying objects using a hierarchical structure of objects included in an image.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present disclosure belongs from the following description. I will be able to.

An image analysis method according to an aspect of the present invention includes obtaining an image including one or more objects, extracting a plurality of detailed regions from the image, analyzing a hierarchical structure between the plurality of detailed regions, and And classifying the object included in the image using the hierarchical structure.

In the present invention, the image may be a three-dimensional image.

In the present invention, the step of extracting a plurality of detailed regions from the image may include binarizing the image to extract an initial detailed region, and subdividing the initial detailed region to extract a final detailed region. have.

In the present invention, the subdividing of the initial detailed region may be performed based on values of voxels included in the initial detailed region.

In the present invention, the subdividing of the initial detailed region may be performed by using variance of values of the voxels.

In the present invention, the subdividing of the initial detailed region comprises, if the variance of values of the voxels is greater than a predetermined threshold, the initial detailed region is subdivided into two or more subregions, and the subregion obtained by subdividing is initially As a detailed region, the initial region may be subdivided.

In the present invention, in the subdividing of the initial detailed region, if the variance of values of the voxels is less than or equal to a predetermined threshold, the initial detailed region may be determined as the final detailed region.

In the present invention, the step of analyzing the hierarchical structure between the plurality of sub-areas comprises determining whether two or more sub-areas among the plurality of sub-areas can be merged into one area, and if mergeable, the two or more sub-areas It may include merging regions into the one region, and determining a hierarchical structure in which the merged region is an upper region and the two or more detailed regions are a lower region.

In the present invention, the step of analyzing the hierarchical structure may be performed recursively until the plurality of detailed regions are merged into one region.

In the present invention, the step of determining whether two or more detailed regions among the plurality of detailed regions can be merged into one region comprises: pixel values in the detailed region, the location of the detailed region, the size of the detailed region, the shape of the detailed region, and merging It may be performed based on at least one of the shape of the formed area and the degree of similarity of adjacent portions between detailed areas.

In the present invention, the step of classifying the object included in the image using the hierarchical structure is performed using a classification result of an object corresponding to an upper region of the object or an object corresponding to a lower region of the object. Can be.

In the present invention, a probability to be applied to the classification result of the object is predefined according to the classification result of the object corresponding to the upper region of the object or the object corresponding to the lower region of the object, and the image In the step of classifying the object included in, the object may be classified using the probability.

An image analysis apparatus according to another aspect of the present invention may include one or more microprocessors that perform the image analysis method according to the present disclosure.

A recording medium according to another aspect of the present invention may store a program for executing an image analysis method according to the present disclosure.

The features briefly summarized above with respect to the present invention are merely exemplary aspects of the detailed description of the present invention to be described later, and do not limit the scope of the present invention.

According to the present disclosure, an image analysis method and apparatus for classifying objects included in an image may be provided.

In addition, according to the present disclosure, an image analysis method and apparatus for classifying an object using a hierarchical structure of objects included in an image may be provided.

The effects that can be obtained in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those of ordinary skill in the art from the following description. will be.

1 is a diagram illustrating an article search system according to an exemplary embodiment of the present disclosure.
2 is a block diagram illustrating a configuration of an image analysis apparatus 200 according to an embodiment of the present disclosure.
3 is a diagram for explaining an image reading process.
4 is a diagram for explaining an application range of artificial intelligence in an image reading process according to an embodiment of the present disclosure.
5 is a diagram showing an embodiment of an image enhancement apparatus for performing image enhancement according to the present disclosure.
6 is a diagram illustrating a process of classifying an object and a background from an image including a single object, and generating location information of the object, according to an embodiment of the present disclosure.
7 is a diagram illustrating an image in which colors are expressed based on physical properties of an object according to an embodiment of the present disclosure.
8 is a diagram for describing a process of generating an output image based on color distribution information of an image according to an exemplary embodiment of the present disclosure.
FIG. 9 is a diagram for describing a process of obtaining a final output image obtained by combining an image obtained by using color distribution information and an image obtained by applying edge-based filtering or smoothing filtering according to an embodiment of the present disclosure.
10 is a diagram for explaining a process of obtaining a final output image using a graphical model according to an embodiment of the present disclosure.
11 is a diagram for describing an image enhancement method according to an embodiment of the present disclosure.
12 is a diagram for describing a context analysis according to an embodiment of the present disclosure.
13 is a diagram illustrating a process of generating and analyzing context information of an image according to an embodiment of the present disclosure.
14 is a diagram illustrating a process of identifying an object by analyzing an image by an image analysis apparatus according to an embodiment of the present disclosure.
15 is a diagram for describing an operation of an image analysis apparatus according to an exemplary embodiment of the present disclosure.
16 is a diagram for explaining an embodiment of a convolutional neural network that generates a multi-channel feature map.
17 is a diagram for describing an embodiment of a pooling technique.
18 is a block diagram illustrating a configuration of an image synthesizing apparatus according to an embodiment of the present disclosure.
19 is a diagram illustrating a process of generating a multi-object image using two images including a single object according to an embodiment of the present disclosure.
20 is a diagram illustrating a process of learning a convolutional neural network using a multi-object image according to an embodiment of the present disclosure.
21 is a diagram for explaining a process of analyzing an actual image using an image synthesizing apparatus according to an embodiment of the present disclosure.
22 is a diagram for describing an image synthesis method according to an embodiment of the present disclosure.
23 is a diagram illustrating an embodiment of the present invention for detecting an object included in a 3D image.
24 is a diagram for describing an embodiment of merging and/or dividing detailed regions.
25 is a diagram for describing an embodiment of detecting an object from a traveler's bag according to the present invention.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the embodiments. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

In describing an embodiment of the present disclosure, when it is determined that a detailed description of a known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, parts not related to the description of the present disclosure in the drawings are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" with another component, it is not only a direct connection relationship, but an indirect connection relationship in which another component exists in the middle. It can also include. In addition, when a certain component "includes" or "have" another component, it means that other components may be further included rather than excluding other components unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of the components unless otherwise stated. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment is a first component in another embodiment. It can also be called.

In the present disclosure, components that are distinguished from each other are intended to clearly describe each feature, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to be formed in one hardware or software unit, or one component may be distributed in a plurality of hardware or software units. Therefore, even if not stated otherwise, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment consisting of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other elements in addition to the elements described in the various embodiments are included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

1 is a diagram illustrating an article search system according to an exemplary embodiment of the present disclosure.

The article retrieval system 100 may include a reading unit 110 and/or a learning unit 120. The reading unit 110 may include an image analysis device 112 and/or an output device 114. The learning unit 120 may include a database 122, a deep learning learning unit 124, an algorithm verification unit 126, and/or a learned model storage unit 128. The reading unit 110 can function as a reading interface, and the learning unit 120 can function as a centrally managed AI data center.

Hereinafter, a case where the article retrieval system according to the present disclosure is used in an electronic customs clearance system or a security retrieval system will be described as an example. However, the product search system according to the present disclosure is not limited to such use. In addition, the product search system according to the present disclosure may be utilized in a system that performs a role of identifying a specific product according to various purposes.

The input 130 of the product search system 100 may include an image, product information and/or control information.

The image may be an image of an article including at least one object. For example, it may be an X-ray image of an article photographed by an X-ray reading device. The image may be a raw image captured by an X-ray imaging device, or may be an image in any form (format) for storing or transmitting the raw image. The image may be obtained by capturing image information that is captured by an X-ray reader and transmitted to an output device such as a monitor, and converting it into data. The image may be enhanced before being output to the output device 114 or before being input to the image analysis device 112. A method of enhancing the image will be described later. The output device 114 may output an image or an enhanced image. The image analysis device 112 may receive an image or an enhanced image and perform an operation of the image analysis device 112 to be described later.

The article information may be information on an article included in a corresponding image. For example, when an article input to the image analysis device is a cargo in an electronic customs clearance system, the article information may include import declaration information and/or customs clearance list list information. As another example, when the article input to the video analysis device is an article in the security inspection system, the article information may include identification information of a passerby, security level of a passerby, and/or information on an authorized item of the passerby.

The product information may undergo a predetermined pre-processing process before being input to the image analysis device 112. For example, an item name refinement operation may be performed on an item list, carry-in information, etc. included in the item information. The refining operation of the product name may refer to an operation of unifying the names of various items entered for the same or similar items.

Entry of the article information may be optional. For example, the product retrieval system 100 of the present disclosure may operate by receiving only an image as an input even without input of product information. The article may be an article to be inspected or read, and may include all kinds of articles. For example, when the image analysis device according to the present disclosure is used in an electronic customs clearance system, the article may be at least one of express cargo, mail cargo, container cargo, traveler transport cargo, and traveler himself. As another example, when the image analysis device according to the present disclosure is used in a security search system, the article may be at least one of a passerby's belongings and a passerby himself.

For example, if the electronic customs clearance system reads the traveler, and the read traveler is a critical traveler with a history of transporting abnormal or dangerous objects in the past, the traveler's cargo will be at a higher level than that of other travelers. Analysis and/or reading may be performed. For example, information that a specific article is a cargo of a tourist of interest may be provided to the reader.

As another example, as a result of reading a passer-by, if the passerby is a passerby with a high security level, the passer's belongings may be analyzed and/or read at a higher level than other passers-by. For example, it is possible to provide information to the reader that a specific article is a possession of a passerby with a high level of security.

The control information may be information for controlling image reading or for controlling the read image. As an example, the control information may be input by the reader 140. For example, the control information may include reader information, manager information, operation mode information, read sensitivity information and/or user interface information. The specific use of the control information will be described later.

The product search system 100 may receive an image, product information, and/or control information 130 and transmit it to the output device 114 or to the image analysis device 112. The image analysis device 112 may analyze the input image using a deep learning-based model that has been learned in advance. The image analysis device 112 may transmit the analyzed result to the output device 114. The output device 114 outputs the input image, product information and/or control information 130, the image analysis result and/or user interface transmitted from the image analysis device 112, and the reader 140 is an output device. The output result of (114) can be read. As described above, the refining operation may be performed on the product information 130, and the image to be analyzed is imaged before being input to the image analysis device 112 and/or before being output to the output device 114. Reinforcement can be performed.

The output device 114 outputs all types of signals that can be detected by humans, such as a device that outputs visual information such as a monitor and a warning light, a device that outputs sound information such as a speaker, and a device that outputs tactile information such as a vibrator. Includes devices capable of. A user interface may be provided through the output device 114, and the reader may control the operation of the product retrieval system 100 using the user interface. For example, the reader 140 may control the operation of the image analysis apparatus by inputting control information using an output user interface.

When the image analysis result of the image analysis device 112 includes an object to be detected, an object with an abnormality, or an object with a risk greater than or equal to a threshold in the corresponding image, related information is output through the output device 114 as the image analysis result. And the reader 140 can confirm this. The image analysis device 112 may perform various processes of analyzing an image to be analyzed. For example, the image analysis device 112 may perform context analysis in order to more accurately analyze an image to be analyzed. Various processes and context analysis performed by the image analysis device 112 will be described later.

The reader 140 may determine whether to perform an additional test based on the image analysis result output through the output device 114. The additional inspection may include a retrofit inspection of directly opening an article related to the image to check an object included in the article. In the present specification, the object to be searched may mean an object having an abnormality or an object having a risk of a threshold value or more, as described above. However, the present disclosure is not limited thereto, and various objects to be detected or searched for by the system of the present disclosure may be included.

The image analysis result of the image analysis device, the remodeling test result input after the reader directly performs the refurbishment test, and/or the matching result information obtained by matching the image and product information by the image analysis device will be transmitted to the learning unit 120. I can. The learning unit 120 may store newly received information in the database 122, and the deep learning learning unit 124 may perform deep learning learning using information stored in the database 122. Alternatively, the deep learning learning unit 124 may directly receive all or part of the training data without being stored in the database 122. The result learned by the deep learning learning unit 124 is verified by the algorithm verification unit 126, and the verified model may be stored as an updated model in the learned model storage unit 128. The model stored in the learned model storage unit 128 is transmitted to the image analysis device 112 again, and the image analysis device 112 may update and use the received model as the previously learned deep learning-based model. . The learning unit 120 may generate one synthesized image by receiving and synthesizing a plurality of images. In addition, a virtual image analysis result corresponding to the composite image, remodeling test result and/or matching result information may be generated using image analysis results, remodeling test results, and/or matching result information for each of the plurality of images. I can. The learning unit 120 may use the synthesized image and the generated virtual information as learning data. According to this, even if the number of training data is absolutely small, by synthesizing or merging these training data, a sufficient amount of training data required for training of an artificial intelligence model can be generated. Synthesis of images and generation of virtual information on the synthesized images will be described later.

The reading unit 110 and the learning unit 120 may be implemented as separate devices, or may be implemented in the same device. In addition, some or all of the configurations included in the reading unit 110 and the learning unit 120 may be composed of hardware or software.

Artificial intelligence technology allows computers to learn data and make decisions on their own like humans. Artificial neural networks are mathematical models inspired by neural networks in biology, and artificial neural networks are formed by combining synapses. By changing the strength of synaptic coupling through learning, neurons can mean the overall model with problem solving ability. Artificial neural networks are generally composed of an input layer, a hidden layer, and an output layer, and neurons included in each layer are connected through weights, and linear combination of weights and neuron values and nonlinearity. Through the activation function, the artificial neural network can have a form that can approximate complex functions. The purpose of artificial neural network training is to find a weight that minimizes the difference between the calculated output and the actual output in the output layer.

A deep neural network is an artificial neural network consisting of several hidden layers between the input layer and the output layer. Complex nonlinear relationships can be modeled through many hidden layers, and advanced abstraction is possible by increasing the number of layers. The structure is called deep learning. Deep learning learns a very large amount of data, and when new data is input, it selects the highest probabilistic answer based on the learning result, so it can operate adaptively according to the image and build a model based on the data. Feature factors can be found automatically during the learning process.

According to an embodiment of the present disclosure, a deep learning-based model is a fully convolutional neural network (a fully convolutional neural network), a convolutional neural network (convolutional neural network), a recurrent neural network (regression It may include at least one of a neural network, a recurrent neural network, a restricted Boltzmann machine (RBM), and a deep belief neural network (DBN), but is not limited thereto. Alternatively, machine learning methods other than deep learning may also be included. Alternatively, a hybrid model that combines deep learning and machine learning may be included. For example, a deep learning-based model may be applied to extract features of an image, and a machine learning-based model may be applied when classifying or recognizing an image based on the extracted features. The machine learning-based model may include, but is not limited to, a support vector machine (SVM), AdaBoost, and the like.

In addition, according to an embodiment of the present disclosure, a method of learning a deep learning-based model may include at least one of supervised learning, unsupervised learning, and reinforcement learning. , Is not limited thereto. Supervised learning is performed by using a series of training data and a corresponding label (label, target output value), and a neural network model based on supervised learning is a model that infers a function from training data. I can. Supervised learning receives a series of training data and the corresponding target output value, finds errors through learning that compares the actual output value of the input data with the target output value, and corrects the model based on the result. do. Supervised learning can be further divided into regression, classification, detection, semantic segmentation, and the like according to the shape of the result. Functions derived through supervised learning can be used to predict new outcomes again. In this way, the neural network model based on supervised learning optimizes the parameters of the neural network model through learning a number of training data.

According to an embodiment of the present disclosure, a deep learning-based model may use input images and information on an article for learning, and even after generating the learned model, the image and information on the article acquired by the device of the present disclosure are Can be used to update the neural network model. In addition, the deep learning-based model according to an embodiment of the present disclosure searches for an analysis result output by the method of the present disclosure, for example, whether there is an abnormality or risk of an identified object, information about the object, and the identified object. The neural network model may be updated using a prediction result such as whether or not the object is a target object, comparison information on the prediction result and the final remodeling test result, and evaluation or reliability information on the prediction result.

2 is a block diagram illustrating a configuration of an image analysis apparatus 200 according to an embodiment of the present disclosure. The image analysis apparatus 200 of FIG. 2 is an embodiment of the image analysis apparatus 112 of FIG. 1.

The image analysis apparatus 200 may include an image reception unit 210, an article information matching unit 220 and/or an image analysis unit 230. As described above, since the input of the article information is optional, the image analysis apparatus 200 may not include the article information matching unit 220. Description of the input of the article information is as described with reference to FIG.

The image receiver 210 may receive an image of an article including one or more objects. A description of the image received by the image receiving unit 210 is as described with reference to FIG. 1.

The article information matching unit 220 may perform matching between article information and the image by receiving article information and an image received from the image receiving unit 210 as inputs. Description of the article information is as described with reference to FIG. Matched images and article information can be output to the reader to assist the reader's reading task. Alternatively, the matched image and product information may be transmitted to the learning unit 120 of FIG. 1 and used for learning a deep learning model. The matched image and product information are stored in the database 122 of the learning unit 120 of FIG. 1, and then refined for each reading object and/or reading task, and the deep learning learning unit 124 is used for each reading object and/or Alternatively, learning can be performed using data refined for each reading task to be applied. The reading object may include express cargo, postal cargo, container cargo, traveler transport cargo, and traveler. In addition, the reading object may include passer-by's belongings and passer-by. The reading task is to determine whether the object contained in the object is abnormal or dangerous, determine whether the identified object is the object to be searched, determine whether the identified object matches information on the object, whether the object has been reported, or It may include a judgment as to whether it has not been reported. As described above, the model learned by the learning unit 124 is input to the image analysis unit 230 to update the existing model. At this time, appropriate artificial intelligence may be updated according to the object to be read. In addition, as described above, the learning unit 124 may generate new learning data using existing learning data and use it for learning. As described above, new learning data can be generated by synthesizing an existing image and merging data.

The image analysis unit 230 may receive an image (analysis target image) or image and product information, analyze the image using a pre-learned deep learning-based model, and output the analyzed result to an output device. have.

When only an image is received, the image analysis unit 230 may identify an object included in the image and determine whether there is an abnormality or a risk of the identified object. The image analysis unit 230 may improve the accuracy of object identification by performing a context analysis process to be described later.

For example, when it is determined that the identified object is prohibited or inappropriate, the image analysis unit 230 may determine that the object is abnormal or dangerous. The risk may be expressed as a numerical value, and it may be determined whether the object is a dangerous object through comparison with a predetermined threshold. The risk value and/or the predetermined threshold may be adaptively determined according to a reading object and/or a reading task.

When the image and product information are received together, the image analysis unit 230 may perform more precise analysis on an object included in the image by using the image and product information. For example, the type, quantity, and/or size information of the item listed in the item list list, the security level of the passerby, and/or the passer's authorized item information may be additionally used to identify the object from the image. When there is a discrepancy between the object and product information identified by analyzing the image, it may be output as a result of the image analysis.

The image analysis result output by the image analysis unit 230 may include at least one of a risk, type, amount, number, size, and location of the object. When the image analysis result is the location of the object, the location of the corresponding object may be displayed on the analysis target image and output to an output device. The location of the object may be displayed in coordinates, but the object can be highlighted and displayed at the corresponding location in the output image so that the reader can easily read it. For example, the edge of the object may be emphasized or the object may be emphasized by displaying a rectangular box surrounding the object. In addition, a predetermined object area may be strengthened so that the reader can more easily identify the object through the image enhancement process described later. For example, by enhancing an area corresponding to a predetermined color, the image may be transformed so that the area can be more clearly identified.

Alternatively, the image analysis unit 230 may determine whether an object to be searched (eg, an object for which customs clearance is prohibited or inappropriate) is included in the image to be analyzed. To this end, the image analysis unit 230 may receive or store information on the object to be searched in advance. Also, the image analysis unit 230 may identify an object included in the image and determine whether the identified object is a search target object.

3 is a diagram for explaining an image reading process.

3A is a flowchart illustrating a conventional reading process, and FIG. 3B is a flowchart illustrating a reading process according to an embodiment of the present disclosure.

As shown in (a) of FIG. 3, according to the existing reading process, when image and/or product information is input 311, it is provided 312 as information to the reader. The reader selects (313) an article requiring remodeling inspection based on the image and/or article information. The result of performing the retrofit inspection is input 314 as the inspection result.

As shown in (b) of FIG. 3, according to the reading process according to an embodiment of the present disclosure, when an image and/or product information is input 321, the image analysis device 322 is pre-learned deep learning. The image is analyzed using the base model, and the analyzed result is provided as information to the reader (324). In addition, the image analysis device 322 may transmit the training data to the artificial intelligence data center 323, and the artificial intelligence data center 323 may learn the training data. The artificial intelligence data center 323 may transmit the learned model to the image analysis device 322 as an auxiliary artificial intelligence for reading tasks for each reading object in the future.

The reader may select 325 an article requiring remodeling inspection based on an analysis result of the image analysis device 322, an image and/or article information. The result of performing the retrofit inspection may be input 326 as an inspection result. The test result may be transmitted to the artificial intelligence data center 323 and used as learning data.

4 is a diagram for explaining an application range of artificial intelligence in an image reading process according to an embodiment of the present disclosure.

As shown in FIG. 4, a sample 420 randomly extracted from among all the articles 410 may be selected 450 as a management target. Alternatively, a risk analysis 440 may be performed on all items 410 by using the screening auxiliary artificial intelligence 430 for selecting a management target, through which the management target may be selected 450.

The use of artificial intelligence is not limited to the risk analysis 440 of the above-described article. For example, if the management target is selected 450, it may be used as a test assistant artificial intelligence 460 to assist the test. For example, by applying the inspection assistant artificial intelligence 460, it is possible to assist the inspection of the reader by identifying the object, determining the existence of an abnormality or the risk of the identified object, and/or providing information on the object to be searched to the reader. . The reader may perform a detailed inspection 470 using information provided by the inspection assistant artificial intelligence.

Hereinafter, an embodiment of a method of enhancing an image before being input to the image analysis device 112 of FIG. 1 and/or before being output to the output device 114 will be described with reference to FIGS. 5 to 11.

5 is a diagram showing an embodiment of an image enhancement apparatus for performing image enhancement according to the present disclosure.

The image enhancement device of FIG. 5 may be configured separately from the image analysis device 112 of FIG. 1 or may be configured as a part thereof.

The image enhancing apparatus 500 may include an image receiving unit 510, an object image extracting unit 520, a color distribution analysis unit 530 and/or an image enhancing unit 540. However, this is only showing some components necessary for describing the present embodiment, and components included in the image enhancement apparatus 500 are not limited to the above-described examples. For example, two or more constituent units may be implemented in one constituent unit, or an operation executed in one constituent unit may be divided and implemented to be executed in two or more constituent units. In addition, some components may be omitted or additional components may be added.

The image enhancement apparatus 500 according to an embodiment of the present disclosure receives the input image 550, extracts an object included in the input image 550, and converts the object image including the object into one or more regions. Segmentation, obtaining color distribution information for each of the one or more regions, determining one or more weights for at least some of the one or more regions based on the color distribution information, and determining one or more weights among the one or more regions The first output image 560 for the object image may be generated by applying it to at least a portion.

Each pixel constituting an image may have a predetermined brightness and color by a combination of a luminance value indicating luminance (brightness) and a color value indicating color. In this case, the color value may be expressed by a combination of values of three or more color elements according to various ways of expressing the color. For example, the color value may be expressed as an RGB value that is a combination of three color elements (Red(R), Green(G), Blue(B)). For example, each of R, G, and B has a value of 0 to 255, and thus the intensity of each color element may be expressed. The range of values that each of R, G, and B can have may be determined based on the number of bits representing each of R, G, and B. For example, when represented by 8 bits, each of R, G, and B may have a value of 0 to 255.

Acquiring color distribution information may mean that various statistical values that can be obtained are obtained by analyzing color elements of color values of pixels included in the corresponding region. For example, the statistic value may be information on which color element has the largest average value among color elements of color values of pixels included in the corresponding area. For example, based on a value obtained by adding values of R, G, and B of all pixels included in the corresponding region, it may be determined which color element has the largest sum or average among R, G, and B. Alternatively, for each pixel, the color element having the largest value among R, G, and B is determined as the dominant color of the corresponding pixel, and which color is most often determined as the dominant color for all pixels included in the corresponding area. I can judge. In this way, it may be determined what the dominant color of the predetermined area is. For example, if R has the largest value among the three color elements (R, G, B) for color values of the majority of pixels included in a predetermined area, the dominant color of the predetermined area is considered to be red. I can judge. In the above description, color distribution information or dominant color was analyzed based on each of R, G, and B. However, the present invention is not limited thereto, and analysis may be performed based on various colors expressed by a combination of two or more of R, G, and B. For example, if the color to be identified is orange, it may be determined whether or not the dominant color of a pixel in the corresponding region is orange based on a combination of some or all of R, G, and B representing orange color.

Hereinafter, it is assumed that the red region of the dominant color is the target of the image enhancement, and an embodiment of a process of enhancing the image by applying a weight will be described in detail. If it is determined that the dominant color of a predetermined area is red based on the color distribution information, one or more weights may be determined for the corresponding area. The weights can be determined for R, G, B and all or part of the luminance. For example, in the case of enhancing red, the weight for R may be greater than 1. Applying a weight may mean multiplying a color element value of a pixel of a corresponding region by a corresponding weight. In this case, the weight for G and/or B may be a value less than 1. By doing so, the red dominant region can be reinforced to a redder region.

In the above, enhancement of a specific color of an image has been described. However, the enhancement of the image of the present disclosure is not limited thereto, and may include all changes in color values or changes in brightness values. Therefore, if necessary, the image can be enhanced by applying a weight to the luminance value as well.

Hereinafter, each component of the image enhancement device 500 will be described.

The image receiver 510 may receive an input image 550 including one or more objects. The input image 550 may be an image before being input to the image analysis device 112 and/or an image before being output to the output device 114.

The object image extraction unit 520 may extract an object included in the input image received by the image receiving unit 510 and divide the object image including the object into one or more regions. For example, the object image extractor 520 may compare the pixel value of the analysis target image with a predetermined threshold value to binarize the pixel value, and then group the binarized pixel values to extract the object included in the input image. Here, extracting an object may mean that the object and the background are distinguished, the object may mean a specific object in the image, and the background may mean a portion of the image excluding the object. The background of the image may be expressed in a predetermined color according to the image capturing method or the capturing device. For example, the predetermined color may be white. When a color representing the background of the image is specified, the background and the object may be separated based on the specified background color. For example, an object may be identified by deleting a specified background color region from the input image 550.

Also, for example, the object image may be obtained by specifying a bounding box surrounding the object area, and the object image extracting unit 520 may generate location information of the divided object based on the specified rectangular box. have. That is, the square box may mean an object recognition box.

According to an embodiment of the present disclosure, if the input image is an X-ray image of an article photographed by an X-ray reading device, the background portion other than the article is unnecessary, so the background portion is cut out and the article is present. It can be analyzed only by domain. In particular, it can be said that it is important to acquire an area for an article in a real environment where articles are continuously passed through an X-ray reader through a conveyor belt. A detailed process of classifying an object and a background and generating location information of the object will be described in detail with reference to FIG. 6.

6 is a diagram illustrating a process of classifying an object and a background from an image including a single object, and generating location information of the object, according to an embodiment of the present disclosure. The object image extracting unit 600 of FIG. 6 may be an embodiment of the object image extracting unit 520 of FIG. 5. The input image 610 may be the input image 550 described with reference to FIG. 5, and may be, for example, an image of an article including the bag 612 as a single object.

The object image extracting unit 600 first roughly cuts the surrounding area based on the bag 612 by performing a cropping operation on the input image 610 including one bag 612. A discarded and cropped image 620 may be obtained. Then, the object image extractor 600 may obtain the binarized image 630 by comparing the pixel value of the cropped image 620 with a predetermined threshold value to binarize the pixel value. In addition, the object image extracting unit 600 may obtain a grouped image 640 by grouping (clustering, morphology, closing) adjacent pixels in order to select a portion of an object in the binarized image 630. Then, the object image extracting unit 600 performs labeling and hole filling operations on the grouped images 640 and converts the pixel group formed in the largest shape into the area 652 for the object. The image 650 from which the object is extracted may be obtained by determining and determining the remainder as the region 654 for the background. In addition, the object image extractor 600 may determine the location of the object in the input image 610 by using information on the extracted object image. For example, the object image extractor 600 may specify a rectangular box surrounding the object area, and generate location information of the object based on the specified rectangular box. Referring to FIG. 6, the object image extraction unit 600 may specify a rectangular box 662 surrounding the bag 612 and obtain location information of the bag 612 based on the specified rectangular box. . For example, the location information of the bag 612 may be location information of four vertices forming the rectangular box 662, but is not limited thereto. For example, the location information may be expressed by the coordinates (x, y) of one vertex of the rectangular box 662 and the width and height of the rectangular box. The coordinates (x, y) of the one vertex may be the coordinates of the upper left vertex of the rectangular box 662. The coordinates (x, y) of the vertex may be specified based on the coordinates (0, 0) of the upper left vertex of the input image 610.

Referring again to FIG. 5, the object image extractor 520 may divide the object image into one or more regions based on the size of the object image. Each of the one or more regions may be square. For example, the object image extractor 520 may determine the number or size of regions for dividing the object image based on the size of the object image. For example, when the object image is relatively large or has a size greater than or equal to a predetermined threshold, it may be divided to have more divided areas. Also, the sizes of regions that divide the object image may not be the same.

In addition, when the object image is not square, the object image extracting unit 520 converts the object image to a square by up-sampling or down-sampling the object image, and then converts one or more object images. Can be divided into areas. For example, the object image is obtained based on a rectangular box surrounding the object with respect to the object extracted by the object image extracting unit 520, so the object image may not be square. In this case, the object image extraction unit 520 may divide the object image into one or more regions, but obtains and acquires a square object image by upsampling or downsampling the object image in the horizontal or vertical direction. The square object image may be divided into one or more regions.

For example, referring to FIG. 8, the object image 800 may be composed of 9 pixels horizontally and 12 pixels vertically, and thus may not be a square. In this case, according to the present disclosure, the object image 800 may be divided into a 3x3 square area (in this case, (the horizontal size of the object image/the horizontal size of the divided area) x (the vertical size/division of the object image) The vertical size of the area) = (9/3) x (12/3) = 12, has a total of 12 areas), an image consisting of 12 horizontal pixels and 12 vertical pixels by upsampling the horizontal of the object image 800 It is also possible to obtain and divide it into a 3x3 sized area and divide it into a total of 16 areas. The shape of one or more areas for dividing the object image is not limited to a square shape. For example, the region may have a shape of n x m in which n and m are different positive integers. In this case, the above-described up-sampling or down-sampling may not be performed.

Referring back to FIG. 5, the color distribution analysis unit 530 obtains color distribution information for each of the areas divided by the object image extracting unit 520, and determines at least some of the areas based on the color distribution information. One or more weights can be determined.

The color distribution information may include information on each of n (n is an integer greater than 1) color expression ranges. The color expression range may vary depending on the type and performance of the image acquisition device. In addition, for example, "color" for R(red) may mean only the color of a pixel having a pixel value of (R, G, B) = (255,0,0) in an 8-bit image. For "color expression range", it is meant not only when the pixel value is (R, G, B) = (255, 0, 0), but also includes a similar color within a predetermined range based on the pixel value. For example, the "color expression range" for R may be in the range of (R, G, B) = (150 to 255, 0 to 100, 0 to 100). That is, a pixel with (R, G, B) = (150, 100, 100) may also be defined as being included in the color expression range of red (R). The "color expression range" may be defined for a color to be identified. In the above example, the description has been made based on the color expression range of red, but the color expression range of green (G) or blue (B) may be defined. Alternatively, a color expression range for an arbitrary color (yellow, orange, sky blue, etc.) expressed by combining some or all of R, G, and B may be defined. In the case of reinforcing the area of an object expressed in orange, for example, among the objects included in the image, as a result of analyzing the color distribution information, by applying a weight to the area where the number of pixels included in the color expression range of orange is multiple or dominant Image enhancement according to the present disclosure may be performed. The method of applying the weight is as described above.

If the image acquisition apparatus expresses color by a combination of three color elements of R, G (green), and B (blue), the color distribution information may include information on some or all of the three color elements. If the color elements are R, G, B, Y (yellow), and P (purple), the color distribution information may include information on some or all of the five color elements.

In the X-ray image of an article photographed by an X-ray reader, different color expression ranges are determined depending on the physical properties of the objects included in the image (for example, whether the object is an organic material, inorganic material, metal, etc.). The applied X-ray image is being used. By reading the color-added X-ray image, the reader can identify not only the shape of the object included in the image, but also the physical properties of the object to some extent. The image enhancement of the present disclosure uses an X-ray image to which a color according to the physical property of an object is added as an input image, analyzes color distribution information, and reinforces a specific color region based on this, thereby detecting objects included in the image. It can improve the accuracy and readability of the reader reading the image.

7 is a diagram illustrating an image in which colors are expressed based on physical properties of an object according to an embodiment of the present disclosure.

Referring to FIG. 7, a bag image 700, a medicine container image 710, and a traveler's luggage carrier image 720 captured by an X-ray reader are shown. In the case of the bag loop 702, the bag zipper 704, the medicine 712, and the bottle 722, it can be seen that the color expression range (applied color) is different according to the physical properties of the object. On the other hand, the bag loop 702, the bag zipper 704, the medicine 712, and the bottle 722 are colored relatively clearly so that they can be distinguished from other objects, while any contents 724 in the luggage In the case of ), it can be seen that it is difficult to confirm what the arbitrary contents 724 are in the traveler's luggage image 720 and it is not easy to distinguish them from other objects. This is due to the physical properties of the object. For example, metals or inorganic materials are expressed in a relatively clear and clear color so that they can be clearly distinguished from the background, while organic materials are expressed in a light color, so that the distinction from the background is not clear. In the area of the color expressing organic matter, the color can be strengthened to a clear and distinct color that can be clearly distinguished from the background by enhancing the color.

In other words, it is necessary to vary the degree of enhancement of the image according to the color expression range by using the feature that the color expression range is different according to the physical properties of the object. To this end, a weight may be applied to at least some of the regions by analyzing a color distribution for each of the divided regions.

The one or more weights may include weights for at least some of n color expression ranges or n color elements representing colors. For example, if one region has n color expression ranges or color elements, the number of weights in the corresponding region may have from 1 to n.

For example, when one weight is determined for one region, the determined weight may be applied to all color elements or all color expression ranges included in the one region. Alternatively, the determined weight may be applied to at least a part of all color elements or all color expression ranges included in the one area. For example, in order to enhance an image, the determined weight may be applied only to a predetermined color element among n color elements or a predetermined color expression range among n color expression ranges.

Or, for example, a weight may be determined for each of n color elements or n color expression ranges. That is, the number of weights for one region may be n. In this case, a weight corresponding to each of all color elements or color expression ranges included in the region may be applied to a corresponding color element or color expression range. As for the weight, a relatively high weight may be assigned to a predetermined color element or a color expression range that is an object of image enhancement. For example, a weight greater than 1 may be assigned and multiplied by a value of a corresponding color element or a pixel value belonging to a corresponding color expression range.

Alternatively, for example, a weight may be determined for each of m color elements or color expression ranges greater than 1 and less than n. That is, the number of weights for one region may be m. In this case, the assigned weight may be applied only to a color element or color expression range to which a weight is assigned among the color elements or color expression ranges included in the area. As described above, a relatively high weight is assigned to a predetermined color element or color expression range that is an object of image enhancement.

As described above, the weight may be determined to be relatively high for a predetermined color element or color expression range among n color elements or color expression ranges. For example, when an object included in an X-ray image is an organic material, it is often expressed in an image with a relatively less clear boundary than an object having other physical properties (metal, inorganic, etc.). This is because the colors of organic objects are not clearly expressed so that they can be distinguished from other objects or backgrounds around them. For example, by being expressed in light orange, there are cases where it is difficult to distinguish well from a white background. Accordingly, by assigning a relatively high weight to a portion of the divided area corresponding to the color expression range representing the organic matter, the corresponding color can be enhanced, and, for example, light orange can be changed to dark orange. By reinforcing the image in this way, it is possible to more clearly distinguish between the surrounding object or the background and the object to be reinforced.

The predetermined color element or color expression range to which a relatively high weight is assigned may be one or more. For example, when the total number of color elements or color expression ranges is n, the predetermined color elements or color expression ranges to which a relatively high weight is assigned may be 1 to n. When there are a plurality of predetermined color elements or color expression ranges, the degree of image enhancement required for each may be different, and different weights may be assigned to each accordingly. For example, when images are clearly expressed in the order of metal->inorganic->organic substances, a relatively high weight may be given only to the color element or color expression range for organic substances, but for inorganic substances and organic substances It can also be given a high weight. In this case, a relatively high weight may be given to organic substances rather than inorganic substances.

A detailed process of obtaining color distribution information for each of the divided regions and determining a weight will be described in detail with reference to FIG. 8.

8 is a diagram for describing a process of generating an output image based on color distribution information of an image according to an exemplary embodiment of the present disclosure.

Referring to FIG. 8, the object image 800 may be divided into one or more regions such as a first region 810 and a second region 820. The process of dividing regions in the object image 800 is the same as described for the object image extracting unit 520 of FIG. 5. Hereinafter, a process of obtaining color distribution information and determining a weight in the first region 810 will be described in detail. It is assumed that the first region 810 has a total of 9 pixels as a 3x3 size region and has 5 (n=5) color expression ranges (hereinafter, the color expression range may be replaced by color elements). . The image enhancement apparatus according to an embodiment acquires color distribution information including information on five color expression ranges for the first area 810, and at least a part of the 3x3 sized area based on the acquired color distribution information. One or more weights can be determined for.

Alternatively, only information on a predetermined color expression range, which is an object of image enhancement, may be acquired and used as color distribution information. For example, when distribution information on a predetermined color expression range is greater than or equal to a predetermined threshold, the corresponding area is determined as a target for enhancement, and a relatively high weight may be assigned to the corresponding area.

Looking at an embodiment of determining a weight based on color distribution information in more detail, n color channel images reflecting information on each of n color expression ranges may be obtained for each of the divided regions. For example, since the first region 810 has information on five (n=5) color expression ranges, five color channel images can be obtained, and the first color channel image 830 and the second It is referred to as a color channel image 840, a third color channel image 850, a fourth color channel image 860, and a fifth color channel image 870. In addition, for example, when the X-ray reader supports five color elements of R, G, B, Y, and P, the first color channel image 830, the second color channel image 840, and the third color channel The image 840, the fourth color channel image 860, and the fifth color channel image 870 may correspond to color elements of R, G, B, Y, and P, respectively.

Each of the first to fifth color channel images 830 to 870 is generated by mapping each pixel to a color channel image corresponding to the corresponding color information based on the color information of each of the constituent pixels of the first region 810. I can. For example, the first pixel 812 is mapped to the pixel 852 at the corresponding position of the third color channel image 850, and the second pixel 814 is the pixel at the corresponding position of the first color channel image 830. 832, and the third pixel 816 is mapped to the pixel 872 at the corresponding position of the fifth color channel image 870, and the fourth pixel 818 is mapped to the second color channel image 840 ), and the fifth pixel 820 is mapped to the pixel 874 at the corresponding position of the fifth color channel image 870, and the sixth pixel 822 is The five color channel image 870 is mapped to the pixel 876 at the corresponding position, and the seventh pixel 824 is mapped to the pixel 844 at the corresponding position of the second color channel image 840, and The pixel 826 is mapped to the pixel 878 at the corresponding position of the fifth color channel image 870, and the ninth pixel 828 is mapped to the pixel 880 at the corresponding position of the fifth color channel image 870. By mapping, the first to fifth color channel images 830 to 870 may be generated.

Meanwhile, when the color expression range is at most n, a number of color channel images less than n may be obtained. For example, in the case of the first region 810, a pixel having a color corresponding to the fourth color channel image 860 Since this does not exist, a total of four color channel images excluding the fourth color channel image 860 may be obtained.

When color channel images are acquired, a first color channel image 830, a second color channel image 840, a third color channel image 850, a fourth color channel image 860, and a fifth color channel image 870 are obtained. ) Can be applied to weights a1, a2, a3, a4, and a5, respectively.

The weight may be determined in consideration of the color distribution of pixels constituting each region, and for example, the weight may be determined in proportion to the color distribution of the pixels. Alternatively, the weight may be determined to have a relatively high weight for a predetermined color expression range and a relatively low weight for the remaining color expression range.

Referring back to FIG. 5, the image enhancement unit 540 may generate a first output image for the object image by applying one or more weights determined by the color distribution analysis unit 530 to at least some of the one or more regions. .

Referring to FIG. 8, a first color channel image 830, a second color channel image 840, a third color channel image 850, a fourth color channel image 860, and a fifth color channel image 870. The weights a1, a2, a3, a4, and a5 are applied to, and the weighted first region 810-1 may be obtained by combining the first to fifth color channel images to which the weight is applied. In addition, the first output image may be finally generated by repeating the above process for the remaining areas of the object image 800. The weight may be determined in consideration of color distribution of pixels constituting each region, and a relatively high weight may be determined for a predetermined color expression range, and a relatively low weight may be determined for the remaining color expression range. For example, the part corresponding to the color representing the organic matter in each divided area is not clearly distinguished from the background, so the border part is not expressed relatively clearly in the image, so the weight is determined relatively high, and the color corresponding to the color representing the metal Since the part has a relatively clear distinction from the background, the weight can be determined relatively low because the boundary part is expressed relatively clearly in the image. As described above, applying a weight may mean replacing a pixel in an area to be enhanced with a new pixel value multiplied by the weight.

Alternatively, as described above, as a result of analyzing the color distribution of the area 810 included in the object image 800, when a predetermined color expression range to be image enhancement is dominant or has a distribution greater than a predetermined threshold, the corresponding area ( 810) can be set to a relatively high weight.

For example, when the color expression range to be reinforced is red (R) and the dominant color of a predetermined region is red, one or more weights may be applied to the corresponding region. For example, when (R, G, B) = (120, 10, 10) of a pixel in the predetermined area, by applying a weight of 2, it can be enhanced to (R, G, B) = (240, 20, 20). have. Alternatively, a weight of 2 may be applied only to some of R, G, and B, for example, red, and may be enhanced to (R, G, B) = (240, 10, 10). Alternatively, a weight of 2 is applied to some of R, G, B, such as red, and a weight of 0.5 is applied to other parts, such as green and blue, so that (R, G, B) = (240, 5, 5) It can also be strengthened with. The above example relates to a case of enhancing red, but is not limited thereto, and an arbitrary color may be determined as a color to be enhanced.

The predetermined threshold and/or weight may be arbitrarily determined or may be determined based on accumulated image processing information. Alternatively, by performing learning on the threshold and/or weight through an artificial intelligence-based learning model, the optimal threshold and/or weight may be continuously updated.

Also, the image enhancement unit 540 may generate a second output image for the object image by applying edge-based filtering or smoothing filtering to at least a portion of one or more regions. In addition, the image enhancement unit 540 may generate a third output image for the object image based on the generated first and second output images.

Edge-based filtering or smoothing filtering is a technique for enhancing the contrast of an image, and includes, for example, Wiener filtering, Unsharp mask filtering, Histogram equalization, and linear contrast adjustment techniques, but is not limited thereto. It may include techniques to reinforce.

FIG. 9 is a diagram for describing a process of obtaining a final output image obtained by combining an image obtained by using color distribution information and an image obtained by applying edge-based filtering or smoothing filtering according to an embodiment of the present disclosure. The object image 900, the first region 910, and the weighted first region 910-1 of FIG. 9 are the object image 800, the first region 810, and the weighted first region of FIG. 8 Each may correspond to (810-1). Referring to FIG. 9, the image enhancement unit 540 may generate a first region 910-2 to which the filtering has been applied for a first region 910, and a first region 910-1 to which a weight is applied. The final first region 910-3 may be generated by combining the filtered first region 910-2. Further, the image enhancement unit 540 may generate a second output image to which the above filtering techniques are applied to the remaining regions as well as a third output image by combining the first output image and the second output image.

A process of generating a weighted area (eg, 910-1), a filtering area (eg, 910-2), and/or the final area 910-3 using the two may be performed in units of areas. However, the present invention is not limited thereto, and the process may be performed on an object image basis. For example, an object image (first output image) to which the weight is applied may be obtained by performing a process of applying a weight to each of the regions included in the object image. In addition, an object image (second output image) may be obtained by performing a process of applying the filtering to each of the regions included in the object image. In addition, the final image (the third output image) may be generated by combining the weighted object image and the edge-enhanced object image.

On the other hand, for example, when less organic matter is contained in the divided area than other materials, the influence on the first output image may be relatively small by combining the second output image with the first output image. In this case, the color representing the organic matter It is possible to determine a relatively higher weight for distribution information. In addition, for example, by combining the first output image and the second output image, even when several objects in the image overlap, more accurate object recognition may be possible.

10 is a diagram for explaining a process of obtaining a final output image using a graphical model according to an embodiment of the present disclosure.

The image enhancement apparatus according to an embodiment determines each of the color expression ranges included in the color distribution information as individual nodes, and determines a relative relationship between the determined individual nodes and a first output image, a second output image, and a third output image. By using the relative relationship of, a hierarchical structure of a graphical model can be created.

Referring to FIG. 10, in each divided area, when n color expression ranges are included in the color distribution information, the lowest node is the first color distribution information 1010-1 to the nth color distribution information 1010 -n. ) Can contain up to n nodes. Then, the first output image 1020 may be obtained by applying a weight to each of the divided region or color expression ranges of the divided region based on each color distribution information. The first output image 1020 may be determined as the final output image. Alternatively, a second output image 1030 obtained by applying an image contrast enhancement technique is further generated, and a third output image 1040 is generated based on the first output image 1020 and the second output image 1030 You may.

11 is a diagram for describing an image enhancement method according to an embodiment of the present disclosure. The image enhancement method of FIG. 11 is a method performed by the image enhancement apparatus of FIG. 5, and a description of the image enhancement apparatus of FIG. 5 may be applied to the image enhancement method of FIG. 11.

In step S1100, an input image may be received.

In step S1110, an object included in the input image may be extracted. For example, an object included in the analysis target image may be extracted by binarizing the pixel value by comparing the pixel value of the input image with a predetermined threshold value, and grouping the binarized pixel values.

In step S1120, an object image including an object may be divided into one or more regions. For example, the number or size of regions for dividing the object image may be determined based on the size of the object image. Also, the sizes of regions that divide the object image may not be the same. In addition, for example, when the object image is not square, the object image is converted into a square by up-sampling or down-sampling the object image, and then the object image can be divided into one or more regions. have.

In step S1130, color distribution information may be obtained for each of one or more areas. For example, the color distribution information may include information on each of n (n is an integer greater than 1) color expression ranges.

In operation S1140, one or more weights may be determined for at least some of the one or more regions based on the color distribution information. For example, one or more weights may include weights for at least some of the n color expression ranges. For example, if one region has n color expression ranges, the number of weights in the corresponding region may range from 1 to n.

In step S1150, a first output image for an object image may be generated by applying the determined one or more weights to at least a portion of one or more regions.

Although not illustrated in FIG. 11, a second output image for an object image may be generated by applying edge-based filtering or smoothing filtering to at least a portion of one or more regions. In addition, for example, a third output image for an object image may be generated based on the generated first and second output images.

In the embodiment described with reference to FIGS. 5 to 11, an example of separating an object and a background by receiving an image including a single object has been described. However, the present invention is not necessarily limited thereto, and the input image may be an image including two or more objects. In this case, two or more objects and backgrounds may be distinguished from the input image, and location information may be generated and used for each of the two or more objects. Further, in this case, in the description with reference to FIG. 6, when a plurality of pixel groups are formed, it may be determined that not only the pixel group formed in the largest shape but also the other pixel groups are regions for the object. The process of generating the location information of each determined object is the same as described for an image including one object.

At least some of the components of the image enhancement apparatus of the present disclosure and steps of the image enhancement method may be performed using an artificial intelligence-based or deep learning-based model. For example, the weight determined based on the size, number, and color distribution information of the area generated by dividing the object image, various thresholds mentioned in the present disclosure, whether or not to generate the second output image, etc. are based on artificial intelligence or deep learning. It can be learned using a model, and information according to the learned model can be used.

Hereinafter, an embodiment of a context analysis method performed by the image analysis apparatus 112 will be described with reference to FIGS. 12 to 17.

The image analysis apparatus 1200 of FIG. 12 may be an embodiment of the image analysis apparatus 112 of FIG. 1. Alternatively, the image analysis apparatus 1200 of FIG. 12 may be a device included in the image analysis apparatus 112 of FIG. 1 or configured separately to perform context analysis.

Referring to FIG. 12, the image analysis apparatus 1200 may include a feature extraction unit 1210, a context generation unit 1220, and/or a feature and context analysis unit 1230. However, this is only showing some components necessary for describing the present embodiment, and components included in the image analysis apparatus 1200 are not limited to the above-described examples.

The image analysis apparatus 1200 may extract features of an input image (analysis target image), generate context information based on the extracted features, and analyze the analysis target image based on the extracted features and the generated context information. have. For example, the image analysis apparatus 1200 may classify an image or find a location of an object of interest using the extracted feature and the generated context information.

The input image of the image analysis device 1200 may be the same as the input image of the image analysis device 112 of FIG. 1.

The feature extractor 1210 may analyze the input image and extract features of the image. For example, the feature may be a local feature for each area of an image. The feature extractor 1210 according to an embodiment may extract features of an input image using a general convolutional neural network (CNN) technique or a pooling technique. The pooling technique may include at least one of a max pooling technique and an average pooling technique. However, the pooling technique referred to in the present disclosure is not limited to the max pooling technique or the average pooling technique, and includes any technique for obtaining a representative value of an image region of a predetermined size. For example, the representative value used in the pooling technique may be at least one of a variance value, a standard deviation value, a mean value, a most frequent value, a minimum value, and a weighted average value, in addition to the maximum value and the average value.

The convolutional neural network of the present disclosure may be used to extract “features” such as a border, a line color, and the like from input data (image), and may include a plurality of layers. Each layer may receive input data and may generate output data by processing the input data of the layer. The convolutional neural network may output an input image or a feature map generated by convolving an input feature map with filter kernels as output data. The initial layers of the convolutional neural network can be operated to extract low-level features such as edges or gradients from the input. The next layers of the neural network can gradually extract more complex features such as eyes, nose, etc. A detailed operation of the convolutional neural network will be described later with reference to FIG. 16.

The convolutional neural network may include a convolutional layer in which a convolution operation is performed, as well as a pooling layer in which a pooling operation is performed. The pooling technique is a technique used to reduce the spatial size of data in the pooling layer. Specifically, the pooling technique includes a max pooling technique that selects the maximum value in a corresponding area and an average pooling technique that selects an average value of the area. In the image recognition field, the max pooling technique is generally used. do. In the pooling technique, generally, the pooling window size and interval (stride) are set to the same value. Here, the stride refers to an interval to be moved when a filter is applied to input data, that is, an interval to which the filter is moved, and stride may also be used to adjust the size of output data. A detailed operation of the pooling technique will be described later with reference to FIG. 17.

The feature extractor 1210 according to an embodiment of the present disclosure may apply filtering to an analysis target image as pre-processing for extracting a feature of an analysis target image. The filtering may be Fast Fourier Transform (FFT), histogram equalization, motion artifact removal, or noise removal. However, the filtering of the present disclosure is not limited to the above-described methods, and may include any type of filtering capable of improving image quality. Alternatively, the image enhancement described with reference to FIGS. 5 to 11 may be performed as a pre-processing.

The context generator 1220 may generate context information of the input image (analysis target image) by using the features of the input image extracted from the feature extractor 1210. For example, the context information may be a representative value indicating the entire or partial area of the image to be analyzed. Further, the context information may be global context information of the input image. The context generator 1220 according to an embodiment may generate context information by applying a convolutional neural network technique or a pooling technique to a feature extracted from the feature extractor 1210. The pooling technique may be, for example, an average pooling technique.

The feature and context analysis unit 1230 may analyze an image based on the feature extracted from the feature extraction unit 1210 and context information generated by the context generation unit 1220. The feature and context analysis unit 1230 according to an embodiment concatenates the local features of each region of the image extracted by the feature extraction unit 1210 and the global context reconstructed by the context generation unit 1220 It can be used together in such a way as to classify the input image or find the location of an object of interest included in the input image. Since the information at a specific two-dimensional position in the input image includes not only local feature information but also global context information, the feature and context analysis unit 1230 uses these information, so that the actual content is different but local feature information. It is possible to more accurately recognize or classify input images that have similar input images.

As described above, the invention according to an embodiment of the present disclosure enables more accurate and efficient learning and image analysis by using global context information as well as local features used by a general convolutional neural network technique. do. From this point of view, the neural network to which the invention according to the present disclosure is applied may be referred to as a'deep neural network through context analysis'.

13 is a diagram illustrating a process of generating and analyzing context information of an image according to an embodiment of the present disclosure.

The feature extraction unit 1310, the context generation unit 1320, and the feature and context analysis unit 1330 of FIG. 13 are each of the feature extraction unit 1210, the context generation unit 1220, and the feature and context analysis of FIG. It may be an embodiment of the unit 1230.

Referring to FIG. 13, the feature extractor 1310 may extract a feature from the input image 1312 using the input image 1312 and generate a feature image 1314 including the extracted feature information. The extracted feature may be a feature of a local region of the input image. The input image 1312 may include an input image of an image analysis device or a feature map of each layer in a convolutional neural network model. In addition, the feature image 1314 may include a feature map and/or feature vector obtained by applying a convolutional neural network technique and/or a pooling technique to the input image 1312.

The context generator 1320 may generate context information by applying a convolutional neural network technique and/or a pooling technique to the feature image 1314 extracted by the feature extractor 1310. For example, the context generator 1320 may generate context information of various scales, such as the entire image, the quadrant area, and the 9 equal part area by variously adjusting the pooling stride. Referring to FIG. 13, a full context information image 1322 including context information for an image of the entire image size, and a fourth context information image including context information for a quadrant image having a size obtained by dividing the entire image into quarters ( 1324) and a ninth-section contextual information image 1326 including context information for a nine-segmented image having a size obtained by dividing the entire image into nines may be obtained.

The feature and context analysis unit 1330 may more accurately analyze a specific region of the analysis target image using both the feature image 1314 and the context information images 1322, 1324, and 1326.

For example, when an image including a boat having a shape similar to a car is an input image, the identified object from the feature image 1314 including the local feature extracted by the feature extraction unit 1310 It cannot be accurately determined whether is a car or a boat. That is, although the feature extraction unit 1310 may recognize the shape of an object based on a local feature, there are cases in which it is not possible to accurately identify and classify the object only with the shape of the object.

The context generator 1320 according to an exemplary embodiment of the present disclosure generates context information 1322, 1324, and 1326 based on the analysis target image or the feature image 1314 to more accurately identify and classify objects. I can. For example, a feature extracted for the entire image is recognized or classified as "natural landscape", a feature extracted for a quadrant image is recognized or classified as a "lake", and a feature extracted for a ninth image is recognized as "water" When recognized or classified as, the extracted features such as "natural landscape", "lake", and "water" can be created and utilized as context information.

The feature and context analysis unit 1330 according to an embodiment of the present disclosure may identify the object having the shape of the boat or vehicle as a “boat” by utilizing the context information.

In the embodiment described with reference to FIG. 13, it has been described that context information for the entire image, context information for a quadrant image, and context information for a 9 segment image are generated and utilized, but the size of the image for extracting the context information is It is not limited to this. For example, context information for an image having a size other than the above-described image may be generated and utilized.

A convolutional neural network technique and pooling according to an embodiment of the present disclosure will be described later with reference to FIGS. 16 and 17.

14 is a diagram illustrating a process of identifying an object by analyzing an image by an image analysis apparatus according to an embodiment of the present disclosure.

For example, the image analysis apparatus 1400 may accurately identify and/or classify an object included in the image 1410 by receiving the image 1410 and generating information on an image region of various sizes. The input image 1410 may be, for example, an X-ray image including a bag. The image analysis device 1400 analyzes the input image 1410 as described above, extracts features for the entire image and features for a partial area of the image, and accurately identifies the object included in the image 1410 by using the same. can do. The feature 1422 of the entire image may be, for example, a feature of the shape of the bag. The features of the partial area of the image may include, for example, a feature 1424 for a handle, a feature 1426 for a zipper, and a feature 1428 for a ring.

The image analysis apparatus 1400 may accurately identify that the object included in the image 1410 is a "bag" by using the generated features 1422, 1424, 1426, and 1428 as context information.

If some of the generated features are features that are not related to the "bag", the image analysis apparatus 1400 cannot identify the object included in the image 1410 as a "bag" or the image 1410 It can provide an analysis result that the object included in the "bag" cannot be identified. Alternatively, if some of the context information is not related to other context information, an abnormality of the object may be output. For example, when an atypical space, a space having a certain thickness or more, etc., which are not related to the usual characteristics of the "bag", is detected, a signal indicating that the "bag" is a bag with abnormality may be output.

As described above, if context information that is not related to normal context information is included, such fact may be output to the reader, and the reader may perform a detailed inspection or remodeling inspection of the article or object of the video. You can do it.

15 is a diagram for describing an operation of an image analysis apparatus according to an exemplary embodiment of the present disclosure.

In step S1500, the image analysis apparatus may extract features of an image to be analyzed.

The image analysis apparatus according to an embodiment may extract features of an input image using a general convolutional neural network technique or a pooling technique. The characteristic of the analysis target image may be a local characteristic for each region of the image, and the pooling technique may include at least one of a max pooling technique and an average pooling technique.

In step S1510, the image analysis apparatus may generate context information based on the features extracted in step S1500.

The image analysis apparatus according to an embodiment may generate context information by applying a convolutional neural network technique and/or a pooling technique to the features extracted in step S1500. The context information may be a representative value indicating the entire or partial region of the image to be analyzed. Further, the context information may be global context information of the input image. In addition, the pooling technique may be, for example, an average pooling technique.

In step S1520, the image analysis apparatus may analyze the analysis target image based on the feature extracted in step S1500 and the context information generated in step S1510.

For example, the image analysis apparatus may classify the input image or find the location of the object of interest included in the input image by combining the local features of each region of the image extracted in step S1500 and the global context reconstructed in step S1510. have. Accordingly, since information at a specific two-dimensional position in an input image is included from local information to a global context, it is possible to more accurately recognize or classify input images that have different actual contents but similar local information. Alternatively, it is possible to detect an object containing context information that is not related to other context information.

16 is a diagram for explaining an embodiment of a convolutional neural network that generates a multi-channel feature map.

Image processing based on convolutional neural networks can be used in various fields. For example, an image processing device for object recognition of an image, an image processing device for image reconstruction, an image processing device for semantic segmentation, and image processing for scene recognition It can be used for devices and the like.

The input image 1610 may be processed through the convolutional neural network 1600 to output a feature map image. The output feature map image can be used in various fields described above.

The convolutional neural network 1600 may be processed through a plurality of layers 1620, 1630, and 1640, and each layer may output multi-channel feature map images 1625 and 1635. The plurality of layers 1620, 1630, and 1640 according to an embodiment may extract features of an image by applying a filter having a predetermined size from the upper left to the lower right of the input data. For example, the plurality of layers 1620, 1630, and 1640 multiply the upper left NxM pixel of the input data by a weight and map it to one neuron at the upper left of the feature map. In this case, the weight to be multiplied will also be NxM. The NxM may be, for example, 3x3, but is not limited thereto. Thereafter, in the same process, the plurality of layers 1620, 1630, and 1640 scan the input data from left to right and from top to bottom by k cells, multiplying the weights and map them to neurons in the feature map. The k field denotes a stride to move the filter when performing convolution, and may be appropriately set to adjust the size of output data. For example, k may be 1. The NxM weight is called a filter or filter kernel. That is, the process of applying a filter in the plurality of layers 1620, 1630, 1640 is a process of performing a convolution operation with the filter kernel, and the resultant extracted result is "feature map" or "feature map" It is called "map image". Also, the layer on which the convolution operation is performed may be referred to as a convolutional layer.

The term “multiple-channel feature map” refers to a set of feature maps corresponding to a plurality of channels, for example, may be a plurality of image data. The multi-channel feature maps are convolutional neural networks. It may be an input from an arbitrary layer or an output according to a result of a feature map operation such as a convolution operation, etc. According to an embodiment, the multi-channel feature maps 1625 and 1635 are the "feature extraction layer" of the convolutional neural network. Is generated by a plurality of layers 1620, 1630, 1640, also referred to as "convolutional layers" or "convolutional layers." Each layer sequentially receives multi-channel feature maps generated in the previous layer, and then as output Finally, the L (L is an integer)-th layer 1640 receives the multi-channel feature maps generated in the L-1 layer (not shown), and multi-channel feature maps (not shown) are received. You can create maps.

Referring to FIG. 16, feature maps 1625 having K1 channels are output according to a feature map operation 1620 at layer 1 for an input image 1610, and feature map operation 1630 at layer 2 It becomes an input for ). In addition, the feature maps 1635 having K2 channels are output according to the feature map operation 1630 at layer 2 for the input feature maps 1625, and feature map operation at layer 3 (not shown). Becomes the input for

Referring to FIG. 16, multi-channel feature maps 1625 generated in a first layer 1620 include feature maps corresponding to K1 (K1 is an integer) number of channels. In addition, the multi-channel feature maps 1635 generated in the second layer 1630 include feature maps corresponding to K2 (K2 is an integer) number of channels. Here, K1 and K2 representing the number of channels may correspond to the number of filter kernels used in the first layer 1620 and the second layer 1630, respectively. That is, the number of multi-channel feature maps generated in the M-th layer (M is an integer of 1 or more and L-1 or less) may be the same as the number of filter kernels used in the M-th layer.

17 is a diagram for describing an embodiment of a pooling technique.

As shown in FIG. 17, the window size of the pooling is 2x2 and the stride is 2, and the output image 1790 may be generated by applying max pooling to the input image 1710.

In (a) of FIG. 17, a 2x2 window 1710 is applied to the upper left of the input image 1710, and a representative value (here, the maximum value 4) among values in the window 1710 is calculated, and the output image 1790 ) In the corresponding position 1720.

Thereafter, in FIG. 17B, the window is moved by the stride, that is, by 2, and the maximum value 3 of the values in the area of the window 1730 is input to the corresponding position 1740 of the output image 1790.

When it is no longer possible to move the window to the right, the above process is repeated again from the position below the stride from the left side of the input image. That is, as shown in (c) of FIG. 17, the maximum value 5 of the values in the area of the window 1750 is input to the corresponding position 1760 of the output image 1790.

Thereafter, as shown in (d) of FIG. 17, the window is moved by the stride, and the maximum value 2 of the values in the area of the window 1770 is input to the corresponding position 1780 of the output image 1790.

The above process is repeatedly performed until a window is positioned in the lower right area of the input image 1710, thereby generating an output image 1790 in which pooling is applied to the input image 1710.

Hereinafter, an embodiment of a method of generating a new composite image and/or virtual article information corresponding thereto by using a plurality of images and/or article information will be described with reference to FIGS. 18 to 22.

18 is a block diagram illustrating a configuration of an image synthesizing apparatus according to an embodiment of the present disclosure.

Referring to FIG. 18, the image synthesizing apparatus 1800 includes an object image extraction unit 1810, an object location information generation unit 1820, an image synthesis unit 1830 and/or an object detection deep learning model learning unit 1840. Can include. However, this is only showing some components necessary for describing the present embodiment, and components included in the image synthesizing apparatus 1800 are not limited to the above-described example. For example, two or more constituent units may be implemented in one constituent unit, or an operation executed in one constituent unit may be divided and implemented to be executed in two or more constituent units. In addition, some components may be omitted or additional components may be added. Or, among the components of the image analysis device 112 of FIG. 1, the image enhancement device 500 of FIG. 5, the image analysis device 1200 of FIG. 12, and the image synthesis device 1800 of FIG. 18, the same function or similar A component that performs a function may be implemented as one component.

The image synthesizing apparatus 1800 according to an embodiment of the present disclosure receives a first image including a first object and a second image including a second object, and provides an object for each of the first image and the second image. And background, generate location information of the first and second objects that are divided, and based on location information of the first object and location information of the second object, a second object including the first object and the second object 3 An image may be generated, and an object detection deep learning model may be trained using the location information of the first object, the location information of the second object, and the third image.

Referring to FIG. 18, an input image 1850 may include an image including a single object. The description of the input image 1850 is the same as the description of the input image described with reference to FIG. 1 and the like.

The object image extractor 1810 may receive an image 1850 including a single object and divide the received image into an object and a background. The description of the object image extracting unit 1810 is the same as the description of the object image extracting unit 520 described with reference to FIGS. 5 and 6.

The object location information generator 1820 may determine the location of the object extracted from the object image extractor 1810. For example, the object location information generation unit 1820 specifies a bounding box surrounding the object area, and generates location information of the object classified by the object image extracting unit 1810 based on the specified rectangular box. can do. The description of the method of generating the location information of the object is the same as the description of the method with reference to FIG. 6.

According to an embodiment of the present disclosure, since the location information of the object included in the image can be automatically generated, it is possible to avoid the hassle of the reader having to directly input the location information of the object for each image for artificial intelligence learning. I can.

Referring back to FIG. 18, the image synthesizing unit 1830 uses a plurality of single object images obtained by obtaining location information of the object through the object image extracting unit 1810 and the object location information generating unit 1820. Can be created. For example, for a first image including a first object and a second image including a second object, respectively, through the object image extraction unit 1810 and the object position information generation unit 1820, the location information of the first object and The location information of the second object is obtained, and the image synthesizer 1830 generates a third image including the first object and the second object based on the obtained location information of the first object and the location information of the second object. can do. A detailed process of generating a multi-object image will be described in more detail with reference to FIG. 19.

19 is a diagram illustrating a process of generating a multi-object image using two images including a single object according to an embodiment of the present disclosure. The image synthesizer 1900 of FIG. 19 is an embodiment of the image synthesizer 1830 of FIG. 18. Referring to FIG. 19, the image synthesizing unit 1900 includes a first single object image 1910, a second single object image 1920, and a first single object image acquired through an object image extraction unit and an object location information generation unit. 1910) and the location information of the second single object image 1920, the first single object image 1910 and the second single object image are combined in the multi-object image 1940 and the multi-object image 1940 Location information 1950 on the created objects may be obtained. Meanwhile, the image synthesizing unit 1900 may also use an image 1930 for a background separated from an object when synthesizing the first single object image 1910 and the second single object image 1920. When synthesizing a plurality of images, location information of the first single object image 1910 and location information of the second single object image 1920 may be arbitrarily modified. In addition, image synthesis may be performed based on the modified location information. By doing so, it is possible to generate a myriad of composite images and virtual location information.

As described above, it is possible to generate as many synthesized images necessary for learning and/or virtual location information corresponding thereto through synthesis. Therefore, even when the absolute number of images and/or location information required for learning is small, a sufficient number of training data to train the artificial intelligence model can be generated.

Referring back to FIG. 18, the object detection deep learning model training unit 1840 may train an object detection deep learning model using location information of a first object, location information of a second object, and a third image. For example, the object detection deep learning model training unit 1840 may train a convolutional neural network model. For training of the convolutional neural network model, location information of the first object, location information of the second object, and a third image may be used.

20 is a diagram illustrating a process of learning a convolutional neural network using a multi-object image according to an embodiment of the present disclosure. The object detection deep learning model learning unit 2000 of FIG. 20 is an embodiment of the object detection deep learning model learning unit 1840 of FIG. 18. Referring to FIG. 20, as data necessary for learning, a multi-object image 2010 synthesized using single object images and location information of the objects may be used. The object detection deep learning model training unit 2000 may train the convolutional neural network 2020 by projecting position information of each single object on the multi-object image 2010 together. According to an embodiment, when a plurality of objects exist in an object passing through an X-ray scanner in an electronic search system, an X-ray image in which a plurality of objects are overlapped may be obtained. According to the present disclosure, Since the convolutional neural network is trained by using the shape of each object together with the location information of the object of, a more accurate detection result can be obtained even if the overlapping between objects occurs.

21 is a diagram illustrating a process of analyzing an actual image using an image synthesizing apparatus according to an embodiment of the present disclosure.

The image synthesizing apparatus 2100 of FIG. 21 is an embodiment of the image synthesizing apparatus 1800 of FIG. 18. The operation of the object image extraction unit 2104, the object location information generation unit 2106, the image synthesis unit 2108, and the object detection deep learning model learning unit 2110 included in the image synthesis apparatus 2100 of FIG. It is the same as the operation of the object image extraction unit 1810, the object location information generation unit 1820, the image synthesis unit 1830, and the object detection deep learning model learning unit 1840 included in the image synthesis apparatus 1800 of 18. . Accordingly, the image synthesizing apparatus 2100 includes an object image extraction unit 2104, an object location information generation unit 2106, an image synthesis unit 2108, and an object detection deep learning model learning unit for a plurality of single object images 2102. By performing the operation at (2110), a learned convolutional neural network model can be generated. The object detection apparatus 2120 may detect each object using a convolutional neural network model learned by the image processing apparatus 2100 for an image 2122 including multiple objects in an actual environment.

According to an embodiment, when the invention of the present disclosure is applied to a product retrieval system, the image synthesizing apparatus 2100 of the present disclosure may generate an image including multiple objects newly based on extraction of a single object region in an X-ray image. I can. In addition, the object detection device 2120 may find a region in which multiple objects included in the article passing through the X-ray scanner exist. Therefore, by automatically extracting the position of the object with respect to the X-ray image, the reader can perform the image inspection more easily, and also includes information on the extracted object and the quantity of the object within the object. It can be used for comparing computerized information.

22 is a diagram for describing an image synthesis method according to an embodiment of the present disclosure.

In step S2200, a first image including a first object and a second image including a second object are received, and an object and a background may be distinguished for each of the first image and the second image. For example, by comparing the pixel value of the input image with a predetermined threshold value to binarize the pixel value, and group the binarized pixel values, objects included in the input image may be distinguished.

In step S2210, location information of the divided first object and second object may be generated. For example, a rectangular box surrounding the object area may be specified, and location information of an object classified in step S2200 may be generated based on the specified rectangular box.

In operation S2220, a third image including the first object and the second object may be generated based on the location information of the first object and the location information of the second object. For example, a third image including the first object and the second object may be generated based on the location information of the first object and the location information of the second object acquired in step S2210.

In step S2230, an object detection deep learning model may be trained using location information of the first object, location information of the second object, and a third image. For example, a convolutional neural network model may be trained, and the location information of the first object generated in step S2210 and the location information of the second object, and the third image generated in step S2220 may be used for learning the convolutional neural network model. I can.

In the embodiment described with reference to FIGS. 18 to 22, an example of separating an object and a background by receiving an image including a single object has been described. However, the present invention is not necessarily limited thereto, and the input image may be an image including two or more objects. In this case, two or more objects and backgrounds may be distinguished from the input image, and location information may be generated and used for each of the two or more objects.

In addition, in the above-described embodiment, it has been described that a third image is generated based on two single object images and location information of each object. However, the present invention is not necessarily limited thereto, and a third image may be generated using two or more single object images and location information of each object. That is, the image processing method and apparatus according to the present disclosure may generate a third image based on two or more images each including one or more objects and location information of each object.

In the above description, an image analysis method, an object detection method, and an image enhancement method for a 2D image have been described. However, various embodiments of the present invention targeting a 2D image may also target a 3D image.

Existing 2D images (eg, 2D X-ray images) and 3D images (eg, 3D CT images) have the following differences.

First, compared to a 2D image, in a 3D image, as the dimension increases, the data becomes vast, and the amount of computation accordingly increases significantly.

Second, in the case of a 2D image, for example, objects are superpositioned in a photographing direction, whereas in the case of a 3D image, overlapping does not occur.

In view of the above differences, the present invention provides various embodiments for efficiently detecting an object included in a 3D image.

Various embodiments of the present invention targeting a 3D image will be described in detail with reference to FIGS. 23 to 25.

For example, a 3D image may mean a 3D security CT image. However, the present invention is not limited thereto, and may include all images expressed in a three-dimensional form. For example, an image composed of one or more voxels having three-dimensional coordinates (x, y, z) and having luminance information and/or color difference information may be a three-dimensional image. Voxels may mean pixels constituting a 3D image, and may correspond to pixels in a 2D image. It is the same that the voxel and the pixel may include luminance information and/or color difference information. The voxel may additionally have z coordinates to define a position on the 3D.

23 is a diagram illustrating an embodiment of the present invention for detecting an object included in a 3D image.

In step S2300, a 3D image may be obtained.

In step S2310, pre-processing on the image may be performed. The pre-processing of the image may include a pre-processing of the 2D image. The preprocessing process for a 2D image may be applied as it is to a 3D image or may be partially modified so that it may be applied to a 3D image.

In step S2320, detailed region extraction may be performed.

First, an initial detailed region may be extracted. The initial detailed areas may be areas that start the algorithm according to the present invention.

In order to extract an initial detailed region, a 3D connection element may be extracted after image binarization. Image binarization may be the same as or similar to image binarization for a 2D image. Each voxel may have a value of 0 or 1 through image binarization. A voxel having a value of 1 among voxels adjacent to an arbitrary voxel having a value of 1 may be identified. A two-dimensional pixel may be represented in a square shape on a two-dimensional plane. Since a voxel is a unit of a pixel in a 3D space, it can be expressed in a hexahedral shape in a 3D space. An adjacent voxel may mean a voxel that shares at least one surface with a current voxel, shares at least one line, or shares at least one point. All adjacent voxels having a value of 1 may be identified, and a voxel group consisting of these identified voxels may be defined as an initial detailed region or extracted as an initial detailed region.

A segmentation operation may be performed based on the extracted initial detailed region. The segmentation operation may be performed based on a voxel value included in the detailed region. The subdivision operation may be performed until the variance of voxel values included in one detailed region becomes less than or equal to a predetermined value (a predetermined threshold). For example, when the variance of voxel values included in one detailed region is greater than or equal to a predetermined value, voxels included in one detailed region may be subdivided into two or more detailed regions according to a predetermined criterion.

For example, as a result of checking the distribution of luminance values of voxels included in one sub-region, if voxels of one group have luminance values of 50 to 60, and voxels of another group have luminance values of 200 to 250, the corresponding sub-region Voxels included in may be subdivided into a first group (luminance values 50 to 60) and a second group (luminance values 200 to 250). The region constituting each voxel group subdivided as described above is again defined as a detailed region, and the subdivision operation may be performed recursively and/or repeatedly.

The variance of voxel values included in the detailed region obtained as a result of performing the subdivision operation becomes less than a certain value. Accordingly, through the subdivision operation, regions having the same physical properties in the 3D image can be identified. The detailed area obtained as a result of performing the subdivision operation may be referred to as a final detailed area to distinguish it from the initial detailed area.

For example, in the case of a 3D security CT image, if an initial detailed area is limited only by a simple connection element extraction method, objects having different properties are merged into one area (object) due to equipment limitations such as resolution. Therefore, in the sub-region based on the connection element extracted first, it can be subdivided so that the difference in voxel values is not large (that is, the variance is less than a certain value). Through this, it is possible to relatively accurately detect object regions having the same/similar properties in consideration of the properties and density of the object.

However, it is not limited to the above example, and detailed regions may be uniformly extracted for the entire three-dimensional region. That is, after binarization, the initial detailed region extracted through the Connected Component extraction method can be used as the final detailed region.

Region integration and/or region division may be performed in step S2330.

Measurement values of voxels within a detailed area (initial detailed area or final detailed area) (e.g., the original signal value directly recognized by CT equipment) or display value (e.g., a value to be output to a display device based on the measured value), details Detailed regions may be hierarchically merged based on criteria such as a three-dimensional position, a size, and a shape of the region, a shape after merging to be described later, and similarity of adjacent portions between regions.

24 is a diagram for describing an embodiment of merging and/or dividing detailed regions.

In FIG. 24, A to G indicate a detailed region included in the image or a merged detailed region.

For example, in the case of a bottom-up approach, regions may be merged in the directions of FIGS. 24A to 24D. Conversely, in the case of the top-down approach, division of the region in the direction of (a) from (d) of FIG. 24 may be performed. Through this process, a hierarchical structure between the detailed area and the merged (divided) detailed area may be derived.

In (a) of FIG. 24, if, for example, the detailed area A and the detailed area B meet a predetermined merging condition, they may be merged into the detailed area E of FIG. 24(b). The new detailed area E may include all voxels included in the detailed area A and the detailed area B. In addition, a detailed area A and a detailed area B are located as sub areas of the detailed area E. This hierarchical relationship may be represented by a "hierarchical structure" corresponding to (b) of FIG. 24.

Subsequently, for example, if the detailed area E and the detailed area C meet a predetermined merging condition, they may be merged into the detailed area F of FIG. 24C. The new detailed area F may include all voxels included in the detailed area E and the detailed area C. In addition, the detailed area E and the detailed area C are located as sub areas of the detailed area F. This hierarchical relationship may be represented by a "hierarchical structure" corresponding to (c) of FIG. 24.

Subsequently, for example, if the detailed area F and the detailed area D meet a predetermined merging condition, they may be merged into the detailed area G of FIG. 24D. The new detailed area G may include all voxels included in the detailed area F and the detailed area D. In addition, a detailed area F and a detailed area D are located as sub areas of the detailed area G. This hierarchical relationship may be represented by a "hierarchical structure" corresponding to (d) of FIG. 24.

By repeatedly performing the process of merging the detailed regions as described above, when all tasks are completed, the relationship to the detailed regions can be expressed in the form of a tree structure.

The predetermined merging condition is a criterion for merging the correlated areas, rather than corresponding to completely different objects. The predetermined merging conditions may be, for example, as follows.

The merging may be performed in a form of preferentially merging detailed areas having a small size among the current detailed areas. For example, in (a) of FIG. 24, since the detailed area A and the detailed area B are the smallest, merging of the detailed area A and the detailed area B may be performed first. For similar reasons, it may not proceed in the form of combining the detailed area A and the detailed area D, or combining the detailed area A and the detailed area C.

Merging may be performed based on the location (distance) between the sub-areas to be merged. For example, detailed regions at a close distance may be preferentially merged. In (a) of FIG. 24, the detailed area A and the detailed area B are close, and the detailed area B and the detailed area C are far. Accordingly, merging of the detailed area A and the detailed area B may be performed first.

Merging may be performed based on a shape between the sub-regions to be merged (the shape of the merged surface or the similarity of adjacent portions). For example, the sub-area A and the sub-area B are characterized by a merge surface, and the shapes of adjacent portions are similar. Accordingly, merging of the detailed area A and the detailed area B may be performed preferentially.

Merging may be performed based on the shape after merging. In (a) of FIG. 24, the shape in which the detailed area A and the detailed area B are merged becomes an almost perfect circle (a sphere in a 3D image). In this case, there is a high probability that the detailed area A and the detailed area B are detailed areas included in the same object. Accordingly, merging of the detailed area A and the detailed area B may be performed preferentially.

In addition to the above-mentioned criteria, criteria conforming to the spirit of the present invention (ie, criteria capable of being judged to be included in one object) may be applied.

According to another embodiment of the present invention, the bottom-up process may be reversed, and the entire area may be divided into detailed areas (top-down method).

In (c) and (d) of FIG. 24, for example, if the detailed area F and the detailed area D included in the area G meet a predetermined merging condition, the area G of FIG. 24(d) is It can be divided into a detailed area F and a detailed area D in (c) of.

Subsequently, in (b) and (c) of FIG. 24, for example, if the detailed area E and the detailed area C included in the detailed area F meet a predetermined merging condition, the details of FIG. 24(c) The area F may be divided into a detailed area E and a detailed area C of FIG. 24B.

Subsequently, in (a) and (b) of FIG. 24, for example, if the detailed area A and the detailed area B included in the detailed area E meet a predetermined merging condition, the details of FIG. 24(b) The area E may be divided into a detailed area A and a detailed area B of FIG. 24A.

24 shows a 2D image, but this embodiment may be performed on a 3D image. In the case of the bottom-up method, 3D detailed regions included in the 3D image may be merged according to the above criteria. In the case of the top-down method, the 3D image can be divided into 3D detailed regions.

In the case of the bottom-up method, for example, as shown in FIG. 24, the detailed area A and the detailed area B may be merged into the detailed area E (FIG. 24(b)). After that, the detailed area E and the detailed area C may be merged into the detailed area F (Fig. 24(c)). After that, the detailed area F and the detailed area D may be merged into the detailed area G (Fig. 24(d)). Finally, when the region G having the same size as the 3D image is obtained by merging, a hierarchical structure according to the order of merging may be obtained.

In the case of the top-down method, an area G having the same size as a 3D image can be divided into a detailed area F and a detailed area D (Fig. 24(c)). After that, the detailed area F can be divided into a detailed area E and a detailed area C (Fig. 24(b)). After that, the detailed area E can be divided into a detailed area A and a detailed area B (FIG. 24A). In the case of the top-down method, after dividing an area to a level that cannot be divided anymore, a hierarchical structure according to the order of division may be obtained.

For the same 3D image including the same object, a layer structure obtained according to a bottom-up method and a layer structure obtained according to a top-down method may be the same.

In step S2340, classification for the detailed area may be performed. In addition, in step S2350, a hierarchical structure according to the integration/division of regions may be analyzed. The final result may be output in step S2360 by using the classification result for the detailed region and the result of analyzing the hierarchical structure. The final result (final analysis result) may be to identify the type of object (object).

More specifically, in step S2340, the types of objects (objects) corresponding to each detailed area may be automatically classified. To this end, various object detection/classification algorithms described above can be applied. Various methods according to an embodiment of the present invention for detecting and classifying an object from a 2D image may be used identically to detect an object from a 3D image or may be used after being deformed according to an increase in dimension. Alternatively, in order to detect and classify an object from a 3D image, a 3D neural network structure or a semi-3D neural network structure may be used. Alternatively, a recursive neural network (RNN) structure for using the hierarchical tree structure may be applied. Alternatively, the reliability may be corrected by using the probability of the inclusion relationship of the object according to the hierarchical tree structure as a weight for the classification result. The probability can be defined in advance.

According to the present disclosure, a hierarchical tree structure of S2350 may be used to improve classification performance of objects included in a 3D image. For example, since a smartphone contains a lithium battery, when a lower node of the tree is a lithium battery, the upper node is likely to be a smartphone. However, the reverse case is not possible and thus may be excluded from the classification candidate. For example, when the lower node is a battery, the probability that the higher node is a laptop or mobile phone may be defined high. However, the probability that the upper node is a knife or USB can be defined low. This probability can be used as a weight for classifying an object corresponding to each node.

25 is a diagram for explaining an embodiment of detecting an object from a traveler's bag according to the present invention.

As a result of analyzing a 3D image according to the present invention, a hierarchical tree structure as shown in FIG. 25 may be obtained. Nodes that contain batteries and gunpowder in lower nodes are likely to be explosives. Another sub-node included in the explosive is likely to be classified as a detonator. In addition, nodes that include battery, CPU, and camera as sub-nodes are likely to be classified as smartphones. Conversely, a lower node of a node classified as a smartphone is likely to be classified as one of the objects (battery, CPU, camera, etc.) generally included in the smartphone.

In this way, the object of the upper node and the object of the lower node may have a predetermined containment relationship, and classification of the object can be performed more accurately by using the containment relationship. For example, a result of classifying objects of a lower node may be used to classify an object of an upper node. Alternatively, the result of classifying the objects of the upper node may be used to classify the objects of the lower node. Alternatively, the object of the upper node is classified using the classification result of the object of the lower node, and again, the classification result of the object of the upper node is used to classify the object of the lower node, or the object of the lower node is classified. You can also verify the results. In addition to the above-described examples, all methods capable of improving the classification result of an object corresponding to each node of the tree using a hierarchical tree structure may be included in the scope of the present invention.

This approach can also be applied to 2D images. However, the present invention may have an optimal effect on a 3D image. This is because, according to the non-destructive inspection method, in the case of a 2D image, detailed areas are often overlapped, and thus a difference in transmittance may occur in an overlapped portion and a non-overlapping portion even if the object is the same. This is because when a difference in transmittance occurs, each portion is expressed in a range of different pixel values, and consequently, each portion can be determined as a different detailed region. On the other hand, in the case of a 3D image, since detailed areas do not overlap, each detailed area can be clearly separated. Accordingly, an embodiment of the present invention using the inclusion relationship between detailed areas can be effectively applied.

Exemplary methods of the present disclosure are expressed as a series of operations for clarity of explanation, but this is not intended to limit the order in which steps are performed, and each step may be performed simultaneously or in a different order if necessary. In order to implement the method according to the present disclosure, the illustrative steps may include additional steps, other steps may be included excluding some steps, or may include additional other steps excluding some steps.

Various embodiments of the present disclosure are not intended to list all possible combinations, but to describe representative aspects of the present disclosure, and matters described in the various embodiments may be applied independently or may be applied in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, or the like.

The scope of the present disclosure is software or machine-executable instructions (e.g., operating systems, applications, firmware, programs, etc.) that cause an operation according to the method of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium (non-transitory computer-readable medium) which stores instructions and the like and is executable on a device or a computer.

Claims (25)

Obtaining an image including one or more objects;
Extracting a plurality of detailed regions from the image;
Analyzing a hierarchical structure between the plurality of detailed regions; And
And classifying the object included in the image using the hierarchical structure,
Analyzing the hierarchical structure between the plurality of detailed regions,
In consideration of the location of the sub-region, the size of the sub-region, the shape of the sub-region, the shape of the merged region, and the similarity of adjacent parts between sub-regions, the hierarchical structure is formed by merging two or more sub-regions into the one region. Image analysis method comprising the step of configuring. The method of claim 1,
The image is a three-dimensional image analysis method. The method of claim 1,
The step of extracting a plurality of detailed regions from the image,
Binarizing the image to extract an initial detailed region; And
And subdividing the initial detailed region to extract a final detailed region. The method of claim 3,
The step of extracting the final detailed region,
An image analysis method for subdividing the initial detailed region based on values of voxels included in the initial detailed region. The method of claim 4,
The step of extracting the final detailed region,
An image analysis method performed by using the variance of values of the voxels. The method of claim 5,
The step of extracting the final detailed region,
If the variance of the values of the voxels is greater than a predetermined threshold, the initial detailed region is subdivided into two or more detailed regions,
An image analysis method for performing the step of subdividing the initial detailed region by using the detailed region obtained by subdividing as an initial detailed region. The method of claim 5,
The step of extracting the final detailed region,
When the variance of the values of the voxels is less than or equal to a predetermined threshold, the initial detailed region is determined as the final detailed region. The method of claim 1,
Analyzing the hierarchical structure between the plurality of detailed regions,
Determining whether two or more of the plurality of detailed areas can be merged into one area;
Merging the two or more sub-areas into the one area if mergeable; And
And determining a hierarchical structure including the merged region as an upper region and the two or more detailed regions as a lower region. The method of claim 8,
Analyzing the hierarchical structure,
An image analysis method performed recursively until the plurality of detailed regions are merged into one region. delete The method of claim 1,
Classifying the object included in the image using the hierarchical structure,
An image analysis method performed using a classification result of an object corresponding to an upper area of the object or an object corresponding to a lower area of the object. The method of claim 11,
A probability to be applied to the classification result of the object is predefined according to the classification result of the object corresponding to the upper region of the object or the object corresponding to the lower region of the object,
Classifying the object included in the image using the hierarchical structure may include classifying the object using the probability. An image analysis apparatus comprising at least one microprocessor,
The processor is
Obtaining an image including one or more objects;
Extracting a plurality of detailed regions from the image;
Analyzing a hierarchical structure between the plurality of detailed regions; And
Classifying the object included in the image using the hierarchical structure,
Analyzing the hierarchical structure between the plurality of detailed regions,
In consideration of the location of the sub-region, the size of the sub-region, the shape of the sub-region, the shape of the merged region, and the similarity of adjacent parts between sub-regions, the hierarchical structure is formed by merging two or more sub-regions into the one region. An image analysis device to perform, including the step of configuring. The method of claim 13,
The image analysis device is a three-dimensional image. The method of claim 13,
The step of extracting a plurality of detailed regions from the image,
Binarizing the image to extract an initial detailed region; And
And subdividing the initial detailed region to extract a final detailed region. The method of claim 15,
The step of extracting the final detailed region,
An image analysis apparatus for subdividing the initial detailed region based on values of voxels included in the initial detailed region. The method of claim 16,
The step of extracting the final detailed region,
An image analysis apparatus performed by using the variance of values of the voxels. The method of claim 17,
The step of extracting the final detailed region,
If the variance of the values of the voxels is greater than a predetermined threshold, the initial detailed region is subdivided into two or more detailed regions,
An image analysis apparatus for performing the step of subdividing the initial detailed region by using the subdivided and acquired detailed region as an initial detailed region. The method of claim 17,
The step of extracting the final detailed region,
When the variance of the values of the voxels is less than or equal to a predetermined threshold, the image analysis apparatus determines the initial detailed region as the final detailed region. The method of claim 13,
Analyzing the hierarchical structure between the plurality of detailed regions,
Determining whether two or more of the plurality of detailed areas can be merged into one area;
Merging the two or more sub-areas into the one area if mergeable; And
And determining a hierarchical structure in which the merged region is an upper region and the two or more detailed regions are a lower region. The method of claim 20,
Analyzing the hierarchical structure,
An image analysis apparatus that is performed recursively until the plurality of detailed regions are merged into one region. delete The method of claim 13,
Classifying the object included in the image using the hierarchical structure,
An image analysis apparatus performed by using a classification result of an object corresponding to an upper area of the object or an object corresponding to a lower area of the object. The method of claim 23,
A probability to be applied to the classification result of the object is predefined according to the classification result of the object corresponding to the upper region of the object or the object corresponding to the lower region of the object,
The step of classifying the object included in the image using the hierarchical structure may include classifying the object using the probability. Obtaining an image including one or more objects;
Extracting a plurality of detailed regions from the image;
Analyzing a hierarchical structure between the plurality of detailed regions; And
Classifying the object included in the image using the hierarchical structure,
Analyzing the hierarchical structure between the plurality of detailed regions,
In consideration of the location of the sub-region, the size of the sub-region, the shape of the sub-region, the shape of the merged region, and the similarity of adjacent parts between sub-regions, the hierarchical structure is formed by merging two or more sub-regions into the one region. A computer-readable recording medium recording a program to be performed, including the step of configuring.

Images