WO2020116923A1

WO2020116923A1 - Image analysis apparatus and method

Info

Publication number: WO2020116923A1
Application number: PCT/KR2019/016987
Authority: WO
Inventors: 김원태; 강신욱; 이명재; 김동민; 김신곤
Original assignee: (주)제이엘케이
Priority date: 2018-12-07
Filing date: 2019-12-04
Publication date: 2020-06-11
Also published as: KR102032796B1

Abstract

An image analysis method according to one embodiment of the present invention comprises the steps of: receiving an image to be analyzed; using a pre-learned deep learning-based model so as to detect a container area for a liquid material from the image to be analyzed; detecting a liquid area from the container area; extracting a feature vector for the liquid area; and analyzing the liquid material by comparing the feature vector with a feature vector included in a database.

Description

Image analysis device and method

The present disclosure relates to an image analysis apparatus and method. More specifically, the present disclosure detects whether an input image contains a liquid-like substance to be searched using a previously learned deep-learning-based model, and is required for training a deep-learning-based model for detecting liquid-like substances. It relates to an apparatus and method for efficiently generating training data.

In security facilities such as ports, airports, or research facilities, there is a need to search for belongings of passers-by to enhance security and prevent technology leakage. At this time, as a technique for more quickly and efficiently searching for belongings for a large number of passengers, an article search system using radiation (X-ray) or the like is often used.

Such a product retrieval system is particularly widely used in customs electronic customs clearance systems or security inspection systems. For example, the customs electronic customs clearance system is a computerized customs clearance service for imports and exports, and through this, it is possible to improve the efficiency of customs administration tasks between multiple parties.

In addition, the security inspection system is a computerized security inspection task to determine whether there is a product that may cause a safety or security problem in the passenger's belongings, thereby enhancing the security of the security area.

On the other hand, deep learning (deep learning) is learning a very large amount of data, and when new data is input, it selects the highest answer probability based on the learning result. In the process of learning a model based on data, the characteristic factors are automatically found, and thus, attempts to utilize them in the artificial intelligence field are increasing.

Existing product retrieval systems lack research on more efficient and accurate data analysis using technologies such as deep learning. In addition, in the goods search system such as customs duties and customs clearance, the necessity of searching for liquid materials that do not have a fixed form has been steadily caused. Accordingly, research on a liquid article search system incorporating deep learning is required.

The technical problem of the present disclosure is to provide an article search system to which a deep learning technique is applied.

Another technical problem of the present disclosure is to provide an apparatus and method for analyzing an image acquired in an article search system using a pre-trained deep learning based model.

Another technical problem of the present disclosure is to provide an apparatus and method for retrieving liquid substances using a pre-trained deep learning based model.

Another technical problem of the present disclosure is to provide an apparatus and method for generating a database used to search for liquid substances.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the description below. Will be able to.

According to an aspect of the present disclosure, receiving an analysis target image, detecting a container region of a liquid substance in the analysis target image using a pre-trained deep learning based model, and detecting the liquid region in the container region. An image analysis method may be provided that includes detecting, extracting a feature vector for the liquid region, and comparing the feature vector with a feature vector included in the database to analyze the liquid substance.

The image analysis method further includes adding a feature vector to the database, wherein adding the feature vector comprises: receiving a read image containing an additional target liquid substance, the additional target liquid substance Receiving information of the container containing the and the information of the liquid substance to be added, extracting a feature vector of the image of the liquid substance to be added, and the feature vector of the image of the liquid substance to be added to the liquid This may include adding to the database.

According to another aspect of the present disclosure, a receiver for receiving an analysis target image and an image analysis unit for analyzing the analysis target image to detect a liquid substance, wherein the image analysis unit uses a pre-trained deep learning based model. A liquid container detection unit for detecting a container region of the liquid substance in the analysis target image, a liquid region detection unit for detecting a liquid region in the container region, a feature extraction unit for extracting a feature vector for the liquid region, and the feature vector And a feature comparison unit that analyzes the liquid-like substance by comparing the feature vectors included in the database.

The receiving unit receives at least one of a readout image containing the liquid substance to be added, information on a container containing the liquid substance to be added, and information on the liquid substance to be added, and the feature extraction section is to add the The feature vector of the image of the target liquid substance may be extracted, and the image analysis unit may further include a database adding unit to add the feature vector of the image of the target substance to the liquid flow database.

According to another aspect of the present disclosure, receiving an analysis target image, detecting a container region of a liquid substance in the analysis target image using a pre-trained deep learning-based model, the liquid region in the container region A computer-readable recording medium recording a program for performing a step of detecting, extracting a feature vector for the liquid region, and comparing the feature vector with a feature vector included in a database to analyze the liquid material. Can be provided.

The features briefly summarized above with respect to the present disclosure are merely illustrative aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present disclosure, an article search system to which a deep learning technique is applied may be provided.

In addition, according to the present disclosure, an apparatus and method for analyzing an image acquired in an article search system using a pre-trained deep learning-based model may be provided.

In addition, according to the present disclosure, an apparatus and method for retrieving liquid substances using a deep learning-based model previously learned may be provided.

Further, according to the present disclosure, an apparatus and method for generating a database used to search for liquid-like substances may be provided.

The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. will be.

2 is a block diagram showing the configuration of an image analysis apparatus according to an embodiment of the present disclosure.

3 is a view for explaining an image reading process.

4 is a diagram for explaining an application range of artificial intelligence in an image reading process according to an embodiment of the present disclosure.

5 is a diagram illustrating an embodiment of an image enhancement device that performs image enhancement according to the present disclosure.

6 is a diagram for explaining a process of classifying an object and a background from an image including a single object and generating location information of the object, according to an embodiment of the present disclosure.

7 is a diagram illustrating an image in which colors are expressed based on physical properties of an object according to an embodiment of the present disclosure.

8 is a view for explaining a process of generating an output image based on color distribution information of an image according to an embodiment of the present disclosure.

FIG. 9 is a diagram for explaining a process of obtaining a final output image that combines an image obtained by using color distribution information and an image obtained by applying edge-based filtering or smoothing filtering according to an embodiment of the present disclosure.

10 is a view for explaining a process of obtaining a final output image using a graphical model according to an embodiment of the present disclosure.

11 is a view for explaining an image enhancement method according to an embodiment of the present disclosure.

12 is a diagram for explaining context analysis according to an embodiment of the present disclosure.

13 is a diagram illustrating a process of generating and analyzing context information of an image according to an embodiment of the present disclosure.

14 is a diagram for explaining a process in which an image analysis apparatus according to an embodiment of the present disclosure analyzes an image to identify an object.

15 is a view for explaining the operation of the image analysis apparatus according to an embodiment of the present disclosure.

16 is a diagram for explaining an embodiment of a multi-product neural network generating a multi-channel feature map.

17 is a view for explaining an embodiment of a pooling technique.

18 is a block diagram showing the configuration of an image synthesizing apparatus according to an embodiment of the present disclosure.

19 is a diagram illustrating a process of generating a multi-object image using two images including a single object according to an embodiment of the present disclosure.

20 is a diagram illustrating a process of training a convolutional neural network using a multi-object image according to an embodiment of the present disclosure.

21 is a view for explaining a process of analyzing an actual image using an image synthesizing apparatus according to an embodiment of the present disclosure.

22 is a diagram for explaining a method for synthesizing an image according to an embodiment of the present disclosure.

23 is a flowchart illustrating a method for detecting a liquid substance according to an embodiment of the present disclosure.

24 is a flowchart illustrating a method of building a liquid flow database according to an embodiment of the present disclosure.

25 is a diagram for describing an image alignment and interpolation method according to an embodiment of the present disclosure.

26 is a view for explaining a database construction result according to an embodiment of the present disclosure.

27 is a flowchart illustrating a method for detecting a liquid substance using a liquid flow database, according to some embodiments of the present disclosure.

28 is a view for explaining a method for detecting a liquid container according to an embodiment of the present disclosure.

29 is a diagram for describing an image post-processing method according to an embodiment of the present disclosure.

30 is another block diagram showing the configuration of an image analysis apparatus according to an embodiment of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily carry out the embodiments. However, the present disclosure can be implemented in many different forms and is not limited to the embodiments described herein.

In describing the embodiments of the present disclosure, when it is determined that a detailed description of known configurations or functions may obscure the subject matter of the present disclosure, detailed description thereof will be omitted. In the drawings, parts irrelevant to the description of the present disclosure are omitted, and similar reference numerals are used for similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" with another component, this is not only a direct connection relationship, but also an indirect connection relationship in which another component exists in the middle. It may also include. Also, when a component is said to "include" or "have" another component, this means that other components may be further included, not specifically excluded, unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance of components, etc., unless otherwise specified. Accordingly, within the scope of the present disclosure, the first component in one embodiment may be referred to as a second component in another embodiment, and likewise the second component in one embodiment may be the first component in another embodiment It can also be called.

In the present disclosure, the components that are distinguished from each other are for clarifying each feature, and the components are not necessarily separated. That is, a plurality of components may be integrated to be composed of one hardware or software unit, or one component may be distributed to be composed of a plurality of hardware or software units. Accordingly, such integrated or distributed embodiments are included within the scope of the present disclosure, unless otherwise stated.

In the present disclosure, components described in various embodiments are not necessarily essential components, and some may be optional components. Accordingly, an embodiment composed of a subset of components described in one embodiment is also included in the scope of the present disclosure. Also, embodiments including other elements in addition to the elements described in various embodiments are included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

1 is a view for explaining an article search system according to an embodiment of the present disclosure.

The article retrieval system 100 may include a reading unit 110 and/or a learning unit 120. The reading unit 110 may include an image analysis device 112 and/or an output device 114. The learning unit 120 may include a database 122, a deep learning learning unit 124, an algorithm verification unit 126, and/or a trained model storage unit 128. The reading unit 110 may function as a reading interface, and the learning unit 120 may function as a centrally managed artificial intelligence data center.

Hereinafter, a case where the article search system according to the present disclosure is utilized in an electronic customs clearance system or a security search system will be described as an example. However, the article search system according to the present disclosure is not limited to such applications. In addition, the article search system according to the present disclosure may be utilized in a system that serves to identify a specific article according to various purposes.

The input 130 of the article search system 100 may include images, article information and/or control information.

The image may be an image of an article including at least one object. For example, it may be an X-Ray image of an article photographed by an X-Ray reading device. The image may be a raw image photographed by an X-Ray imaging device or an image in an arbitrary format (format) for storing or transmitting the raw image. The image may be obtained by capturing and dataizing image information captured by an X-Ray reading device and transmitted to an output device such as a monitor. The image may be enhanced before being output to the output device 114 or before being input to the image analysis device 112. The method of enhancing the image will be described later. The output device 114 may output an image or an enhanced image. The image analysis device 112 may receive an image or an enhanced image and perform an operation of the image analysis device 112 described later.

The article information may be information about the article included in the corresponding image. For example, when the product input to the video analysis device is a cargo in the electronic customs clearance system, the product information may include import declaration information and/or customs inventory list information. As another example, when the product input to the image analysis device is a product in a security inspection system, the product information may include passer identification information, passer security level and/or passer authorized item information.

The product information may be subjected to a predetermined pre-processing process before being input to the image analysis device 112. For example, a refining operation of a product name may be performed on a product list, import information, and the like included in the product information. Purification work of the product name may refer to a work of unifying the names of various items input for the same or similar items.

Input of article information may be optional. For example, the article retrieval system 100 of the present disclosure can operate by receiving only an image as an input even if there is no entry of article information. The article may include all kinds of articles as objects to be inspected or read. For example, when the image analysis apparatus according to the present disclosure is used in an electronic customs clearance system, the article may be at least one of express cargo, postal cargo, container cargo, traveler transport cargo, and traveler himself. As another example, when the image analysis apparatus according to the present disclosure is used in a security search system, the article may be at least one of a passenger's belongings and the passenger's own.

For example, if the electronic customs clearance system reads a traveler, and the traveler is a major traveler with a history of transporting anomalous or dangerous objects in the past, the traveler's cargo has a higher level than that of other travelers. Analysis and/or reading may be performed. For example, the reader may be provided with information that a particular item is the cargo of a traveler of interest.

As another example, as a result of reading a passer-by, when the passer-by is a passer with a high security level, it is possible to perform a higher level of analysis and/or reading on the belongings of the passer-by than other passers-by. For example, it is possible to provide the reader with information that a specific item is belonging to a passer with a high security level.

The control information may be information for controlling image reading or controlling the read image. In one example, control information may be input by the reading source 140. For example, control information may include source information, manager information, operation mode information, read sensitivity information, and/or user interface information. The detailed use of control information will be described later.

The article search system 100 may receive an image, article information, and/or control information 130 and transmit it to the output device 114 or transmit it to the image analysis device 112. The image analysis device 112 may analyze the input image using a pre-trained deep learning-based model. The image analysis device 112 may transmit the analyzed result to the output device 114. The output device 114 outputs the input image, product information and/or control information 130, the video analysis result and/or user interface received from the video analysis device 112, and the reader 140 is an output device The output result of 114 can be read. As described above, a refining operation may be performed on the article information 130, and also, before being input to the image analysis device 112 and/or before being output to the output device 114, the image for the analysis target image Consolidation can be performed.

The output device 114 outputs all types of signals that can be detected by humans, such as a device that outputs visual information such as a monitor or a warning light, a device that outputs sound information such as a speaker, or a device that outputs tactile information such as a vibrator Includes a device that can. A user interface may be provided through the output device 114, and a reader may control the operation of the article retrieval system 100 using the user interface. For example, the reading source 140 may control the operation of the image analysis device by inputting control information using an output user interface.

When an image analysis result of the image analysis device 112 includes an object to be detected, an object with an abnormality, or an object whose risk level is greater than or equal to a threshold, the related information is output through the output device 114 as an image analysis result. And the reader 140 can confirm this. The image analysis device 112 may perform various processes of analyzing an image to be analyzed. For example, the image analysis device 112 may perform context analysis to more accurately analyze an analysis target image. Various processes and context analysis performed by the image analysis device 112 will be described later.

The reader 140 may determine whether to perform an additional test based on an image analysis result output through the output device 114. The additional inspection may include an opening inspection to directly open an article related to the corresponding image and check an object included in the corresponding article. In the present specification, the object to be searched may refer to an object with an abnormality or an object with a risk greater than or equal to a threshold as described above. However, the present invention is not limited thereto, and may include various objects to be detected or searched by the system of the present disclosure.

The image analysis result of the image analysis device, the remodeling inspection result input by the reader after performing the remodeling inspection directly, and/or matching result information obtained by matching the image and product information by the image analysis device may be transmitted to the learning unit 120. Can be. The learning unit 120 may store newly received information in the database 122, and the deep learning learning unit 124 may perform deep learning learning using the information stored in the database 122. Alternatively, without being stored in the database 122, the deep learning learning unit 124 may directly receive all or part of the learning data. The results learned by the deep learning learning unit 124 are verified by the algorithm verification unit 126, and the verified models may be stored as updated models in the trained model storage unit 128. The model stored in the trained model storage unit 128 is transmitted to the image analysis device 112 again, and the image analysis device 112 may update and use the received model as the above-described pre-trained deep learning-based model. . The learning unit 120 may generate a composite image by receiving and synthesizing a plurality of images. In addition, virtual image analysis results, remodeling inspection results and/or matching result information corresponding to the composite image may be generated using image analysis results, remodeling inspection results, and/or matching result information for each of the plurality of images. Can be. The learning unit 120 may use the composite image and the generated virtual information as learning data. According to this, even if the number of training data is absolutely small, it is possible to generate sufficient amount of training data necessary for training the artificial intelligence model by synthesizing or merging these training data. Synthesis of the image and generation of virtual information on the composite image will be described later.

The reading unit 110 and the learning unit 120 may be implemented as separate devices or may be implemented within the same device. In addition, some or all of the components included in the reading unit 110 and the learning unit 120 may be configured by hardware or software.

Artificial intelligence technology allows computers to learn data and make decisions on their own as if they were humans. Artificial neural networks are mathematical models inspired by biological neural networks. By changing the intensity of synaptic binding through learning, neurons can mean an overall model with problem-solving skills. Artificial neural networks are generally composed of an input layer, a hidden layer, and an output layer. Neurons included in each layer are connected through weights, linear combination of weights and neuron values, and nonlinearity. Through the activation function, the artificial neural network may have a form capable of approximating a complex function. The purpose of artificial neural network learning is to find a weight that minimizes the difference in value between the output calculated from the output layer and the actual output.

Deep neural network (deep neural network) is an artificial neural network consisting of several hidden layers between the input layer and the output layer, and can model complex nonlinear relationships through many hidden layers. The structure is called deep learning. Deep learning learns a very large amount of data, and when new data is input, it can operate adaptively according to the image because it selects the highest answer probability based on the learning result. In the process of learning, characteristic factors can be automatically found.

According to an embodiment of the present disclosure, a deep learning-based model includes a fully convolutional neural network, a fully convolutional neural network, and a cyclic neural network (regression). It may include at least one of a neural network, a recurrent neural network, a restricted Boltzmann machine (RBM), and a deep belief neural network (DBN), but is not limited thereto. Alternatively, a machine learning method other than deep learning may also be included. Alternatively, a hybrid model combining deep learning and machine learning may be included. For example, a feature of an image based on deep learning may be extracted, and a machine learning based model may be applied when classifying or recognizing an image based on the extracted feature. Models based on machine learning may include, but are not limited to, Support Vector Machines (SVM), AdaBoost, and the like.

Further, according to an embodiment of the present disclosure, a method for learning a model based on deep learning may include at least one of supervised learning, unsupervised learning, or reinforcement learning. , But is not limited to this. Supervised learning is performed using a series of learning data and a corresponding label (target output value), and the neural network model based on supervised learning is a model in which a function is inferred from training data. Can be. Supervised learning receives a series of training data and its corresponding target output value, finds errors through learning to compare the actual output value with the target output value for input data, and corrects the model based on the result. do. Supervised learning can be divided into regression, classification, detection, and semantic segmentation. The function derived through supervised learning can be used to predict new results. As described above, the neural network model based on supervised learning optimizes the parameters of the neural network model through learning a lot of training data.

According to an embodiment of the present disclosure, a model based on deep learning may use information about an input image and an article for learning, and after generating a trained model, information about an image and an article acquired by the apparatus of the present disclosure Can be used to update the neural network model. In addition, in the deep learning based model according to an embodiment of the present disclosure, an analysis result output by the method of the present disclosure, for example, anomalies or risks for the identified object, information about the object, and the identified object are searched The neural network model may be updated using prediction results, such as whether the object is an object, comparison information on the prediction result and the final remodeling test result, and evaluation or reliability information on the prediction result.

2 is a block diagram showing the configuration of an image analysis apparatus 200 according to an embodiment of the present disclosure. The image analysis device 200 of FIG. 2 is an embodiment of the image analysis device 112 of FIG. 1.

The image analysis device 200 may include an image receiving unit 210, an article information matching unit 220, and/or an image analyzing unit 230. As described above, since the input of the article information is optional, the image analysis apparatus 200 may not include the article information matching unit 220. Description of the input of the article information is as described with reference to FIG. 1.

The image receiving unit 210 may receive an image of an article including one or more objects. The description of the image received by the image receiving unit 210 is as described with reference to FIG. 1.

The article information matching unit 220 may receive the image received from the article information and the image receiving unit 210 as an input and perform matching of the article information and the image. The description of the article information is as described with reference to FIG. 1. Matched images and article information may be output to a reader to assist the reader in reading. Alternatively, the matched image and article information may be transmitted to the learning unit 120 of FIG. 1 to be used for learning the deep learning model. The matched image and article information are stored in the database 122 of the learning unit 120 of FIG. 1, and then refined for each reading object and/or reading task, and the deep learning learning unit 124 is for each reading object and/or Alternatively, learning can be performed using the refined data for each read task to be applied. The objects to be read may include express cargo, postal cargo, container cargo, traveler transport cargo, and traveler. Also, the object to be read may include a passenger's belongings and a passenger. The reading task includes determining whether an object included in the object is abnormal or dangerous, determining whether the identified object is an object to be searched, determining whether information on the identified object matches the object, whether the object is reported, or It may include a judgment as to whether or not it has been reported. As described above, the model trained in the learning unit 124 may be input to the image analysis unit 230 to update the existing model. At this time, suitable artificial intelligence may be updated according to the object to be read. In addition, as described above, the learning unit 124 may generate new learning data using the existing learning data and use it for learning. As described above, new learning data can be generated by combining existing images and merging data.

The image analysis unit 230 may receive an image (image to be analyzed) or image and article information, analyze the image using a pre-trained deep learning-based model, and output the analyzed result to the output device have.

When only an image is received, the image analysis unit 230 may identify an object included in the image, and determine whether there is an abnormality or risk for the identified object. The image analysis unit 230 may improve the accuracy of object identification by performing a context analysis process described below.

For example, when it is determined that the identified object is prohibited or inappropriate, the image analysis unit 230 may determine that the object is abnormal or dangerous. The risk may be expressed as a numerical value, and it may be determined whether the object is a dangerous object through comparison with a predetermined threshold. The numerical value related to the risk and/or the predetermined threshold may be adaptively determined according to a read target and/or a read task.

When the image and the article information are received together, the image analysis unit 230 may more accurately perform analysis on the object included in the image using the image and article information. For example, the type, quantity and/or size information of the items listed in the item list list, the security level of the passer, and/or the authorized item information of the passer may be additionally used to identify the object from the image. When there is a discrepancy between the object and the item information identified by analyzing the image, it may be output as a result of the image analysis.

The image analysis result output by the image analysis unit 230 may include at least one of an object's risk, type, amount, number, size, and location. When the image analysis result is the location of the object, the location of the object may be displayed on the image to be analyzed and output to the output device. The position of the object may be displayed in coordinates, but the object may be highlighted and displayed at the corresponding position in the output image so that the reader can easily read it. For example, the object may be emphasized by highlighting the edge of the object or by displaying a square box surrounding the object. In addition, a predetermined object area may be enhanced so that a reader can more easily identify the object through an image enhancement process described later. For example, an image corresponding to a predetermined color may be enhanced to convert an image so that the region can be more clearly identified.

Alternatively, the image analysis unit 230 may determine whether an object to be searched (eg, an object for which customs clearance is prohibited or inappropriate) is included in the analysis target image. To this end, the image analysis unit 230 may receive or store information about an object to be searched in advance. In addition, the image analysis unit 230 may identify an object included in the image and determine whether the identified object is a search target object.

3 is a view for explaining an image reading process.

FIG. 3A is a flowchart of a conventional reading process, and FIG. 3B is a flowchart of a reading process according to an embodiment of the present disclosure.

As shown in (a) of FIG. 3, according to an existing reading process, when image and/or article information is input 311, it is provided 312 to the reader as information. The reader selects (313) an article requiring remodeling inspection based on the image and/or article information. The result of performing the remodeling inspection is input 314 as the inspection result.

As shown in (b) of FIG. 3, according to a reading process according to an embodiment of the present disclosure, when image and/or article information is input 321, the image analysis apparatus 322 may learn deep learning in advance. The image is analyzed using the base model, and the analyzed result is provided as information to the reader (324). Also, the image analysis device 322 may transmit the training data to the AI data center 323, and the AI data center 323 may learn the training data. The artificial intelligence data center 323 may transmit the trained model to the image analysis device 322 as a read assistant assistive AI for each reading object.

The reader may select 325 an item requiring remodeling inspection based on an analysis result of the image analysis device 322, an image, and/or item information. The result of performing the remodeling test may be input 326 as a test result. The test result may be transmitted to the AI data center 323 and used as learning data.

As illustrated in FIG. 4, a sample 420 randomly extracted from all the items 410 may be selected 450 as a management object. Alternatively, the risk analysis 440 for all the products 410 may be performed using the screening assistant artificial intelligence 430 for selecting the management object, and the management object may be selected 450 through this.

The use of artificial intelligence is not limited to the risk analysis 440 of the aforementioned articles. For example, when the management target is selected 450, it may be used as an inspection aid artificial intelligence 460 to assist the examination. For example, by applying the inspection aid artificial intelligence 460, the inspection of the inspection source can be assisted by identifying the object, determining whether an identified object is abnormal or dangerous, and/or providing information about the object to be searched to the inspection source. . The reader may perform a precise inspection 470 using information provided by the inspection assistant artificial intelligence.

Hereinafter, with reference to FIGS. 5 to 11, an embodiment of a method of enhancing an image before being input to the image analysis device 112 of FIG. 1 and/or before being output to the output device 114 will be described.

The image enhancement device of FIG. 5 may be configured separately from the image analysis device 112 of FIG. 1 or may be configured as a part thereof.

The image enhancement device 500 may include an image reception unit 510, an object image extraction unit 520, a color distribution analysis unit 530, and/or an image enhancement unit 540. However, this only shows some components necessary to describe the present embodiment, and the components included in the image enhancement apparatus 500 are not limited to the above-described examples. For example, two or more components may be implemented in one component, or an operation executed in one component may be divided and implemented to be executed in two or more components. In addition, some components may be omitted or additional components may be added.

The image enhancement apparatus 500 according to an embodiment of the present disclosure receives the input image 550, extracts an object included in the input image 550, and extracts an object image including the object into one or more regions. Divide, obtain color distribution information for each of the one or more areas, and determine one or more weights for at least some of the one or more areas based on the color distribution information, and determine the determined one or more weights among the one or more areas The first output image 560 for the object image may be generated by applying to at least a part.

Each pixel constituting the image may have a predetermined brightness and color by a combination of a luminance value representing luminance (brightness) and a color value representing color. At this time, the color value may be represented by a combination of values of three or more color elements according to various ways of expressing color. For example, the color value may be expressed as an RGB value that is a combination of three color elements (Red(R), Green(G), Blue(B)). For example, each of R, G, and B has a value from 0 to 255, so that the intensity of each color element can be expressed. The range of values that each of R, G, and B can have may be determined based on the number of bits representing each of R, G, and B. For example, when represented by 8 bits, each of R, G, and B may have a value from 0 to 255.

Acquiring color distribution information may mean acquiring various statistical values that can be obtained by analyzing color components of color values of pixels included in a corresponding region. For example, the statistical value may be information on a color element having an average largest value among color elements of color values of pixels included in a corresponding region. For example, based on the sum of the values of R, G, and B of all pixels included in the corresponding area, it may be determined which color element has the largest sum or average among R, G, and B. Alternatively, for each pixel, the color element having the largest value among R, G, and B is determined as the dominant color of the corresponding pixel, and which color is determined as the dominant color for all pixels included in the corresponding region I can judge. In this way, it can be determined what the dominant color of a given area is. For example, for the color values of the majority of pixels included in a predetermined region, if R among the three color elements R, G, and B has the largest value, the dominant color of the predetermined region is red. I can judge. In the above description, color distribution information or dominant color was analyzed based on each of R, G, and B. However, the present invention is not limited thereto, and may be analyzed based on various colors expressed by a combination of two or more of R, G, and B. For example, if the color to be identified is orange, it may be determined whether the dominant color of the pixel in the corresponding area is orange based on a combination of some or all of R, G, and B representing orange.

Hereinafter, it is assumed that a region in which the dominant color is red is an object of image enhancement, and an embodiment of a process of enhancing the image by applying a weight will be described in detail. When it is determined that the dominant color of a predetermined region is red based on the color distribution information, one or more weights may be determined for the corresponding region. Weights can be determined for all or part of R, G, B and luminance. For example, when enhancing red, the weight for R may be a value greater than one. Applying a weight may mean multiplying a color element value of a pixel in a corresponding area by a corresponding weight. In this case, the weight for G and/or B may be a value less than one. By doing so, the region where red is dominant can be strengthened to a region that is more red.

In the above, it was described to enhance a specific color of the image. However, the enhancement of the image of the present disclosure is not limited to this, and may include both a change in color value or a change in brightness value. Therefore, if necessary, an image may be enhanced by applying a weight to a luminance value.

Hereinafter, each component of the image enhancement device 500 will be described.

The image receiving unit 510 may receive an input image 550 including one or more objects. The input image 550 may be an image before being input to the image analysis device 112 and/or an image before being output to the output device 114.

The object image extracting unit 520 may extract an object included in the input image received from the image receiving unit 510 and divide the object image including the object into one or more regions. For example, the object image extractor 520 compares the pixel value of the analysis target image with a predetermined threshold to binarize the pixel values and group the binarized pixel values to extract objects included in the input image. Here, extracting an object may mean distinguishing an object from a background, an object may mean a specific object in an image, and the background may mean a portion excluding an object from an image. The background of the image may be expressed in a predetermined color according to a method of photographing or a photographing device. For example, the predetermined color may be white. When a color representing the background of the image is specified, the background and the object may be separated based on the specified background color. For example, an object may be classified by deleting the specified background color area from the input image 550.

Also, for example, an object image may be obtained by specifying a bounding box surrounding an object area, and the object image extracting unit 520 may generate location information of the separated object based on the specified rectangle box. have. That is, the rectangular box may mean an object recognition box.

According to an embodiment of the present disclosure, if the input image is an X-Ray image of an article photographed by an X-Ray reading device, since the background portion other than the article is unnecessary, the background portion is cut out and the article exists It can be analyzed with only the domain. In particular, it can be said that it is important to obtain an area for an article in a real environment in which the article continuously passes through the X-Ray reading device through the conveyor belt. A detailed process of distinguishing the object from the background and generating the location information of the object will be described in detail with reference to FIG. 6.

6 is a diagram for explaining a process of classifying an object and a background from an image including a single object and generating location information of the object, according to an embodiment of the present disclosure. The object image extraction unit 600 of FIG. 6 may be an embodiment of the object image extraction unit 520 of FIG. 5. The input image 610 may be the input image 550 described with reference to FIG. 5, for example, an image related to an article including the bag 612 as a single object.

The object image extraction unit 600 first roughly cuts the surrounding area based on the bag 612 by performing a cropping operation on the input image 610 including one bag 612. A discarded, cropped image 620 may be obtained. Then, the object image extractor 600 may obtain the binarized image 630 by binarizing the pixel value by comparing a pixel value of the cropped image 620 with a predetermined threshold. Then, the object image extractor 600 may obtain a grouped image 640 by grouping (clustering, morphology, closing) adjacent pixels to select a portion of the object in the binarized image 630. Then, the object image extractor 600 performs labeling and hole filling operations on the grouped image 640 to convert the group of pixels formed in the largest shape into an area 652 for the object. The image 650 from which the object is extracted may be obtained by determining and determining the rest as the region 654 for the background. Also, the object image extraction unit 600 may determine the location of the object in the input image 610 using information on the extracted object image. For example, the object image extraction unit 600 may specify a rectangular box surrounding the object area, and generate location information of the object based on the specified rectangular box. Referring to FIG. 6, the object image extraction unit 600 may specify a rectangular box 662 surrounding the bag 612 and obtain location information of the bag 612 based on the specified rectangular box. . For example, the location information of the bag 612 may be location information of four vertices forming the rectangular box 662, but is not limited thereto. For example, the location information may be represented by the coordinates (x, y) of one vertex of the rectangular box 662, and the width and height of the rectangular box. The coordinates (x, y) of the one vertex may be the coordinates of the upper left corner of the rectangular box 662. The coordinates (x, y) of the vertex may be specified based on the coordinates (0, 0) of the upper left corner of the input image 610.

Referring back to FIG. 5, the object image extractor 520 may divide the object image into one or more regions based on the size of the object image. Each of the one or more regions may be square. For example, the object image extraction unit 520 may determine the number or size of regions for dividing the object image based on the size of the object image. For example, when the object image is relatively large or has a size larger than a predetermined threshold, the object image may be divided to have more divided areas. Also, the sizes of the regions dividing the object image may not be the same.

In addition, the object image extractor 520 converts the object image into a square by up-sampling or down-sampling the object image when the object image is not square, and then converting the object image into one or more squares. It can be divided into regions. For example, since the object image is obtained based on a rectangular box surrounding the object for the object extracted by the object image extraction unit 520, the object image may not be square. In this case, the object image extractor 520 may divide the object image into one or more regions, but acquires and obtains a square object image by upsampling or downsampling in the horizontal or vertical direction of the object image. The divided square object image may be divided into one or more regions.

For example, referring to FIG. 8, the object image 800 may not be square because it is composed of 9 pixels horizontally and 12 pixels vertically. In this case, according to the present disclosure, the object image 800 may be divided into a 3x3 sized square area (in this case, (horizontal size of object image/horizontal size of divided area)) x (vertical size/division of object image) Vertical size of the area) = (9/3) x (12/3) = 12, which has 12 areas in total), upsampling the horizontal of the object image 800, and consisting of 12 horizontal pixels and 12 vertical pixels It is also possible to obtain and divide it into 3x3 sized areas to divide them into a total of 16 areas. The shape of one or more regions dividing the object image is not limited to a square. For example, the region may have a form of n x m where n and m are different positive integers. In this case, the aforementioned upsampling or downsampling may not be performed.

Referring back to FIG. 5, the color distribution analysis unit 530 acquires color distribution information for each of the regions divided by the object image extraction unit 520, and based on the color distribution information, for at least some of the regions One or more weights can be determined.

The color distribution information may include information for each of n (n is an integer greater than 1) color expression ranges. The color expression range may vary depending on the type and performance of the image acquisition device. Further, for example, "color" for R(red) may mean only the color of a pixel having a pixel value of (R, G, B) = (255,0,0) in an 8-bit image, but R The term "color expression range" for "meaning that the pixel value is (R, G, B) = (255,0,0)" is meant to include similar colors within a predetermined range based on the pixel value. For example, the “color expression range” for R may be in the range of (R, G, B) = (150-255, 0-100, 0-100). That is, a pixel having (R, G, B) = (150, 100, 100) can also be defined as being included in the color expression range of red (R). The "color expression range" may be defined for a color to be identified. In the above-described example, the color expression range of red is described as a reference, but the color expression range of green (G) or blue (B) may be defined. Alternatively, a range of color expression for arbitrary colors (yellow, orange, sky blue, etc.) expressed by combining some or all of R, G, and B may be defined. When an object included in the image is to be enhanced, for example, an area expressed in orange, as a result of analyzing color distribution information, by applying a weight to a region in which a number of pixels included in the range of orange color expression are predominant or dominant, Image enhancement according to the present disclosure may be performed. The method of applying the weight is as described above.

If the image acquisition apparatus expresses a color by a combination of three color elements: R, G (green), and B (blue), the color distribution information may include information on some or all of the three color elements. If there are five color elements R, G, B, Y (yellow), and P (purple), the color distribution information may include information on some or all of the five color elements.

In an X-Ray image of an object photographed by an X-Ray reading device, different color expression ranges are determined according to properties of objects included in the image (for example, whether the object is an organic substance, an inorganic substance, a metal, or the like). The applied X-Ray image is used. By reading a color-added X-Ray image, the reader can discriminate not only the shape of the object included in the image, but also the physical properties of the object. The image enhancement of the present disclosure analyzes color distribution information using an X-Ray image to which color is added according to the physical properties of an object as an input image, and strengthens a region of a specific color based on this, thereby detecting an object included in the image. It can improve the accuracy and readability of the reader reading the image.

Referring to FIG. 7, a bag image 700 taken by an X-Ray reading device, a medicine container image 710 and a traveler luggage carrier image 720 are shown. In the case of the bag loop 702, the bag zipper 704, the medicine 712, and the bottle 722, it can be confirmed that the color expression range (the applied color) is different depending on the properties of the object. On the other hand, the bag loop 702, the bag zipper 704, the medicine 712, and the bottle 722 are relatively clearly colored so that they can be distinguished from other objects, while any content in the luggage 724 In the case of ), it can be seen that it is difficult to determine what the arbitrary content 724 is in the traveler's luggage image 720 and it is not easy to distinguish it from other objects. This is due to the physical properties of the object. For example, a metal or an inorganic material is expressed in a relatively clear and distinct color so that it can be clearly distinguished from a background, whereas an organic material is expressed in a light color so that the distinction from the background is not clear. The area of color representing organic matter can be enhanced with a clear and clear color that can be clearly distinguished from the background through a method of enhancing the corresponding color.

In other words, it is necessary to vary the degree of enhancement of the image according to the color expression range by using a feature that the color expression range is different according to the physical properties of the object. To this end, the color distribution for each of the divided regions may be analyzed to apply weights to at least some regions.

The one or more weights may include weights for at least some of n color expression ranges or n color elements representing colors. For example, if one region has n color expression ranges or color elements, the number of weights in the region may have 1 to n.

For example, when one weight is determined for one area, the determined weight may be applied to all color elements or all color expression ranges included in the one area. Alternatively, the determined weight may be applied to at least a portion of all color elements or all color expression ranges included in the one area. For example, in order to enhance the image, the determined weight may be applied only to a predetermined color element among n color elements or a predetermined color expression range among n color expression ranges.

Or, for example, a weight may be determined for each of n color elements or n color expression ranges. That is, the number of weights for one region may be n. In this case, a weight corresponding to each color element or color expression range included in the region may be applied to the corresponding color element or color expression range. The weight may be given a relatively high weight for a predetermined color element or color expression range that is an object of image enhancement. For example, a weight greater than 1 may be given and multiplied by a value of a corresponding color element or a pixel value belonging to a corresponding color expression range.

Alternatively, for example, a weight may be determined for each of m color elements greater than 1 and less than n or a color expression range. That is, the number of weights for one region may be m. In this case, the weighted weight may be applied only to a weighted color element or color expression range among color elements or color expression ranges included in the region. It is as described above that a relatively high weight is given to a predetermined color element or color expression range that is an object of image enhancement.

As described above, the weight may be relatively high for a predetermined color element or color expression range among n color elements or color expression ranges. For example, when an object included in an X-Ray image is an organic material, a boundary is often less clearly defined in an image than an object having different physical properties (metal, inorganic, etc.). This is because the color of the object, which is an organic material, is not vivid enough to be distinguished from other objects or backgrounds. For example, by being expressed in light orange, it may not be well distinguished from a white background. Therefore, by applying a relatively high weight to a portion corresponding to a color expression range representing an organic material among the divided regions, the corresponding color can be enhanced to change, for example, light orange to dark orange. By strengthening the image in this way, it is possible to more clearly distinguish between the surrounding objects or the background and the object to be strengthened.

The predetermined color element or color expression range to which a relatively high weight is assigned may be one or more. For example, when the total color element or color expression range is n, the predetermined color element or color expression range to which a relatively high weight is assigned may be 1 to n. When the predetermined color element or color expression range is plural, the degree of image enhancement required for each may be different, and accordingly, different weights may be assigned to each. For example, when an image is clearly expressed in the order of metal->inorganic->organic, a relatively high weight may be given only to a color element or a color expression range for an organic material, but for inorganic and organic materials, it is relatively more than a metal. You can also give it a high weight. At this time, a relatively high weight may be given to the organic material rather than the inorganic material.

A detailed process of obtaining color distribution information for each of the divided areas and determining a weight will be described in detail with reference to FIG. 8.

Referring to FIG. 8, the object image 800 may be divided into one or more regions, such as the first region 810 and the second region 820. The process of dividing regions in the object image 800 is as described with respect to the object image extractor 520 of FIG. 5. Hereinafter, a process of obtaining color distribution information and determining weights in the first area 810 will be described in detail. It is assumed that the first region 810 is a 3x3 sized region and has a total of 9 pixels, and has 5 (n=5) color expression ranges (hereinafter, the color expression range may be replaced with color elements). . The image enhancement device according to an embodiment acquires color distribution information including information on five color expression ranges for the first area 810, and based on the obtained color distribution information, at least a part of the 3x3 sized area One or more weights can be determined for.

Alternatively, only information on a predetermined color expression range targeted for image enhancement may be obtained and used as color distribution information. For example, when the distribution information for a predetermined color expression range is greater than or equal to a predetermined threshold, the corresponding area is determined as a target for enhancement, and a relatively high weight can be given to the corresponding area.

Looking at an embodiment of determining the weight based on the color distribution information, n color channel images reflecting information on each of the n color expression ranges for each of the divided regions may be obtained. For example, since the first region 810 has information on five (n=5) color expression ranges, five color channel images can be obtained, which are the first color channel image 830 and the second. Referred to as a color channel image 840, a third color channel image 850, a fourth color channel image 860, and a fifth color channel image 870. In addition, for example, when the X-Ray reading device supports five color elements of R, G, B, Y, and P, the first color channel image 830, the second color channel image 840, and the third color channel The image 840, the fourth color channel image 860, and the fifth color channel image 870 may correspond to color elements of R, G, B, Y, and P, respectively.

Each of the first to fifth color channel images 830 to 870 is generated by mapping each pixel to a color channel image corresponding to the corresponding color information based on color information of each of the constituent pixels of the first region 810. Can be. For example, the first pixel 812 is mapped to the pixel 852 at the corresponding position of the third color channel image 850, and the second pixel 814 is the pixel at the corresponding position of the first color channel image 830. Mapped to 832, and the third pixel 816 is mapped to the pixel 872 at the corresponding position of the fifth color channel image 870, and the fourth pixel 818 is the second color channel image 840. ) Is mapped to the pixel 842 at the corresponding position, and the fifth pixel 820 is mapped to the pixel 874 at the corresponding position of the fifth color channel image 870, and the sixth pixel 822 is It is mapped to the pixel 876 at the corresponding position in the fifth color channel image 870, and the seventh pixel 824 is mapped to the pixel 844 at the corresponding position in the second color channel image 840, and also the eighth The pixel 826 is mapped to the pixel 878 at the corresponding position in the fifth color channel image 870, and the ninth pixel 828 is mapped to the pixel 880 at the corresponding position in the fifth color channel image 870. By mapping, the first to fifth color channel images 830 to 870 may be generated.

Meanwhile, when the color expression range is up to n, fewer color channel images than n may be obtained. For example, in the case of the first region 810, pixels having a color corresponding to the fourth color channel image 860 may be obtained. Since this does not exist, a total of four color channel images can be obtained except for the fourth color channel image 860.

When the color channel images are acquired, the first color channel image 830, the second color channel image 840, the third color channel image 850, the fourth color channel image 860, and the fifth color channel image 870 ) Can be applied to the weights a1, a2, a3, a4, and a5, respectively.

The weight may be determined in consideration of the color distribution of pixels constituting each area, and for example, the weight may be determined to be proportional to the color distribution of pixels. Alternatively, the weight may be determined to have a relatively high weight for a predetermined color expression range and a relatively low weight for the rest of the color expression range.

Referring back to FIG. 5, the image enhancement unit 540 may generate a first output image for the object image by applying one or more weights determined by the color distribution analysis unit 530 to at least some of the one or more regions. .

Referring to FIG. 8, a first color channel image 830, a second color channel image 840, a third color channel image 850, a fourth color channel image 860, and a fifth color channel image 870 Weighted a1, a2, a3, a4, and a5 may be applied to the weighted first region 810-1 by combining the weighted first to fifth color channel images. And, by repeating the above process for the remaining regions of the object image 800, the first output image may be finally generated. The weight may be determined in consideration of the color distribution of pixels constituting each region, and a relatively high weight may be determined for a predetermined color expression range and a relatively low weight for the remaining color expression ranges. For example, the portion corresponding to the color representing the organic material in each divided region is not clearly distinguished from the background, so the boundary portion is not relatively clearly expressed in the image, so the weight is determined relatively high, and the color corresponding to the color representing the metal Since the portion is relatively distinct from the background, the weight of the border portion may be relatively low because the border portion is relatively clearly expressed in the image. As described above, applying the weight may mean replacing a pixel in the enhanced region with a new pixel value multiplied by the weight.

Alternatively, as described above, as a result of analyzing the color distribution of the region 810 included in the object image 800, when a predetermined color expression range that is an object of image enhancement is dominant or has a distribution of a predetermined threshold or more, the corresponding region ( 810), a relatively high weight can be set.

For example, when the color expression range targeted for enhancement is red (R) and the dominant color of a predetermined region is red, one or more weights may be applied to the corresponding region. For example, when (R, G, B) = (120, 10, 10) of the pixels in the predetermined area, by applying a weight of 2, (R, G, B) = (240, 20, 20) can be enhanced have. Alternatively, a weight of 2 is applied only to some of R, G, and B, for example, red, and may be enhanced to (R, G, B) = (240, 10, 10). Alternatively, a weight of 2 is applied to some of R, G, and B, such as red, and a weight of 0.5 is applied to other parts, such as green and blue, (R, G, B) = (240, 5, 5) It can also be strengthened. The above example relates to a case in which red is enhanced, but is not limited thereto, and any color may be determined as a target color.

The predetermined threshold and/or weight may be arbitrarily determined, or may be determined based on accumulated image processing information. Alternatively, by performing learning on the threshold and/or weight through an AI-based learning model, the optimal threshold and/or weight may be continuously updated.

Also, the image enhancement unit 540 may generate a second output image for the object image by applying edge-based filtering or smoothing filtering on at least some of the one or more regions. Also, the image enhancement unit 540 may generate a third output image for the object image based on the generated first output image and second output image.

Edge-based filtering or smoothing filtering is a technique for enhancing the contrast of an image, including, but not limited to, Wiener filtering, Unsharp mask filtering, Histogram equalization, linear contrast adjustment, and the like. May include techniques to enhance.

FIG. 9 is a diagram for explaining a process of obtaining a final output image that combines an image obtained by using color distribution information and an image obtained by applying edge-based filtering or smoothing filtering according to an embodiment of the present disclosure. The object image 900 of FIG. 9, the first area 910 and the weighted first area 910-1 are the object image 800 of FIG. 8, the first area 810, and the weighted first area It may correspond to (810-1), respectively. Referring to FIG. 9, the image enhancement unit 540 may generate the first region 910-2 to which the filtering is applied to the first region 910, and the first region 910-1 to which the weight is applied. And the first region 910-2 to which filtering has been applied may be combined to generate a final first region 910-3. In addition, the image enhancement unit 540 may generate a second output image in which the above filtering techniques are applied to the remaining regions, and a third output image combining the first output image and the second output image.

The process of generating a weighted area (eg, 910-1), a filtered area (eg, 910-2), and/or a final area 910-3 using the two may be performed in units of areas. However, the present invention is not limited thereto, and the process may be performed in units of object images. For example, a weighted object image (first output image) may be obtained by performing a process of applying a weight to each of the regions included in the object image. Also, an object image (second output image) may be obtained by performing the process of applying the filtering to each of the regions included in the object image. In addition, the final image (third output image) may be generated by combining the weighted object image and the edge-enhanced object image.

On the other hand, if the organic matter in the divided region is less than other materials, for example, the influence on the first output image may be relatively small by combining the second output image with the first output image. In this case, the color representing the organic substance The weight for the distribution information can be determined relatively higher. Also, for example, by combining the first output image and the second output image, more accurate object recognition may be possible even when multiple objects in the image overlap.

The image enhancement apparatus according to an embodiment determines each of the color expression ranges included in the color distribution information as an individual node, and determines a relative relationship between each determined individual node and a first output image, a second output image, and a third output image. Using the relative relationship of, a graphical model of a hierarchical structure can be generated.

Referring to FIG. 10, in each of the divided regions, if n color expression ranges are included in the color distribution information, the lowest node is the first color distribution information 1010-1 to the nth color distribution information 1010-n ) Up to n nodes. Then, based on each color distribution information, a first output image 1020 may be obtained by applying a weight to each of the corresponding divided regions or the color expression ranges of the divided regions. The first output image 1020 may be determined as the final output image. Alternatively, the second output image 1030 obtained by applying the contrast enhancement technique of the image is further generated, and the third output image 1040 is generated based on the first output image 1020 and the second output image 1030. You may.

11 is a view for explaining an image enhancement method according to an embodiment of the present disclosure. The image enhancement method of FIG. 11 is a method performed by the image enhancement apparatus of FIG. 5, and the description of the image enhancement apparatus of FIG. 5 may be applied to the image enhancement method of FIG. 11.

In step S1100, an input image may be received.

In step S1110, an object included in the input image may be extracted. For example, by comparing the pixel value of the input image with a predetermined threshold, the pixel value is binarized and the binarized pixel value is grouped to extract an object included in the analysis target image.

In step S1120, the object image including the object may be divided into one or more regions. For example, the number or size of regions for dividing the object image may be determined based on the size of the object image. Also, the sizes of the regions dividing the object image may not be the same. In addition, for example, when the object image is not square, the object image can be divided into one or more regions after up-sampling or down-sampling to convert the object image into a square. have.

In step S1130, color distribution information may be obtained for each of the one or more regions. For example, the color distribution information may include information for each of n (n is an integer greater than 1) color expression range.

In step S1140, one or more weights may be determined for at least some of the one or more regions based on the color distribution information. For example, the one or more weights may include weights for at least some of the n color expression ranges. For example, if one region has n color expression ranges, the number of weights in the region may have 1 to n.

In operation S1150, a first output image for the object image may be generated by applying the determined one or more weights to at least some of the one or more regions.

Although not illustrated in FIG. 11, a second output image for the object image may be generated by applying edge-based filtering or smoothing filtering to at least some of the one or more regions. Also, for example, a third output image for the object image may be generated based on the generated first output image and second output image.

In the embodiment described with reference to FIGS. 5 to 11, an example of receiving an image including a single object and separating the object from the background has been described. However, the present invention is not limited thereto, and the input image may be an image including two or more objects. In this case, two or more objects and a background can be distinguished from the input image, and location information can be generated and used for each of the two or more objects. In addition, in this case, in the description with reference to FIG. 6, when a plurality of pixel groups are formed, it may be determined that each pixel group is an area for an object as well as other pixel groups formed in the largest shape. The process of generating location information of each determined object is the same as described for an image including one object.

At least some of the components of the image enhancement apparatus and steps of the image enhancement method of the present disclosure may be performed using an AI-based or deep learning-based model. For example, the size, number of regions generated by dividing an object image, weights determined based on color distribution information, various thresholds mentioned in the present disclosure, whether a second output image is generated, or the like is based on artificial intelligence or deep learning. It can be learned using a model, and information according to the trained model can be used.

Hereinafter, an embodiment of a context analysis method performed by the image analysis apparatus 112 will be described with reference to FIGS. 12 to 17.

The image analysis device 1200 of FIG. 12 may be an embodiment of the image analysis device 112 of FIG. 1. Alternatively, the image analysis device 1200 of FIG. 12 may be included in the image analysis device 112 of FIG. 1, or may be configured separately to perform context analysis.

Referring to FIG. 12, the image analysis apparatus 1200 may include a feature extraction unit 1210, a context generation unit 1220, and/or a feature and context analysis unit 1230. However, this only shows some components necessary to describe the present embodiment, and the components included in the image analysis apparatus 1200 are not limited to the above-described examples.

The image analysis apparatus 1200 may extract characteristics of an input image (analysis target image), generate context information based on the extracted characteristics, and analyze an analysis target image based on the extracted characteristics and the generated context information. have. For example, the image analysis apparatus 1200 may classify an image using the extracted feature and the generated context information or locate the object of interest.

The input image of the image analysis apparatus 1200 may be the same as the input image of the image analysis apparatus 112 of FIG. 1.

The feature extraction unit 1210 may analyze the input image to extract features of the image. For example, the feature may be a local feature for each region of the image. The feature extraction unit 1210 according to an embodiment may extract characteristics of an input image using a general convolutional neural network (CNN) technique or a pooling technique. The pooling technique may include at least one of a max (max) pooling technique and an average pooling technique. However, the pooling technique referred to in the present disclosure is not limited to the Max pooling technique or the average pooling technique, and includes any technique for obtaining a representative value of an image region of a predetermined size. For example, the representative value used in the pooling technique may be at least one of a variance value, a standard deviation value, a mean value, a most frequent value, a minimum value, and a weighted average value, in addition to the maximum value and the average value.

The convolutional neural network of the present disclosure can be used to extract “features” such as borders, line colors, and the like from input data (images), and may include a plurality of layers. Each layer may receive input data and process input data of the corresponding layer to generate output data. The convolutional neural network may output a feature map generated by convolution of an input image or an input feature map with filter kernels as output data. The initial layers of the convolutional neural network can be operated to extract low level features such as edges or gradients from the input. The next layers of the neural network can extract progressively more complex features, such as the eyes and nose. The detailed operation of the convolutional neural network will be described later with reference to FIG. 16.

The convolutional neural network may include a pooling layer in which a pooling operation is performed in addition to a convolutional layer in which a convolution operation is performed. The pooling technique is a technique used to reduce the spatial size of data in the pooling layer. Specifically, the pooling technique includes a max pooling technique that selects a maximum value in a corresponding region and an average pooling technique that selects an average value in a corresponding region. In the field of image recognition, a max pooling technique is generally used. do. In the pooling technique, the pooling window size and spacing (stride, stride) are generally set to the same value. Here, the stride means adjusting an interval to move when applying a filter to input data, that is, an interval to move the filter, and stride can also be used to adjust the size of the output data. The detailed operation of the pulling technique will be described later with reference to FIG. 17.

The feature extraction unit 1210 according to an embodiment of the present disclosure is pre-processing for extracting features of an analysis target image, and filtering may be applied to the analysis target image. The filtering may be a Fast Fourier Transform (FFT), histogram equalization, motion artifact removal, or noise removal. However, the filtering of the present disclosure is not limited to the above-listed methods, and may include all types of filtering capable of improving the image quality. Alternatively, as the pre-processing, enhancement of the image described with reference to FIGS. 5 to 11 may be performed.

The context generation unit 1220 may generate context information of the input image (analysis target image) using the features of the input image extracted from the feature extraction unit 1210. For example, the context information may be a representative value representing all or part of an image to be analyzed. Also, the context information may be global context information of the input image. The context generation unit 1220 according to an embodiment may generate context information by applying a convolutional neural network technique or a pooling technique to features extracted from the feature extraction unit 1210. The pooling technique may be, for example, an average pooling technique.

The feature and context analysis unit 1230 may analyze an image based on the feature extracted by the feature extraction unit 1210 and the context information generated by the context generation unit 1220. The feature and context analysis unit 1230 according to an embodiment concatenates local features and local contexts reconstructed by the context generation unit 1220 for each region of the image extracted by the feature extraction unit 1210. It can be used together to classify the input image or to find the location of the object of interest included in the input image. Since information at a specific 2D position in the input image includes not only local feature information but also global context information, the feature and context analysis unit 1230 uses these information, so that the actual content is different but local feature information. It is possible to more accurately recognize or classify similar input images.

As described above, the invention according to an embodiment of the present disclosure enables more accurate and efficient learning and image analysis by using global context information as well as local features used by general convolutional neural network techniques. do. From this point of view, the neural network to which the invention according to the present disclosure is applied may be referred to as'deep neural network through context analysis'.

The feature extraction unit 1310, the context generation unit 1320, and the feature and context analysis unit 1330 of FIG. 13 are the feature extraction unit 1210, the context generation unit 1220, and the feature and context analysis of FIG. 12, respectively. It may be an embodiment of the unit 1230.

Referring to FIG. 13, the feature extractor 1310 may extract a feature from the input image 1312 using the input image 1312 and generate a feature image 1314 that includes the extracted feature information. The extracted feature may be a feature for a local area of the input image. The input image 1312 may include an input image of an image analysis device or a feature map at each layer in a convolutional neural network model. Also, the feature image 1314 may include a feature map and/or feature vector obtained by applying a convolutional neural network technique and/or a pooling technique to the input image 1312.

The context generation unit 1320 may generate context information by applying a convolutional neural network technique and/or a pooling technique to the feature image 1314 extracted by the feature extraction unit 1310. For example, the context generating unit 1320 may generate context information of various scales, such as an entire image, a quadrant area, and a 9-section area, by variously adjusting the spacing of the pooling. Referring to FIG. 13, an entire context information image 1322 including context information for an image of a full-size image, and a quadrant context information image including context information for a quarter image having a size that is divided into four parts of the entire image ( 1324) and a 9-part context information image 1326 may be obtained, including context information for a 9-part image of a size divided into 9 parts.

The feature and context analysis unit 1330 may more accurately perform analysis on a specific region of an analysis target image using both the feature image 1314 and the

context information images

1322, 1324, and 1326.

For example, when an image including a boat having a shape similar to a car is an input image, the identified object is obtained from the feature image 1314 including local features extracted by the feature extractor 1310. It is impossible to accurately determine whether is a car or a boat. That is, the feature extraction unit 1310 may recognize the shape of the object based on the local feature, but may not accurately identify and classify the object using only the shape of the object.

The context generation unit 1320 according to an embodiment of the present disclosure can more accurately identify and classify objects by generating

context information

1322, 1324, and 1326 based on the analysis target image or the feature image 1314. Can be. For example, the feature extracted for the entire image is recognized or classified as "natural landscape", the feature extracted for the quarter image is recognized or classified as "lake", and the feature extracted for the 9-part image is "water" When recognized or classified as, the extracted features “natural landscape”, “lake”, and “water” may be generated and utilized as context information.

The feature and context analysis unit 1330 according to an embodiment of the present disclosure may identify an object having a shape of the boat or vehicle as a "boat" by utilizing the context information.

In the embodiment described with reference to FIG. 13, it has been described to generate and utilize context information for an entire image, context information for a quarter image, and context information for a ninth image, but the size of the image for extracting context information is It is not limited to this. For example, context information for an image having a size other than the above-described image may be generated and utilized.

The convolutional neural network technique and pooling according to an embodiment of the present disclosure will be described later with reference to FIGS. 16 and 17.

For example, the image analysis device 1400 may accurately identify and/or classify objects included in the image 1410 by receiving the image 1410 and generating information on image regions of various sizes. The input image 1410 may be, for example, an X-ray image including a bag. The image analysis device 1400 analyzes the input image 1410 as described above, extracts features for the entire image, and features for some areas of the image, and accurately identifies the objects included in the image 1410 using the image analysis. can do. The feature 1422 for the entire image may be, for example, a feature for the shape of the bag. Features for some areas of the image may include, for example, features 1424 for handles, features 1426 for zippers, features 1428 for rings, and the like.

The image analysis apparatus 1400 can accurately identify that the object included in the image 1410 is a "bag" by using the generated features 1422, 1424, 1426, and 1428 as context information.

If some of the generated features are not related to the "bag", the image analysis device 1400 cannot identify that the object included in the image 1410 is a "bag" or the image 1410 It may provide an analysis result that the object included in the "bag" cannot be identified. Alternatively, when some of the context information is not related to other context information, an abnormality of the corresponding object may be output. For example, when an irregular space, a space of a certain thickness, or the like, which is not related to the normal characteristics of the "bag", is detected, the corresponding "bag" may output a signal that there is an abnormal bag.

As described above, when contextual information that is not related to the normal contextual information is included, such fact may be output to the reader, and the reader may, based on this, perform a close inspection or remodeling inspection of the object or object of the corresponding image. Can be done.

In step S1500, the image analysis device may extract characteristics of the image to be analyzed.

The image analysis apparatus according to an embodiment may extract characteristics of an input image using a general convolutional neural network technique or a pooling technique. The feature of the analysis target image may be a local feature for each region of the image, and the pooling technique may include at least one of a max pooling technique and an average pooling technique.

In step S1510, the image analysis device may generate context information based on the feature extracted in step S1500.

The image analysis apparatus according to an embodiment may generate context information by applying a convolutional neural network technique and/or a pooling technique to features extracted in step S1500. The context information may be a representative value representing all or part of an image to be analyzed. Also, the context information may be global context information of the input image. Further, the pooling technique may be, for example, an average pooling technique.

In step S1520, the image analysis device may analyze the analysis target image based on the feature extracted in step S1500 and the context information generated in step S1510.

For example, the image analysis apparatus may classify the input image by combining local features of each region of the image extracted in step S1500 and the global context reconstructed in step S1510, or find the location of the object of interest included in the input image. have. Accordingly, since information at a specific 2D position in the input image is included from the local information to the global context, more accurate recognition or classification of input images having different local contents but similar local information is possible. Alternatively, it is possible to detect an object containing contextual information that is not related to other contextual information.

The image processing based on the convolutional neural network can be used in various fields. For example, an image processing device for object recognition of an image, an image processing device for image reconstruction, an image processing device for semantic segmentation, and image processing for scene recognition It can be used for devices and the like.

The input image 1610 may be processed through the convolutional neural network 1600 to output a feature map image. The output feature map image can be utilized in various fields described above.

The convolutional neural network 1600 may be processed through a plurality of

layers

1620, 1630, and 1640, and each layer may output multi-channel

feature map images

1625 and 1635. The plurality of

layers

1620, 1630, and 1640 according to an embodiment may extract a feature of an image by applying a filter having a constant size from the upper left to the lower right of the received data. For example, the plurality of

layers

1620, 1630, and 1640 multiply the weights of the upper left NxM pixels of the input data and map them to one neuron in the upper left of the feature map. In this case, the multiplied weight will also be NxM. The NxM may be, for example, 3x3, but is not limited thereto. Subsequently, in the same process, the plurality of

layers

1620, 1630, and 1640 scan input data from left to right and from top to bottom by multiplying the weights by k cells to map to the neurons of the feature map. The k column means a stride to move the filter when performing the convolution, and may be appropriately set to adjust the size of the output data. For example, k may be 1. The NxM weight is called a filter or filter kernel. That is, the process of applying a filter in a plurality of

layers

1620, 1630, and 1640 is a process of performing a convolution operation with the filter kernel, and as a result, the extracted result is a "feature map" or "feature. It is called "map image". In addition, the layer on which the convolution operation is performed may be referred to as a convolutional layer.

The term “multiple-channel feature map” refers to a set of feature maps corresponding to a plurality of channels, and may be, for example, a plurality of image data. It may be an input from an arbitrary layer, or an output according to a result of a feature map operation such as a convolution operation, etc. According to an embodiment, the

multi-channel feature maps

1625 and 1635 are feature extraction layers of the convolutional neural network. Fields" or "convolutional layers", which are created by a plurality of

layers

1620, 1630, 1640. Each layer sequentially receives multi-channel feature maps generated in the previous layer, and then as outputs Multi-channel feature maps may be generated.Lastly, the L (L is an integer) layer 1640 receives the multi-channel feature maps generated by the L-1-th layer (not shown), and the multi-channel features of not shown. You can create maps.

Referring to FIG. 16, feature maps 1625 having K1 channels are outputs according to feature map operation 1620 in layer 1 for input image 1610, and feature map operation 1630 in layer 2 ). In addition, feature maps 1635 having K2 channels are outputs according to feature map operation 1630 in layer 2 for input feature maps 1625, and feature map operation in layer 3 (not shown) It becomes the input for.

Referring to FIG. 16, the multi-channel feature maps 1625 generated in the first layer 1620 include feature maps corresponding to K1 (K1 is an integer) channels. In addition, the multi-channel feature maps 1635 generated in the second layer 1630 include feature maps corresponding to K2 (K2 is an integer) channels. Here, K1 and K2 representing the number of channels may correspond to the number of filter kernels used in the first layer 1620 and the second layer 1630, respectively. That is, the number of multi-channel feature maps generated in the M (M is an integer of 1 or more and L-1 or less) layer may be the same as the number of filter kernels used in the M layer.

17 is a view for explaining an embodiment of a pooling technique.

As illustrated in FIG. 17, the window size of the pooling is 2×2 and the stride is 2, and Max pooling may be applied to the input image 1710 to generate the output image 1790.

In FIG. 17A, a 2x2 window 1710 is applied to the upper left of the input image 1710, and a representative value (here, maximum value 4) among the values in the window 1710 area is calculated to output the image 1790. ) In the corresponding position 1720.

Thereafter, in FIG. 17B, the window is moved by stride, that is, by 2, and a maximum value 3 of the values in the window 1730 area is input to a corresponding position 1740 of the output image 1790.

If the window can no longer be moved to the right, the process is repeated from the position below the stride from the left of the input image. That is, as illustrated in (c) of FIG. 17, the maximum value 5 of the values in the window 1750 area is input to the corresponding position 1760 of the output image 1790.

Thereafter, as shown in FIG. 17D, the window is moved by the stride, and a maximum value 2 of the values in the window 1770 area is input to a corresponding position 1780 of the output image 1790.

The above process may be repeatedly performed until a window is located in the lower right area of the input image 1710, thereby generating an output image 1790 that applies pooling to the input image 1710.

Hereinafter, an embodiment of a method of generating a new composite image and/or virtual article information corresponding thereto by using a plurality of images and/or article information will be described with reference to FIGS. 18 to 22.

Referring to FIG. 18, the image synthesis device 1800 includes an object image extraction unit 1810, an object location information generation unit 1820, an image synthesis unit 1830, and/or an object detection deep learning model learning unit 1840. It can contain. However, this only shows some components necessary to describe the present embodiment, and the components included in the image synthesizing apparatus 1800 are not limited to the above-described examples. For example, two or more components may be implemented in one component, or an operation executed in one component may be divided and implemented to be executed in two or more components. In addition, some components may be omitted or additional components may be added. Or, among the components of the image analysis device 112 of FIG. 1, the image enhancement device 500 of FIG. 5, the image analysis device 1200 of FIG. 12, and the image synthesis device 1800 of FIG. 18, the same function or similar The component performing the function may be implemented as one component.

The image synthesizing apparatus 1800 according to an embodiment of the present disclosure receives a first image including a first object and a second image including a second object, and objects for each of the first image and the second image And a background, the first object and the second object are generated, and the first object and the second object are based on the first object and the second object. 3 images can be generated, and an object detection deep learning model can be trained using location information of a first object, location information of a second object, and a third image.

Referring to FIG. 18, the input image 1850 may include an image including a single object. The description of the input image 1850 is the same as the description of the input image described with reference to FIG. 1 and the like.

The object image extraction unit 1810 may receive an image 1850 including a single object and distinguish the received image into an object and a background. The description of the object image extraction unit 1810 is the same as the description of the object image extraction unit 520 described with reference to FIGS. 5 and 6.

The object location information generation unit 1820 may determine the location of the object extracted from the object image extraction unit 1810. For example, the object location information generating unit 1820 specifies a bounding box surrounding the object area, and generates location information of the object classified by the object image extraction unit 1810 based on the specified square box. can do. The description of the method for generating the location information of the object is the same as the description for the method with reference to FIG. 6.

According to an embodiment of the present disclosure, since location information of an object included in an image may be automatically generated, a hassle of having to manually input location information of an object for each image is avoided by a reader for artificial intelligence learning. Can be.

Referring back to FIG. 18, the image synthesizing unit 1830 uses a plurality of single object images obtained through the object image extraction unit 1810 and the object location information generation unit 1820 to obtain multi-object images. Can generate For example, for the first image including the first object and the second image including the second object, the location information of the first object through the object image extraction unit 1810 and the object location information generation unit 1820, respectively, and The location information of the second object is obtained, and the image synthesis unit 1830 generates a third image including the first object and the second object based on the obtained location information of the first object and the location information of the second object can do. A detailed process of generating a multi-object image will be described in more detail with reference to FIG. 19.

19 is a diagram illustrating a process of generating a multi-object image using two images including a single object according to an embodiment of the present disclosure. The image synthesizing unit 1900 of FIG. 19 is an embodiment of the image synthesizing unit 1830 of FIG. 18. Referring to FIG. 19, the image synthesizing unit 1900 includes the first single object image 1910, the second single object image 1920, and the first single object image (obtained through the object image extraction unit and the object location information generation unit). Using the location information of 1910) and the second single object image 1920, the first single object image 1910 and the second single object image are included in the synthesized multi-object image 1940 and the multi-object image 1940. It is possible to obtain location information 1950 for the objects. Meanwhile, the image synthesizing unit 1900 may also use an image 1930 for a background separated from an object when synthesizing the first single object image 1910 and the second single object image 1920. When synthesizing a plurality of images, the location information of the first single object image 1910 and the location information of the second single object image 1920 may be arbitrarily modified. In addition, image synthesis may be performed based on the corrected location information. By doing so, it is possible to generate a myriad of synthetic images and virtual location information.

As described above, through synthesis, as many synthetic images as necessary for learning and/or virtual location information corresponding thereto may be generated. Therefore, even if the absolute number of images and/or location information necessary for learning is small, it is possible to generate any number of learning data sufficient to train the AI model.

Referring back to FIG. 18, the object detection deep learning model learning unit 1840 may train an object detection deep learning model using location information of a first object, location information of a second object, and a third image. For example, the object detection deep learning model learning unit 1840 may train the convolutional neural network model. The location information of the first object, the location information of the second object, and the third image may be used for training the convolutional neural network model.

20 is a diagram illustrating a process of training a convolutional neural network using a multi-object image according to an embodiment of the present disclosure. The object detection deep learning model learning unit 2000 of FIG. 20 is an embodiment of the object detection deep learning model learning unit 1840 of FIG. 18. Referring to FIG. 20, a multi-object image 2010 synthesized by using single object images and location information of objects may be used as data necessary for learning. The object detection deep learning model learning unit 2000 may train the convolutional neural network 2020 by projecting the location information of each single object with respect to the multi-object image 2010. According to an embodiment, if a plurality of objects are present in an object passing through an X-Ray searcher in an electronic search system, an X-Ray image in which a plurality of objects are overlapped may be obtained. According to the present disclosure, a plurality of images may be obtained. Since the convolutional neural network is trained by using the shape of each object together with the positional information of the object, more accurate detection results can be obtained even if overlapping occurs between objects.

The image synthesizing apparatus 2100 of FIG. 21 is an embodiment of the image synthesizing apparatus 1800 of FIG. 18. The operation of the object image extraction unit 2104, the object location information generation unit 2106, the image synthesis unit 2108, and the object detection deep learning model learning unit 2110 included in the image synthesis device 2100 of FIG. The operations of the object image extraction unit 1810, the object location information generation unit 1820, the image synthesis unit 1830, and the object detection deep learning model learning unit 1840 included in the image synthesis device 1800 of 18 are the same. . Accordingly, the image synthesizing apparatus 2100 includes an object image extraction unit 2104, an object location information generation unit 2106, an image synthesis unit 2108, and an object detection deep learning model learning unit for a plurality of single object images 2102. By performing the operation in 2110, a trained convolutional neural network model can be generated. The object detection device 2120 may detect each object using the convolutional neural network model trained by the image processing device 2100 for the image 2122 including multiple objects in a real environment.

According to an embodiment, when the invention of the present disclosure is applied to an article search system, the image synthesizing apparatus 2100 of the present disclosure may generate a new multi-object-included image based on a single object region extraction in an X-Ray image. Can be. Also, the object detection device 2120 may find an area where multiple objects included in an article passing through the X-Ray searcher exist. Therefore, by automatically extracting the position of the object with respect to the X-Ray image, a reader can perform the image inspection task more easily, and also includes information on the quantity of the extracted object and the object in the object. It can be used for tasks such as comparing computerized information.

In step S2200, the first image including the first object and the second image including the second object may be input to distinguish an object and a background for each of the first image and the second image. For example, by comparing the pixel value of the input image with a predetermined threshold, the pixel value is binarized, and the objects included in the input image can be distinguished by grouping the binarized pixel values.

In step S2210, location information of the separated first object and the second object may be generated. For example, a rectangular box surrounding the object area may be specified, and based on the specified rectangular box, location information of the object classified in step S2200 may be generated.

In step S2220, based on the location information of the first object and the location information of the second object, a third image including the first object and the second object may be generated. For example, a third image including the first object and the second object may be generated based on the position information of the first object and the position information of the second object obtained in step S2210.

In step S2230, the object detection deep learning model may be trained using the location information of the first object, the location information of the second object, and the third image. For example, the convolutional neural network model may be trained, and the location information of the first object and the location information of the second object generated in step S2210 and the third image generated in the step S2220 may be used for training the convolutional neural network model. Can be.

In the embodiment described with reference to FIGS. 18 to 22, an example of receiving an image including a single object and separating the object from the background has been described. However, the present invention is not limited thereto, and the input image may be an image including two or more objects. In this case, two or more objects and a background can be distinguished from the input image, and location information can be generated and used for each of the two or more objects.

In addition, in the above-described embodiment, it has been described as generating a third image based on two single object images and location information of each object. However, the present invention is not limited thereto, and a third image may be generated using two or more single object images and location information of each object. That is, the image processing method and apparatus according to the present disclosure may generate a third image based on two or more images each including one or more objects and location information of each object.

Hereinafter, a method of detecting a liquid-like substance and a method of configuring a database for detecting the liquid-like substance will be described with reference to FIGS. 23 to 30.

In the conventional article retrieval system, the detection of liquid-like materials was mostly performed depending on the experience of an experienced reader. Objects with a predetermined shape can be detected by reading the characteristics of the object, but the liquid substance has no shape, and the density of the liquid to be detected varies depending on the container it contains, making it easy to detect according to a consistent algorithm. Because I did not.

In the field of customs clearance, there are frequent cases where dangerous substances or prohibited substances are transferred to other containers and transported, and thus, despite the above-mentioned problems, the need to detect liquid substances has been emerging.

The present invention has been proposed to solve the detection of liquid substances that rely on reading by highly skilled readers, and according to the present invention, an inexperienced reader also helps with an image analysis device to which the method for detecting liquid substances is applied. It can perform quick and accurate detection of liquid substances.

According to FIG. 23, a method for detecting a liquid substance according to an embodiment of the present disclosure may be performed by the article search system 100 or the image analysis device 112. Hereinafter, for convenience, it will be described that the image analysis device 112 performs a liquid substance detection method. However, in a broad sense, the method for detecting liquid substances according to the present disclosure may also be interpreted as being performed by the article retrieval system 100.

A method for detecting a liquid substance according to an embodiment of the present disclosure includes receiving an image to be analyzed from an X-ray apparatus (S2300), detecting a container region in the analyzed image (S2310), and liquid in the detected vessel region Step (S2320) of detecting a region, step of extracting features of the detected liquid region (S2330), and/or comparing features of the extracted liquid region with feature information included in the liquid flow database to analyze the image to be analyzed. It may include the step (S2340) of determining what the liquid-like substance is included.

In step S2300, the image analysis apparatus may receive an image to be analyzed from the X-ray apparatus. Here, the analysis target image may mean an image or a photograph containing a liquid substance to be detected. The liquid in the present invention may freely flow, such as water or oil, to change its shape according to the shape of the container, does not have a constant shape, and may mean a material whose volume change is not large even when compressed. In addition, the container in the present invention may mean an object used to contain a liquid material. The method for detecting a liquid substance according to the present disclosure may mean a method for detecting a liquid substance contained in a container.

In step S2310, the image analysis device may detect the container region in the image to be analyzed. Here, the container region may refer to a cross-sectional space on an image generated by the closed space of the container. The container area may be different depending on the inclination of the container or the angle of rotation during X-ray imaging. In addition to the detection of the container area, the image analysis device may generate several candidate lists for the type and/or volume of the container using a pre-trained deep learning based model. Furthermore, the image analysis apparatus may derive probability information related to which of the candidates included in the candidate list is the type and/or volume of containers in the image.

In step S2320, the image analysis device may detect a liquid region in the container region detected in step S2310. Here, the liquid region may mean a region in which liquid is contained in an enclosed space created by the container region. The liquid region on the X-ray image may be shown by the container, in a form included in the container area.

In step S2330, the image analysis device may extract characteristics of the liquid region detected in step S2320. Here, the feature may include color information, color pattern information, brightness information, brightness pattern information, and/or interference information of a liquid region appearing through an image, but is not limited thereto. As another example, the feature for the liquid region may mean a feature vector represented by the container and/or liquid on the image. As another example, the feature vector may mean a vector parameterizing features for the liquid region.

As another example, the feature vector may be derived through features of a liquid region on an image to be analyzed. For example, the feature vector may be derived using at least one of color information of a liquid region, color pattern information, brightness information, brightness pattern information, and distribution information of a liquid component in a container region.

In order to extract the feature vector from the liquid region on the analysis target image, an average or variance value of color or brightness in the analysis target image may be used. In addition, the image analysis apparatus may extract a feature vector from the liquid region using at least one of a Scale Invariant Feature Transform (SIFT) or a Speeded Up Robust Features (SURF) algorithm.

As another example, the image analysis apparatus is provided with an output vector of a convolutional neural network (CNN) using the context of a liquid region, and an optimal output vector through a dimensional change algorithm of a vector such as principal component analysis. It can be converted to an output vector with complexity.

The above-described algorithm is an example of an algorithm used to derive the feature vectors of the present disclosure, and does not limit the scope of the present invention. The image analysis apparatus may extract or derive a feature vector from the liquid region by using various algorithms that analyze or extract the features of the image.

Here, SIFT may mean an algorithm for extracting feature points that are invariant to the size and rotation of an image. In addition, SURF may refer to an algorithm for finding feature points that are invariant to environmental changes by considering environmental changes such as scale and lighting point of view from multiple images. Also, PCA may refer to an algorithm for converting a high-dimensional feature vector to a low-dimensional feature vector.

In step S2340, the image analysis device may compare the feature extracted in step S2330 with feature information included in the liquid flow database, and determine what liquid flow material is included in the analysis target image.

As an example, the image analysis apparatus may perform similarity comparison between a feature vector included in the liquid flow database and a feature vector derived using features extracted through the liquid region. The image analysis device may determine the type of the liquid substance included in the image to be analyzed according to the similarity comparison result. Furthermore, the image analysis apparatus may generate a list of several candidates for the type and/or volume of liquid substances in the image. Furthermore, the image analysis device may derive probability information related to which of the candidate types included in the candidate list is the type and/or volume of the liquid substance in the image.

Referring to Fig. 24, a description will be given of a method of constructing a video analysis apparatus configuring a database that can be used in step S2340 described above. The method for constructing liquid substance data according to an embodiment of the present disclosure may be performed by the article retrieval system 100, the image analysis device 112, or the learning unit 120. Hereinafter, for convenience, it will be described that the image analysis device 112 constructs a liquid substance database. However, in a broad sense, the method for detecting liquid substances according to the present disclosure may also be interpreted as being performed by the article search system 100 or the learning unit 120.

According to an embodiment of the present disclosure, a method of constructing a liquid flow database includes receiving a read image containing a liquid flow substance (S2400), arranging and/or interpolating the read image (S2410), and a liquid flow substance image. It may include the step of extracting the characteristics of (S2430) and / or adding the characteristics of the extracted liquid material image to the liquid database.

In step S2400, the image analysis device may receive a read image for adding a liquid flow database. As an example, the read image may be input by a reader or administrator who wishes to build a liquid database. Here, the read image may include a liquid material. Since the read image is an image used to add a database, unlike the image to be analyzed previously described, it may be configured to include only liquid materials for building a database.

The liquid-like substance included in the readout image may be an additional target liquid-like substance. In the present disclosure, the liquid substance to be added may indicate a liquid substance, which is used to add features of the liquid substance substance image to the liquid substance database. That is, in the present disclosure, the additional liquid-like substance may refer to the liquid-like substance included in the readout image.

The image analysis apparatus can receive and data the container information and the liquid substance contained in the read image together with the read image. As an example, the information of the container and the information of the liquid substance may be input by a reading source or a manager who inputs the read image. It may contain at least one. Further, the liquid information may include at least one of a liquid type, a liquid characteristic, a liquid density, a liquid viscosity, and a liquid amount. The above-described container information and liquid information are examples, and various information capable of representing the container and the liquid may be included in the container information or the liquid information.

Since the X-ray image of the liquid substance varies depending on the type of container containing the liquid, the amount of liquid contained in the container, and the degree of inclination or rotation of the container, the image analysis device is the information of the container and the type of liquid. It is necessary to establish a liquid flow database that can be determined according to. As an example, the liquid flow database according to the present disclosure may be in a form in which information on a container and liquid information on a specific liquid flow material included in a read image are parameterized.

In step S2410, the image analysis device may align and/or interpolate the read image. Here, the alignment of the read image may refer to a process of aligning the liquid materials that are arranged or rotated at various angles at a reference angle. Interpolation of the read image may mean a process of obtaining an image for each angle of the liquid substance by rotating the aligned liquid substance in a certain angle unit.

Since the liquid material on the X-ray image is shown differently depending on the angle at which the container is placed, the image analysis device needs to align the read image in a form that can be used as a basis for building a database.

The image analysis apparatus may obtain an aligned liquid substance image 2530 by rotating the liquid substance 2520 on the read image. For example, alignment of the read image may be performed by the image alignment unit 2500. For example, the image alignment unit 2500 may align the liquid material on the read image by rotating the lid portion of the container facing upward.

The image analysis apparatus rotates the liquid substance material image 2530 aligned by the image aligning unit 2500 at a predetermined angle interval to obtain the liquid substance substance images 2540 shown at various angles. The image analysis apparatus may obtain a database according to the rotation of the liquid substance by using various rotated liquid substance images 2540. For example, interpolation of the aligned liquid material image 2530 may be performed by the interpolation unit 2510.

Since the rotation of the image does not change the characteristics of the container and the characteristics of the liquid, the image analysis apparatus can acquire a liquid flow database arranged at various angles under the same conditions according to the interpolation method described above.

In step S2420, the image analysis device may extract characteristics of the liquid material image that can correspond to the container information and the liquid information. The feature extraction in step S2420 may correspond to the operations of the feature extraction unit 1210, the context generation unit 1220, and the feature and/or context analysis unit 1230 described above. The image analysis device may extract characteristics of the liquid substance on the read image, generate context information based on the extracted features, and data information about the read image based on the extracted features and the generated context information. . Here, the feature may mean a local feature for each region of the image. Details of the feature extraction of the liquid material are as described in FIG. 12 and will be omitted.

In step S2430, the image analysis device may add a feature of the liquid material image extracted in S2420 to the liquid flow database.

26 is a view for explaining a result of building a liquid flow database according to an embodiment of the present disclosure.

Since the X-ray image of the liquid substance changes greatly depending on the characteristics of the container containing the liquid and the characteristics of the liquid, when the image analysis device database features the characteristics of the liquid substance substance image, the aforementioned container information and liquid information Can be considered simultaneously.

For example, the feature database of the liquid substance image according to the present disclosure may be constructed based on the characteristics of the container. For example, the database can be categorized by material, shape, and characteristics of the container. For example, when the container is a glass bottle, the characteristics of the image according to the capacity of the glass bottle, the characteristics of the image according to the thickness of the glass bottle, the characteristics of the image according to the type of liquid contained, and the characteristics of the image according to the amount of liquid contained therein Each can be added to a database.

As another example, the image analysis device may parameterize the characteristics of the extracted liquid substance image into a feature vector. In this case, the feature database of the liquid substance image may be a feature vector database of the liquid substance image.

26 shows an example of clustering various feature vectors and feature vectors included in the liquid flow database into a plurality of

clusters

2600, 2610, 2620, and 2630. The x-axis and the y-axis are respectively indicated by length and weight, but this is only an example of the present disclosure, and such description does not limit the feature vector parameter of the present invention. The feature vector can be expressed by various types of parameters for representing the characteristics of the liquid substance image. Each feature vector included in the liquid flow database can be clustered by comparing similarities between feature vectors. Here, the clusters may be used to determine the type of the liquid substance using feature vectors extracted from the analysis target image.

The database generated according to the method for constructing a liquid flow database described with reference to FIG. 24 may be stored in the database 122 included in the learning unit 120 or may be stored in a separate database existing inside the image analysis device. have. The base produced here can be used for the detection of liquid substances described in FIG. 23.

Referring to Fig. 27, a description will be given of a method in which an image analysis device provides a detection result to a reader. The method for detecting a liquid substance according to an embodiment of the present disclosure includes receiving an image to be analyzed (S2700), pre-processing an image to be analyzed (S2710), and detecting a container region in the image to be analyzed (S2720). , Extracting the liquid region through post-treatment for the container region (S2730), extracting the feature vector for the liquid region (S2740), creating a liquid flow database (S2750), and extracting the feature vector It may include the step of determining what the liquid substance of the image to be analyzed is using the liquid flow database (S2760) and/or providing the detection result to the reader (S2770).

Step S2700, step S2720, step S2740, and step S2760 may correspond to steps S2300, S2310, S2330, and S2340 of the image analysis apparatus described in FIG. 23, respectively. In addition, step S2750 may correspond to steps S2400 to S2430 of FIG. 24.

In step S2710, the image analysis device may perform pre-processing of the input image. The pre-processing of the analysis image may mean a process of removing interference on the entire analysis image or a process of setting color information of the X-ray input image as a preset for detection of the image analysis apparatus.

In step S2720, the image analysis device may detect the container region 2830 included in the analysis image 2810. The image analysis apparatus may utilize a convolutional neural network (CNN) technique, a detection algorithm through serial configuration (adaboost), and the like to detect the container region in the analysis image.

The convolutional neural network of the present disclosure may be used to extract features such as a border, thickness, line color, and region color of a container region from an analysis target image, and may include a plurality of layers. Each layer can receive input data and process input data of the corresponding layer to generate output data. As the hierarchy increases, neural networks can extract more specific features.

For example, the image analysis apparatus may detect the container region in the image to be analyzed using the convolutional neural network described in FIG. 16.

28 shows an example in which a bag of a passenger is included in an image to be analyzed. The image analysis apparatus may receive an analysis target image 2810 in which a passenger's bag is photographed from the X-ray apparatus. In one example, step S2720 may be performed by the liquid container detector 2800. The liquid container detection unit may detect an object having the same shape as the container in the analysis target image 2810 as the container area 2830. The container detection unit 2800 may detect the container region 2830 in the analysis image using the above-described convolutional neural network or a pre-trained deep learning-based model.

As another example, the liquid container detector 2800 may detect the container area 2830 and the area 2820 including some background 2831 around the container area, as shown in FIG. 28. Here, the region 2820 including some background may mean a rectangular box used for object identification.

In step S2730, the image analysis device may perform detection of the liquid region 2930 through post-processing of the detected

container region

2920 or 2940. Post-processing the container area may include removing some background or background interference 2942 around the container area and/or removing internal interference 2950 to the container area.

When the upper and lower portions overlap by an image of a liquid substance shown in the analysis image input through the X-ray or another object, interference 2950 on the image may be illustrated. In particular, since the liquid material has a large color change on the image due to the density or occlusion of the liquid, interference is likely to occur compared to an object having a predetermined shape.

FIG. 29 shows an area 2920 in which some background or background interference 2942 is included in the container area 2940 detected by the liquid container detection unit 2800. The image analysis apparatus may remove some background or background interference 2942 excluding the container region 2940 in the region 2920 including some background or background interference 2942. For example, step S2730 may be performed by the background interference canceling unit 2900 and/or the container internal interference canceling unit 2910.

The background interference removing unit 2700 may remove some background or background interference 2701 from the detected container region 2920. The removal of some background or background interference 2701 may be achieved through the above-described convolutional neural network or a pre-trained deep learning based model.

29 illustrates a situation in which an internal interference 2950 exists in a portion of the container region 2930 due to the ID Tag object included in the passenger's bag. Here, the internal interference may include a region caused by the thickness of the container or a region caused by other objects disposed on the upper and lower ends of the liquid material. Alternatively, the internal interference may include interference caused by fluidity or image interference generated during X-ray imaging.

The internal interference removing unit 2910 may remove internal interference 2950 included in the container region 2930 from which some background or background interference 2942 is removed. The final liquid region 2930 may be detected by removing the internal interference of the container internal interference removal unit 2910.

In order to remove the internal interference, the internal interference canceling unit 2910 may utilize an interference cancellation method in consideration of a video frequency analysis method or a context for arranging surrounding items. For example, the interference removal unit 2910 inside the container may extract the interference-removed liquid region 2930 using KR application 10-2017-0183857 "Image analysis apparatus and method based on image characteristics and context". .

In step S2740, the image analysis device may extract a feature vector for the liquid region detected in step S2730. As described above, the feature vector may mean a vector parameterizing characteristics of the extracted liquid region. The image analysis apparatus may extract a feature vector represented by a predetermined parameter using features of the liquid region.

In step S2760, the image analysis device compares the similarity between the extracted feature vector and the feature vector included in the liquid flow database, and compares at least one of the type of the liquid substance, the type of the container, and/or the probability included in the analysis target image. Can decide.

For example, the image analysis apparatus determines which cluster among the plurality of

clusters

2800, 2810, 2820, and 2830 in which the extracted feature vector is illustrated in FIG. 28 or performs clustering on the extracted feature vector can do. Here, the cluster in which the feature vector should be included can be determined by comparing the similarity between the representative feature vector of the cluster and the extracted feature vector. As another example, the image analysis apparatus may derive a probability value related to which cluster the extracted feature vector is included.

For example, each of the first cluster 2800, the second cluster 2810, the third cluster 2820, and the fourth cluster 2830 of FIG. 28 is placed in a water (2800) and a 1.5L plastic bottle in a 500 ml plastic bottle. It may be a cluster including a cola 2810 in a glass, alcohol 2820 in a 500 ml glass bottle, and water 2830 in a 2 L plastic bottle. The data shown in each cluster may be characteristic vector data obtained in a database creation process.

As an example, in connection with FIG. 28, when it is determined that the feature vector of the liquid substance material image included in the image to be analyzed is included in the first cluster, the image analysis apparatus performs liquid flow included in the image to be analyzed It can be determined that the substance is water in a 500 ml plastic bottle.

As another example, the feature vector of the liquid substance image included in the image to be analyzed is included in the first cluster 2800 with a probability of 25% and in the third cluster 2820 with a probability of 70%. If it is determined that the image analysis apparatus, the image analysis device may determine that the liquid substance contained in the image to be analyzed is water contained in a 500 ml plastic bottle with a probability of 25%, and alcohol contained in a 500 ml glass bottle with a probability of 70%.

In step S2770, the image analysis device may provide the detection result to the reader. The image analysis device may provide the detection result of the detection of the liquid substance through the output device 114.

Here, the read result may include the presence or absence information of liquid substances, container information, liquid information, similar data information, remodeling inspection necessity information, risk information, etc., but the above-described read result is an example and according to the present disclosure. The read result is not limited to this. The image analysis device may provide the reader with a variety of information related to the liquid-like substance conducive to the reader.

For example, the image analysis device may provide the reader with container information as a candidate list for the container and a probability that the container shown in the analysis target image corresponds to each candidate. Here, the candidate for the container may be expressed by the type of container and the capacity of the container. For example, the image analysis device reads information that the container included in the image to be analyzed is a 200 ml glass bottle with a probability of 90%, a 180 ml glass bottle with a probability of 5%, and a 200 ml plastic bottle with a probability of 2%. Can be provided.

For example, the image analysis apparatus may provide the reader with liquid information as a candidate list for a liquid type and a probability that the liquid type shown in the analysis target image corresponds to each candidate. For example, the image analysis device may provide the reader with information that the liquid contained in the image to be analyzed is water with a probability of 60% and alcohol with a probability of 38%.

The similar data information may mean information related to data having a feature vector most similar to a feature vector for a liquid substance included in an image to be analyzed. For example, the image analysis apparatus may provide an image source for a liquid flow material having a feature vector most similar to the liquid flow material included in the current analysis target image.

The information on whether a remodeling inspection is necessary may be information on whether a remodeling inspection of a reading source for the analysis target image is necessary. That is, the image analysis device may provide the reader with information related to whether remodeling inspection is required for the liquid substance included in the image to be analyzed. Determination of whether to perform remodeling analysis of the image analysis apparatus is the same as described above, so a description thereof will be omitted.

The risk information may refer to the risk of the liquid substances included in the analysis target image derived based on container information and/or liquid information. For example, risk information can be provided to the reader in the form of a number.

Referring to FIG. 30, the image analysis device 3000 may include an input unit 3010 and/or an image analysis unit 3020. However, this only shows some components necessary to describe the present embodiment, and the components included in the image analysis apparatus 3000 are not limited to the above-described examples. For example, two or more components may be implemented in one component, or an operation executed in one component may be divided and implemented to be executed in two or more components. In addition, some components may be omitted or additional components may be added. In addition, among the components of the image analysis device 112 of FIG. 1, the image enhancement device 500 of FIG. 5, the image analysis device 1200 of FIG. 12, and the image synthesis device 1800 of FIG. 18, the same function or similar The component performing the function may be implemented as one component.

The input unit 3000 provides an image to be analyzed 3030 to an analysis target image provided from an X-ray reading device and an input read image, container information, and/or liquid information to build a database. Here, the input unit 3010 may be defined as a receiving unit.

The image analysis unit 3020 may perform a series of operations for detecting a liquid substance in the analysis target image. The image analysis unit 3020 may detect a liquid substance included in the analysis target image using the liquid flow database 3040 or the liquid flow database provided from the learning unit 120. In addition, the image analysis unit 3020 may provide the detection result to the reader through the output device 3030.

As another example, although not illustrated in FIG. 30, the image analysis unit 3020 uses the readout image, container information, and/or liquid information to characterize the characteristics of the extracted additional liquid substance material image in the liquid flow database 3040. It may include a database addition to add to. Here, the liquid flow database 3040 may mean the learning unit 120 of FIG. 1 or the database 122 included in the learning unit. The liquid flow database 3040 disclosed by FIG. 30 is illustrated as being configured as a separate device from the image analysis device 3000, but the scope of rights of the present disclosure is not limited thereto. A database having the same function as the liquid flow database 3040 may be configured as part of the image analysis apparatus 3000.

The image analysis unit 3020 includes a pre-processing unit 3021, a liquid region detection unit 3022, a liquid container detection unit 3023, a feature extraction unit 3024, a post-processing unit 3025, and/or a feature comparison unit 3026. can do.

In addition, although not illustrated in FIG. 30, the image analysis device may further include an image alignment unit, an interpolation unit, and an interference removal unit. Here, the interference canceling unit may include a background interference canceling unit and/or an internal interference canceling unit.

The detailed operation of each component included in the image analysis apparatus 3000 is the same as that described with reference to FIGS. 23 to 29, and thus description thereof will be omitted.

The output device 3030 may display information processed by the image analysis device 3000. For example, the output device 3030 may display execution screen information of an application program driven by the image analysis device 3000, or a user interface or GUI (Graphic User Interface) information according to the execution screen information.

The output device 3030 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible) display), a three-dimensional display (3D display), an electronic ink display (e-ink display).

The output device 3030 disclosed by FIG. 30 is illustrated as being configured as a separate device from the image analysis device 3000, but the scope of rights of the present disclosure is not limited thereto. The output unit having the same function as the output device may be configured as a part of the image analysis device 3000.

Exemplary methods of the present disclosure are expressed as a series of operations for clarity of description, but are not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. To implement the method according to the present disclosure, the steps illustrated may include other steps in addition, other steps may be included in addition to the other steps, or additional other steps may be included in addition to some of the steps.

The various embodiments of the present disclosure are not intended to list all possible combinations, but are intended to describe representative aspects of the present disclosure, and the details described in the various embodiments may be applied independently or in combination of two or more.

Further, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), Universal It can be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor.

The scope of the present disclosure includes software or machine-executable instructions (eg, operating systems, applications, firmware, programs, etc.) that cause actions according to the methods of various embodiments to be executed on a device or computer, and such software or Instructions include a non-transitory computer-readable medium that is stored and executable on a device or computer.

The present invention can be utilized in the field of analyzing images or images.

Claims

Receiving an analysis target image;

Detecting a container region of a liquid substance in the analysis target image using a previously learned deep learning-based model;

Detecting a liquid region in the container region;

Extracting a feature vector for the liquid region; And

And analyzing the liquid material by comparing the feature vector and the feature vector included in the database.
According to claim 1,

The feature vector is determined based on at least one of color information, color pattern information, brightness information, brightness pattern information, and interference information of the liquid region.
According to claim 1,

The step of detecting the liquid region is

Removing the interference to the container area,

The interference with the container area includes at least one of background interference and internal interference of the container area.
According to claim 1,

The step of analyzing the liquid material,

Determining at least one of the type of the container containing the liquid-like substance and the type of the liquid-like substance, by using the similarity between the extracted feature vector and the feature vector included in the database. .
According to claim 1,

The step of analyzing the liquid material,

And comparing the extracted feature vector with the feature vector included in the database to measure the amount of the liquid substance.
According to claim 1,

Further comprising the step of outputting the analysis results for the liquid material,

The analysis result includes at least one of the type of the container containing the liquid-like material, the volume of the container, and the type of the liquid-like material.
According to claim 1,

The step of analyzing the liquid material,

And determining a probability that the container containing the liquid-like substance corresponds to a specific container included in the database.
The method of claim 7,

Further comprising the step of outputting the analysis results for the liquid material,

The analysis result includes information related to the probability that the container containing the liquid substance corresponds to the specific container.
According to claim 1,

The step of analyzing the liquid material,

And determining a probability that the type of the liquid substance corresponds to a specific liquid substance contained in the database.
The method of claim 9,

Further comprising the step of outputting the analysis results for the liquid material,

The analysis result includes information related to the probability that the type of the liquid substance corresponds to the specific substance.
According to claim 1,

The method further includes adding a feature vector to the database,

The step of adding the feature vector,

Receiving a readout image containing an additional target liquid substance;

Receiving information of a container containing the liquid substance to be added and information of the liquid substance to be added;

Extracting a feature vector of the image of the liquid substance to be added; And

And adding a feature vector of the image of the liquid substance to be added to the liquid flow database.
The method of claim 11,

The step of adding the feature vector,

And aligning the read image according to a predetermined angle.
The method of claim 11,

The step of adding the feature vector,

And rotating the read image according to a predetermined angular interval to obtain an image for each angle of the read image.
A receiver configured to receive an analysis target image; And

Including the image analysis unit to detect the liquid material by analyzing the analysis target image,

The image analysis unit,

A liquid container detector configured to detect a container region of the liquid-like substance in the analysis target image using a previously learned deep learning-based model;

A liquid region detection unit for detecting a liquid region in the container region;

A feature extraction unit for extracting a feature vector for the liquid region; And

And a feature comparison unit that analyzes the liquid material by comparing the feature vector and the feature vector included in the database.
The method of claim 14,

The feature vector is determined based on at least one of color information, color pattern information, brightness information, and interference information of the liquid region.
The method of claim 14,

The receiving unit receives at least one of a readout image including the liquid substance to be added, information on a container containing the liquid substance to be added, and information on the liquid substance to be added,

The feature extraction unit extracts a feature vector of the image of the liquid substance to be added,

The image analysis unit, the image analysis apparatus further comprising a database addition unit for adding a feature vector of the image of the liquid substance to be added to the database.
Receiving an analysis target image;

Detecting a container region of a liquid substance in the analysis target image using a previously learned deep learning-based model;

Detecting a liquid region in the container region;

Extracting a feature vector for the liquid region; And

A computer-readable recording medium recording a program for performing the step of analyzing the liquid-like substance by comparing the feature vector and the feature vector included in the database.