KR102583675B1 - Method and system for classifying image - Google Patents

Abstract

An image classification system according to an aspect according to the technical idea of the present disclosure includes an image characteristic estimation module for estimating three-dimensional characteristics of an input image, and an image classification system based on the three-dimensional characteristics estimated by the image characteristic estimation module. and an image classification module that classifies, wherein the image characteristic estimation module is trained to estimate the three-dimensional characteristics by reflecting three-dimensional characteristics of the input image and color changes between pixels in the input image. Includes a first network based on deep learning.

Description

Image classification method and system {METHOD AND SYSTEM FOR CLASSIFYING IMAGE}

The technical idea of the present disclosure relates to an image classification method and system, and more specifically, to a method and system that can estimate three-dimensional characteristics of a two-dimensional image and classify the image based on the estimated characteristics. will be.

Computer vision is a field of artificial intelligence and refers to a field of research that reproduces general human visual recognition abilities using computers. In order for a device such as a computer to understand the content of an image, image classification must be applied to extract meaning from the image using the computer vision and machine learning algorithms described above.

A typical image represents an object in a two-dimensional form, but most real objects have a three-dimensional shape, so there is a limitation that it is difficult to accurately classify the image using only two-dimensional information. Therefore, in order to improve accuracy when classifying images, attempts are increasing to estimate the 3D shape of an object from a 2D image and then classify the image using the estimated result.

One problem that the present invention seeks to solve is to provide a method to improve accuracy when classifying images by estimating three-dimensional characteristics of images.

In order to achieve the above object, an image classification system according to an aspect according to the technical idea of the present disclosure includes an image characteristic estimation module for estimating three-dimensional characteristics of an input image, and the image characteristic estimation module. and an image classification module that classifies the image based on three-dimensional characteristics estimated by, wherein the image characteristic estimation module classifies three-dimensional characteristics of the input image and color changes between pixels in the input image. It includes a deep learning-based first network learned to estimate the three-dimensional characteristics by reflecting .

According to one embodiment, the image characteristic estimation module estimates the positions of a plurality of 3D points corresponding to the 3D characteristics of the image, and outputs values corresponding to the estimated positions as an estimation result of the 3D characteristics. You can.

According to one embodiment, the image classification system estimates a mesh representing 3D characteristics of the image, and the plurality of 3D points may correspond to a plurality of vertices included in the mesh.

According to one embodiment, the image classification system further includes a learning module for learning the first network, and the learning module provides a rendering result using the estimated 3D characteristics and 3D characteristics of the input image. The first network can be updated based on the difference between the correct answer data corresponding to .

According to one embodiment, the learning module may update the first network based on color differences between corresponding pixels between the image reconstructed based on the estimated 3D characteristics and the input image.

According to one embodiment, the image classification module may further include a deep learning-based second network learned to classify the image based on values corresponding to the positions of the plurality of 3D points.

According to one embodiment, the image classification module includes a deep learning-based second network learned to classify the input image based on the depth image and the input image rendered based on the plurality of 3D points. More may be included.

According to one embodiment, the image classification system is implemented with at least one computing device, and each of the at least one computing device may include a processor and memory.

An image classification method according to an aspect according to the technical spirit of the present disclosure includes estimating three-dimensional characteristics of an input image, and classifying the input image based on the estimated three-dimensional characteristics, The step of estimating the 3D characteristics includes a deep learning-based first network learned to estimate the 3D characteristics by reflecting the 3D characteristics of the input image and color changes between pixels in the input image. It includes the step of estimating three-dimensional characteristics of the input image using .

According to an embodiment of the present disclosure, images can be more accurately classified through 3D characteristics estimated from 2D images using a deep learning-based learned network.

In addition, according to an embodiment of the present disclosure, the estimation accuracy of 3D characteristics can be maximized by training the network to consider color changes of pixels in a 2D image when estimating 3D characteristics.

The effects according to the technical idea of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

In order to more fully understand the drawings cited in this disclosure, a brief description of each drawing is provided.
1 is a diagram schematically illustrating an image classification operation using an image classification system according to an exemplary embodiment of the present disclosure.
FIG. 2 is a diagram illustrating a schematic configuration of a convolutional neural network as an example of a network included in each of the image characteristic estimation module and the image classification module of the image classification system shown in FIG. 1.
Figure 3 is a flowchart for explaining an image classification method according to an exemplary embodiment of the present disclosure.
FIG. 4 is a diagram for explaining a mesh-based 3D characteristic estimation operation according to an exemplary embodiment of the present disclosure.
Figure 5 is a diagram showing examples of an operation for classifying an image based on estimated 3D characteristics.
FIG. 6 is a flowchart illustrating an operation in which a learning module of an image classification system performs learning on a network of an image feature estimation module according to an exemplary embodiment of the present disclosure.
Figures 7 and 8 are exemplary diagrams related to the learning operation of the learning module shown in Figure 6.
Figure 9 is a schematic block diagram of a device performing an image classification method according to an exemplary embodiment of the present disclosure.

Illustrative embodiments according to the technical idea of the present disclosure are provided to more completely explain the technical idea of the present disclosure to those skilled in the art, and the examples below can be modified into various other forms. may be, and the scope of the technical idea of the present disclosure is not limited to the examples below. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to completely convey the technical idea of the present invention to those skilled in the art.

Although the terms first, second, etc. are used in this disclosure to describe various members, regions, layers, portions, and/or components, these members, parts, regions, layers, portions, and/or components are referred to by these terms. It is obvious that it should not be limited by . These terms do not imply any particular order, superiority, inferiority, or superiority or inferiority, and are used only to distinguish one member, region, region, or component from another member, region, region, or component. Accordingly, the first member, region, portion, or component to be described in detail below may refer to the second member, region, portion, or component without departing from the teachings of the technical idea of the present disclosure. For example, a first component may be referred to as a second component without departing from the scope of the present disclosure, and similarly, the second component may also be referred to as a first component.

Unless otherwise defined, all terms used herein, including technical terms and scientific terms, have the same meaning as commonly understood by those skilled in the art in the technical field to which the concept of the present disclosure pertains. Additionally, commonly used terms, as defined in dictionaries, should be interpreted to have meanings consistent with what they mean in the context of the relevant technology, and should not be used in an overly formal sense unless explicitly defined herein. It should not be interpreted.

In cases where an embodiment can be implemented differently, a specific process sequence may be performed differently from the described sequence. For example, two processes described in succession may be performed substantially at the same time, or may be performed in an order opposite to the order in which they are described.

In the accompanying drawings, variations of the depicted shape may be expected, depending, for example, on manufacturing techniques and/or tolerances. Accordingly, embodiments based on the technical spirit of the present disclosure should not be construed as being limited to the specific shape of the area shown in the present disclosure, but should include, for example, changes in shape that occur during the manufacturing process. The same reference numerals are used for the same components in the drawings, and duplicate descriptions thereof are omitted.

As used herein, the term 'and/or' includes each and every combination of one or more of the mentioned elements.

Hereinafter, embodiments based on the technical idea of the present disclosure will be described in detail with reference to the attached drawings.

1 is a diagram schematically illustrating an image classification operation using an image classification system according to an exemplary embodiment of the present disclosure. FIG. 2 is a diagram illustrating a schematic configuration of a convolutional neural network as an example of a network included in each of the image characteristic estimation module and the image classification module of the image classification system shown in FIG. 1.

Referring to FIG. 1, the image classification system 100 according to an embodiment of the present disclosure classifies the image by recognizing or identifying the specific object included in the input image when an image for a specific object is input. The action can be performed.

This image classification system 100 may include at least one computing device. For example, each of the at least one computing device corresponds to a hardware-based device including a processor, memory, communication interface, input unit, and/or output unit. In this case, components (modules) included in the image classification system 100 may be implemented as hardware, software, or a combination thereof, and may be implemented by being integrated or divided into the at least one computing device.

The image classification system 100 according to an embodiment of the present disclosure may include an image characteristic estimation module 110, an image classification module 120, and a learning module 130, but is not limited thereto and has many more components. It may also include .

The image characteristic estimation module 110 can estimate three-dimensional characteristics of the input unknown. Generally, the input image contains only two-dimensional information, and the image characteristic estimation module 110 estimates three-dimensional characteristics of the input image by estimating depth information from the input image. You can. The image characteristic estimation module 110 according to an embodiment of the present disclosure includes a network (artificial neural network) learned based on deep learning, and can estimate three-dimensional characteristics from an input image using the network. For example, when an image of a 'small intestine' is input, the image characteristic estimation module 110 may provide information on a plurality of 3D points representing 3D characteristics of the input image. The plurality of 3D points will be described in more detail later with reference to FIG. 4.

The image classification module 120 may perform a classification operation on the input image based on the 3D characteristics estimated by the image characteristic estimation module 110. The image classification module 120 also includes a network learned based on deep learning, and can provide image classification results from the 3D characteristic information using the network. For example, the network outputs a probability value for each item that can be provided as a classification result, and the image classification module 120 may be implemented to output the item with the highest probability value as the image classification result. For example, when 3D characteristic information for 'small intestine' is input, the image classification module 120 may classify the image as an image for 'small intestine' based on the input information.

The network included in the image characteristic estimation module 110 and the image classification module 120 may be implemented as a convolutional neural network (CNN), but is not limited to this and may be implemented as various types of known neural networks. You can. Referring to FIG. 2, the CNN may be composed of one or more convolutional layers, a pooling layer, and a fully connected layer. The CNN repeats the convolution and pooling process for two-dimensional data such as images, acquires features from the data (obtains an output feature map), and uses the extracted features. The results can be output by performing a classification process.

Continuing to refer to FIG. 1 , the learning module 130 may perform a learning operation of the network included in each of the image characteristic estimation module 110 and the image classification module 120. The learning module 130 may include an objective function for learning each of the networks.

In relation to learning of the network included in the image characteristic estimation module 110, the learning module 130 renders and/or reconstructs the image based on the three-dimensional characteristics estimated by the image characteristic estimation module 110, and renders And/or learning (updating the network) for the network of the image characteristic estimation module 110 may be performed through comparison between the reconstructed image and the correct answer data (correct answer image, etc.). The objective function may be implemented to learn the network in a way that minimizes the difference between the rendered and/or reconstructed image and the correct answer data (correct answer image, etc.). Learning a network may mean updating the network by modifying the weights between nodes included in the network based on the objective function. The learning operation of the network of the image characteristic estimation module 110 will be described in more detail later with reference to FIGS. 6 to 8.

In addition, with regard to learning of the network included in the image classification module 120, the objective function will be implemented to learn the network in a way that minimizes cross-entropy loss based on the predicted classification result and the ground truth. However, it is not limited to this and can be implemented to learn the network according to various known methods.

Meanwhile, 'learning' used in this specification may have the same meaning as learning or training (or training), and performing learning means that the network performs learning or trains the network. may include.

Figure 3 is a flowchart for explaining an image classification method according to an exemplary embodiment of the present disclosure. FIG. 4 is a diagram for explaining a mesh-based 3D characteristic estimation operation according to an exemplary embodiment of the present disclosure. Figure 5 is a diagram showing examples of an operation for classifying an image based on estimated 3D characteristics.

Referring to FIG. 3 , when an image is input (S300), the image classification method according to an exemplary embodiment of the present disclosure may include estimating three-dimensional characteristics of the input image (S310).

In this regard, referring to FIG. 4 , the image characteristic estimation module 110 of the image classification system 100 may estimate 3D characteristics of an input image and output an estimation result. As described above, the image characteristic estimation module 110 includes a deep learning-based network such as CNN, and can estimate the 3D characteristics through the network.

Meanwhile, according to an embodiment of the present disclosure, the image characteristic estimation module 110 may be implemented to estimate information on the 3D characteristics in a mesh form. The mesh includes a plurality of vertices 402 and 412, a plurality of edges 404 and 414 connected between two of the vertices, and a plurality of vertices and a plurality of sides. It may be composed of faces 406 and 416 representing the area formed by. The plurality of vertices may correspond to 3D points (coordinates) representing 3D characteristics. In this specification, the information on the three-dimensional characteristics is shown as being expressed in the form of a square mesh, but the shape of the mesh may be expressed in various polygonal forms such as triangles.

The image characteristic estimation module 110 may be implemented to estimate the 3D characteristics of the input image by transforming the initial mesh 400 to correspond to the 3D characteristics of the input image. Specifically, the image characteristic estimation module 110 changes the position of at least some of the vertices 402 of the initial mesh 400 according to three-dimensional characteristics such as the outline or depth of the input image, thereby The deformed mesh 410 can be estimated from. In addition, the network of the image characteristic estimation module 110 according to an embodiment of the present disclosure may be trained to estimate the deformed mesh 410 by further reflecting the color variance of the input image. Depending on the embodiment, the network may be trained to prevent overlap between faces 416 of the deformed mesh 410. To this end, the network may be trained so that the length deviation of the sides 414 of the deformed mesh 410 is minimized or the area difference between the sides 416 is minimized.

The image characteristic estimation module 110 may provide information about the three-dimensional characteristics of the input image by outputting information about the vertices 412 of the deformed mesh 410. For example, the output information is provided in the form of a position change amount (coordinate change amount, etc.) 412a from each of the vertices 402 of the initial mesh 400, or each of the vertices 412 of the deformed mesh 410. It may be provided in the form of a position value (coordinate value, etc.) 412b.

Figure 3 will be described again.

The image classification method may include classifying an image based on the estimated 3D characteristics (S320) and providing a classification result (S330).

The image classification module 120 of the image classification system 100 identifies and classifies the input image (object included in the image) based on the 3D characteristic information estimated by the image characteristic estimation module 110. You can.

In this regard, referring to (a) of FIG. 5, as described above in FIG. 4, the image characteristic estimation module 110 calculates the amount of position change (coordinate change amount, etc.) from each of the vertices 402 of the initial mesh 400 ( The three-dimensional characteristic information may be provided in the form of 412a) or in the form of position values (coordinates, etc.) 412b of each vertex 412 of the deformed mesh 410.

The image classification module 120 may obtain a classification result for the image by inputting the 3D characteristic information into a network. For example, if the output value (probability value, etc.) corresponding to 'small intestine' is the highest among the output values of the network, the image classification module 120 may provide 'small intestine' as the classification result.

Referring to the embodiment of FIG. 5 (b), the image classification module 120 provides a classification result for the input image based on the input image and a depth image rendered based on the 3D characteristic information. You can also obtain . In this case, the network included in the image classification module 120 may be a network implemented as a CNN.

Through the above-described operation, the image classification system 100 can provide classification results for the input image. Meanwhile, in order to improve the accuracy of classification results, a learning process of the network included in each of the image characteristic estimation module 110 and the image classification module 120 is required.

FIG. 6 is a flowchart illustrating an operation in which a learning module of an image classification system performs learning on a network of an image feature estimation module according to an exemplary embodiment of the present disclosure. Figures 7 and 8 are exemplary diagrams related to the learning operation of the learning module shown in Figure 6.

Referring to FIG. 6, the learning operation for the network of the image characteristic estimation module 110 performs rendering using the 3D characteristics estimated for the image (S600) and depth information between the rendering result and the correct answer data. Based on the difference, a step (S602) of performing learning of the network included in the image characteristic estimation module 110 may be included.

Referring to FIG. 7 in relation to steps S600 to S602, the image classification system 100 (or learning module 130) has the image characteristic estimation module 110 estimate three-dimensional characteristics from images (training images, etc.). When 3D characteristic information is output, rendering can be performed based on the output 3D characteristic information. The image classification system 100 may obtain depth data 702 from the rendering result 700. For example, the depth data 702 may correspond to depth data when the rendering result 700 is viewed from the direction of the camera that acquired the image.

Additionally, the image classification system 100 may obtain depth data 712 from the correct answer data 710 representing the three-dimensional characteristics of the image. Like the depth data 702, the depth data 712 may correspond to depth data when looking at the correct answer data 710 from the direction of the camera that acquired the image.

The learning module 130 performs the image characteristic estimation module 110 based on the difference between the depth data 702 obtained from the rendering result 700 and the depth data 712 obtained from the correct answer data 710. Learning about the network can be performed. That is, the objective function for network learning of the image characteristic estimation module 110 may be implemented to learn the network in a way that minimizes the difference between the depth data 702 and 712. Through the above-mentioned learning, the image characteristic estimation module 110 can improve the estimation accuracy of 3D characteristics from the image, especially characteristics related to depth.

In addition, the learning operation for the network of the image characteristic estimation module 110 reconstructs the image based on the estimated three-dimensional characteristics (S610), and determines the network based on the pixel value difference between the reconstructed image and the correct image. It may include a step of performing learning (S612).

Referring to FIG. 8 in relation to steps S610 to S612, the image classification system 100 (or learning module 130) calculates three-dimensional characteristics from the image that the image characteristic estimation module 110 inputs (learning image, etc.). If the estimated 3D characteristic information is output, the image can be reconstructed based on the output 3D characteristic information.

Specifically, the image classification system 100 may acquire the deformed mesh 410 described above in FIG. 4 based on the estimation result of the image characteristic estimation module 110. A plurality of pixels may correspond to the face 416 of the deformed mesh 410. The image classification system 100 may generate a reconstructed image 800 by ensuring that pixels corresponding to the same side 416 of the deformed mesh 410 have the same pixel value (color). For example, the pixel value of the pixels corresponding to the same side 416 in the reconstructed image 800 may correspond to the average pixel value of the pixels of the corresponding side 416 among the pixels of the input image, but is limited to this. It doesn't work.

The image classification system 100 may perform learning on the network of the image characteristic estimation module 110 based on the color difference (pixel value difference) between the reconstructed image 800 and the correct image 810. That is, the objective function for network learning of the image characteristic estimation module 110 can be implemented to learn the network in a way that minimizes the color difference between the reconstructed image 800 and the correct image 810. That is, the image characteristic estimation module 110 can be trained to estimate 3D characteristics by more accurately considering the color variance of pixels in the image when estimating 3D characteristics.

Therefore, the image characteristic estimation module 110 according to an embodiment of the present disclosure estimates the 3D characteristics by considering color changes within the image as well as depth information when estimating the 3D characteristics of the image, thereby providing more accurate estimation results and The accuracy of classification can be improved.

Although not shown, depending on the embodiment, the image characteristic estimation module 110 may be trained in a direction that minimizes the length difference between the sides of the deformed mesh 410, or may be trained in a direction that minimizes the area difference between the sides.

Figure 9 is a schematic block diagram of a device performing an image classification method according to an exemplary embodiment of the present disclosure.

Referring to FIG. 9, a device 900 according to an embodiment of the present disclosure may correspond to any one of at least one computing device constituting the image classification system 100 described above in FIG. 1. In this case, the device 900 may correspond to a device that performs a 3D characteristic estimation operation of an image, an image classification operation based on the estimated 3D characteristics, and/or a learning operation described above in this specification.

This device 900 may include a processor 910 and memory 920. However, the components of the device 900 are not limited to the examples described above. For example, device 900 may include more or fewer components than the components described above. Additionally, there may be at least one processor 910 and there may be at least one memory 920. Additionally, two or more of the processor 910 and the memory 920 may be combined into one chip.

According to one embodiment, the processor 910 corresponds to at least one of the above-described image feature estimation module 110, image classification module 120, and learning module 130, or executes at least one of the modules. You can control it.

This processor 910 includes hardware such as a CPU, an application processor (AP), an integrated circuit, a microcomputer, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and/or a neural processing unit (NPU). can do.

According to an embodiment of the present disclosure, the memory 920 may store programs and data necessary for the operation of the device 900.

Additionally, the memory 920 may store at least one of data generated or acquired through the processor 910. Depending on the embodiment, the memory 920 may store data, commands, algorithms, etc. related to the image characteristic estimation module 110, the image classification module 120, and/or the learning module 130.

The memory 920 may be composed of a storage medium such as ROM, RAM, flash memory, SSD, or HDD, or a combination of storage media.

The description of the above-described embodiments is merely an example with reference to the drawings for a more thorough understanding of the present disclosure, and should not be construed as limiting the technical idea of the present disclosure.

In addition, it will be clear to those skilled in the art to which this disclosure pertains that various changes and modifications can be made without departing from the basic principles of the present disclosure.

Claims (15)

In an image classification system,
An image feature estimation module that estimates three-dimensional features of an input image; and
An image classification module that classifies the image based on three-dimensional characteristics estimated by the image characteristic estimation module,
The image feature estimation module is,
A first network based on deep learning learned to estimate the three-dimensional characteristics by reflecting the outline of the input image, depth, and color changes between pixels in the input image,
The image classification system further includes a learning module for learning the first network,
The learning module is,
Update the first network based on color differences between corresponding pixels between the image reconstructed based on the estimated three-dimensional characteristics and the input image,
Is the image reconstructed based on the three-dimensional characteristics an image reconstructed so that pixels corresponding to the same side of the deformed mesh have the same pixel value?
Image classification system.

According to paragraph 1,
The image feature estimation module is,
Estimating the positions of a plurality of 3D points corresponding to the 3D characteristics of the image, and outputting values corresponding to the estimated positions as estimation results of the 3D characteristics,
Image classification system.
According to paragraph 2,
Estimate a mesh representing the three-dimensional characteristics of the image,
The plurality of three-dimensional points correspond to a plurality of vertices included in the mesh,
Image classification system.
According to paragraph 1,
The learning module is,
Updating the first network based on the difference between a rendering result using the estimated 3D characteristics and the correct answer data corresponding to the 3D characteristics of the input image,
Image classification system.

delete According to paragraph 3,
The image classification module is,
Further comprising a deep learning-based second network learned to classify the image based on values corresponding to the positions of the plurality of 3D points,
Image classification system.
According to paragraph 3,
The image classification module is,
Further comprising a deep learning-based second network learned to classify the input image based on the depth image and the input image rendered based on the plurality of 3D points,
Image classification system.
According to paragraph 1,
The image classification system is implemented with at least one computing device,
Each of the at least one computing device includes a processor and memory,
Image classification system.
estimating three-dimensional characteristics of the input image; and
Classifying the input image based on estimated three-dimensional characteristics,
The step of estimating the three-dimensional characteristics is,
Using a deep learning-based first network learned to estimate the three-dimensional characteristics by reflecting the outline, depth, and color changes between pixels in the input image, 3 for the input image This is the step of estimating dimensional characteristics,
The step of updating the first network is,
Updating the first network based on color differences between corresponding pixels between the image reconstructed based on the estimated three-dimensional characteristics and the input image,
The image reconstructed based on the three-dimensional characteristics is an image reconstructed so that pixels corresponding to the same side of the deformed mesh have the same pixel value,
Image classification method.
According to clause 9,
The step of estimating the three-dimensional characteristics is,
estimating the positions of a plurality of 3D points corresponding to 3D characteristics of the input image; and
Including outputting values corresponding to the estimated position as an estimation result of the three-dimensional characteristic,
Image classification method.
According to clause 10,
The plurality of 3D points correspond to a plurality of vertices included in a mesh estimated to represent the 3D characteristics of the input image.
Image classification method.
According to clause 9,
Further comprising updating the first network based on the difference between a rendering result using the estimated 3D characteristics and the correct answer data corresponding to the 3D characteristics of the input image,
Image classification method.
delete According to clause 11,
The step of classifying the input image is,
A step of classifying the input image using a second deep learning-based network learned to classify the image based on values corresponding to the positions of the plurality of 3D points,
Image classification method.
According to clause 11,
The step of classifying the input image is,
generating a depth image based on the plurality of 3D points;
A step of classifying the input image using a deep learning-based second network learned to classify the input image based on the generated depth image and the input image,
Image classification method.

Images