KR102556235B1 - Method and apparatus for content based image retrieval - Google Patents

Abstract

According to an embodiment of the present disclosure, a content-based image retrieval method using a neural network performed by a computing device including at least one processor is disclosed. The method may include: inputting an input image to a learned analysis model including at least one neural network to obtain a global feature analysis result and a local feature analysis result for the input image; obtaining a first search result including at least one candidate image similar to an input image based on the global feature analysis result; obtaining a second search result including at least one candidate image similar to an input image from the first search result based on a result of the regional feature analysis; and providing the second search result.

Description

Content-based video search method and apparatus {METHOD AND APPARATUS FOR CONTENT BASED IMAGE RETRIEVAL}

The present invention relates to a content-based image search method, and more particularly, to an image search method using a neural network.

In the conventional art, a Greulich-Pyle (GP) method and a Tanner-Whitehouse 3 (TW3) method have been used as methods for reading the degree of bone development (bone age) from a bone image. The GP method compares and analyzes the overall degree of bone development based on a standard atlas containing pictures of hand bones standardized by bone age, and measures bone age as a result. The TW3 measurement method determines the grade of bone maturity for each of the 13 parts of the hand bone, and calculates the bone age by adding the scores corresponding to that grade. The GP method and the TW3 method each have advantages and disadvantages depending on the bone age calculation method, and there is a difference in whether to view the input bone image as a whole or in part. Furthermore, there was a problem that both the GP and TW3 methods could affect the accuracy of bone age reading according to differences in proficiency between readers.

Therefore, there has been a continuous demand in the art to maximize the advantages of each method by simultaneously using the GP method and the TW3 method, and to solve the problems caused by the difference in skill level as described above. In addition, in addition to the bone age reading operation, the user's demand to receive a bone image similar to the input image to use the result of the reading program is also increasing.

Korean registered patent "KR2097742" discloses an artificial intelligence-based medical image search system and its driving method.

The present disclosure has been made in response to the above background art, and aims to provide an image search method using a neural network.

According to an embodiment of the present disclosure for realizing the above object, a content-based image retrieval method using a neural network performed by a computing device including at least one processor is disclosed. The method may include: inputting an input image to a learned analysis model including at least one neural network to obtain a global feature analysis result and a local feature analysis result for the input image; obtaining a first search result including at least one candidate image similar to an input image based on the global feature analysis result; obtaining a second search result including at least one candidate image similar to an input image from the first search result based on a result of the regional feature analysis; and providing the second search result.

In an alternative embodiment, the image is a bone image subject to bone age determination, and the analysis model includes a common feature analysis model and may further include at least one of a global feature analysis model and a regional feature analysis model. .

In an alternative embodiment, the global feature analysis model included in the analysis model may include at least one pooling layer, and the local feature analysis model included in the analysis model may include a region-of-interest detection module.

In an alternative embodiment, the global feature analysis result may include at least one global feature vector generated in a process of calculating a global feature analysis model for the input image; Alternatively, global meta information included in the final output of the global feature analysis model for the input image may be included.

In an alternative embodiment, the local feature analysis result may include at least one local feature vector generated in a process of calculating a local feature analysis model for the input image; Alternatively, regional meta information included in the final output of the regional feature analysis model for the input image may be included.

In an alternative embodiment, the feature vector may be generated based on at least one of a max pooling method, an average pooling method, and a generalized mean pooling method.

In an alternative embodiment, the first search result is obtained by comparing a plurality of similarities calculated between a global feature vector of the input image and a global feature vector of each of a plurality of candidate images, or the input image It may be obtained by comparing a plurality of coincidence degrees calculated between the global meta information for and the global meta information for each of a plurality of candidate images.

In an alternative embodiment, the second search result compares a plurality of similarities calculated between a local feature vector of the input image and a local feature vector of each of a plurality of candidate images included in the first search result. Alternatively, it may be obtained by comparing a plurality of coincidence degrees calculated between regional meta information on the input image and regional meta information on each of a plurality of candidate images included in the first search result.

In an alternative embodiment, the degree of similarity may be calculated based on at least one of a cosine similarity between feature vectors, an L1 distance between feature vectors, and an L2 distance between feature vectors.

A computer program stored in a computer readable storage medium is disclosed according to an embodiment of the present disclosure for realizing the above object. When the computer program is executed on one or more processors, the following operations for content-based image retrieval using a neural network are performed, and the operations include: inputting an input image to a learned analysis model including at least one neural network. obtaining a global feature analysis result and a local feature analysis result for the input image; obtaining a first search result including at least one candidate image similar to an input image based on the global feature analysis result; obtaining a second search result including at least one candidate image similar to an input image from the first search result based on a result of the regional feature analysis; and providing the second search result.

According to an embodiment of the present disclosure for realizing the above object, an apparatus for searching an image based on content using a neural network is disclosed. The device may include one or more processors; a memory for storing an analysis model including one or more neural networks; and a network unit, and the one or more processors input an input image to a learned analysis model including at least one neural network to obtain a global feature analysis result and a local feature analysis result for the input image, and to obtain a global feature analysis result and a local feature analysis result for the input image. A first search result including at least one candidate image similar to the input image is obtained based on a feature analysis result, and a second search result including at least one candidate image similar to the input image based on the regional feature analysis result may be obtained from the first search result, and the second search result may be provided.

The present disclosure may provide an image search method using a neural network.

1 is a block diagram of a computing device for content-based image retrieval according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.
3 is an exemplary diagram illustrating the structure of an analysis model according to an embodiment of the present disclosure.
4 is an exemplary diagram illustrating a method for obtaining a search result including an image similar to an input image by using an analysis model by a computing device according to the present disclosure.
5 is a flowchart illustrating a process of providing, by a computing device, a second search result including a candidate image similar to an input image according to an embodiment of the present disclosure.
6 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may be connected, for example, via signals with one or more packets of data (e.g., data and/or signals from one component interacting with another component in a local system, distributed system) to other systems and over a network such as the Internet. data being transmitted) may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

The term "image" or "image data" as used throughout the description and claims of the present invention refers to multidimensional data composed of discrete image elements (e.g., pixels in the case of a two-dimensional image); in other words, ( A term used to refer to a visible object (e.g., displayed on a video screen) or a digital representation of that object (e.g., a file corresponding to the pixel output of a CT, MRI detector, etc.).

For example, in the present disclosure, “image” or “image” refers to computed tomography (CT), magnetic resonance imaging (MRI), fundus imaging, ultrasound, or any known art in the art of the present invention. may refer to a medical image of a subject collected by another medical imaging system of . The image does not necessarily have to be provided in a medical context, but may be provided in a non-medical context, such as X-ray imaging for security screening.

The term "image" used throughout the detailed description and claims of the present invention may be used as a concept including all types of medical images that can be input to a bone age interpretation model, including input images or candidate images. Therefore, in the present disclosure, an input image or candidate image, which is a target of bone age determination, may include, for example, at least one of an X-ray image, a CT image, and an MRI image related to a bone provided to determine bone age. . In addition, the input image or candidate image may include any bone-related image that can be a target of bone age determination, such as a hand bone, an elbow bone, a knee bone, and the like, without limitation. The input image may be an image input to a computing device to search for a similar image. The candidate image is an image similar to the input image and may mean an image that can be searched or searched by a computing device. Candidate images may be stored in memory. The memory may be internal to or external to the computing device.

Throughout the detailed description and claims of the present invention, the 'DICOM (Digital Imaging and Communications in Medicine)' standard is a term that collectively refers to various standards used for digital image expression and communication in medical devices, The DICOM standard is published by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

Throughout the detailed description and claims of the present invention, 'Picture Archiving and Communication System (PACS)' is a term that refers to a system that stores, processes, and transmits in accordance with the DICOM standard, and includes X-ray, CT, and MRI. Medical imaging images obtained using digital medical imaging equipment such as are stored in a DICOM format and can be transmitted to terminals inside and outside the hospital through a network, to which reading results and medical records can be added.

1 is a block diagram of a computing device for content-based image retrieval according to an embodiment of the present disclosure.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 may include a processor 110 , a memory 130 , and a network unit 150 .

The processor 110 may include one or more cores, and includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. unit), data analysis, and processors for deep learning. The processor 110 may read a computer program stored in the memory 130 and process data for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 is used for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating neural network weights using backpropagation. calculations can be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, the CPU and GPGPU can process learning of network functions and data classification using network functions. In addition, in an embodiment of the present disclosure, the learning of a network function and data classification using a network function may be processed by using processors of a plurality of computing devices together. In addition, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

According to an embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the network unit 150 .

According to an embodiment of the present disclosure, the memory 130 is a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) -Only Memory), a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The above description of the memory is only an example, and the present disclosure is not limited thereto.

The network unit 150 according to an embodiment of the present disclosure includes a Public Switched Telephone Network (PSTN), x Digital Subscriber Line (xDSL), Rate Adaptive DSL (RADSL), Multi Rate DSL (MDSL), and VDSL ( Various wired communication systems such as Very High Speed DSL), Universal Asymmetric DSL (UADSL), High Bit Rate DSL (HDSL), and Local Area Network (LAN) may be used.

In addition, the network unit 150 presented in this specification includes Code Division Multi Access (CDMA), Time Division Multi Access (TDMA), Frequency Division Multi Access (FDMA), Orthogonal Frequency Division Multi Access (OFDMA), SC-FDMA ( Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit 150 may be configured regardless of its communication mode, such as wired and wireless, and may be configured with various communication networks such as a personal area network (PAN) and a wide area network (WAN). can In addition, the network may be the known World Wide Web (WWW), or may use a wireless transmission technology used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth.

The techniques described herein may be used in the networks mentioned above as well as other networks.

According to an embodiment of the present disclosure, a content-based image retrieval method using a neural network performed by a computing device including at least one processor includes a learned analysis of an input image including at least one neural network. obtaining a global feature analysis result and a local feature analysis result for the input image by inputting the data to the model; obtaining a first search result including at least one candidate image similar to an input image based on the global feature analysis result; obtaining a second search result including at least one candidate image similar to an input image from the first search result based on a result of the regional feature analysis; and providing the second search result.

In an embodiment of the present disclosure, the processor 110 may obtain a global feature analysis result and a local feature analysis result of the input image by inputting an input image to a learned analysis model including at least one neural network. The analysis model may include at least one neural network. A neural network included in the analysis model may be trained. Learning of the neural network includes updating at least one weight or bias value included in the neural network.

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes at least one node. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. can A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), auto encoders, generative adversarial networks (GANs), and restricted boltzmann machines (RBMs). machine), a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

In one embodiment of the present disclosure, the network function may include an autoencoder. An autoencoder may be a type of artificial neural network for outputting output data similar to input data. An auto-encoder may include at least one hidden layer, and an odd number of hidden layers may be disposed between input and output layers. The number of nodes of each layer may be reduced from the number of nodes of the input layer to an intermediate layer called the bottleneck layer (encoding), and then expanded symmetrically with the reduction from the bottleneck layer to the output layer (symmetrical to the input layer). Autoencoders can perform non-linear dimensionality reduction. The number of input layers and output layers may correspond to dimensions after preprocessing of input data. In the auto-encoder structure, the number of hidden layer nodes included in the encoder may decrease as the distance from the input layer increases. If the number of nodes in the bottleneck layer (the layer with the fewest nodes located between the encoder and decoder) is too small, a sufficient amount of information may not be conveyed, so more than a certain number (e.g., more than half of the input layer, etc.) ) may be maintained.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

In the detailed description and claims of the present invention, a convolutional neural network may include a sequence of neural network layers. The neural network layer may include a convolution layer, a pooling layer, a fully connected layer, and the like. The initial input to the convolutional neural network may be received by the first lowest layer in the sequence. A convolutional neural network can sequentially input an initial input into layers in a sequence to generate a final output from the initial input. The initial input may be, for example, an image, and a final output thereof may be a score set for each category in a category set including one or more categories.

In the description and claims of the present invention, each neural network layer in a convolutional neural network includes a set of nodes. Each neural network layer may receive as an input an initial input to a convolutional neural network or an output of a previous neural network layer. For example, in a sequence composed of a plurality of neural network layers, an N-th neural network layer may receive an output of an N-1-th neural network layer as an input. Each neural network layer in a convolutional neural network produces an output from an input. If the layer is the highest layer in the sequence, it is treated as the final output of the convolutional neural network.

In the detailed description and claims of the present invention, the term “feature map” may refer to output data generated by a neural network layer. The feature map may be output data generated by a neural network layer and data input to a next layer. In other words, the feature map may be a concept including inputs and outputs of each neural network layer in the computation process of the artificial neural network model. The “feature map” may be a 3D array having size factors for three axes of height, width, and channel. The feature map may include at least one 2D array. The two-dimensional array may be referred to as a matrix, grid, or matrix. In addition, in the detailed description and claims of the present invention, a “feature vector” may be a one-dimensional array having one data for each channel on the channel axis in the above-described “feature map”. The feature vector may include data as much as the size of the channel. One matrix included in the feature map output by the neural network layer may correspond to one type of filter.

The structure of the analysis model is described below. The learning method of the analysis model will be described in detail below.

3 is an exemplary diagram illustrating the structure of an analysis model 300 according to an embodiment of the present disclosure. In an embodiment of the present disclosure, the analysis model 300 includes a common feature analysis model 310 and may include at least one of a global feature analysis model 350 and a local feature analysis model 330 . The common feature analysis model 310 may output a common feature map representing an input image. The common feature analysis model 310 may include at least one convolutional layer. A convolution layer may include at least one filter. In an embodiment of the present disclosure, the processor 110 uses a common feature analysis model 310 to determine a common feature having a three-dimensional size in which various features are expressed through a plurality of channels from an input image having three channels according to RGB values. A feature map can be output.

In an embodiment of the present disclosure, the global feature analysis model 350 includes at least one convolutional neural network, and may analyze global features of an input image through the convolutional neural network layer and then output a global feature analysis result. there is. The global feature analysis model 350 may include one or more pooling layers to output global feature analysis results. The global feature may be a feature calculated from all pixels of the input image. For example, the global feature analysis model 350 may output a predicted bone age or a predicted age from an input image by analyzing global features as described above. The global feature analysis model 350 may output a predicted bone age by performing an operation through a Greulich-Pyle (GP) method from all pixels included in a bone image given as an input image.

In an embodiment of the present disclosure, the regional feature analysis model 330 includes at least one convolutional neural network, and may output a regional feature analysis result for an input image through a convolutional neural network layer. The regional feature analysis model 330 may extract one or more regions of interest from the input image and analyze regional features of the region of interest in order to output a regional feature analysis result. The regional feature analysis model 330 may include a region of interest detection module for extracting the one or more regions of interest. The region feature analysis model 330 may calculate a predicted partial bone grade, a predicted partial bone maturity score, and the like for each of one or more regions of interest. For example, when a bone image is used as an input image, the local feature analysis model 330 extracts at least one region of interest from the input bone image, and generates Tanner-Whitehouse 3 (TW3) data according to the location of each region of interest. By applying the method, a predicted partial bone grade or a predicted partial bone maturity score can be output. In one embodiment of the present disclosure, the predicted partial bone maturation score may mean a radius-ulna-short (RUS) score on the TW3 method. For example, the regional feature analysis model 330 may extract a radius portion from the input medulla bone image and output a predicted partial bone grade of B. The regional feature analysis model 330 may output a predicted partial bone maturity score of 23 by extracting a radius portion from the input medulla bone image. Subsequently, the regional feature analysis model 330 may extract the ulna portion from the input marrow bone image and output a predicted partial bone grade of C. The regional feature analysis model 330 may extract the ulna portion from the input medulla bone image and output a predicted partial bone maturation score of 33. In the continued embodiment, the processor 110 outputs a predicted bone maturation score of the input image based on the predicted partial bone maturation score or the predicted partial bone maturation grade for at least one region of interest calculated by the regional feature analysis model 330. You may.

Referring to FIG. 3 , an analysis model 300 of the present disclosure includes a common feature analysis model 310 and may additionally include at least one of a global feature analysis model 350 and a local feature analysis model 330 . According to an embodiment of the present disclosure, the processor 110 may input the common feature map output from the common feature analysis model 310 to the global feature analysis model 350 or the local feature analysis model 330 . The global feature analysis model 350 or the local feature analysis model 330 receives as an input the common feature map in which various features of the input image having three channels according to RGB values are expressed abundantly, and the global feature analysis model ( 350) and the regional feature analysis model 330 respectively process common tasks to be performed at once, and have an effect of sharing the result. The common task includes, for example, a general filter application or a Gaussian filter application for noise removal. The general filter may include, for example, a filter according to a direction of a line segment located in an image, a filter for finding a circular shape in an image, and the like. The above-described example of the common operation is only an example and does not limit the present disclosure. According to the present disclosure, the processor 110 may input the common feature map output from the common feature analysis model 310 to the global feature analysis model 350 and the local feature analysis model 330, respectively. The global feature analysis model 350 and the local feature analysis model 330 may be processed in parallel. This has an effect of improving overall processing performance and image search speed.

The global feature analysis model 350 included in the analysis model according to an embodiment of the present disclosure may include at least one pooling layer. The pooling layer may reduce the height and width of the input feature map by a predetermined pooling method. For example, when a feature map having a size of 64x64xC is input to the pooling layer, the pooling layer may generate an output feature map having a size of 32x32xC. At this time, the pooling layer uses a window having a height and width of 2 for the feature map having a size of 64x64xC, but when the window is moved, the size of the stride is set to 2, and the output feature having a size of 32x32xC You can create maps. The window size means the number of data that a filter used for pooling can read at one time. The stride means a movement interval when the window is moved in the direction of the “height” and “width” axes in the feature map. The types of pooling methods in the pooling layer will be described in detail below.

The global feature analysis model 350 of the present disclosure includes at least one pooling layer to determine an image by integrating a wider range of data, and thus can analyze global features. In addition, since a wide range of data is synthesized and judged, there is an advantage in that the characteristics can be constantly analyzed even if the location of the main region in the image changes.

The regional feature analysis model 330 included in the analysis model according to an embodiment of the present disclosure may include a region of interest detection module.

In the present disclosure, the ROI detection module may include a Region Proposal Network (RPN) module. The RPN module may include a convolutional neural network or a deep neural network. The RPN module may detect one or more regions of interest from a feature map of an input image. The feature map of the input image may be expressed as a set of two-dimensional array matrices. The RPN module may search a feature map of the input image with a window having a predetermined size and determine at least one region of interest. The term “region of interest” may be used interchangeably with the term “region of interest (RoI)”. The processor 110 may determine one or more regions of interest through the RPN module, and may obtain regional feature maps of the one or more regions of interest. Output For example, the RPN module included in the analysis model of the present disclosure may generate a hand bone image. When is received as an input image, a partial region of at least one knuckle may be extracted as a region of interest. Further detailed description of the RPN module described above is provided in the prior paper “Shaoqing Ren, 　Kaiming He, 　Ross Girshick, 　Jian Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” which is incorporated by reference in its entirety in this application. It is discussed in more detail in “arXiv:1506.01497, 2015”.

In the present disclosure, the ROI detection module may include an object detection model trained for object detection. The object detection model may include at least one neural network layer. The object detection model may return a feature map of the detected object as its output. The object detection model may detect one or more regions of interest in the input image and return regional feature maps for the one or more detected regions of interest. The object detection model may have been previously supervised to detect one or more regions of interest in the input image. As an embodiment, when an object detection model is trained to detect regions of interest corresponding to each node of the skull by receiving a bone image, the object detection model includes a first region of interest, a second region of interest, ..., an Nth region of interest. A region of interest may be pre-learned by a plurality of labeled medulla images. N may be an arbitrary integer according to the number of regions of interest. The processor 110 of the present disclosure may detect one or more regions of interest in the input image by using the object detection model and obtain a regional feature map of the region of interest. Results of global feature analysis according to an embodiment of the present disclosure may include at least one global feature vector generated during the operation of the global feature analysis model 350 for the input image or global meta information included in the final output of the global feature analysis model for the input image. The at least one global feature vector may be generated from a global feature map generated between a plurality of neural network layers included in the global feature analysis model 350 . A global feature vector may be generated from a global feature map through a pooling method. The pooling method may include a global pooling method. For example, the pooling method may include max pooling, average pooling, or generalized average pooling. The processor 110 of the present disclosure may generate a global feature vector of an input image from a global feature map of the input image and compare the global feature vector of another image. The other image may be one of candidate images stored in a memory.

Global meta information according to the present disclosure may include at least one piece of output data included in a final output output by the global feature analysis model. The global meta information may include at least one of a predicted bone age and a predicted age of the input image. Throughout the present specification, bone age means bone age determined from bones included in an image, age may mean a universal age determined based on a birthday, and bone age and age may be distinguished from each other. The example of the type of output data included in the aforementioned global meta information is only an example and does not limit the present disclosure. The processor 110 of the present disclosure may use a global feature analysis model learned based on a GP scheme to output the global meta information.

The local feature analysis result according to an embodiment of the present disclosure is obtained from at least one local feature vector generated in the process of calculating the regional feature analysis model 330 for the input image or the final output of the regional feature analysis model for the input image. May contain included local meta information.

At least one regional feature vector included in the regional feature analysis result of the present disclosure may exist in each of one or more regions of interest extracted from an input image by the region of interest detection module. The at least one regional feature vector may include a first regional feature vector, a second regional feature vector, ..., an Nth regional feature vector according to the type of the region of interest. The N value may be predetermined as a natural number according to the number of regions of interest. At least one regional feature vector may be generated from feature maps of one or more regions of interest extracted by the above-described region of interest detection module through a pooling method. The pooling method may include max pooling, average pooling, or generalized average pooling. The processor 110 of the present disclosure may generate a regional feature vector of an input image from one or more regional feature maps of the input image, and compare the regional feature vector of another image with each other. The other image may be one of candidate images stored in a memory. The other image may be one of candidate images included in the first search result.

Regional meta information included in the regional feature analysis result of the present disclosure may exist in each of one or more regions of interest extracted from an input image. The region meta information may include a predicted partial bone grade or a predicted partial bone maturity score for each of one or more regions of interest included in the input image. One or more regions of interest included in the input image may include, for example, a region according to a Tanner-Whitehouse (TW) scheme. The processor 110 of the present disclosure may use a regional feature analysis model learned based on the TW3 scheme to output the regional meta information. The regional feature analysis model learned based on the TW3 method may output a predicted partial bone grade or a predicted partial bone maturity score according to the type of region of interest in the hand bone image. The processor 110 according to the present disclosure may calculate a predicted bone maturation score for the input image using the predicted partial bone maturation score for each of one or more ROIs. The predicted bone maturation score for the input image may mean the sum of predicted partial bone maturation scores for each of one or more ROIs.

A feature vector according to an embodiment of the present disclosure may be generated based on at least one of a max pooling method, an average pooling method, and a generalized mean pooling method. The feature vector may include a global feature vector and a local feature vector. The processor 110 may perform pooling through a pooling layer among neural network layers included in the analysis model 300 . Specifically, the pooling method refers to a method of reducing the size of a two-dimensional array (matrix) included in a feature map and present for each channel. For example, the pooling method for each matrix may include a method of reducing an NxN-sized matrix to an MxM (M < N) matrix. When the size of M is 1, an output of the pooling layer may be a feature vector. Accordingly, the processor 110 of the present disclosure may generate a global feature vector and a local feature vector from the global feature map and the local feature map, respectively, through a pooling method.

The Max Pooling method, one of the pooling methods, maintains the number of channels of the input feature map when generating the output feature map, moves a window of a predetermined size in the width and height directions, and obtains the maximum value within the window range. It is a dimensionality reduction method that leaves the data with For example, when the size of the window is 2x2, when the processor 110 generates an output feature map from an input feature map, one data having the largest value among a total of 4 data corresponding to a 2x2 window in a matrix is selected. A 1x1 matrix corresponding to a window can be created. When the processor 110 performs pooling on the input feature map through the neural network, when the size of the window is equal to the height and width of the input feature map, a feature vector may be generated as an output.

The average pooling method is similar to the maximum pooling method, but matches an area included in a window with an average value of data within the window range. For example, when the size of the window is 3x3, the processor 110 can generate a 1x1 matrix corresponding to the window with an average value of a total of 9 data corresponding to the 3x3 window in the matrix included in the input feature map.

The generalized mean pooling method may use a generalized formula rather than average pooling or maximum pooling in matching a plurality of data corresponding to a window to one data. The generalized formula may include learnable factors. As an embodiment, the generalized equation is equal to Equation 1.

[Equation 1]

represents a feature vector generated by the pooling layer. That is, in the example of Equation 1, the size of the window for generalized average pooling may be equal to "height" x "width" of the input feature map. represents a number for indicating the order of channels. is the feature vector represents the second data. represents a feature map input to the pooling layer. Among the channels of the feature map input to the pooling layer, represents the matrix corresponding to the second. Is Indicates the number of elements in the matrix. said As a factor for calculation, it may be a learnable factor. The average pooling method described above in terms of Equation 1 is In the case of substituting 1 for , the above-described maximum pooling method is This is the case when an infinity value is assigned to . Since Equation 1 is differentiable, in the present disclosure, when the generalized pooling method is used, the processor 110 can also differentiate the calculation process of the pooling layer. Therefore, when a model is learned using the generalized pooling method, parameters included in the pooling layer can also be updated through learning.

The analysis model 300 according to an embodiment of the present disclosure may be learned by using a first training data set labeled with at least one of an actual bone age and an actual age for one or more training hand bone images. The analysis model 300 may be trained by a second training data set labeled with real partial bone grades and real partial bone maturity scores for each region of interest of the TW3 method for one or more training bone images. The first training data set and the second training data set may share at least one training bone image. The processor 110 inputs the first training data set or the second training data set to the analysis model 300, and learns the analysis model 300 based on a total loss function according to a combination of a regression loss function or a classification loss function. can make it The analysis model 300 may be trained in a direction in which the total loss value is reduced. Learning the model in a direction in which the total loss value is reduced is a concept including updating weights and biases through error backpropagation of the entire model of the final error. The regression loss function may be based on a residual (difference value) between a predicted value and an actual value. The actual value means a labeled value and means a ground truth for an input in a model to be learned. For example, the regression loss function may be calculated based on the square of the residual, the absolute value of the residual, and the like. The classification loss function may be, for example, a sigmoid-based cross entropy function or a softmax-based cross entropy function. The example of the loss function described above is only an example and does not limit the present disclosure.

The global feature analysis model 350 included in the analysis model 300 according to the present disclosure outputs the predicted bone age or predicted age of the input image according to the GP method using the first training data set, and the labeled actual bone age Alternatively, it may be learned to reduce the difference based on the comparison result with the labeled actual age. Reducing the difference based on the comparison result includes updating at least one weight and bias value included in the global feature analysis model in order to reduce an error between the predicted value and the actual value.

The regional feature analysis model 330 included in the analysis model 300 according to the present disclosure obtains a predicted partial bone grade or a predicted partial bone maturity score for one or more regions of interest according to the TW3 method using the second training data set. It can be learned to output. The regional feature analysis model may be learned in a direction of reducing the difference based on a comparison result between the output predicted partial bone grade or predicted partial bone maturity score and the labeled actual partial bone grade or labeled actual partial bone maturity score. The learning of the regional feature analysis model includes an operation of the processor 110 updating at least one weight or bias value included in the regional feature analysis model in a direction in which the value of the cross entropy function related to the output of the regional feature analysis model decreases. can include

FIG. 4 is an exemplary diagram illustrating a method of obtaining a search result including an image similar to the input image 410 by using the analysis model 300 by the computing device 100 according to the present disclosure. The processor 110 performing a content-based image search according to an embodiment of the present disclosure generates a first search result 430 including at least one candidate image similar to the input image 410 based on the global feature analysis result. can be obtained The processor 110 may obtain (450) a second search result 470 including at least one candidate image similar to the input image 410 from the first search result, based on a regional feature analysis result. The processor 110 may provide the second search result 470 to the user. The at least one candidate image may be a bone image, such as the input image 410, which is a target for bone age reading. At least one candidate image may be stored in the memory 130 of the computing device 100 . At least one candidate image may be stored in a database server external to the computing device. If the candidate image is stored in an external server, the computing device 100 may receive the candidate image through the network unit 150 .

In the present disclosure, the first search result obtained by the processor 110 is obtained by comparing a plurality of similarities calculated between a global feature vector of an input image and a global feature vector of each of a plurality of candidate images, or an input image. It may be obtained by comparing a plurality of coincidence degrees calculated between global meta information on an image and global meta information on each of a plurality of candidate images. At this time, the plurality of candidate images that are searched for by the processor 110 to obtain the first search result include some or all of the images stored in a memory existing inside or outside the computing device 100 . In addition, in the present disclosure, the second search result obtained by the processor 110 includes a plurality of regions calculated between a local feature vector of the input image and a local feature vector of each of a plurality of candidate images included in the first search result. It may be obtained by comparing similarities or by comparing a plurality of coincidences calculated between regional meta information of an input image and regional meta information of each of a plurality of candidate images included in the first search result. A plurality of local feature vectors for an input image or candidate image may exist for one image according to the number of regions of interest.

The term "correspondence" according to the present disclosure may mean a numerical value indicating whether or not to match according to an error between two pieces of meta information. The processor 110 may determine the degree of agreement by considering an error of one or more pieces of data included in the meta information.

As an embodiment, when comparing the matching degree of global meta information, the processor 110 compares the predicted bone age of the input image with the predicted bone age of each of one or more candidate images, and determines a predetermined number of values in order of smallest difference. A candidate image may be obtained as a first search result. For a specific example, assuming that the predetermined number is set to 3, the predicted bone age of the input image is 10.4, and the predicted bone age of candidate images stored in memory is [9, 10.3, 10.4, 10.5, 11], the processor ( 110) may obtain three candidate images having predicted bone ages of 10.3, 10.4, and 10.5, respectively, as a first search result. In addition, the processor 110 calculates the degree of agreement with a plurality of candidate images by considering the predicted bone age and the predicted age included in the global meta information together, and then selects a predetermined number of candidate images according to the order of highest degree of agreement as the first search result. You may. When determining the degree of agreement using two or more pieces of data included in the meta information, the processor 110 may set different or equal weights according to types of data when determining the degree of agreement by summing errors of each data. For example, when determining the degree of agreement using the predicted bone age and the predicted age included in the global meta information, the processor 110 may determine the degree of agreement by multiplying errors of each data by the same weight and summing them.

As another embodiment, when comparing the degree of agreement on regional meta information, the processor 110 may use the predicted partial bone grade or predicted partial bone maturity score for each of one or more regions of interest included in the input image and the one included in the candidate image. A second search result may be obtained by comparing predicted partial bone grades or predicted partial bone maturation scores for each of the above regions of interest. The candidate image may be a candidate image included in the first search result. For example, when the degree of agreement is calculated using the predicted partial goal ratings and the second search result is obtained accordingly, the processor 110 sums the errors of the predicted partial goal ratings for each of one or more regions of interest so that the sum of the errors is the largest. A second search result including a predetermined number of candidate images in a small order may be obtained. The processor 110 may replace the grade interval with a constant sequence in order to calculate the error of the predicted partial goal grade. The sequence includes an arbitrary sequence and may be, for example, an arithmetic sequence. Specifically, when the distribution of predicted partial goal grades included in the regional meta information exists from A to I, the processor 110 may replace one grade difference with a number having a predetermined size in alphabetical order. For another example, when the degree of agreement is calculated using the predicted partial bone maturity score and the second search result is obtained accordingly, the processor 110 determines the error of the predicted partial bone maturity score for one or more regions of interest according to the TW calculation method. It is also possible to obtain a second search result including a predetermined number of candidate images in the order of decreasing total sum of errors by summing . Candidate images included in the second search result may be a subset of candidate images included in the first search result. The foregoing specific example is only an example and does not limit the present disclosure.

The term “similarity” according to the present disclosure may refer to a numerical value representing a degree of similarity between two feature vectors including at least one element. The similarity according to the present disclosure may be calculated based on at least one of a cosine similarity between feature vectors, an L1 distance between feature vectors, and an L2 distance between feature vectors.

The cosine similarity between feature vectors can be expressed as Equation 2, for example.

[Equation 2]

and denotes two different feature vectors, respectively. and denotes the size of each feature vector. and is within two different feature vectors. represents the value of the second element. The cosine similarity calculated as above may have a value between -1 and 1. The cosine similarity value may have a value of -1 when the directions of the two feature vectors are completely different, and may have a value of 1 when they completely coincide.

The L1 distance between feature vectors can be expressed as in Equation 3.

[Equation 3]

represents the L1 distance function. and denotes different feature vectors, respectively. and are respectively and Within the feature vector expressed as represents the value of the first element. is the total number of elements. That is, the L1 distance means a result obtained by converting the difference between each element of the feature vector into an absolute value and then adding them up. It can be interpreted that the larger the value of the L1 distance, the larger the error between the two feature vectors. Conversely, the closer the value of the L1 distance is to 0, the smaller the error between the two feature vectors can be interpreted. Throughout this specification, “L1 distance” calculated as in Equation 3 may be used interchangeably as a term having the same meaning as “Manhattan distance”. Since the L1 distance is made up of only a subtraction operation and the computation burden on the processor 110 or the CPU is small, the computation speed is fast.

The L2 distance between feature vectors can be expressed as Equation 4.

[Equation 4]

represents the L2 distance function. and denotes different feature vectors, respectively. and are respectively and Within the feature vector expressed as represents the value of the first element. is the total number of elements. That is, the L2 distance refers to a result of performing a root operation on the total sum after squaring and summing the differences of each element of the feature vector. The larger the value of the L2 distance, the larger the error between the two feature vectors can be interpreted, and the closer the value of the L2 distance is to 0, the smaller the error between the two feature vectors can be interpreted. Throughout this specification, “L2 distance” calculated as in Equation 4 may be used interchangeably as a term having the same meaning as “Euclidean distance”.

The processor 110 according to the present disclosure calculates a plurality of similarities between a global feature vector of an input image and a global feature vector of each of a plurality of candidate images, and compares the plurality of similarities to a predetermined number of candidate images. may be obtained as the first search result. In addition, the processor 110 of the present disclosure calculates a plurality of similarities between a local feature vector of an input image and a local feature vector of each of a plurality of candidate images, and compares the plurality of similarities to a predetermined number of candidates. Images may be obtained from the first search result as the second search result. The processor 110 may obtain N candidate images having the highest similarity to the input image by calculating similarities between two regional feature vectors corresponding to the same region of interest in the input image and the candidate image, and summing them for each region of interest. there is. As described above, the similarity may be calculated based on at least one of the cosine similarity, the L1 distance, and the L2 distance between feature vectors. The degree of similarity may be used when obtaining a first search result for global meta information and may be used when obtaining a second search result for local meta information.

In an embodiment of the present disclosure, a process in which the processor 110 obtains candidate images having a high degree of similarity based on similarities between a plurality of regional feature vectors of an input image and a plurality of regional feature vectors of one or more candidate images Illustratively describe it. In the continued example, it is assumed that the similarity is calculated based on the cosine similarity, and one or more candidate images include candidate image 'A' and candidate image 'B'. It is assumed that the region of interest also includes the first region of interest and the second region of interest. In the above embodiment, the cosine similarity between the local feature vector of the first ROI included in the input image and the local feature vector of the first ROI included in the candidate image 'A' may be 0.9. The cosine similarity between the local feature vector of the first ROI included in the input image and the regional feature vector of the first ROI included in the candidate image 'B' may be 0.8. Also, the cosine similarity between the local feature vector of the second ROI included in the input image and the regional feature vector of the second ROI included in the candidate image 'A' may be 0.88. The cosine similarity between the local feature vector of the second ROI included in the input image and the regional feature vector of the first ROI included in the candidate image 'B' may be 0.85. In this case, the processor 110 may calculate a similarity between the input image and the candidate image based on similarities existing for each of a plurality of regions of interest included in the candidate image. For example, the processor 110 may calculate the similarity between the input image and the candidate image based on an average, a maximum value, or a sum of similarities existing for each of a plurality of regions of interest included in the candidate image. In the above example, the similarity between the input image and the candidate image 'A' may be (0.9+0.88)/2 = 0.89. In the above example, the similarity between the input image and the candidate image 'B' may be (0.8+0.85)/2 = 0.825. The processor 110 may determine that 'A' is more similar to the input image based on the comparison result among candidate images 'A' and 'B'. The above example is only an example of a method for selecting one or more candidate images similar to an input image from among a plurality of candidate images based on the similarity between feature vectors, and the present disclosure is not limited thereto.

Hereinafter, with reference to FIG. 5, a process of obtaining a first search result and a second search result using similarity and matching will be described. 5 is a flowchart illustrating a process of providing, by a computing device, a second search result including a candidate image similar to an input image according to an embodiment of the present disclosure. The processor 110 according to the present disclosure may obtain a global feature analysis result and a local feature analysis result for the input image by inputting the input image to a learned analysis model including at least one neural network (510). The global feature analysis result may include at least one global feature vector generated in the process of calculating the global feature analysis model for the input image or global meta information included in a final output of the global feature analysis model for the input image. The global meta information may include at least one of a predicted bone age and a predicted age of the input image. The regional feature analysis result may include at least one regional feature vector generated in the process of calculating the regional feature analysis model for the input image or regional meta information included in the final output of the regional feature analysis model for the input image. Regional meta-information may include predicted partial bone grades or predicted partial bone maturity scores for each of one or more regions of interest included in the input image. The processor 110 may obtain a first search result including at least one candidate image similar to the input image based on the global feature analysis result (530). The processor 110 may obtain a second search result including at least one candidate image similar to the input image from the first search result based on the regional feature analysis result (550). In obtaining the first search result or the second search result, the processor 110 may compare at least one candidate image similar to the input image based on the degree of coincidence or similarity. The processor 110 may obtain a first search result by comparing the degree of correspondence between the input image and the at least one candidate image. In one embodiment, when the degree of agreement is calculated based on the predicted bone age included in the global meta information, the processor 110 selects one or more candidate images having the smallest difference from the predicted bone age of the input image among one or more pre-stored candidate images. may be obtained as a first search result. After that, the processor 110 may obtain a second search result by comparing similarities between the input image and the at least one candidate image. The second search result may be obtained from the first search result. For example, when the similarity is the cosine similarity, the processor 110 calculates the cosine similarity between the local feature vector of the input image and the local feature vector of one or more candidate images included in the second search result, and calculates the cosine similarity according to the result. Top N candidate images having a high similarity may be obtained as a second search result. N is an arbitrary natural number and may be predetermined by a user. The processor 110 may provide the second search result to a user or an external entity (570). The above-described example of the order and method of acquiring the first search result or the second search result is only an example and does not limit the present disclosure.

As described above, according to the present disclosure, the processor 110 may provide a search result to a user through filtering a total of two times. According to the present disclosure, the processor 110 may acquire and provide a second search result again through additional comparison after obtaining a first search result from all pre-stored candidate images. According to an embodiment of the present disclosure, the processor 110 obtains a first search result by comparing a plurality of matching degrees between global meta information of an input image and global meta information of each of a plurality of candidate images, A second search result may be obtained by comparing a plurality of similarities between a local feature vector of an input image and a plurality of local feature vectors of a plurality of candidate images included in the first search result. That is, the processor 110 may obtain the first search result by comparing the global meta information of the input image and candidate images. Comparison of global meta-information can be performed by comparing numerical values included in the output of the global feature analysis model, so it has an advantage of faster computation speed than feature vector similarity comparison. Then, the processor 110 may obtain a second search result by comparing similarities between local feature vectors of candidate images included in the first search result and local feature vectors of the input image. Since the similarity of regional feature vectors includes an abstract concept inherent in feature vectors, when the second search result is obtained through this, there is an advantage in that a candidate image having a similar bone shape and degree of bone maturity to the input image can be accurately selected. In summary, according to the present disclosure, the processor 110 quickly obtains a first search result by comparing the degree of agreement based on global meta information, and then compares the degree of similarity based on a local feature vector to obtain a more detailed comparison result within the first search result. By obtaining 2 search results, it has the effect of increasing the search performance while improving the search speed of the entire model.

6 is a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented. Although the present disclosure has been described above as being generally embodied by a computing device, those skilled in the art will understand that the present disclosure may be combined with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or may be implemented in hardware and software. It will be appreciated that it can be implemented as a combination.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. It will also be appreciated by those skilled in the art that the methods of the present disclosure may be used in single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which is It will be appreciated that other computer system configurations may be implemented, including those that may be operative in connection with one or more associated devices.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media. Computer readable storage media are volatile and nonvolatile media, transitory and non-transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Including all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 1100 implementing various aspects of the present disclosure is shown including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106 , to processing unit 1104 . Processing unit 1104 may be any of a variety of commercially available processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

System bus 1108 may be any of several types of bus structures that may additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112 . A basic input/output system (BIOS) is stored in non-volatile memory 1110, such as ROM, EPROM, or EEPROM, and is a basic set of information that helps transfer information between components within computer 1102, such as during startup. contains routines. RAM 1112 may also include high-speed RAM, such as static RAM, for caching data.

The computer 1102 may also include an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) - the internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes—a magnetic floppy disk drive (FDD) 1116 (e.g., for reading from or writing to a removable diskette 1118), and an optical disk drive 1120 (e.g., a CD-ROM) for reading disc 1122 or reading from or writing to other high capacity optical media such as DVDs). The hard disk drive 1114, magnetic disk drive 1116, and optical disk drive 1120 are connected to the system bus 1108 by a hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to The interface 1124 for external drive implementation includes at least one or both of USB (Universal Serial Bus) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the description of computer readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art can use zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other tangible media readable by the computer, such as the like, may also be used in the exemplary operating environment and any such media may include computer executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored on the drive and RAM 1112, including an operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 through one or more wired/wireless input devices, such as a keyboard 1138 and a pointing device such as a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 1104 through an input device interface 1142 that is connected to the system bus 1108, a parallel port, IEEE 1394 serial port, game port, USB port, IR interface, may be connected by other interfaces such as the like.

A monitor 1144 or other type of display device is also connected to the system bus 1108 through an interface such as a video adapter 1146. In addition to the monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148 via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, handheld computer, microprocessor-based entertainment device, peer device, or other common network node, and generally includes It includes many or all of the components described for, but for simplicity, only memory storage device 1150 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 1102 connects to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communications to LAN 1152, which also includes a wireless access point installed therein to communicate with wireless adapter 1156. When used in a WAN networking environment, computer 1102 may include a modem 1158, be connected to a communicating computing device on WAN 1154, or establish communications over WAN 1154, such as over the Internet. have other means. A modem 1158, which may be internal or external and a wired or wireless device, is connected to the system bus 1108 through a serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored on remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1102 is any wireless device or entity that is deployed and operating in wireless communication, eg, printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communication satellites, wireless detectable tags associated with It operates to communicate with arbitrary equipment or places and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) makes it possible to connect to the Internet without wires. Wi-Fi is a wireless technology, such as a cell phone, that allows such devices, eg, computers, to transmit and receive data both indoors and outdoors, i.e. anywhere within coverage of a base station. Wi-Fi networks use a radio technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band) .

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols and chips that may be referenced in the above description are voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields s or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein are electronic hardware, (for convenience) , may be implemented by various forms of program or design code (referred to herein as software) or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical disks (eg, CDs, DVDs, etc.), smart cards, and flash memory devices (eg, EEPROM, cards, sticks, key drives, etc.), but are not limited thereto. Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of example approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of this disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure, and the general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present disclosure is not to be limited to the embodiments presented herein, but is to be interpreted in the widest scope consistent with the principles and novel features presented herein.

Claims (11)

A content-based image retrieval method using a neural network performed by a computing device including at least one processor, comprising:
acquiring a global feature analysis result and a local feature analysis result for the input image by inputting an input image into a learned global feature analysis model and a local feature analysis model each including at least one neural network;
obtaining a first search result based on the global feature analysis result;
obtaining a second search result from the first search result based on the regional feature analysis result; and
providing the second search result;
including,
The first search result and the second search result include an image similar to the input image,
Content-based image retrieval method using neural network.
According to claim 1,
The image is a bone image that is a target for bone age reading,
The analysis model includes a common feature analysis model and further comprises at least one of a global feature analysis model or a regional feature analysis model.
Content-based image retrieval method using neural network.
According to claim 1,
The global feature analysis model included in the analysis model includes at least one pooling layer,
The regional feature analysis model included in the analysis model includes a region of interest detection module.
Content-based image retrieval method using neural network.
According to claim 1,
The global feature analysis result is,
at least one global feature vector generated in a process of calculating a global feature analysis model for the input image; or
Including global meta information included in the final output of the global feature analysis model for the input image,
Content-based image retrieval method using neural network.
According to claim 1,
As a result of the regional feature analysis,
at least one local feature vector generated in a process of calculating a local feature analysis model for the input image; or
Including regional meta information included in a final output of a regional feature analysis model for the input image,
Content-based image retrieval method using neural network.
According to claim 4 or 5,
The feature vector is
Generated based on at least one pooling method of the Max Pooling method, the Average Pooling method, or the Generalized Mean Pooling method,
Content-based image retrieval method using neural network.
According to claim 1,
The first search result is,
It is obtained by comparing a plurality of similarities calculated between the global feature vector of the input image and the global feature vector of each of a plurality of candidate images, or
Obtained by comparing a plurality of coincidences calculated between global meta information on the input image and global meta information on each of a plurality of candidate images,
Content-based image retrieval method using neural network.
According to claim 1,
The second search result is,
It is obtained by comparing a plurality of similarities calculated between a local feature vector of the input image and a local feature vector of each of a plurality of candidate images included in the first search result, or
Acquired by comparing a plurality of coincidences calculated between regional meta information on the input image and regional meta information on each of a plurality of candidate images included in the first search result.
Content-based image retrieval method using neural network.
According to claim 7 or 8,
The degree of similarity is
Calculated based on at least one of the cosine similarity between feature vectors, the L1 distance between feature vectors, or the L2 distance between feature vectors,
Content-based image retrieval method using neural network.
A computer program stored on a computer-readable storage medium, which, when executed in one or more processors, performs the following operations for content-based image retrieval using a neural network, the operations comprising:
acquiring a global feature analysis result and a local feature analysis result for the input image by inputting the input image into a learned global feature analysis model and a local feature analysis model each including at least one neural network;
obtaining a first search result based on the global feature analysis result;
obtaining a second search result from the first search result based on the regional feature analysis result; and
providing the second search result;
including,
The first search result and the second search result include an image similar to the input image,
A computer program stored on a computer-readable storage medium.
As a content-based image retrieval device using a neural network,
one or more processors;
a memory for storing an analysis model including one or more neural networks; and
network unit;
includes, and
The one or more processors,
inputting an input image into a learned global feature analysis model and a local feature analysis model including at least one neural network, respectively, to obtain a global feature analysis result and a local feature analysis result for the input image;
obtaining a first search result based on the global feature analysis result;
obtaining a second search result from the first search result based on the regional feature analysis result; and
providing the second search result;
The first search result and the second search result include an image similar to the input image,
A content-based image retrieval device using a neural network.

Images