JP2011048481A - Image processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To retrieve a relevant image from among managed images without making it necessary to designate an area for paying attention about a processing object image. <P>SOLUTION: An image processing apparatus 10 is configured to learn a main classification rule to classify an image area into main classifications by using the characteristic information of a sample image area and the information of main classification designated about the sample image area as learning information and a sub-classification rule to classify the characteristic information of one or more image areas whose main classifications are matched into a plurality of sub-classifications, and to identify the main classification of each partial image area set about the processing object image based on the characteristic information extracted from the partial image area and the main classification rule, and to identify the sub-classification of each partial image area based on the main classifications of the partial image areas, the characteristic information extracted from the partial image areas and the sub-classification rule, and to generate the characteristic information based on the main classifications and the sub-classifications identified for each partial image area, and to retrieve the image having the characteristic information relevant to the generated characteristic information from among managed images. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program.

  An image obtained by a medical apparatus such as CT or MRI may be managed, and the managed image may be used for medical treatment or the like. For example, Patent Document 1 below discloses a technique for searching for an image having image characteristics similar to a new diagnosis target image from images that have been examined in the past.

JP 2001-155019 A

  One of the objects of the present invention is to provide an image processing apparatus and program capable of searching related images from images to be managed without having to specify a region of interest for the processing target image. .

  In order to achieve the above object, the invention of the image processing device according to claim 1 uses the feature information of the specimen image area and the information of the main classification designated for the specimen image area as learning information. Main classification learning means for learning main classification rules to be classified into main classifications, and sub-class learning for learning sub-classification rules for classifying feature information of one or a plurality of sample image regions having the same main classification into a plurality of sub-classes Means, setting means for setting a plurality of partial image areas for an image to be processed, main classification for each of the plurality of partial image areas, feature information extracted from the partial image areas, and the main classification rule Main classification identifying means for identifying based on the sub-classification for each of the plurality of partial image areas, based on the main classification of the partial image area, feature information extracted from the partial image area, and the sub-classification rule Identify A classification identifying unit; a feature information generating unit configured to generate feature information representing a feature of the image to be processed based on a main classification and a sub-classification identified for each of the plurality of partial image regions; and the feature information generation Search means for searching for management information in which one or a plurality of images and feature information are managed in association with each other, and an image having feature information related to the feature information generated by the means is included. .

  According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the feature information generating unit is characterized based on a main classification and a sub-classification identified for each of the plurality of partial image areas. A feature is generated by generating a vector and generating feature information representing the feature of the image to be processed based on a vector obtained by adding the feature vectors generated for the plurality of partial image regions.

  According to a third aspect of the present invention, in the image processing apparatus according to the first aspect, the plurality of partial image regions are classified into one or a plurality of image regions based on the main classification identified by the main classification identifying unit. The feature information generating means includes a position for each of the one or a plurality of image area groups, and a main classification and a sub classification identified for the partial image areas included in the image area group. Based on this, feature information representing the feature of the image to be processed is generated.

  According to a fourth aspect of the present invention, the computer classifies the image area into the main classification using the feature information of the sample image area and the information of the main classification designated for the sample image area as learning information. Main classification learning means for learning main classification rules, sub-classification learning means for learning sub-classification rules for classifying feature information of one or a plurality of image regions having the same main classification into a plurality of sub-classes, Setting means for setting a plurality of partial image areas for an image, and a main classification for identifying a main classification for each of the plurality of partial image areas based on feature information extracted from the partial image area and the main classification rule Sub-classification identification for identifying sub-classification for each of the plurality of partial image areas based on a main classification of the partial image area, feature information extracted from the partial image area, and the sub-classification rule And feature information generating means for generating feature information representing the characteristics of the image to be processed based on the stage and the main classification and sub-classification identified for each of the plurality of partial image areas, and the feature information generating means This is a program for causing an image having feature information related to the generated feature information to function as search means for searching from management information in which one or a plurality of images and feature information are managed in association with each other.

  According to the first and fourth aspects of the invention, it is possible to search for a related image from among the images to be managed without having to specify a region of interest for the processing target image.

  According to the second aspect of the present invention, the feature information of each partial image region can be reflected in the feature information of the image to be processed.

  According to the third aspect of the present invention, the position information of the image feature can be reflected in the feature information of the image to be processed.

It is a functional block diagram of an image processing apparatus. It is a figure which shows an example of a learning model. It is a figure which shows an example of a subclassification. It is a figure which shows an example of the process result with respect to a process target image. It is a figure which shows an example of management data. It is a flowchart of a learning process. It is a flowchart of a similar image search process. It is a figure explaining 2nd Embodiment. It is a figure which shows the comparison result of the similar image search result of 1 aspect of this invention, and a prior art.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments (hereinafter referred to as embodiments) for carrying out the invention will be described with reference to the drawings.

  FIG. 1 shows a functional block diagram of an image processing apparatus 10 according to the present embodiment. As shown in FIG. 1, the image processing apparatus 10 includes a sample image acquisition unit 12, a teacher information acquisition unit 14, a main classification learning unit 16, a sub classification learning unit 18, a processing target image acquisition unit 20, and a partial image region setting unit. 22, an image feature extraction unit 24, a main classification identification unit 26, a sub classification identification unit 28, a feature information generation unit 30, an image management unit 32, a related image search unit 34, and a search result presentation unit 36.

  The functions of the above-described units are such that a computer having a control unit such as a CPU, a storage unit such as a memory, and an input / output unit that transmits and receives data to and from an external device reads a program stored in a computer-readable information storage medium. It may be realized by executing. The program may be supplied to the image processing apparatus 10 which is a computer by an information storage medium, or may be supplied via a data communication unit such as the Internet.

[Learning process]
First, a function for realizing the learning process performed by the image processing apparatus 10 will be described.

  The sample image acquisition unit 12 acquires a sample image used for learning processing. In the present embodiment, an image region identified as a lesion site by a doctor is used as a specimen image. For example, in the case of a pulmonary disease, the sample image acquisition unit 12 samples an image including a part corresponding to a disease such as thickening of the bronchus, honeycomb cell, tear, small nodule, large nodule, ground glass, calcification, cystic disease, or the like. It may be acquired as an image.

  The teacher information acquisition unit 14 acquires main classification information corresponding to each of the sample images acquired by the sample image acquisition unit 12 as teacher information. In the present embodiment, the teacher information acquisition unit 14 acquires a disease (honeycomb lung file glass or the like) corresponding to each specimen image as a main classification (label information), and the main classification is an input device such as a keyboard or a mouse. It is good also as inputting (designating) using.

  The main classification learning unit 16 uses the feature information of each sample image acquired by the sample image acquisition unit 12 and the information of the main classification acquired by the teacher information acquisition unit 14 for each sample image as learning information. A main classification rule for classifying image feature information into main classifications is learned. The main classification learning unit 16 performs supervised learning using the feature information of the sample image and the main classification information of the sample image as a set. For example, a support vector machine may be used as the learning algorithm.

  Further, as image feature information, data obtained by averaging calculation results such as a vector having a luminance value of each pixel in the image area as a component, and a Gabor filter calculated in each pixel in the image area in the image area. And using various image feature data, such as texture features using luminance level co-occurrence matrix, and local feature quantities obtained by quantizing a predetermined filter result calculated at local points in the image area and histogramming them. Good.

  The sub-classification learning unit 18 learns a sub-classification rule for classifying an image into a plurality of sub-classifications based on the feature information of images with the same main classification. In the present embodiment, the sub-classification learning unit 18 classifies each piece of feature information of the sample image with the matching main classification into a plurality of clusters (sub-classifications) using an unsupervised clustering algorithm. As described above, the sub-classification in the present embodiment corresponds to each cluster when images classified into the main classification are further classified into clusters.

  The sub-classification learning unit 18 may use a k-means method or the like as the clustering algorithm if the feature information of the image is a continuous amount, or LDA (if the feature information of the image is quantized. A Latent Dirichret Allocation) model or the like may be used.

  FIG. 2 shows an example of a learning model when the LDA model 40 is adopted as the clustering algorithm. In FIG. 2, α and β are parameters, W is feature information (feature vector), Z is a cluster (subclass), and θ is a probability that W belongs to the cluster. In this LDA model 40, the values of α and β are learned on the basis of the feature information of each sample image having the same main classification, and the values of α and β are learned for each main classification. The details of the algorithm of the LDA model 40 are described in known literature (for example, DMBlei, AYNg, MIJordan, “Latent Dirichlet Allocation” JMLR, vol. 3, pp. 993-1022 (2003)). May be used.

  FIG. 3 shows an example of the sub-classification learned based on an image in which the main classification is the honeycomb lung. In the example shown in FIG. 3, the image is classified into five sub-classifications (clusters). However, the number of sub-classifications is merely an example and is not limited to this number. Then, the number of subclasses may be determined in advance for each main class, or the number of subclasses may be determined according to a predetermined criterion.

[Similar image search processing]
Next, a function for realizing a process for searching for an image similar to a diagnostic image to be processed, which is performed by the image processing apparatus 10, will be described.

  The processing target image acquisition unit 20 acquires an image to be processed (hereinafter referred to as a processing target image). In the present embodiment, the processing target image is an image to be diagnosed and serves as a search key. In the present embodiment, the processing target image may be an image obtained by imaging the lung by CT or MRI, for example.

  The partial image region setting unit 22 sets a plurality of partial image regions for the processing target image acquired by the processing target image acquisition unit 20. In this embodiment, the partial image area setting unit 22 divides the processing target image into rectangular areas of a predetermined size and sets each rectangular area as a partial image area. The partial image areas may be set so that the feature amounts are substantially equal, or the partial image areas may be set so as to partially overlap each other. Further, the partial image area setting unit 22 may set the partial image area for the entire processing target image, or may set the partial image area for a part thereof.

  The image feature extraction unit 24 extracts image features for each partial image region set by the partial image region setting unit 22. As the image feature, data of the same type as the image feature obtained for the sample image during the learning process is extracted from each partial image region.

  The main classification identifying unit 26 identifies the main classification of each partial image area based on the image features extracted for the partial image area. Specifically, the main classification identifying unit 26 identifies, based on the main classification rule learned by the main classification learning unit 16, which main classification the image feature extracted for the partial image region belongs to. .

  The sub-classification identification unit 28 identifies the sub-classification of each partial image area based on the image features extracted for each partial image area and the main classification identified by the main classification identification unit 26 for each partial image area. Is. Specifically, the sub-classification identifying unit 28 selects a sub-classification rule (model) based on the main classification identified for the partial image region, and extracts the partial image region based on the selected sub-classification rule. Identify subclasses of image features. For example, when the LDA model 40 is used, the identification result by the sub-classification identifying unit 28 is expressed as a corresponding probability to a plurality of sub-classes belonging to the main class.

  The feature information generation unit 30 generates feature information of the processing target image based on the identification result by the main classification identification unit 26 and the sub classification identification unit 28 for each partial image region set in the processing target image. is there. In the present embodiment, the feature vector of the partial image area is generated based on the identification result of the main classification and the sub classification for the partial image area.

  FIG. 4 shows an example of the processing result for the processing target image 50 (diagnostic image). As shown in FIG. 4, a lung cross section is captured in the processing target image 50, and a plurality of partial image areas are set in the processing target image by the partial image area setting unit 22. Then, the main classification is identified by the main classification identifying unit 26 for the image features extracted for each partial image region, and the sub-classification is further identified by the sub-classifying identification unit 28 based on the identification result.

  As shown in FIG. 4, when the main classification for the partial image area 52 is “normal” and the sub classification is “0.2, 0.1, 0.7”, the partial image area 52 The feature vector is represented as (0.2, 0.1, 0.7, 0, 0, 0, 0, 0, 0). In the subclass, the number of clusters is 3, and if each cluster is A, B, C, “α, β, γ” indicates the probability that the image features belong to A, B, C, respectively, and α + β + γ = 1. .

  When the main classification for the partial image area 54 is “ground glass” and the sub-classification is “0.5, 0.3, 0.2”, the feature vector for this partial image area 54 is (0, 0, 0, 0.5, 0.3, 0.2, 0, 0, 0). When the main classification for the partial image area 56 is “honeycomb lung” and the sub-classification is “0.3, 0.3, 0.4”, the feature vector for the partial image area 56 is (0 , 0, 0, 0, 0, 0, 0.3, 0.3, 0.4).

  The feature information generating unit 30 adds the feature vectors for each partial image region, and normalizes the added vector by the number of partial image regions (may be the area of the partial image region), whereby the feature information of the processing target image is obtained. Generate information.

  The image management unit 32 manages feature information, diagnosis results, and the like associated with images that have already been processed. In the present embodiment, the image management unit 32 includes a partial image region setting unit 22, an image feature extraction unit 24, a main classification identification unit 26, a sub classification identification unit 28, and feature information generation for images that have been diagnosed in the past. It is assumed that management data including feature information obtained as a result of processing by the unit 30 and link information to a report indicating a diagnosis result diagnosed by a doctor for the image is managed.

  FIG. 5 shows an example of management data. As shown in FIG. 5, the management data is configured as a table in which image feature information indicated by the image ID and link information to the diagnosis report are stored in association with the image ID.

  The related image search unit 34 searches for an image related to the processing target image from the images managed by the image management unit 32 based on the feature information (search key) generated for the processing target image. is there. In the present embodiment, the related image search unit 34 refers to management data managed by the image management unit 32, and includes feature information (feature vector) of the search key and feature information (feature vector) of each managed image. Each of the similarities may be calculated, and an image related to the image serving as a search key may be searched from the managed images based on the calculated similarity. The similarity may be calculated based on the Euclidean distance between the vectors, the Cullback library distance, the correlation coefficient, the histogram intersection, and the like. In addition, when using the distance between vectors for similarity, it is supposed that similarity is so high that the distance between vectors is small.

  The search result presentation unit 36 displays or prints out the image searched for by the related image search unit 34 and the diagnostic report associated with the image, and presents it to the user.

  Next, the flow of each of the learning process and the similar image search process in the image processing apparatus 10 according to the present embodiment will be described with reference to FIGS.

  FIG. 6 shows a flowchart of the learning process. As shown in FIG. 6, the image processing apparatus 10 acquires a specimen image corresponding to a lesion site (S101), and further acquires information on the main classification of the specimen image (S102). The image processing apparatus 10 learns a main classification rule for classifying image feature information into a main classification based on the acquired learning data (S103). Then, the image processing apparatus 10 learns a sub-classification rule for classifying each feature information into a plurality of sub-classifications (clusters) based on the feature information of each image having the same main classification (S104).

  FIG. 7 shows a flowchart of the similar image search process. As illustrated in FIG. 7, the image processing apparatus 10 acquires, for example, an image obtained by CT or MRI as a processing target image (S201), divides the acquired processing target image, and includes a plurality of processing target images. A partial image area is set (S202).

  The image processing apparatus 10 selects one unprocessed partial image area that has been set (S203), and extracts the image features of the selected partial image area (S204). The image processing apparatus 10 identifies the main classification to which the extracted image feature belongs based on the learned main classification rule (S205). Furthermore, the image processing apparatus 10 identifies the subclass to which the extracted image feature belongs based on the subclass rule corresponding to the identified main class (S206). The image processing apparatus 10 determines whether or not there is an unprocessed partial image area set as the processing target image (S207). If it is determined that there is, the process returns to S203 to perform subsequent processing. If it is determined that the process is not repeated, the feature information of the processing target image is generated based on the information of the main classification and the sub classification obtained for each partial image area (S208).

  The image processing apparatus 10 searches for an image having feature information similar to the feature information of the search key from images obtained in the past using the feature information of the generated processing target image as a search key (S209). The searched image and information linked to the image are presented to the user (S210).

  Next, a second embodiment of the present invention will be described with reference to FIG. In the second embodiment, the feature information generation process for the processing target image and the image feature information comparison process are different from the first embodiment in the other points, and are the same in other respects. Differences from the embodiment will be described.

  As shown in FIG. 8, after identifying the main classification for each partial image area set for the processing target image 50, the image processing apparatus 10 uses the same partial classification of adjacent partial image areas that match the main classification. Image region cluster. Then, the image processing apparatus 10 generates, for each image region cluster, feature information of the image region cluster based on the addition average of the image features of the partial image regions included in the image region cluster and the barycentric position of the image region cluster. .

  Specifically, with reference to the example illustrated in FIG. 8, the image processing apparatus 10 calculates the average of the barycentric coordinates (x, y) of the image area cluster and the image features of the partial image areas included in the image area cluster. An 11-dimensional vector obtained by concatenating the 9-dimensional feature vector generated by the above is generated as a feature vector of the image region cluster.

The image processing apparatus 10 generates a feature vector for each image area cluster, and calculates an evaluation value using the following EMD (Earth Mover's Distance) when comparing the similarity with other images. First, the cluster P = {(p ₁ , w _p1 ),..., (P _m , w _pm )} obtained for the first image, and the cluster Q = {(q ₁ ) obtained for the second image. , W _q1 ),..., (Q _n , w _qn )}, the correlation between the first image and the second image is to obtain a flow f _ij that minimizes the following equation (1). The cost calculated by (1) and normalized by the flow becomes EMD.

In this embodiment, p and q are feature vectors, w is the size of an image area cluster, and _dij is the distance between p _i and q _j .

  The image processing apparatus 10 according to the second embodiment calculates the distance between the processing target image and the past image to be managed based on the above formula (1), and the calculated one having the smallest calculated distance is similar. It is good also as searching as an image. The allowable range for the positional deviation of the image region cluster to be compared may be adjusted by changing the barycentric coordinates at the distance dij and the size (weight) of the image region cluster.

  In the image processing apparatus 10 according to the above-described second embodiment, information on the position distribution of the image feature is included in the image feature vector, so that the position information is added to the similar image search.

  FIG. 9 shows a similar image search result performed by the image processing apparatus 10 according to the first embodiment of the present invention for 100 case images, and a similar image search result based on a conventional feature amount of the entire image. An example of the result of sensory evaluation by a doctor was shown. As shown in FIG. 9, it can be said that the image searched as a similar image in one embodiment of the present invention is closer to the judgment of a doctor who is an expert than the image searched by the conventional similar image search.

  The present invention is not limited to the embodiment described above. For example, the present invention may be used for similar image retrieval other than medical use.

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 12 Sample image acquisition part, 14 Teacher information acquisition part, 16 Main classification learning part, 18 Sub classification learning part, 20 Process target image acquisition part, 22 Partial image area setting part, 24 Image feature extraction part, 26 Main classification identification unit, 28 Sub classification identification unit, 30 Feature information generation unit, 32 Image management unit, 34 Related image search unit, 36 Search result presentation unit, 40 LDA model, 50 Processing target image, 52, 54, 56 Partial image region.

Claims (4)

Main classification learning means for learning a main classification rule for classifying an image area into a main classification, using the feature information of the sample image area and the information of the main classification designated for the sample image area as learning information;
Sub-classification learning means for learning sub-classification rules for classifying the feature information of one or a plurality of sample image regions having the same main classification into a plurality of sub-classifications;
Setting means for setting a plurality of partial image areas for an image to be processed;
Main classification identifying means for identifying the main classification for each of the plurality of partial image areas based on the feature information extracted from the partial image area and the main classification rule;
Sub-classification means for identifying the sub-classification for each of the plurality of partial image areas based on a main classification of the partial image area, feature information extracted from the partial image area, and the sub-classification rule;
Feature information generating means for generating feature information representing the characteristics of the image to be processed based on the main classification and the sub-classification identified for each of the plurality of partial image areas;
Search means for searching for an image having feature information related to the feature information generated by the feature information generation means from management information in which one or a plurality of images and the feature information are managed in association with each other. An image processing apparatus. The feature information generation means generates a feature vector based on the main classification and the sub classification identified for each of the plurality of partial image areas, and adds the feature vectors generated for the plurality of partial image areas, respectively. The image processing apparatus according to claim 1, wherein feature information representing a feature of the image to be processed is generated based on a vector. Dividing the plurality of partial image areas into one or a plurality of image area groups based on the main classification identified by the main classification identifying means,
The feature information generation unit is configured to determine the processing target image based on the position of each of the one or more image region groups and the main classification and sub-classification identified for the partial image areas included in the image area group. The image processing apparatus according to claim 1, wherein feature information representing a feature is generated. Computer
Main classification learning means for learning a main classification rule for classifying an image area into a main classification, using the feature information of the sample image area and the information of the main classification designated for the sample image area as learning information;
Sub-classification learning means for learning sub-classification rules for classifying the feature information of one or a plurality of image regions having the same main classification into a plurality of sub-classes
Setting means for setting a plurality of partial image areas for an image to be processed;
Main classification identifying means for identifying the main classification for each of the plurality of partial image areas based on the feature information extracted from the partial image area and the main classification rule;
Sub-classification means for identifying the sub-classification for each of the plurality of partial image areas based on a main classification of the partial image area, feature information extracted from the partial image area, and the sub-classification rule;
Feature information generating means for generating feature information representing the characteristics of the image to be processed based on the main classification and the sub-classification identified for each of the plurality of partial image areas;
An image having feature information related to the feature information generated by the feature information generating means, and functioning as a search means for searching from management information managed by associating one or a plurality of images with the feature information. program.

Images