JP6642132B2 - Image processing system, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing system and the like.

  2. Description of the Related Art In the current medical field, images of a diseased part taken at different times for the same patient are compared, and a disease state of the patient is diagnosed based on a difference between the compared images. Diagnosis by comparing images related to a diseased part of a patient is called comparative reading.

  In the task of comparative image interpretation, a similar case search is performed for a case under diagnosis by searching for similar cases in the past and referring to the search results. For example, a past case having a similar image feature of a lesion is retrieved from a past case database and displayed. As a conventional technology related to these tasks, there is, for example, a conventional technology 1.

JP-A-2004-5364

  However, the above-described related art has a problem that the similarity between a query case and another case cannot be calculated with high accuracy.

  FIG. 14 is a diagram for explaining the problem. For example, it is assumed that a case image, which is an image related to a diseased part, is captured and acquired at a plurality of timings for one diseased part of one patient. For example, in a CT image or an MRI image, a plurality of images are often collected at one timing for one diseased part, and therefore, FIG. 14 shows a state in which a plurality of images are overlapped as an image. . As a result, a set of images respectively taken at a plurality of timings is formed for one patient. In FIG. 14, for example, an encircling line 21 indicates a set of case images captured at five times. Similarly, it is assumed that case image sets 22 and 23, which are a time-series image group of a plurality of past imaging results, are accumulated for other patients or other diseased sites. When each of the case image sets 21 to 23 is referred to as a case image set group, in order to perform comparative image reading, a change in a diseased part of a patient to be diagnosed this time is stored in the case image set stored in the database 20. The physician determines which case image set in the group is similar, and compares the case image set with the past case similar to the case image set to determine the state of the patient this time.

  Here, if the case image sets 21 to 23 and the search target image set 25 to be searched, that is, the shooting intervals of the images included in the query case are the same, the comparison will be easy. However, in general, the timing at which an image of a diseased part is to be taken differs from patient to patient, and the number of times at which the image is taken also differs. In the example of FIG. 14, an image in which the search target image set 25 includes images captured at three times is present. Therefore, in order to be able to determine the similarity, a mechanism for absorbing the difference in the imaging timing for each patient is required.

  FIG. 15 is a diagram for explaining that the progress of the lesion, that is, the way of changing the lesion portion is non-linear. Assuming that each lesion image is represented as a feature vector in the feature space, the trajectory of the feature vector is generally non-linear. Therefore, the trajectory of each feature vector 31 in the linear space 30 is non-linear. For example, cancers tend to grow non-linearly, so that the larger they grow, the faster they grow.

  FIG. 16 is a diagram for explaining a problem. For example, when the feature vectors at times t1 to t3 are obtained by linear interpolation from the feature vectors at time t0 and time t4, each feature vector moves along the line segment 32. However, since the actual feature vector moves nonlinearly along the curved surface 33, a deviation occurs between each feature vector by linear interpolation and each actual feature vector.

  Since the tumor changes nonlinearly while taking various shapes, the high-dimensional curved surface formed by the feature vector is extremely complicated, and it is difficult to estimate the curve of the trajectory related to the feature vector of the image by learning.

  In one aspect, an object of the present invention is to provide an image processing system, an image processing method, and an image processing program that can accurately calculate the similarity between a query case and another case.

  In the first plan, the image processing system has a specifying unit, a generating unit, and a determining unit. The identification unit is a search target among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts, which are obtained by being imaged a plurality of times at an interval for each patient. When searching for an image set similar to the search target image set, the first image shot at the first time in the search target image set and the first image shot at a second time different from the first time For the set with the second image, at least one set including a set of images having feature amounts similar to the set of the feature amount of the first image and the feature amount of the second image is specified in the case image set group. I do. The generation unit has a shooting time of a first similar image having a feature amount similar to the feature amount of the first image and a feature amount similar to the feature amount of the second image included in the specified one or more sets. An interpolated image corresponding to an image captured at a time between the first image and the second image is obtained by using a feature amount of one or more images captured at a time between the capturing times of the second similar image. One or more feature quantities estimated to have are generated. The determination unit determines the similarity between the search target image set and one or more sets in the case image set group using the feature amount corresponding to the generated interpolation image.

  The similarity between the query case and another case can be calculated with high accuracy.

FIG. 1 is a functional block diagram illustrating the configuration of the image processing apparatus according to the present embodiment. FIG. 2 is a diagram illustrating an example of the data structure of the image table. FIG. 3 is a diagram for explaining the processing of the control unit. FIG. 4 is a diagram for explaining extraction of a feature vector for a query case. FIG. 5 is a diagram for explaining extraction of a time-series image group. FIG. 6 is a diagram illustrating an example of a time-series feature vector group similar to a query case. FIG. 7 is a diagram for explaining mapping by kernel principal component analysis. FIG. 8 is a diagram for explaining the processing of the determination unit. FIG. 9 is a diagram illustrating a process of determining a time to be interpolated. FIG. 10 is a diagram illustrating an example of a correspondence table between times. FIG. 11 is a diagram illustrating an example of a result of correspondence between times by the DP. FIG. 12 is a flowchart illustrating the processing procedure of the image processing apparatus according to the present embodiment. FIG. 13 is a diagram illustrating an example of a computer that executes an image processing program. FIG. 14 is a diagram for explaining a problem. FIG. 15 is a diagram for explaining a problem. FIG. 16 is a diagram for explaining a problem.

  Hereinafter, embodiments of an image processing system, an image processing method, and an image processing program disclosed in the present application will be described in detail with reference to the drawings. The present invention is not limited by the embodiment.

  FIG. 1 is a functional block diagram illustrating the configuration of the image processing apparatus according to the present embodiment. As shown in FIG. 1, the image processing apparatus 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

  The communication unit 110 is a processing unit that executes data communication with another device via a network or the like. For example, the communication unit 110 corresponds to a communication device.

  The input unit 120 is an input device that inputs various types of information to the image processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, and the like. The user operates the input unit 120 to select a case or the like of the query.

  The display unit 130 is a display device that displays a processing result of the control unit 150. The display unit 130 corresponds to a liquid crystal display, a touch panel, or the like. For example, the display unit 130 displays an image of another case similar to the case of the query searched by the control unit 150.

  The storage unit 140 has an image table 141. The storage unit 140 corresponds to a semiconductor memory device such as a random access memory (RAM), a read only memory (ROM), and a flash memory, and a storage device such as a hard disk drive (HDD).

  The image table 141 is a table that stores information on time-series images related to various cases. FIG. 2 is a diagram illustrating an example of the data structure of the image table. As shown in FIG. 2, the number of cases is recorded in the image table 141, and information in which a patient ID (Identification), the number of imaging times, and a data pointer are set in order is recorded for the number of cases. Have been. At the end of the data pointer, information in which the imaging time, the image data, the tumor position information, and the feature vector are set in order is recorded for the number of imaging times.

  For example, each image data associated with the same patient ID corresponds to the case image sets 21 to 23 described with reference to FIG. A set of case image sets of each patient ID corresponds to a case image set group. Further, as described later, a search target image set to be searched is selected from each case image set.

  The feature vector is a vector representing, for example, a feature of a tumor as a lesion site included in the image data. For example, the feature vector is information obtained by digitizing the size of a rectangle surrounding the tumor, the coordinates of the rectangle, the density and color of the tumor, and the like.

  The control unit 150 includes a specifying unit 151, a generating unit 152, and a determining unit 153. The control unit 150 corresponds to an integrated device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 150 corresponds to, for example, an electronic circuit such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit).

  FIG. 3 is a diagram for explaining the processing of the control unit. As illustrated in FIG. 3, the control unit 150 receives the feature vectors 40 and 50 of the time-series image corresponding to the symptom of the query (Step S10). For example, the feature vectors 40 and 50 exist in the feature space 30a.

  The control unit 150 compares the feature vectors 40 and 50 with the respective feature vectors of the image table 141 to obtain similar feature vectors (step S11). For example, feature vectors similar to the feature vector 40 are set as feature vectors 41, 42, 43, and 44. Feature vectors similar to the feature vector 50 are referred to as feature vectors 51, 52, 53, and 54.

  The control unit 150 extracts a time-series feature vector of a time-series image group to which each similar feature vector belongs (step S12). For example, the control unit 150 specifies a patient ID to which a similar feature vector belongs based on the image table 141, and extracts all the feature vectors belonging to the specified patient ID as a time-series vector group. In the example illustrated in step S12 of FIG. 3, the control unit 150 extracts the feature vector groups 61a, 62a, 63a, and 64a using the patient ID to which the feature vectors 41 to 44 and 51 to 54 belong as a key. This example shows an example in which four patient IDs are specified. By the step S12, in the high-dimensional curved surface 31 formed by the feature vector, it is possible to obtain the neighborhood 32 related to the trajectory curve of the query.

  The control unit 150 captures the nonlinear structure of the neighborhood 32 related to the trajectory curve of the query as a linear space 30b by kernel principal component analysis. In fact, by the kernel principal component analysis, the time-series feature vector groups 61a to 64a are regarded as the vector groups 61b to 64b of the linear space 30b via the high-dimensional space.

  The control unit 150 can estimate the approximate straight line 65b for the query case based on the time-series feature vector groups 61b to 64b (step S13). This approximate straight line 65b becomes an approximate curve 65a in the feature space 30a. The similarity between the approximate curve 65a and the other time-series feature vector groups 61a to 64a is calculated based on the calculation of the distance between the approximate straight line 65b on the linear space 30b and the time-series feature vector groups 61b to 64b. It becomes possible. Note that, in the example of FIG. 3, for simplification of description, an example is shown in which the feature vectors 40 and 50 of two images are received among a plurality of time-series images of one patient. The number may be any number as long as it is two or more.

  Subsequently, each processing of the specifying unit 151, the generating unit 152, and the determining unit 153 included in the control unit 150 will be described in order.

  When receiving the patient ID as a case of the query, the identifying unit 151 compares the received patient ID with the image table 141, and determines two feature vectors of each time-series image indicated by the pointer of the patient ID. Extract above. The number of feature vectors extracted by the specifying unit 151 need not be limited to two, and may be any number as long as it is two or more. For example, the feature vector extracted by the specifying unit 151 includes a feature vector of an image taken at the earliest time and a feature vector of an image taken at the last time in each time-series image.

FIG. 4 is a diagram for explaining extraction of a feature vector for a query case. It is assumed that the query case has n time-series images. As described above, when receiving the patient ID, the specifying unit 151 extracts the feature vectors of the n time-series images from the image table 141. As a result, a sequence of the obtained feature vectors is defined as “y _τ1 ,..., Y _τn ”. Here, the subscript τ _i of the feature vector indicates the observed time. As shown in FIG. 4, in the case of n = 2, the time-series images for cases of query, a time-series images of tau _1, becomes time-series images of tau _2, the feature vector, y _.tau.1, y _.tau.2 Becomes As described above, n may be 2 or more.

  The identification unit 151 extracts a time-series image group similar to the time-series image of the query case by calculating the distance between the feature vector related to the query case and all the feature vectors on the image table 141.

FIG. 5 is a diagram for explaining extraction of a time-series image group. The identifying unit 151 determines, for each of the n feature vectors y _τ1 ,..., Y _τn of the query case, among the feature vectors related to the individual images of the cases included in the image table 141, The feature vector up to is calculated. In the example illustrated in FIG. 5, the specifying unit 151 extracts the upper N-th feature vector x ^{τ1,1... Τ1, N} closest to the feature vector y _τ1 . The identification unit 151 extracts the top N-th feature vector x ^{τ2,1... Τ2, N} closest to the feature vector y _τ2 .

  The closest top N feature vectors to the vector related to the query case are defined as in equation (1).

The specifying unit 151 extracts all the feature vector groups of the time-series image of the case to which each feature vector of Expression (1) belongs. A feature vector group of the extracted time-series image is defined as in Expression (2). In equation ^(2), x τi, time-series images ^of cases ^j belongs to a _p ^{k i,} the time-series images consisting of ^j-number.

The identification unit 151 selects a feature vector of a time-series image similar to the feature vector of the query case from the feature vector group defined by Expression (2). For example, the specifying unit 151 ^selects, from among the vectors x _k ^{τi, J} defined by Expression (3), at least two vectors that are similar to the time-series feature vectors of the query cases.

  FIG. 6 is a diagram illustrating an example of a time-series feature vector group similar to a query case. In the example illustrated in FIG. 6, feature vector groups 61a to 64a similar to the query case are extracted. For example, in the feature vector group 61a, at least two feature vectors are similar to any of the feature vectors of the query case. Similarly to the feature vector group 61a, the feature vector groups 62a to 64a have at least two feature vectors similar to any of the feature vectors of the query case.

  In the following description, as an example, it is assumed that the specifying unit 151 extracts m time-series feature vector groups, and defines each feature vector group as shown in Expression (4).

  Subsequently, the process proceeds to the description of the generation unit 152. The generation unit 152 captures a non-linear structure formed by a time-series feature vector group similar to the query case as a linear space via a high-dimensional space by kernel principal component analysis. As a result, the time-series feature vector group is regarded as a vector group in a linear space. Then, it is possible to estimate the distance between the approximate curve for the query case and the time-series feature vector group based on the vector group in the linear space.

The generation unit 152 calculates a kernel matrix K using the kernel function k. Here, for convenience of explanation, a time-series feature vector group similar to the case of the query is represented as x ₁ ,..., X _q . Here, q is the number of time-series feature vector groups similar to the query. As the kernel function, for example, a Gaussian kernel represented by Expression (5) is used. T included in Expression (5) is a certain constant.

  The kernel matrix K calculated by the generator 152 according to the equation (5) is calculated as a q × q square matrix K as shown in the equation (6).

The generation unit 152 calculates, for the kernel matrix K, upper-order eigenvalues having larger r values and eigenvectors corresponding to the upper-order eigenvalues. For example, the generation unit 152 obtains an eigenvalue λ by solving the eigen equation (7) of the kernel matrix K as an equation of an unknown number λ. Further, each eigenvalue λ is substituted into the simultaneous equation (8) to obtain an eigenvector v corresponding to the eigenvalue λ. For example, the eigenvalues of the upper λ _1, ···, and λ _r, the corresponding eigenvectors _v 1, ···, and _{v r.}

  FIG. 7 is a diagram for explaining mapping by kernel principal component analysis. The generation unit 152 obtains a kernel matrix K of the feature vector groups 61a to 64a, and calculates an eigenvalue λ and an eigenvector v based on the kernel matrix K, so that the feature vector groups 61a to 64a are vectors in the linear space 30b. It can be grasped as groups 61b to 64b.

  Subsequently, the process proceeds to the description of the determination unit 153. The determination unit 153 estimates an approximate curve obtained by interpolating the feature vector for the time series vector group of the query case, and calculates the similarity between the approximate curve and another case based on the distance calculation on the linear section 30b. It is a processing unit for calculating.

  FIG. 8 is a diagram for explaining the processing of the determination unit. The determining unit 153 estimates the distance between the query case and another time-series feature vector group based on the feature vector groups 61b to 64b. Specifically, the determination unit 153 calculates the similarity between the approximate curve 65a and the other time-series feature vector groups 61a to 64a by calculating the distance between the approximate straight line 65b in the linear space 30b and the feature vector groups 61b to 64b. Is calculated based on the calculation of.

  Hereinafter, the processing of the determination unit 153 will be described in order. The determination unit 153 determines a time at which a feature vector is interpolated for a query case. For each time corresponding to the feature vector, the length between the times is set to be equal to or smaller than a certain value σ. When the length between adjacent times is larger than σ, the determination unit 153 equally divides the time between s so as to become smaller than σ.

FIG. 9 is a diagram illustrating a process of determining a time to be interpolated. For example, it is assumed that the imaging times τ ₁ , τ ₂ , τ ₃ , τ ₄ , τ ₅ , τ _{6 of} the query case and the predetermined time width σ are given as shown in FIG. The determination unit 153 determines that the interval between τ ₁ and τ _{2 and the} interval between τ ₄ and τ ₅ are larger than σ. The determination unit 153 divides τ ₁ and τ ₂ equally into s, and determines a time to be interpolated based on the length of the s equal when σ is equal to or less than σ for the first time. In the example shown in FIG. 9, times τ ₁₁ and τ ₁₂ to be interpolated are determined between τ ₁ and τ ₂ . The determination unit 153 adds times τ ₁₁ and τ ₁₂ between τ ₁ and τ ₂ .

Similarly, the determination unit 153 divides τ ₄ and τ ₅ equally into s, and determines a time to be interpolated based on the length of the s equal when σ or less becomes σ for the first time. In the example shown in FIG. 9, a time τ ₄₁ to be interpolated is determined between τ ₄ and τ ₅ . The determination unit 153 adds a time τ ₄₁ between τ ₄ and τ ₅ .

The determination unit 153 calculates the distance between the feature vector related to the time-series image of the case in the image table 141 and the feature vector of the case of the query by Dynamic Programming (DP). It is assumed that the feature vectors related to the time-series images of the cases in the image table 141 are _xt1 , ..., _xt1 . The judging unit 153 generates a correspondence table between times, and determines the distance between y _τi and x _tj or y _τik and x _tj at each grid point (τ _i , t _j ) or (τ _ik , t _j ) as follows. Calculate by formula. Note that the feature vector of the query case at the time τ _i is y _τi .

When τ _i is not the interpolation time, the determination unit 153 calculates a distance D (y _τi , x _tj ) between the feature vector y _τi and the feature vector at time t _j based on Expression (9).

When τ _ik is the interpolation time, the determining unit 153 calculates a distance D (y _τik , x _tj ) between the feature vector y _τik and the feature vector at time t _j based on Expression (10). Here, τ _ik is a point that internally divides τ _i and τ _{i + 1} into m: n.

With respect to Expressions (9) and (10), the determination unit 153 calculates the value of v (z) by Expression (11). z corresponds to the _y τi, y τik or _{x tj. V} 1 included in the formula (11), ··· _{v r} corresponds to the eigenvector of the kernel matrix K.

FIG. 10 is a diagram illustrating an example of a correspondence table between times. The vertical axis of the correspondence table 60 indicates the photographing time of the query case and the interpolated time. The horizontal axis of the correspondence table 60 indicates the imaging time of the case in the image table 141. Determination unit 153, the above-mentioned formula (9), based on equation (10), the distance D (y _{.tau.i, x tj)} or the distance D (y _{.tau.i, x tj)} is calculated. For example, the determination unit 153 creates a pair of τ _i and t _j or a pair of τ _ik and t _j and calculates the accumulated value of the distance D of each pair. Determination unit 153, while changing a set of the set or tau _ik and _{t j} of the tau _i and _{t j} comprehensively, the accumulated value is minimum, a set of the tau _i and _{t j,} or tau _ik and _{t j} Is searched based on DP.

  The specific calculation method of DP is described in the literature (Yoshimura et al., "Offline Matching Method for Japanese Signatures by Sequential Application of DP Matching Method," IEICE (D-II), Vol. J81-D-II No.10, pp.2259-2266, "2.2.2 DP matching processing" of October 1998). The determination unit 153 calculates DP based on the above calculation method.

FIG. 11 is a diagram illustrating an example of a result of correspondence between times by the DP. FIG. 11 shows a DP calculation result of a feature vector of a query case and a feature vector of a time-series image associated with a certain patient ID in the image table 141. In the example shown in FIG. 11, the DP, the feature vector y _.tau.1 time tau ₁ which case the query is captured, the time-series images of cases of image table 141 and a feature vector x _t3 of time t ₃ when taken Correlated. A feature vector y _.tau.2 cases of captured time tau ₂ queries, time-series images of cases of image table 141 are associated with the feature vector x _t6 of time t _6, which is photographed. Query feature vector y _.tau.3 cases of by time tau ₃ shooting, time-series images of cases of image table 141 are associated with the feature vector x _t6 of time t _6, which is photographed. The feature vector y _τ4 at the time τ ₄ at which the query case was captured is associated with the feature vector x _t7 at the time t ₇ at which the time-series image of the case in the image table 141 was captured. A feature vector y _.tau.5 time tau ₅ which cases the query is captured, the time-series images of cases of image table 141 are associated with the feature vector x _t9 of time t _9, which is photographed. The feature vector y _τ6 at the time τ ₆ at which the query case was captured is associated with the feature vector x _t9 at the time t ₉ at which the time-series image of the case in the image table 141 was captured.

Further, the interpolated feature vector y _{τ11 at} time τ ₁₁ is associated with the feature vector x _{t4 at} time t ₄ at which the time-series image of the case in the image table 141 was captured. A feature vector y _Tau12 of interpolated time tau _12, time-series images of cases of image table 141 are associated with the feature vector x _t5 of time t ₅ that is photographed. A feature vector y _Tau41 of interpolated time tau _41, time-series images of cases of image table 141 are associated with the feature vector x _t8 of the captured time t ₈ is.

  The determination unit 153 repeatedly executes the above-described processing similarly for the feature vector of the time-series image associated with another patient ID. The determination unit 153 determines the similarity based on the accumulated value of the distance D regarding the feature vector for each patient ID. For example, the determining unit 153 determines that the smaller the cumulative value is, the higher the similarity is.

  For example, based on the similarity between the feature vector of the query case and the feature vector of the time-series image associated with a certain patient ID in the image table 141, the determination unit 153 determines the top N most similarities. The sequence image is displayed on the display unit 130.

  Next, a processing procedure of the image processing apparatus 100 according to the present embodiment will be described. FIG. 12 is a flowchart illustrating the processing procedure of the image processing apparatus according to the present embodiment. As illustrated in FIG. 12, the specifying unit 151 of the image processing apparatus 100 receives a patient ID corresponding to a query case (Step S101). The specifying unit 151 extracts a feature vector for the query case (step S102). The specifying unit 151 extracts a feature vector of a time-series image group similar to the query case (step S103).

  The generation unit 152 of the image processing apparatus 100 calculates an eigenvector by performing kernel principal component analysis on a time-series feature vector group similar to the query case (step S104). The determination unit 153 of the image processing apparatus 100 calculates the similarity between the case of the query and the case of the image table 141 (Step S105). The determination unit 153 outputs a similar case (Step S106).

  Next, effects of the image processing apparatus 100 according to the present embodiment will be described. The image processing device 100 extracts a time-series image group similar to the time-series image of the case of the query, performs kernel principal component analysis on the feature vector of the extracted time-series image group, and dilates with the eigenvector obtained as a result. In the obtained linear space, the image of the query case is interpolated, and the similarity between the time series image of the query case and another time series image is obtained. For this reason, it is necessary to absorb the difference between the photographing time of the time series image of the query case and the photographing time of the other time series images, and to accurately specify the time series image similar to the time series image of the query case. Can be.

  Since the time interval for imaging the lesion portion varies depending on the patient, the time-series images of one case have different imaging time intervals between the images, and it is difficult to determine the similarity of the change in the lesion. However, as described above, since the difference in shooting time interval is interpolated using a similar time-series image group, the degree of similarity between the query case and another case can be determined.

  For example, when a set of time-series images related to a tumor part of a patient is given as an image table 141, when a time-series image of a query case is given, when the image is stored in the image table 141 regardless of the imaging time. The degree of similarity with the series image can be calculated with high accuracy. In particular, even when the progression of the lesion portion changes nonlinearly, the non-linear structure is estimated and collated, so that the collation can be performed with high accuracy.

  The image processing apparatus 100 captures a nonlinear structure formed by the time-series feature vector group as a linear space by mapping a time-series feature vector group similar to the query case onto a linear space based on kernel principal component analysis. This makes it possible to estimate an approximate curve for a query case based on a linear space based on kernel principal component analysis, thereby suppressing the occurrence of a deviation from an actual feature vector.

  The image processing device 100 specifies the similarity based on the distance between the feature vector relating to the time-series image of the case in the image table 141 mapped on the linear space and the feature vector of the case of the query. Therefore, the similarity between the case of the query and another case can be calculated with high accuracy.

  By the way, in the above-described embodiment, an example in which the image processing apparatus 100 includes the storage unit 140, the specifying unit 151, the generation unit 152, and the determination unit 153 has been described. However, the present invention is not limited to this. For example, a first server having the storage unit 140, a second server having the identification unit 151, a third server having the generation unit 152, and a fourth server having the determination unit 153 may be arranged on a network to form an image processing system. . Each of the first to fourth servers included in the image processing system performs data communication with each other, and performs the same processing as that of the image processing apparatus 100.

  Next, an example of a computer that executes an image processing program that realizes the same functions as the image processing apparatus 100 described in the above embodiment will be described. FIG. 13 is a diagram illustrating an example of a computer that executes an image processing program.

  As illustrated in FIG. 13, the computer 200 includes a CPU 201 that executes various types of arithmetic processing, an input device 202 that receives input of data from a user, and a display 203. In addition, the computer 200 includes a reading device 204 that reads a program or the like from a storage medium, and an interface device 205 that exchanges data with another computer via a network. Further, the computer 200 has a RAM 206 for temporarily storing various information, and a hard disk device 207. Each of the devices 201 to 207 is connected to the bus 208.

  The hard disk device 207 has a specific program 207a, a generation program 207b, and a determination program 207c. The CPU 201 reads out the specific program 207a, the generation program 207b, and the determination program 207c and expands them on the RAM 206.

  The specific program 207a functions as the specific process 206a. The generation program 207b functions as the generation process 206b. The judgment program 207c functions as a judgment process 206c.

  The processing of the specifying process 206a corresponds to the processing of the specifying unit 151. The processing of the generation process 206b corresponds to the processing of the generation unit 152. The processing of the determination process 206c corresponds to the processing of the determination unit 153.

  Note that the specific program 207a, the generation program 207b, and the determination program 207c need not always be stored in the hard disk device 207 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card inserted into the computer 200. Then, the computer 200 may read out and execute each of the programs 207a to 207c.

  Regarding the embodiment including each of the above examples, the following supplementary notes are further disclosed.

(Supplementary Note 1) Search results among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts, which are obtained by taking a plurality of images of each patient with a time interval. When searching for an image set similar to the search target image set, a first image shot at a first time in the search target image set is shot at a second time different from the first time. A set including an image set having a feature amount similar to the set of the feature amount of the first image and the feature amount of the second image in the case image set group. A specifying unit for specifying one or more of
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. A generation unit that generates one or more feature values estimated that
An image processing system comprising: a determination unit configured to determine a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image.

(Supplementary Note 2) The generation unit performs kernel principal component analysis on a feature amount similar to a set of a feature amount of the first image and a feature amount of the second image, thereby performing the similarity. The feature amount is mapped to a linear space, and one or more feature amounts estimated to be included in the interpolated image are generated based on the similar feature amounts in the linear space. Image processing system.

(Supplementary Note 3) The determining unit is configured to determine a distance between the similar feature amount in the linear space and a feature amount of the first image in the linear space, and the similar feature amount in the linear space and the second image. 3. The image processing system according to claim 2, wherein the similarity is determined based on a distance from the characteristic amount of the linear image and a distance between the similar characteristic amount on the linear space and the characteristic amount of the interpolation image. .

(Supplementary Note 4) For the search target image set, there are a third image and a fourth image photographed before and after the time corresponding to the interpolation image, and the time corresponding to the interpolation image is the time of the third image. When it is a time located at a point that internally divides the shooting time and the shooting time of the fourth image into m: n, the determining unit calculates a value obtained by multiplying the feature amount vector of the third image by n. And a vector obtained by adding a value obtained by multiplying the vector of the feature amount of the fourth image by m to the vector of the similar feature amount to obtain the similar feature amount on the linear space and the interpolation. 4. The image processing system according to claim 2, wherein a distance between the image and the feature amount is calculated.

(Supplementary Note 5) An image processing method executed by a computer,
A search target image set to be searched among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts obtained by capturing a plurality of times at a time for each patient. When searching for an image set similar to the first image, a first image shot at a first time in the search target image set and a second image shot at a second time different from the first time In the case image set group, at least one set including an image set having a feature amount similar to the set of the feature amount of the first image and the feature amount of the second image is specified. And
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. Generate one or more features that are estimated to be
An image processing method comprising: performing a process of determining a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image.

(Supplementary Note 6) The generating process performs the kernel principal component analysis on a feature amount similar to a set of a feature amount of the first image and a feature amount of the second image, thereby performing the similarity analysis. Appending to the linear space, generating one or more characteristic amounts estimated to have the interpolated image based on the similar characteristic amounts in the linear space. The image processing method described in the above.

(Supplementary Note 7) The process of determining the degree of similarity includes the step of: determining a distance between the similar feature amount in the linear space and a feature amount of the first image in the linear space; The method according to claim 6, wherein the similarity is determined based on a distance between the feature amount of the second image and a distance between the similar feature amount in the linear space and the feature amount of the interpolation image. Image processing method.

(Supplementary Note 8) Regarding the search target image set, there are a third image and a fourth image photographed before and after the time corresponding to the interpolation image, and the time corresponding to the interpolation image is the time of the third image. If it is a time located at a point that internally divides the shooting time and the shooting time of the fourth image into m: n, the process of determining the degree of similarity includes: Is subtracted from a vector obtained by adding a value obtained by multiplying the vector obtained by multiplying the vector of the feature amount of the fourth image by m to obtain the similar feature on the linear space. 8. The image processing method according to claim 6, wherein a distance between the amount and a feature amount of the interpolation image is calculated.

(Supplementary Note 9) An image processing program executed by a computer,
A search target image set to be searched among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts obtained by capturing a plurality of times at a time for each patient. When searching for an image set similar to the first image, a first image shot at a first time in the search target image set and a second image shot at a second time different from the first time In the case image set group, at least one set including an image set having a feature amount similar to the set of the feature amount of the first image and the set of the feature amount of the second image is included in the case image set group. Identify
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. Generate one or more features that are estimated to be
An image processing program for executing a process of determining a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image.

(Supplementary Note 10) The generating process includes performing a kernel principal component analysis on a feature amount similar to a set of a feature amount of the first image and a feature amount of the second image, thereby obtaining the similarity. Appending 9 to the linear space, and generating one or more characteristic amounts estimated to be included in the interpolated image based on the similar characteristic amounts in the linear space. Image processing program as described.

(Supplementary Note 11) The process of determining the degree of similarity includes the step of: determining a distance between the similar feature amount in the linear space and a feature amount of the first image in the linear space; The method according to claim 10, wherein the similarity is determined based on a distance from the feature amount of the second image and a distance between the similar feature amount in the linear space and the feature amount of the interpolation image. Image processing program.

(Supplementary Note 12) Regarding the search target image set, there are a third image and a fourth image photographed before and after the time corresponding to the interpolation image, and the time corresponding to the interpolation image is the time of the third image. If it is a time located at a point that internally divides the shooting time and the shooting time of the fourth image into m: n, the process of determining the degree of similarity includes: Is subtracted from a vector obtained by adding a value obtained by multiplying the vector obtained by multiplying the vector of the feature amount of the fourth image by m to obtain the similar feature on the linear space. 12. The image processing program according to claim 10, wherein a distance between the amount and a feature amount of the interpolation image is calculated.

REFERENCE SIGNS LIST 100 image processing device 140 storage unit 141 image table 150 control unit 151 specifying unit 152 generation unit 153 determination unit

Claims (6)

A search target image set to be searched among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts obtained by capturing a plurality of times at a time for each patient. When searching for an image set similar to the first image, a first image shot at a first time in the search target image set and a second image shot at a second time different from the first time In the case image set group, at least one set including an image set having a feature amount similar to the set of the feature amount of the first image and the feature amount of the second image is specified. A specific part to be
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. A generation unit that generates one or more feature values estimated that
An image processing system comprising: a determination unit configured to determine a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image.   The generator performs a kernel principal component analysis on a feature amount similar to a set of the feature amount of the first image and the feature amount of the second image, thereby linearizing the similar feature amount. The image processing according to claim 1, wherein one or more features that are estimated to be included in the interpolated image are generated based on the similar features in the linear space by mapping to the space. system.   The distance between the similar feature amount on the linear space and the feature amount of the first image on the linear space, the similar feature amount on the linear space and the feature amount of the second image. 3. The image processing system according to claim 2, wherein the similarity is determined based on a distance between the similar feature on the linear space and a feature of the interpolated image. 4.   For the search target image set, there is a third image and a fourth image photographed before and after the time corresponding to the interpolation image, and the time corresponding to the interpolation image is the photographing time of the third image, If the time is a point located at a point that internally divides the shooting time of the fourth image into m: n, the determination unit may calculate a value obtained by multiplying a vector of the feature amount of the third image by n, By subtracting the similar feature vector from the vector obtained by adding the value obtained by multiplying the feature vector of the four images by m, the similar feature in the linear space and the feature of the interpolated image are subtracted. The image processing system according to claim 2, wherein a distance from the image processing apparatus is calculated. An image processing method executed by a computer,
A search target image set to be searched among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts obtained by capturing a plurality of times at a time for each patient. When searching for an image set similar to the first image, a first image shot at a first time in the search target image set and a second image shot at a second time different from the first time In the case image set group, at least one set including an image set having a feature amount similar to the set of the feature amount of the first image and the feature amount of the second image is specified. And
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. Generate one or more features that are estimated to be
An image processing method comprising: performing a process of determining a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image. An image processing program executed by a computer,
A search target image set to be searched among case image set groups including individual case image sets that can be expressed by a plurality of types of feature amounts obtained by capturing a plurality of times at a time for each patient. When searching for an image set similar to the first image, a first image shot at a first time in the search target image set and a second image shot at a second time different from the first time In the case image set group, at least one set including an image set having a feature amount similar to the set of the feature amount of the first image and the set of the feature amount of the second image is included in the case image set group. Identify
The photographing time of a first similar image having a feature amount similar to the feature amount of the first image included in one or more specified sets, and a second similarity having a feature amount similar to the feature amount of the second image. An interpolated image corresponding to an image captured at a time between the first image and the second image is provided by using a feature amount of one or more images captured at a time between the image capturing times. Generate one or more features that are estimated to be
An image processing program for executing a process of determining a degree of similarity between the search target image set and one or more sets in a case image set group using a feature amount corresponding to the generated interpolation image.

Images