JP2010069208A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve visibility in observation of a plurality of time-serial images which are time-serially picked up. <P>SOLUTION: The device includes a standardizing section 51 to standardize each imaging position to a subject in the plurality of time-serial images which an imaging device time-serially imaged of the subject by identifying them as positions on the coordinates for an subject model to which the shape of the subject on each time-serial image matches to an subject model that is preliminarily set in accordance with the subject, and a display control section 41 to display each time-serial image with the imaging position standardized by the standardizing section 51 on a display section 20 to each imaging position as standardized in accordance with the coordinate system for the subject model. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, and an image processing program for observing a plurality of time-series images in which an imaging device such as a capsule endoscope images a subject such as a digestive organ in time series. It is.

  In recent years, a medical device has been realized that can image the intestinal digestive tract and perform endoscopy by swallowing it from the mouth like a capsule endoscope. After being introduced into the body, the capsule endoscope sequentially captures images while moving by a peristaltic motion of the digestive tract, and transmits the captured image data to a receiving device outside the body. The doctor moves the images stored in the receiving apparatus to a diagnostic workstation or the like and observes them. When the affected part is found by observation, medical treatment such as tissue sampling, hemostasis, and excision of the affected part is performed by inserting a medical treatment tool into the patient's body or incising the patient's body as necessary.

  In order to perform such medical treatment efficiently, it is useful to improve the visibility of the image when observing image data captured by the capsule endoscope. In general, the movement of the capsule endoscope in the digestive organs of the body is unstable, and the capsule endoscope moves back and forth in the body due to the influence of the peristaltic movement etc. that occurs in the digestive organs, and repeats imaging around the same part Often happens. In addition, the capsule endoscope may show a behavior such as staying in one place for a long time. Due to such behavior of the capsule endoscope, for example, images near the same part are repeatedly displayed with periodicity, and the visibility of the image is reduced, resulting in a decrease in the efficiency of image observation of the observer by a doctor or the like. Will be.

  As a technique for improving the visibility of image observation for diagnosis by a doctor, Patent Document 1 compares similarities between a plurality of images captured by a capsule endoscope, and as a result of comparison, similarities are determined. A method of integrating images determined to have is used.

  Further, Non-Patent Document 1 proposes a method for improving the visibility of an image by assuming a contracted position of the small intestine as a subject model, obtaining the position by edge analysis or the like, and setting it as the center of the image.

Japanese Patent No. 3639825 “Detection of contraction position of small intestine using capsule endoscope”, IEICE Technical Report, IEICE, no.MI2005-110, pp. 9-12, Okinawa, January, 2006

  However, Patent Document 1 merely compares similarities between a plurality of captured images, and no knowledge about the captured subject is used in the image integration process. Therefore, in the method of comparing only similar points between a plurality of images taken as in Patent Document 1, it often happens that it is difficult to find similar points between images. As a result, the integration processing of a plurality of images becomes difficult, and eventually the visibility is not sufficiently improved during image observation.

  In Non-Patent Document 1, the contraction position of the small intestine is applied to the image as a subject model and the image is normalized. However, since the subject model is limited to the small intestine, it is difficult to deal with other internal organs. There are drawbacks. Thereby, the improvement of the visibility becomes insufficient when observing a plurality of images taken in time series for diagnosis by a doctor or the like.

  The present invention has been made in view of the above, and an image processing apparatus, an image processing method, and an image processing apparatus capable of improving the visibility for observing a plurality of time-series images obtained by imaging an object in time series. An object is to provide an image processing program.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to an aspect of the present invention provides a plurality of time-series images obtained by capturing an image of a subject in time series with respect to the subject. A normalization unit that specifies the position of the subject model that matches the subject shape of each time-series image with respect to a subject model set in advance according to the subject, and the standard A display control unit configured to display each time-series image in which each imaging position is standardized by the standardization unit at a standardized imaging position according to the coordinate system of the subject model and to display on the display unit. Is.

  According to the image processing apparatus according to this aspect, the coordinates of the common subject model set in advance according to the subject imaged by the imaging device are set as the respective imaging positions of the plurality of time-series images captured by the imaging device. Since each time-series image that is normalized at the upper position and each imaging position is normalized is arranged on each normalized imaging position according to the coordinate system of the subject model and displayed on the display unit, a common subject Through the model, it is possible to display images with substantial alignment between each time series image, and the visibility of a plurality of time series images is improved. At this time, a subject model set in advance according to the subject is used. Thus, it is possible to cope with a subject to be imaged without being limited to a specific subject model.

  The image processing method according to another aspect of the present invention is such that an imaging position of each of a plurality of time-series images obtained by capturing an image of a subject in time series with respect to the subject is preset according to the subject. A normalization step for normalizing by specifying the position on the coordinates of the subject model to which the subject shape of each time-series image matches the subject model, and each imaging position is normalized by the normalization step And a display control step of displaying each time-series image at a standardized imaging position according to the coordinate system of the subject model and displaying the image on the display unit.

  An image processing program according to another aspect of the present invention provides a computer included in an image processing device, wherein each imaging position with respect to the subject of a plurality of time-series images in which the imaging device images the subject in time series are A normalization procedure for normalization by specifying a subject model that is preset according to the subject at a coordinate position of the subject model that matches the subject shape of each time-series image, and the normalization procedure A display control procedure in which each time-series image in which each imaging position is standardized is arranged at each standardized imaging position according to the coordinate system of the subject model and displayed on a display unit. is there.

  ADVANTAGE OF THE INVENTION According to this invention, the visibility for observation of the several time series image which the imaging device imaged the to-be-photographed object in time series can be improved.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of an image processing device, an image processing method, and an image processing program according to the invention will be described with reference to the drawings. In the present embodiment, a capsule endoscope, which is an imaging device introduced into a subject, captures a predetermined series of time series images of about 60,000 images taken in time series using a body lumen organ as a subject. An image processing apparatus that performs processing and displays it on the display unit will be described as an example. In the description of the drawings, the same portions or corresponding portions are denoted by the same reference numerals.

  FIG. 1 is a schematic diagram illustrating a configuration example of an in-vivo image information acquisition system including an image processing apparatus according to an embodiment of the present invention. In the present embodiment, a capsule endoscope is illustrated as an imaging apparatus that is introduced into a subject and captures an image using a body lumen organ as a subject. The capsule endoscope has an imaging function, a wireless function, an illumination function that illuminates the imaging part, etc., and after being swallowed from the subject's mouth for examination, until it is discharged naturally or the battery runs out During this time, intraluminal images of the digestive tract (internal lumen organ) of the esophagus, stomach, small intestine, and large intestine are sequentially captured and wirelessly transmitted outside the body.

  As shown in FIG. 1, the in-vivo image information acquisition system of the present embodiment is a capsule endoscope 3 that is an imaging device that is introduced into a subject 1 and images a time-series image of a digestive organ in the body. Reception antenna groups A1 to An that receive image data of time-sequential images wirelessly transmitted from the capsule endoscope 3; reception that processes image data received by these reception antenna groups A1 to An and accumulates image data An apparatus 4 and a workstation 100 that processes image data stored in the receiving apparatus 4 and realizes the image processing apparatus of the present embodiment are provided. For example, a removable portable recording medium 5 is used for transferring image data between the receiving device 4 and the workstation 100.

  FIG. 2 is a block diagram illustrating a schematic configuration of the workstation 100 that realizes the image processing apparatus according to the present embodiment. The workstation 100 is configured by, for example, a general-purpose computer, and includes an input unit 10 that receives input of various types of information, a display unit 20 that displays and outputs various types of information such as processing results, and various types of information such as image data of time-series images. A storage unit 30 for storage and a control unit 40 for controlling processing and operation of each unit of the workstation 100 are provided. Further, the workstation 100 includes an image processing unit 50 that processes a plurality of time-series images in which the capsule endoscope 3 images the in-vivo digestive organs in time series.

  The input unit 10 is realized by a keyboard, a mouse, a data communication interface, and the like, and accepts input of various information directly from an observer such as a doctor, and is digested in the body from a portable recording medium 5 such as various memory cards, CDs, and DVDs. Accepts input of image data of a time-series image obtained by imaging an organ. The input unit 10 includes an operation receiving unit 11 that receives an operation for selecting an image displayed on the display unit 20.

  The display unit 20 is realized by a display device such as an LCD or an ELD, and displays an image processed by the image processing unit 50. The display unit 20 displays a GUI (Graphical User Interface) screen that requests the observer to input various processing information. The storage unit 30 is realized by a hard disk, a ROM, a RAM, and the like, and stores various processing programs such as an image processing program to be executed by the control unit 40 and various information such as processing results.

  The control unit 40 is realized by a CPU and executes various processing programs stored in the storage unit 30 to control processes and operations of each unit included in the workstation 100. In particular, the control unit 40 causes each unit of the image processing unit 50 to execute the image processing procedure according to the present embodiment by executing an image processing program. In addition, the control unit 40 includes a display control unit 41 that performs control for causing the display unit 20 to display a time-series image that has been subjected to predetermined image processing by the image processing unit 50. In particular, the display control unit 41 of the present embodiment displays a time-series image that has been standardized using a predetermined subject model in which the imaging position with respect to the in-body digestive organ (subject) is set in advance by the processing of the standardization unit described later. Then, control is performed so that the image is displayed on the display unit 20 at the imaging position on the coordinate system of the subject model.

  The image processing unit 50 includes a normalization unit 51, a layer creation unit 52, a similarity calculation unit 53, a rearrangement unit 54, an alignment unit 55, and an image quality adjustment unit 56, and the capsule endoscope 3 determines the intestinal digestive organ. Predetermined image processing as described below is performed on a plurality of time-series images captured in time series.

  First, an outline of processing contents of each unit of the image processing unit 50 will be described. The normalization unit 51 captures each imaging position with respect to the digestive organ (subject) in the body with respect to a plurality of time-series images captured in time series by the capsule endoscope 3 and stored in the storage unit 30. A normalization process is executed by specifying the subject model in a time series image that matches the subject model in advance with respect to a subject model set in advance according to the digestive organs in the body. Thus, basically, the display control unit 41 arranges each time-series image whose imaging positions are normalized by the normalization unit 51 at each normalized imaging position according to the coordinate system of the subject model. Thus, it is possible to display on the display unit 20.

  In addition, the layer creation unit 52 displays each time-series image superimposed on the display unit 20 as a layer having a hierarchical structure by the display control unit 41 so that each imaging position is normalized by the normalization unit 51. This is for creating layer data for displaying a time-series image as a layer having a hierarchical structure of a coordinate system corresponding to the coordinate system of the subject model. For this reason, the layer creation unit 52 acquires the standardized information and the data of the time series image regarding the imaging position of each time series image standardized from the standardization unit 51, and coordinates of the subject model for each layer of the hierarchical structure. A process of creating layer data in which each time-series image is arranged at a position according to the normalized information on the coordinate system corresponding to the system is executed. The created layer data also includes a layer number indicating the layer order to be superimposed, and is basically set in the time series order (imaging order) of the time series images.

  Furthermore, the similarity calculation unit 53 acquires layer data for each layer created by the layer creation unit 52, and executes a process of calculating the similarity between time-series images between neighboring layers in the hierarchical structure. Further, the rearrangement unit 54 acquires the similarity information calculated by the similarity calculation unit 53 and the layer data of the time-series image so that the time-series images are arranged in descending order of similarity based on the acquired similarity. In addition, processing for rearranging the layer order of the time-series images in the hierarchical structure is executed.

  In addition, the alignment unit 55 acquires layer data of the time-series image whose layer order has been rearranged by the rearrangement unit 54, and converts each time-series image to be processed into a neighboring layer in the hierarchical structure. A process of aligning the time series images with other time series images so that the similarity between the time series images is maximized is executed.

  Further, the image quality adjustment unit 56 acquires layer data of the time-series images that have been subjected to the alignment process by the alignment unit 55, and connects the joints between the time-series images of neighboring layers that are superimposed and displayed according to the hierarchical structure. A process of adjusting the image quality to eliminate the uncomfortable feeling is executed.

  The layer data of a plurality of time-series images adjusted in image quality by the image quality adjustment unit 56 is output to the display unit 20 by the display control unit 41, and the plurality of time-series images are coordinated corresponding to the coordinate system of the subject model. Overlaid as a hierarchical layer of the system. At this time, an observer such as a doctor can observe a time-series image in which the arrangement on the display is shaped by the image processing by the image processing unit 50 through the display by the display unit 20. The display state on the display unit 20 can be freely changed by an operation on the GUI screen via the operation receiving unit 11. Further, the layer data of the time-series image output from the image quality adjustment unit 56 can be stored in the storage unit 30. The layer data of the time-series image stored in the storage unit 30 can be displayed on the display unit 20. Therefore, the shaped time-series image after image processing can be observed not only at the time of processing but also at an arbitrary time point thereafter.

  Next, details of processing executed by each unit of the image processing unit 50 will be described. First, normalization of the imaging position with respect to the subject of the time-series image by the normalization unit 51 will be described. The standardization unit 51 uses a common subject model set in advance according to the subject to be imaged in order to normalize the imaging position of each time-series image with respect to the subject. In the present embodiment, a circular model that includes a circular subject that is dark at the center and that becomes brighter as it is away from the center is represented in advance as a subject model, and is expressed by a predetermined mathematical expression having the center as the origin of the xy coordinate system. (See Fig. 3-3). This is because the field of application of the present embodiment is the capsule endoscope 3, and the shape of the internal organ as a subject imaged by the capsule endoscope 3 has a substantially tubular shape. That is, the center of the circular shape assumes the deep part of the circular tube of the internal organ, and the outer part of the circular shape represents the side wall portion close to the imaging position of the circular tube of the internal organ.

  The normalization unit 51 indicates which position (imaging position) of each of the time-series images captured by the capsule endoscope 3 is captured in each time-series image as a circular model (subject model). Is specified by applying. That is, the imaging position of each time series image is normalized by specifying the imaging position of each time series image by the position on the common circular model. In the present embodiment, the frame of the time-series image inside the subject 1 captured in time series by the capsule endoscope 3 is maintained in a state that is substantially horizontal and vertical with respect to the x and y coordinate axes. It shall be.

  Here, when a circular model (subject model) having a circular shape is expressed by a mathematical expression, the following expression (1) is obtained.

(Equation 1)
F (x, y) = d {(x−a) ² + (y−b) ² } + c ²

F (x, y) in equation (1) represents a model of luminance information at coordinates (x, y) indicating the pixel position of the image in the xy coordinate system of the subject, and the parameters of the circular model (subject model) are a, b, c, d. In order to solve these parameters a, b, c, and d, in an actually captured image, a set of luminance information and pixel position (F ₁ , x ₁ , y ₁ ), (F ₂ , x ₂ , y ₂ ), (F ₃ , x ₃ , y ₃ ), (F ₄ , x ₄ , y ₄ ) are acquired as pixel information and applied to the expression (1), It is possible to obtain the four relational expressions shown, and obtain the parameters a, b, c, and d of the circular model (subject model) using these four expressions.

(Equation 2)
F ₁ (x ₁ , y ₁ ) = d {(x ₁ −a) ² + (y ₁ −b) ² } + c ²
F ₂ (x ₂ , y ₂ ) = d {(x ₂ −a) ² + (y ₂ −b) ² } + c ²
F ₃ (x ₃ , y ₃ ) = d {(x ₃ −a) ² + (y ₃ −b) ² } + c ²
F ₄ (x ₄ , y ₄ ) = d {(x ₄ −a) ² + (y ₄ −b) ² } + c ²

  However, in reality, estimation may not be stably performed with only four pieces of pixel information due to the influence of noise, etc., so an equation is created using pixel information for more points, and a least square criterion is used. It is desirable to obtain parameters a, b, c, and d. Alternatively, instead of pixel information, processing such as increasing the luminance information at the corresponding position by smoothing pixels in the vicinity of the target pixel may be performed, and the parameters a, b, c, and d may be estimated more robustly. Good.

  Further, in the present embodiment, a circular model whose center is dark and becomes brighter as it is away from the center is used as the subject model, but actually, if a model specific to the subject to be imaged is used, Any subject model may be used. For example, in the present embodiment, a circular model with a single circular shape is used. However, a circular model obtained by combining a plurality of circular shapes may be used as a more detailed subject model. In addition, for example, the subject model may be changed to an elliptical shape in order to more accurately represent the circular shape.

By calculating the parameters a, b, c, and d of the circular model in this way, it is possible to know at which position (imaging position) on the coordinate system of the circular model the actually captured time-series image is captured. For example, when the parameters a, b, c, and d of the circular model are (A, B, C, D), the time series image is (A, B) from the center (origin) in the xy coordinate system of the circular model. It can be seen that the image is taken at an image pickup position where the luminance value is C ² and the luminance value increases at a rate of Dr ^{2 as} the distance r from the position (A, B) increases. That is, the normalization unit 51 calculates these parameters a, b, c, and d using pixel information of a plurality of points in each time series image, so that the subject shape of the time series image is circular on the xy coordinate system. A coordinate position suitable for the model can be specified, and the specified coordinate position is normalized as an imaging position of the time-series image.

The layer creation unit 52 creates layer data for superimposing and displaying as a layer of a hierarchical structure a time-series image whose imaging position is normalized by the coordinate position on the circular model (subject model). Here, each layer of the hierarchical structure for creating the layer data has a coordinate system corresponding to the xy coordinate system of the circular model. That is, the origin (the center of the circle) of the xy coordinate system of the circular model expressed by the equation (1) is set as the center position of each layer, the luminance value of the center position is set to 0, and the distance r from the center position is as follows: luminance value is a coordinate system that will increase at a rate of r ^2. Then, the layer creation unit 52 creates layer data for each layer by arranging each time-series image at the coordinate position according to the normalization information obtained from the normalization unit 51 on such a coordinate system.

For example, in the above example, the time series image in which the parameters (A, B, C, D) are obtained according to the circular model, the (A, B) coordinate position of the time series image is arranged at the center position on the corresponding layer. Then, an offset of −C ² is added to the entire time series image, and the luminance change is adjusted by 1 / D so that the luminance value increases at a rate of r ^{2 as} the distance r increases on the layer. In this way, by arranging the corresponding time-series images at predetermined positions on the corresponding layer, each time-series image is layered in a hierarchical structure for overlay display.

  3A to 3D are explanatory diagrams illustrating an example of time-series image normalization processing and layering processing. The five images 1 to 5 illustrated in FIG. 3A are examples of time-series images actually captured by the capsule endoscope 3 in time-series order. Also, the five images 1 to 5 shown in FIG. 3-2 are expressed by a circular model according to the parameters calculated for the five time-series images 1 to 5 shown in FIG. 3-1. Further, FIG. 3C illustrates a state in which the result illustrated in FIG. 3B is arranged in each of the five layers according to the normalization information on the coordinate system corresponding to the xy coordinate system of the circular model. In FIG. 3C, a portion where the center is black and gradually becomes white as the distance from the center indicates a circular model as a subject model. FIG. 3-4 shows the five time-series images shown in FIG. 3A actually captured on each of the five layers according to the normalization information on the coordinate system corresponding to the xy coordinate system of the circular model. The result of arranging and displaying on the display unit 20 is shown.

  For example, the time-series image indicated by 1 in FIG. 3A becomes a modeled image indicated by 1 in FIG. 3-2 when expressed by the circular model of the equation (1). The modeled image as shown in 1 in FIG. 3-2 has a position shown by 1 on the coordinate system of the circular model shown in FIG. 3-3 according to the parameter calculated from the equation (1). The subject shape is specified as a suitable position. In this way, the normalization unit 51 identifies the imaging position of the time-series image indicated by 1 in FIG. 3A as the position indicated by 1 in FIG. 3C if expressed using a circular model. To standardize. The same applies to other time-series images. That is, by using a common circular model, each time-series image is aligned with the circular model.

  At this time, since only one part of the subject is displayed when only one time-series image is displayed, each layer is displayed in a hierarchical structure as shown in FIG. By arranging and superimposing the images on each layer according to the imaging position, the entire image of the subject can be observed.

  In FIG. 3, since the number of time-series images used is small in order to make it easy to understand the overlapping state of the time-series images arranged on each layer, the display unit 20 shown in FIG. In the display example, a part of the shape of the subject model is visible as a background. However, in reality, it is processed using a larger number of time-series images, and the time-series images on these multiple layers are superimposed and displayed so as to cover the entire area on the subject model, so the background subject model The shape of is almost invisible. Of course, in the display state where the subject model can be seen, the CG (computer graphic) of the subject model may be displayed as it is, or may be displayed in gray.

  In this way, the layer creation unit 52 creates layer data for arranging a plurality of time-series images on the layer, and when displaying on the display unit 20, the plurality of time-series images are superimposed and displayed as hierarchical layers. As a result, only a part of the imaged part of the subject is displayed with only individual time-series images, whereas a plurality of time-series images are displayed simultaneously according to the layer of the hierarchical structure. By substantially expanding to the periphery of a single time-series image, the entire subject is displayed and the lack of information that cannot be displayed with only one time-series image is compensated for, and the visibility is greatly improved. Will be.

Next, similarity calculation processing by the similarity calculation unit 53 and rearrangement processing by the rearrangement unit 54 will be described. First, N sheets of a predetermined range of layers I are acquired in the order of overlapping (I ₁ , I ₂ ,..., I _N ). Then, the similarity calculation unit 53 calculates the similarity between the time-series images between the layer I ₁ located at the head and the target, and the remaining (N−1) layers, respectively. based on the similarity of information, rearranging section 54 performs a process to bring the next target layer I ₁ the most similar layout for layer I ₁ as a target. For example, when the layer having the highest similarity to the target layer I ₁ is I _k , the N layers are arranged in the order of (I ₁ , I _k , I ₂ ,..., I _N ). Be replaced. However, in the processing here, if similarity to the target layer I ₁ is smaller than a predetermined threshold deemed that did not exist the target similar, does not perform the rearrangement process of the layer sequence, and so on May be.

For example, as shown in FIG. 4, a region present on the front side of the target layer I _k, to the target layer I _k and each share in the hierarchical structure of the layer L', M', layer I _l having _N', I _m, and _{I n} is three. Therefore, the similarity calculation unit 53 calculates the similarity between the respective time series images for the region L ′ shared between the target layer I _k and the layer I _l, and the target layer I _k and the layer I _m. Similarity between each time-series image for the region M ′ shared with each other and similarity between each time-series image for the region _N ′ shared between the target layer I _k and the layer In are also calculated. To do. Results of calculation of such similarity, the similarity is the layer _{I l,} I m, if order larger of _{I n,} the rearranging unit 54, the layer _I l to the target layer _{I k,} I m, rearrangement _{I n} without the layer order _{is, I k, I l, I} m, and is maintained at _{I n.} On the other hand the calculated degree of similarity is the layer _{I m,} I l, if order larger of _{I n,} rearranging unit 54, as come Layer _{I m} to the next target layer _{I k,} the layer I to the target layer _{I k l,} rearranges the _I m, _{I n,} the layer sequence _{I k, I m, I l} , made to be _{I n.}

  As a similarity evaluation method, a difference error value between layer images may be used as an evaluation index. Typical indicators of the difference error value include SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), and normalized cross-correlation. Taking SSD as an example

It becomes. Here, Ω _k is a common existence area where the pixel values of the target layer I ₁ and the layer I _k are both present. This expression evaluates SSD, rearranges SSD the lowest layer to the layer position immediately after the target layer I _1. At this time, there may occur a case where the common existence region Ω _k is small and the similarity value such as SSD cannot be evaluated properly. Considering such a case, for example, when the common existence region Ω _k for the target layer is smaller than a predetermined threshold value, a measure such as removal from the target of similarity evaluation may be taken.

As another similarity evaluation method, the common existence region Ω _k between the layers may be used as an evaluation index. Because the inter were evaluated large common existence region Omega _k layer, it is often the imaging position is almost identical, as a result, since the time-series images that closely resembles is likely to be captured.

  Actually, the final similarity value may be obtained by combining the two similarity evaluation indexes as described above. Further, the similarity evaluation index is not limited to the two similarity evaluation indexes as described above, and any similarity evaluation criterion can be used as long as it represents that the “appearance” between images is closer. It may be used.

Further, the similarity evaluation may be performed after weighting processing is performed on the similarity evaluation index. For example, a layer that is distant from the target layer I _{1 in} time can be a layer that is different in “look” from the target layer I ₁ . Therefore, as the layer away from the target layer I ₁ in time, may be take measures such as reducing the degree of similarity by weighting coefficient corresponding to the temporal distance as similarity decreases.

  In the present embodiment, the imaging position of the time series image is standardized by the subject model in this way, so that the time series images whose positional relationships are generally specified through the common subject model are used ( (Similarity between layers) is calculated, so it is easier to extract similar regions than when calculating the similarity between multiple images without using knowledge about the subject to be imaged. Thus, the similarity can be calculated efficiently and with high accuracy, which is effective in providing a display image with good visibility.

After the most similar layer is replaced with the next layer by the method described above with respect to the target layer I ₁ , the process of rearranging the layers one after another is performed again using the next layer as the target first layer again. This process is sequentially performed for all layers. However, the actual layer order rearrangement procedure does not depend on the above method, and any process that rearranges the layer order so that the arrangement of time-series images closer to “look” becomes the same playback display position. Such a method may be used.

  In this way, the rearrangement unit 54 rearranges the layer order based on information such as the degree of similarity of time-series images between adjacent layers in the hierarchical structure calculated by the similarity calculation unit 53. Thus, the time-series images that are closer to each other are displayed in close proximity at the same playback display position in accordance with the standardized imaging position, and can be displayed in a superimposed manner. Become. In particular, in the subject 1, the capsule endoscope 3 may be repeatedly imaged in the vicinity of the same part by moving back and forth due to a peristaltic motion, but is not similar to the imaging order (time-series order) including the preceding and following movements. By rearranging the layer order so as to be in order, movement between images before and after the capsule endoscope 3 is suppressed. Therefore, when displaying the time-series image on the display unit 20, images near the same part are not repeatedly displayed with periodicity, and the moving image display is shaped so that the images near the same part are continuous. Therefore, the visibility of the image is greatly improved.

  The time-series image whose layer order has been rearranged in the rearrangement unit 54 is sent to the alignment unit 55, and alignment processing is performed between the time-series images of other neighboring layers in the hierarchical structure. . By the normalization processing for the subject model of the imaging position of the time-series image by the normalization unit 51, basically, the alignment of each time-series image with respect to the common subject model has already been generally performed. Since alignment between time series images is not performed, such alignment between time series images is supplementarily performed by the alignment unit 55.

Hereinafter, the alignment process in the alignment unit 55 will be described with reference to FIG. Now, consider a case where the positioning processing of the layer for a layer I _k as a target. For the target layer I _k , alignment processing is performed between the target layer I _k and a layer positioned in front of the layer I _k .

For example, in a state after rearrangement of the layer sequence made by the rearranging unit 54, as shown in FIG. 5, present on the front side of the target layer I _k, the target layer I in the stacking order of the hierarchical structure of the layer _k and regions L, M, layer share respectively n _{I l,} and I m, is _{I n} is three. That is, these regions L, M, and N are similar regions that are common between layers (images). Therefore, performing these regions L, M, and images which integrates N (assumed as _{I lmn),} the positioning process between the image of the target layer _{I k} (assumed as _{I k).} In the alignment process, the image I _k or the image I _lmn is deformed to align the positions, the similarity evaluation value between the two images is calculated by SSD, SAD, normalized cross-correlation, etc., and the similarity evaluation value is the highest. The deformation amount is regarded as a deformation amount for alignment, and the image _Ik is deformed by the deformation amount for alignment to complete the alignment.

The deformation for alignment may be any image deformation on the layer according to the xy coordinate system of the circular model, but here, two deformation methods are exemplified. One is a method in which the layer I _k is deformed while rotating around the center (origin) of the circular model in the xy coordinate system of the circular model as shown in FIG. . The other one is a method of deforming the layer _Ik while enlarging or reducing along the radial direction of the circular model in the xy coordinate system of the circular model, as shown in FIG.

As described above, by performing image deformation unique to the circular model in the xy coordinate system of the circular model, it is possible to efficiently realize the image alignment processing. As described above, after performing the alignment process of the target layer I _k and then performing the alignment process such as performing the alignment of the next layer I _{k + 1} , all layers are processed. An alignment process is performed on the image. Here, the time-series images on the layer are generally aligned with respect to the subject model at the stage of normalization of the imaging position on the subject model by the normalization unit 51, but the time-series images actually captured are taken. By performing the alignment between the images by the alignment unit 55, when displaying on the display unit 20, there is little misalignment between the images displayed in an overlapping manner, and it is possible to perform overlay display with higher accuracy and improved visibility. .

The time-series images of the layers subjected to the alignment processing between the images in the alignment unit 55 are sent to the image quality adjustment unit 56, and the visibility at the time of image display is based on the image information between neighboring layers in the hierarchical structure. The image quality is adjusted to improve the image quality. The image quality adjustment section 56, for example, the target layer I _k, and performs image quality adjustment processing between the target layer I _k other adjacent layer. The acquisition method of other layers existing in the vicinity of the target layer I _k may be arbitrary, but here, an acquisition method of two neighboring layers is illustrated.

One is a method for obtaining a layer belonging from the target layer I _k to layer closer than a predetermined number. For example, to obtain the five layers ± from the target layer I _k, 10 sheets of the layer is obtained in the neighborhood layer of the target layer _Ik.

The other is to share the existing area of the target layer I _k and the pixel information in a hierarchical structure, a method of acquiring the layer closest to the target layer I _k. For example, in the front layer of the target layer I _k, layer I _l, I _m, and I _n holds each pixel information to the target layer I _k with the same area in the nearest region L, M, in N . The holding behind the layer of target layer _{I k,} layer _{I o, I p, I q} , I r is the region O, P, Q, each pixel information to the target layer _{I k} with the same area nearest the R Suppose you are. In such a case, a method of obtaining these seven layers _{I l, I m, I n} , I o, I p, I q, the _{I r} as a neighbor layer of the target layer _{I k.}

The image quality adjustment of the time series images is performed between the neighboring layer and the target layer obtained in this way. The image quality adjustment in this case is, for example, time-series image gradation correction processing, color conversion processing, noise removal processing, and the like. After performing the image quality adjustment processing of the time-series image of the target layer _Ik, the image quality adjustment processing of the next target layer is performed sequentially. Image quality adjustment processing is performed on the time-series images. At this time, the image quality adjustment process may be sequentially performed on the next layer, but in reality, the image quality adjustment process may be performed on a skipped layer.

  In this way, by adjusting the image quality of the time-series image between neighboring layers by the image quality adjustment unit 56, it is possible to display an image with higher visibility. In particular, the difference in image quality between time-series images of neighboring layers that are simultaneously displayed by overlapping display becomes inconspicuous, and visibility is improved.

  As described above, each time-series image shaped through image processing by the normalization unit 51, the layer creation unit 52, the similarity calculation unit 53, the rearrangement unit 54, the alignment unit 55, and the image quality adjustment unit 56 is displayed. Under the control of the control unit 41, the layers are displayed in a superimposed manner on the display unit 20 according to the layer data as a layer having a hierarchical structure. Alternatively, the image data shaped in the storage unit 30 may be temporarily stored and held as layer data without performing image display, and may be read from the storage unit 30 and displayed on the display unit 20 at a desired time.

  Here, a method of displaying an image by the display unit 20 will be described. A plurality of time-series images captured by the capsule endoscope 3 and image-processed by the image processing unit 50 are converted into layer data to be superimposed and displayed as hierarchical layers, and the display unit 20 has a hierarchical structure. Overlaid by the layers. An example of the display is shown in FIG. 3-4 described above. The time-series images captured by the capsule endoscope 3 and processed by the image processing unit 50 are arranged in layers in the order in which they are captured (time-series), but some layers are formed by the rearrangement unit 54. There is a place where the order of layer display is changed in order of similarity by the rearrangement process.

  The observer can control display / non-display of each layer manually or automatically. In the case of manual operation, for example, the frontmost layer is hidden by the right key (not shown) of the keyboard in the input unit 10 in the workstation 100, and the currently displayed frontmost layer is displayed by the left key (not shown). A configuration can be adopted in which a non-displayed layer existing in front of one of the layers is displayed. By giving such an operation method to the workstation 100 and pressing the right key continuously, the layers are sequentially hidden from the front (the layer located at the forefront is sequentially erased), so that they are displayed in a superimposed manner. It is possible to stably observe a plurality of time-series images. In addition, by pressing the left key continuously, the layers are displayed sequentially from the back side (the layers on the foreground appear in sequence), allowing you to observe the image as if playing back a time-series image in reverse. It becomes.

  In the case of automatic processing, a forward playback mode and a reverse playback mode are prepared for each, and processing that automatically performs processing in which the right key is pressed in the manual processing / processing in which the left key is pressed is automatically performed. Just take a form.

  It should be noted that each of the operation target layers positioned at the foreground may be displayed / hidden instantaneously as described above. However, the frontmost image can be displayed by assigning transmittance to each layer and continuously changing the transmittance. The display / non-display may be switched so that the display state gradually changes. In other words, by assigning a transmittance to each layer, the image is displayed / hidden while sequentially changing the transmittance from 0% to 100% or from 100% to 0% sequentially. It is also possible to adopt a configuration that enables more natural and highly visible image observation. In addition, automatic / manual switching can be performed in the middle of the display, and if it is desired to take a closer look, the display / non-display of the layer can be manually controlled.

  FIG. 7 is a schematic flowchart showing the above-described image processing procedure executed under the control of the control unit 40. First, the control unit 40 reads the data of the time-series image signal and the header information that are captured by the capsule endoscope 3 and stored in the storage unit 30 through the portable recording medium 5 (step S1). Next, the control unit 40 controls the normalization unit 51 to calculate a parameter of a mathematical expression expressing a circular model (subject model) using pixel information of a plurality of points in each read time-series image ( Step S2). The normalization unit 51 normalizes the imaging position of each time-series image by specifying the position where the subject shape of each time-series image matches the subject model according to the parameters calculated in this way.

  Then, the control unit 40 controls the layer creation unit 52 so that each time-series image whose imaging position is standardized is captured in each layer of the hierarchical structure of the coordinate system corresponding to the coordinate system of the subject model. The layer data is created in accordance with (step S3).

  Thereafter, a variable i for designating a target is set to i = 1 (step S4). Then, the control unit 40 sets the i-th layer in the hierarchical structure as the target layer (step S5), and controls the similarity calculation unit 53 to calculate the similarity using the time-series image of the target layer as reference data. (Step S6). The similarity is calculated by calculating the similarity between the time-series image of the target layer (i-th layer) and each time-series image of the (i + 1) th to (i + N) -th neighboring layers on the front side. I do. As an index of similarity, for example, SSD, SAD, normalized cross-correlation, or the like may be used. The number N of layers may be a natural number of 2 or more. However, (i + N) does not exceed the number of acquired time-series images.

  Subsequently, the control unit 40 controls the rearrangement unit 54 to obtain the above-mentioned N similarities (from the time-series image of the target layer (i-th layer) and the (i + 1) -th image on the front side thereof. Using the (i + N) -th neighbor layer similarity level), the (i + 1) -th to (i + M) -th layer order is rearranged in order of similarity (step S7). . Here, M is a natural number that satisfies M ≦ N. More specifically, a process of rearranging the N layers that can be rearranged from the N similarities described above by the M layers closest to the i-th is performed.

  Further, the control unit 40 controls the alignment unit 55 to perform alignment processing on the time-series images of the layers whose layer order has been rearranged (step S8). In the alignment process, the (i + 2) th layer is aligned based on the (i + 1) th layer, and then the (i + 3) th layer is aligned based on the (i + 2) th layer. In this way, the time-series images of the target layer are aligned in order. Note that the alignment process may be performed, for example, in image deformation unique to the subject model. For example, when the subject is a body lumen organ, the subject model is a circular model, and image transformation is performed by rotating a time-series image in the circumferential direction of the circle according to the coordinate system of the circular model, or the coordinates of the circular model According to the system, the time series image may be enlarged or reduced in the radial direction of the circle.

  Subsequently, the control unit 40 confirms whether all layers have been designated as targets (step S9). However, since the alignment process and the rearrangement process are impossible in the final layer, all the layers indicate the layers excluding the final layer. If all layers have not been completed (step S9: No), the variable i is incremented by +1 (step S10), and the process proceeds to step S5. When all the layers have been completed (step S9: Yes), the control unit 40 controls the image quality adjustment unit 56, so that the time series images of the layers for which similarity calculation, rearrangement processing, and alignment processing have been performed, respectively. Then, image quality adjustment is performed between the time-series images of neighboring layers in the hierarchical structure (step S11). The main contents of image quality adjustment are tone correction processing, color conversion processing, noise removal processing, etc., but in reality, any method that adjusts image quality between layers is independent of these methods. It doesn't matter. In the image quality adjustment processing, image quality adjustment is performed sequentially from the first layer data to the last layer data with the time-series images of neighboring layers.

  Then, the control unit 40 outputs a signal of layer data including a time-series image on which image quality adjustment processing has been performed (step S12). The output signal is sent to, for example, the display unit 20 under the control of the display control unit 41, and each time-series image is superimposed and displayed as a layer having a hierarchical structure according to the layer data, and becomes an observation target of the observer. .

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention. In the present embodiment, the outline of a typical process execution sequence of the present invention has been described. However, the process execution sequence of the present invention is not limited to this, for example, a layer formed by the similarity calculation unit 53 and the rearrangement unit 54 The rearrangement process, the alignment process by the alignment unit 55, and the image quality adjustment process by the image quality adjustment unit 56 may be omitted. In this case, the display form of the plurality of time-series images on the display unit 20 is displayed according to the arrangement on the subject model whose imaging positions are standardized by the normalization unit 51, so that the time-series images are displayed at fixed positions on the display unit. As compared with the case where the images are sequentially displayed, it is possible to perform a display in which the time series images are substantially aligned through a common subject model, and the visibility of a plurality of time series images is improved. In particular, the standardization of the imaging position using the subject model enables rough positioning of each image, so multiple time-series images can be displayed at the same time as a layered display with a layered structure. Compared with the sequential display of the sequence images one by one, the image display has sufficiently good visibility. At this time, since a subject model set in advance according to the subject to be used is used, it is possible to cope with the subject to be imaged without being limited to a specific subject model.

  The layer order rearrangement process by the similarity calculation unit 53 and the rearrangement unit 54, the alignment process by the alignment unit 55, and the image quality adjustment process by the image quality adjustment unit 56 are executed in a mode in which the order of the processes is changed. For example, such a mode may be adopted that such a mode is selected from the operation reception unit 11. Further, it is possible to implement a mode in which a layer order rearrangement process by the similarity calculation unit 53 and the rearrangement unit 54, an alignment process by the alignment unit 55, and an image quality adjustment process by the image quality adjustment unit 56 are performed a plurality of times while being sequentially repeated. Alternatively, for example, a mode in which such a mode is selected from the operation reception unit 11 may be employed.

It is the schematic which shows the structural example of the in-vivo image information acquisition system containing the image processing apparatus concerning embodiment of this invention. It is a block diagram which shows schematic structure of the workstation which implement | achieves the image processing apparatus concerning this Embodiment. It is explanatory drawing which shows an example of the time series image actually imaged with the capsule type | mold endoscope. It is explanatory drawing which shows a mode that the time series image shown to FIGS. 3-1 was expressed with the circular model. It is explanatory drawing which shows a mode that the result shown to FIGS. 3-2 was arrange | positioned in the layer according to the coordinate system of a to-be-photographed model. It is explanatory drawing which shows a mode that an actual time series image is arrange | positioned on a layer on the coordinate system of a to-be-photographed model, and is displayed on a display part. It is a perspective view for demonstrating the example of a similarity calculation and rearrangement process. It is a perspective view which shows the example of a position alignment process. It is explanatory drawing which shows an example of the deformation | transformation for alignment. It is explanatory drawing which shows the other example of the deformation | transformation for alignment. It is a schematic flowchart which shows the image processing procedure performed under control of a control part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Subject 3 Capsule-type endoscope 100 Workstation 20 Display part 41 Display control part 51 Normalization part 52 Layer preparation part 53 Similarity calculation part 54 Rearrangement part 55 Positioning part 56 Image quality adjustment part

Claims (14)

The imaging shape of each time-series image is adapted to a subject model set in advance according to the subject with respect to each imaging position of the plurality of time-series images obtained by the imaging device imaging the subject in time-series. A normalization unit that normalizes by specifying the position of the subject model on the coordinates;
A display control unit configured to display each time-series image in which each imaging position is standardized by the normalization unit at a standardized imaging position according to the coordinate system of the subject model and to display on the display unit;
An image processing apparatus comprising: The subject model set in advance according to the subject is a circular model that includes a circular subject and is expressed by a mathematical formula according to the coordinate system,
The normalization unit calculates a parameter of the mathematical expression representing the circular model using pixel information of a plurality of points in each time-series image, and a subject shape of each time-series image is determined according to the calculated parameter. The image processing apparatus according to claim 1, wherein the imaging position is normalized by specifying a position that matches the circular model on the coordinate system. Layer data for overlaying and displaying as a layer of a hierarchical structure of a coordinate system corresponding to the coordinate system of the subject model is created for each time-series image whose imaging positions are normalized by the normalization unit It also has a layer creation part,
3. The image processing according to claim 1, wherein the display control unit displays the time-series images as the layers having a hierarchical structure in an overlapping manner on the display unit according to the created layer data. apparatus.   The display control unit displays a plurality of time-series images in a time-series superimposed manner by sequentially erasing or causing the display of the layer positioned at the forefront of the superimposed display to appear. Item 4. The image processing apparatus according to Item 3.   The said display control part changes the transmittance | permeability of the said time-sequential image displayed on the said layer located in the forefront of a superimposition display sequentially, or changing it sequentially, The said Claim 4 characterized by the above-mentioned. Image processing device. A similarity calculator that calculates the similarity between the time-series images between neighboring layers in a hierarchical structure;
A rearrangement unit for rearranging the layer order of the time-series images in a hierarchical structure based on the similarity calculated by the similarity calculation unit;
The image processing apparatus according to claim 3, further comprising:   The image processing apparatus according to claim 6, wherein the similarity calculation unit calculates the similarity by performing weighting according to a temporal distance at the time of imaging between the time series images.   The image processing apparatus further includes an alignment unit that aligns each time-series image in which each imaging position is standardized using the circular model with another time-series image of the neighboring layer in the hierarchical structure. The image processing apparatus according to claim 3, wherein:   The alignment unit aligns the time-series images with the time-series images of other layers in the vicinity while rotating the time-series images in a circumferential direction of a circle according to the coordinate system of the circular model. The image processing apparatus according to claim 8.   The alignment unit aligns each time-series image with the time-series images of other layers in the vicinity while expanding or reducing each time-series image in the radial direction of a circle according to the coordinate system of the circular model. The image processing apparatus according to claim 8, wherein the apparatus is an image processing apparatus.   The image processing apparatus according to claim 3, further comprising an image quality adjustment unit that adjusts an image quality of the time-series images between the adjacent layers in a hierarchical structure.   The plurality of time-series images are time-series images in which the imaging device introduced into the subject images a body lumen organ as the subject. An image processing apparatus according to 1. The imaging shape of each time-series image is adapted to a subject model set in advance according to the subject with respect to each imaging position of the plurality of time-series images obtained by the imaging device imaging the subject in time-series. A normalization step of normalizing by specifying the position of the subject model on the coordinates;
A display control step for displaying each time-series image in which each imaging position is standardized in the normalization step, arranged on each standardized imaging position according to the coordinate system of the subject model, and displaying on the display unit;
An image processing method comprising: In the computer provided in the image processing apparatus,
The imaging shape of each time-series image is adapted to a subject model set in advance according to the subject with respect to each imaging position of the plurality of time-series images obtained by the imaging device imaging the subject in time-series. A normalization procedure for normalization by specifying the position of the subject model on the coordinates,
A display control procedure in which each time-series image in which each imaging position is standardized by the normalization procedure is arranged at each imaging position standardized according to the coordinate system of the subject model and displayed on a display unit;
The program for image processing characterized by performing these.