JP2006061274A - Program and endoscope system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technique capable of displaying an image concerning a medical endoscope in a mode in which an affected part is found easily. <P>SOLUTION: A control part 20 sets an image generation condition of volume rendering according to information on the relative position and posture of the endoscope 6 in a three-dimensional area equivalent to the body of a patient. Then, according to the image generation condition, a virtual endoscopic image is generated based on volume data TD. Next, edges are extracted about the virtual endoscopic image, and an image area surrounded by edges whose curvature is larger than a prescribed value among the edges is detected as a first image area satisfying a prescribed shape condition. Then, about a color endoscopic real image obtained by the endoscope 6, an image for display in which the display modes of the respective pixels in a second image area corresponding to the first image area and a surrounding image area are differentiated from each other is generated. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a technique for generating a display image based on three-dimensional data in the medical field.

  As a technique for visualizing and displaying data indicating a three-dimensional structure (three-dimensional data), a technique such as volume rendering or surface rendering is known. These techniques are used, for example, when displaying a three-dimensional image that is a medical image for the purpose of making a diagnosis in the medical field.

  Recently, in surgery using an endoscope, three-dimensional data related to the patient's body is acquired in advance by CT scan (Computed Tomography Scan) or the like, and a virtual endoscope based on the three-dimensional data is acquired. Research has been conducted to perform surgery in a state where both the image of the above and the image of the actual endoscope are displayed side by side on the screen.

  In this virtual endoscope image, a virtual color is generally given (for example, Non-Patent Document 1).

  Prior art documents relating to such technology include the following.

"Anytime, anywhere 3D Real INTAGE basic edition" [online], Sumisho Electronics Co., Ltd. [searched July 8, 2004], Internet <http://www.netscience.ne.jp/banners/real_intage/>.

  However, since the virtual endoscopic image is given a virtual color different from that of the actual endoscopic image, it is difficult to compare the two images, and it is difficult to find the affected part that is the target of surgery.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of displaying an image related to a medical endoscope in an aspect in which an affected part can be easily found.

  In order to solve the above problems, the invention of claim 1 is a program that, when executed in a computer, causes the computer to function as an image processing device that generates a display image based on three-dimensional data. The image processing apparatus detects three-dimensional data related to a three-dimensional region corresponding to a patient's body, a real image related to the three-dimensional region acquired by a predetermined endoscope, and a predetermined position and orientation detection unit. Reading means for reading position and orientation information related to the position and orientation of the predetermined endoscope in the three-dimensional region into predetermined storage means; and condition setting means for setting image generation conditions according to the position and orientation information; A surface image generation means for generating a three-dimensional surface image by surface rendering based on the three-dimensional data according to the image generation conditions; Wherein the scan char mapping by mapping the real image to said three-dimensional surface image, characterized in that it comprises a display image generating means for generating a display image.

  The invention of claim 2 is a program for causing a computer to function as an image processing device that generates an image for display based on three-dimensional data by being executed in the computer, wherein the image processing device is a patient The three-dimensional data relating to the three-dimensional region corresponding to the body, the actual image relating to the three-dimensional region acquired by a predetermined endoscope, and the three-dimensional region detected by the predetermined position and orientation detection means. In accordance with the image generation condition, a reading unit that reads position and orientation information related to a predetermined endoscope position and orientation into a predetermined storage unit, a condition setting unit that sets an image generation condition according to the position and orientation information, A three-dimensional image generating means for generating a three-dimensional image based on the three-dimensional data, and a first image area satisfying a predetermined shape among the three-dimensional images An area detection means for detecting, and for the real image, a display image for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image area And an image generation means.

  The invention of claim 3 is a program that, when executed on a computer, causes the computer to function as an image processing device that generates a display image based on three-dimensional data. The three-dimensional data related to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among the three-dimensional regions corresponding to the body of the user, and the three-dimensional data acquired by a predetermined endoscope A reading unit that reads, into a predetermined storage unit, an actual image related to the region and position and orientation information related to the position and orientation of the predetermined endoscope in the three-dimensional region detected by the predetermined position and orientation detection unit; Condition setting means for setting an image generation condition according to position and orientation information, and generation of a three-dimensional image based on the three-dimensional data according to the image generation condition A three-dimensional image generating means, an area detecting means for detecting a first image area corresponding to the predetermined shape area of the three-dimensional image, and a second corresponding to the first image area for the real image. Display image generating means for generating a display image by changing the image area of the image area to a visual expression format different from that of the surrounding image areas.

  The invention according to claim 4 is the program according to claim 2 or claim 3, wherein the display image generation means uses the second image region and the surrounding image as the display image. An image in which the display mode of each pixel with respect to the region is different from each other is generated.

  The invention according to claim 5 is the program according to claim 4, wherein the display image generation means includes the second image area and the surrounding image area as the display image. A color image in which the pixels are displayed in different color representations is generated.

  The invention according to claim 6 is the program according to claim 2 or claim 3, wherein the display image generating means adds a marking indicating the second image area to the actual image. Generating the display image.

  The invention according to claim 7 is a program for causing a computer to function as an image processing apparatus that generates an image for display based on three-dimensional data by being executed in the computer, wherein the image processing apparatus is a patient The three-dimensional data relating to the three-dimensional region corresponding to the body, the actual image relating to the three-dimensional region acquired by a predetermined endoscope, and the three-dimensional region detected by the predetermined position and orientation detection means. In accordance with the image generation condition, a reading unit that reads position and orientation information related to a predetermined endoscope position and orientation into a predetermined storage unit, a condition setting unit that sets an image generation condition according to the position and orientation information, A three-dimensional image generating means for generating a three-dimensional image based on the three-dimensional data, and a first image area satisfying a predetermined shape among the three-dimensional images The first image area while assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image, based on the area detection means for detecting and the three-dimensional image And a display image generating means for generating a display image by using a visual expression format different from that of the surrounding image area.

  The invention of claim 8 is a program for causing a computer to function as an image processing apparatus that generates an image for display based on three-dimensional data by being executed in the computer, wherein the image processing apparatus is a patient The three-dimensional data related to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among the three-dimensional regions corresponding to the body of the user, and the three-dimensional data acquired by a predetermined endoscope A reading unit that reads, into a predetermined storage unit, an actual image related to the region and position and orientation information related to the position and orientation of the predetermined endoscope in the three-dimensional region detected by the predetermined position and orientation detection unit; Condition setting means for setting an image generation condition according to position and orientation information, and generation of a three-dimensional image based on the three-dimensional data according to the image generation condition Three-dimensional image generation means, area detection means for detecting a first image area corresponding to the predetermined shape area in the three-dimensional image, and based on the three-dimensional image, each pixel value of the real image is obtained. A display for generating a display image by assigning the first image region to a different visual representation format from the surrounding image region while assigning it to each pixel of the three-dimensional image corresponding to each pixel of the real image Image generating means.

  The invention according to claim 9 is the program according to claim 7 or claim 8, wherein the display image generation means uses the first image region and the surrounding image as the display image. An image in which the display mode of each pixel with respect to the region is different from each other is generated.

  The invention according to claim 10 is the program according to claim 9, wherein the display image generation means includes the first image region and the surrounding image region as the display image. A color image in which the pixels are displayed in different color representations is generated.

  The invention according to an eleventh aspect is the program according to the seventh or eighth aspect, wherein the display image generating means adds a marking indicating the first image area to the three-dimensional image. Then, the display image is generated.

  The invention according to claim 12 is the program according to any one of claims 2 to 11, wherein the predetermined shape condition is that the curvature of the contour of the image region to be determined is greater than a predetermined value. It is the condition that it is large.

  The invention of claim 13 is a program for causing a computer to function as an image processing device that generates an image for display based on three-dimensional data by being executed in the computer, wherein the image processing device is a patient. The three-dimensional data relating to the three-dimensional region corresponding to the body, the actual image relating to the three-dimensional region acquired by a predetermined endoscope, and the three-dimensional region detected by the predetermined position and orientation detection means. In accordance with the image generation condition, a reading unit that reads position and orientation information related to a predetermined endoscope position and orientation into a predetermined storage unit, a condition setting unit that sets an image generation condition according to the position and orientation information, The three-dimensional image generating means for generating a three-dimensional image based on the three-dimensional data, and aligning the real image and the three-dimensional image, Characterized in that it comprises a display image generating means for generating a display image obtained by superimposing the actual image and the three-dimensional image.

  The invention of claim 14 is an endoscope system, which is an endoscope that acquires a real image relating to a three-dimensional region corresponding to a patient's body, and the position of the endoscope in the three-dimensional region; A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the image processing device reads the three-dimensional data, the actual image, and the position / orientation information into a predetermined storage unit; and an image generation condition corresponding to the position / orientation information. Condition setting means for setting, surface image generation means for generating a three-dimensional surface image by surface rendering based on the three-dimensional data according to the image generation conditions, and texture mapping Mapping a real image to the three-dimensional surface image to have a display image generating means for generating a display image, and an output means for outputting the display image to the image output device It is characterized by.

  The invention of claim 15 is an endoscope system, wherein an endoscope that acquires a real image relating to a three-dimensional region corresponding to a patient's body, the position of the endoscope in the three-dimensional region, and A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the image processing device reads the three-dimensional data, the actual image, and the position / orientation information into a predetermined storage unit; and an image generation condition corresponding to the position / orientation information. Condition setting means for setting, three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data in accordance with the image generation conditions, and a first image region that satisfies a predetermined shape condition among the three-dimensional images Inspect And a display image for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image areas. It has a generation means and an output means for outputting the display image to the image output device.

  The invention of claim 16 is an endoscope system, wherein an endoscope that acquires a real image relating to a three-dimensional region corresponding to a patient's body, the position of the endoscope in the three-dimensional region, and A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the three-dimensional data is three-dimensional data relating to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among the three-dimensional regions, and the image processing A device that reads the three-dimensional data, the real image, and the position and orientation information into a predetermined storage unit; a condition setting unit that sets an image generation condition according to the position and orientation information; and Raw image In accordance with conditions, a three-dimensional image generation unit that generates a three-dimensional image based on the three-dimensional data, a region detection unit that detects a first image region corresponding to the predetermined shape region in the three-dimensional image, Display image generating means for generating a display image by setting a second image region corresponding to the first image region to a visual representation format different from the surrounding image region for the real image, and the display Output means for outputting an image to the image output device.

  The invention of claim 17 is an endoscope system, wherein an endoscope that acquires a real image relating to a three-dimensional region corresponding to a patient's body, the position of the endoscope in the three-dimensional region, and A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the image processing device reads the three-dimensional data, the actual image, and the position / orientation information into a predetermined storage unit; and an image generation condition corresponding to the position / orientation information. Condition setting means for setting, three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data in accordance with the image generation conditions, and a first image region that satisfies a predetermined shape condition among the three-dimensional images Inspect The first image area is assigned to each pixel of the three-dimensional image corresponding to each pixel of the real image, while assigning each pixel value of the real image to each pixel of the real image based on the three-dimensional image. A display image generating unit configured to generate a display image by using a visual expression format different from that of a surrounding image region; and an output unit configured to output the display image to the image output device. And

  The invention of claim 18 is an endoscope system, wherein an endoscope that acquires a real image relating to a three-dimensional region corresponding to a patient's body, a position of the endoscope in the three-dimensional region, and A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the three-dimensional data is three-dimensional data relating to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among the three-dimensional regions, and the image processing A device that reads the three-dimensional data, the real image, and the position and orientation information into a predetermined storage unit; a condition setting unit that sets an image generation condition according to the position and orientation information; and Raw image In accordance with conditions, a three-dimensional image generation unit that generates a three-dimensional image based on the three-dimensional data, a region detection unit that detects a first image region corresponding to the predetermined shape region in the three-dimensional image, Based on a three-dimensional image, the first image region is different from the surrounding image regions while assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image. The image processing apparatus includes display image generation means for generating a display image by using a visual expression format, and output means for outputting the display image to the image output apparatus.

  The invention of claim 19 is an endoscope system, which is an endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body, the position of the endoscope in the three-dimensional region, and A position / orientation information acquisition apparatus that acquires position / orientation information related to an attitude, an image processing apparatus that generates a display image based on three-dimensional data related to the three-dimensional region, and an image that visually outputs the display image An output device, wherein the image processing device reads the three-dimensional data, the actual image, and the position / orientation information into a predetermined storage unit; and an image generation condition corresponding to the position / orientation information. Condition setting means for setting, three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions, and positioning the real image and the three-dimensional image, Actual picture Characterized in that it has a display image generating means for generating a display image obtained by superimposing the corresponding three-dimensional image, and output means for outputting the display image to the image output device.

  According to the first aspect of the present invention, an image generation condition is set in accordance with information related to the position and posture of the endoscope in a three-dimensional region corresponding to the patient's body, and the three-dimensional image is set according to the image generation condition. A three-dimensional surface image is generated by surface rendering based on the three-dimensional data relating to the region, and the real image acquired by the endoscope is texture-mapped to the three-dimensional surface image, thereby displaying the display image. Since the virtual image generated from the three-dimensional data can be displayed in the same color as the actual image actually acquired by the endoscope, the virtual image generated from the three-dimensional data can be displayed, for example, Comparison with a real image becomes easy. Therefore, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  According to the second aspect of the present invention, the three-dimensional data related to the three-dimensional region is obtained in accordance with the image generation condition according to the information related to the position and posture of the endoscope in the three-dimensional region corresponding to the patient's body. A three-dimensional image is generated based on the first image region, a first image region that satisfies a predetermined shape condition is detected from the three-dimensional image, and a second image region corresponding to the first image region is detected for the real image With a configuration in which a display image is generated by adopting a visual expression format different from that of the surrounding image area, an area that satisfies a predetermined shape condition corresponding to the affected area can be easily found visually. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  According to a third aspect of the present invention, three-dimensional data in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among three-dimensional regions corresponding to a patient's body is read, and the three-dimensional A three-dimensional image is generated based on the three-dimensional data in accordance with an image generation condition corresponding to information related to the position and orientation of the endoscope in the region, and the third shape image corresponding to the predetermined shape region in the three-dimensional image. A configuration in which one image area is detected and a display image is generated for a real image by setting a second image area corresponding to the first image area to a visual representation format different from that of the surrounding image areas. Thus, an area in which a predetermined shape condition corresponding to the affected part is satisfied can be easily found visually. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  According to the fourth aspect of the present invention, a configuration for generating a display image in which the display modes of the respective pixels of the second image region and the surrounding image region are different from each other is generated for the actual image. This makes it easy to visually find a region that satisfies a predetermined shape condition corresponding to the affected area.

  According to the invention described in claim 5, a color image in which the pixels of the second image region and the surrounding image region are displayed with different color representations is generated as a display image for the actual image. With such a configuration, it becomes easier to visually find a region that satisfies a predetermined shape condition corresponding to the affected part.

  According to the sixth aspect of the present invention, a predetermined shape condition corresponding to the affected area is satisfied by a configuration in which a marking for indicating the second image region is added to the actual image to generate a display image. The area is surely easy to find visually.

  According to the seventh aspect of the present invention, the three-dimensional data related to the three-dimensional region according to the image generation condition according to the information related to the position and posture of the endoscope in the three-dimensional region corresponding to the body of the patient. A three-dimensional image is generated based on the three-dimensional image, a first image region satisfying a predetermined shape condition is detected from the three-dimensional image, and an actual image acquired by the endoscope is based on the three-dimensional image. A display image is generated by assigning each pixel value to each pixel of the three-dimensional image corresponding to each pixel of the actual image, and making the first image region a different visual representation format from the surrounding image region. With such a configuration, an area that satisfies a predetermined shape condition corresponding to the affected part is easily visually found. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  According to the invention described in claim 8, three-dimensional data in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition among three-dimensional regions corresponding to the body of a patient is read, and the three-dimensional A three-dimensional image is generated based on the three-dimensional data in accordance with an image generation condition corresponding to information related to the position and orientation of the endoscope in the region, and the third shape image corresponding to the predetermined shape region in the three-dimensional image. The first image area is detected, and each pixel value of the real image acquired by the endoscope is assigned to each pixel of the three-dimensional image corresponding to each pixel of the real image on the basis of the three-dimensional image. By forming the image area into a visual representation format different from that of the surrounding image areas, the display image is generated, so that an area that satisfies the predetermined shape condition corresponding to the affected area can be easily found visually. . That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  According to the ninth aspect of the present invention, based on the three-dimensional image, a display image in which the display modes of the respective pixels of the first image region and the surrounding image region are made different from each other is generated. With such a configuration, it becomes easy to visually find a region that satisfies a predetermined shape condition corresponding to the affected part.

  According to the tenth aspect of the present invention, on the basis of a three-dimensional image, a color image in which the pixels of the first image area and the surrounding image area are displayed in mutually different color representations is displayed. A configuration that generates as an image makes it easier to visually find a region that satisfies a predetermined shape condition corresponding to an affected area.

  According to the eleventh aspect of the present invention, a predetermined shape condition corresponding to the affected area is obtained by a configuration in which a marking for indicating the first image region is added to the three-dimensional image to generate a display image. The filled area is surely easy to find visually.

  According to the twelfth aspect of the present invention, the predetermined shape condition for detecting the first image area is a condition that the curvature of the contour of the image area to be determined is larger than a predetermined value. Thus, for example, it is possible to make it easy to visually find an affected part having a surface shape with a large curvature.

  According to the invention described in claim 13, the three-dimensional data related to the three-dimensional area according to the image generation condition according to the information related to the position and posture of the endoscope in the three-dimensional area corresponding to the body of the patient. The surface shape of each part in the display image is generated by generating a three-dimensional image based on the image and generating a display image in which the three-dimensional image and the actual image acquired by the endoscope are superimposed. It becomes an aspect that is easy to recognize. As a result, the image related to the medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  Further, according to the invention described in claim 14, the same effect as in claim 1 is obtained.

  According to the invention described in claim 15, the same effect as in claim 2 is obtained.

  According to the sixteenth aspect of the invention, the same effect as in the third aspect of the invention can be achieved.

  According to the seventeenth aspect, the same effect as in the seventh aspect is obtained.

  According to the eighteenth aspect of the invention, the same effect as in the eighth aspect of the invention can be achieved.

  According to the nineteenth aspect, the same effect as in the thirteenth aspect is obtained.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
<Outline of endoscope system>
FIG. 1 is a diagram illustrating an overview of an endoscope system 1 according to an embodiment of the present invention.

  The endoscope system 1 includes a personal computer (hereinafter simply referred to as “personal computer”) 2, a monitor 3, an operation unit 4, a mounting unit 5, an endoscope 6, and a position and orientation detector 7. Composed.

  The personal computer 2 includes a control unit 20, an input / output I / F 21, and a storage unit 22.

  The input / output I / F 21 transmits / receives data between the personal computer 2 and each peripheral device of the personal computer 2 (the monitor 3, the operation unit 4, the mounting unit 5, the endoscope 6, and the position and orientation detector 7). It is an interface (I / F) for performing. Further, since the input / output I / F 21 transmits / receives data to / from the control unit 20, data can be transmitted / received between the control unit 20 and peripheral devices of the personal computer 2.

  The storage unit 22 is composed of, for example, a hard disk or the like, and data relating to the body including a three-dimensional internal structure of the patient (for example, internal structure of the esophagus and stomach) detected by CT scan (Computed Tomography Scan) or the like TD (also referred to as “three-dimensional data”) or virtual three-dimensional image data (also referred to as “virtual endoscopic image”) as if it were taken with an endoscope based on the three-dimensional data TD. An image processing program PG for generating or performing other image processing or the like is stored. In the present embodiment, a virtual endoscopic image is generated based on three-dimensional data by a general volume rendering technique.

  Here, the three-dimensional data is, for example, so-called “volume data”. When obtaining this volume data, a technique called voxel expression is used in which a target object is expressed by a set having very small cubes (or rectangular parallelepipeds) as elements. And the cube (or rectangular parallelepiped) as this element is called a voxel.

  In general, by using a CT scan (Computed Tomography Scan) using X-rays or the like, it is possible to obtain an image when a human body is cut into circles. Then, by shifting the position of this ring cutting, and by setting the cutting direction to up and down, front and back, and left and right, data in which the amount of X-ray absorption is stored in the voxel data is obtained. Data representing the density and density distribution in the three-dimensional space is called “volume data”. This volume data is obtained by dividing a three-dimensional region of an object (here, a patient's body) represented by the volume data into a plurality of voxels, and one voxel value (here, an X-ray absorption amount) for each voxel. Is a given data. Then, the three-dimensional area related to the volume data is a target of calculation processing in volume rendering. Here, it is assumed that the three-dimensional region related to the volume data is a three-dimensional region corresponding to the patient's body (including the internal structure).

  The control unit 20 mainly includes a CPU, a ROM 20a, a RAM 20b, and the like, and is a part that performs overall control of each unit of the personal computer 2. The control unit 20 reads the volume data TD stored in the storage unit 22 into the RAM 20b, reads the image processing program PG, and executes it on the CPU, thereby creating a virtual endoscopic image based on the volume data. Other image processing is performed, and various image data are output to the monitor 3 via the input / output I / F 21. The volume data TD may be stored in a storage medium of the memory card 51 described later. As described above, the personal computer 2 generates display image data (also referred to as “display image”) for visual output to the monitor 3 based on the three-dimensional data, and performs other image processing and the like. Works as an image processing device to execute.

  The monitor 3 is configured by a CRT, for example, and visually outputs various images generated by the control unit 20.

  The operation unit 4 includes a keyboard, a mouse, and the like, and transmits various electrical signals to the input / output I / F 21 in accordance with various user operations. The mounting unit 5 can detachably mount a storage medium such as the memory card 51. Various data, programs, and the like stored in the memory card 51 attached to the attachment unit 5 can be taken into the control unit 20 and the storage unit 22 via the input / output I / F 21.

  The endoscope 6 images the inside of the patient's body (for example, the inside of the esophagus and the stomach), so that color image data (“color endoscope actual image”) about the structure of the inner wall in each part of the body. (Also called). In the endoscope 6, for example, by acquiring a color endoscope real image at 30 frames per second, it is possible to obtain a moving image related to the structure of the inner wall in each part of the body. The actual color endoscope image acquired by the endoscope 6 is read and temporarily stored in the RAM 20b of the control unit 20 via the input / output I / F 21 and subjected to image processing to be described later, and then the monitor 3 Is output visually at.

  Further, as shown in FIG. 2, the endoscope 6 includes a coupling optical system 6b at the tip of a tube body 6a using an optical fiber or a relay lens system. In addition, the endoscope 6 is provided with a scissor portion 6c that sandwiches and removes a tissue in the body (for example, a tissue of a tumor portion) on the side surface of the distal end portion of the tube body 6a. Furthermore, the light source 60 is a light source for irradiating an observation target (for example, the inside of the esophagus or stomach) through the tubular body 6a.

  The position / orientation detector 7 detects the position and angle of the coupling optical system 6b provided at the distal end of the endoscope 6 in the patient's body. For example, a general echo using ultrasonic waves or the like is used. It consists of devices. In this position / orientation detector 7, for example, the echo device main body can be freely moved in the vertical and horizontal directions, and in the predetermined three-dimensional coordinate system (XYZ orthogonal coordinate system) with a predetermined position as a reference point, the echo device The coordinate position of the main body can be detected. For example, the echo device main body is installed at the tip of an arm extending from a predetermined fixed point, and the position of the echo device main body is changed by driving the arm electrically. Can be measured, the coordinate position and orientation of the echo device body can be detected.

  Further, the echo device main body detects the position occupied by the coupling optical system 6b in the detection area by the echo and the direction and angle of the coupling optical system 6b by detecting the shape of the characteristic tip of the endoscope 6. be able to. Then, by setting the reference point of the three-dimensional coordinate system, for example, at the position of a specific part of the patient, a coupled optical system in a three-dimensional region corresponding to the patient's body (hereinafter also referred to as “body three-dimensional region”) Information related to the relative position of 6b (position information), information related to the posture such as the direction and angle of the coupling optical system 6b (posture information), and the like can be acquired. As a result, it is possible to detect parameters (also referred to as “external parameters”) such as a relative positional relationship and an angular relationship between the three-dimensional region related to the volume data and the endoscope 6. That is, the relative position and posture of the endoscope 6 in the three-dimensional body region can be detected.

  As the position / orientation detector 7 for detecting an external parameter of the endoscope 6, a sensor such as a gyroscope provided at the distal end of the endoscope 6 may be considered.

  Information relating to the relative position and posture of the endoscope 6 in the three-dimensional body region detected by the position and posture detector 7 as described above (also referred to as “position and posture information”) is input via the input / output I / F 21. Thus, it is read into the RAM 20b of the control unit 20 and temporarily stored.

  The internal parameters (also referred to as “internal parameters”) of the endoscope 6 such as the focal length, the pixel size, the angle formed by the vertical and horizontal lines of the pixel, the center of the image, and the like are the design of the endoscope 6. It depends on.

  Therefore, the control unit 20 performs a three-dimensional image (virtual image) based on the volume data obtained by volume rendering under the conditions according to the external parameters acquired by the position / orientation detector 7, that is, the position / orientation information and the internal parameters. When the endoscope image is generated, the composition of the color endoscope real image acquired by the endoscope 6 and the virtual endoscope image are exactly the same.

  Here, if the internal parameters are fixed and are known, the control unit 20 determines whether the color endoscope actual image and the color endoscope real image are changed according to the external parameters (that is, position and orientation information) that change every time. Conditions (also referred to as “image generation conditions”) for generating a virtual endoscopic image can be set so that the composition (that is, the relative positional relationship of the subject) with the virtual endoscopic image matches. Then, the control unit 20 generates a virtual endoscope image by volume rendering based on the volume data in accordance with the set image generation conditions.

  Note that the projection matrix of the actual color endoscope image can be obtained from the internal parameters and the external parameters of the endoscope 6 when acquiring the actual color endoscope image. This projection matrix is a transformation matrix composed of 3 rows × 4 columns for transforming a predetermined camera coordinate system into a world coordinate system (reference coordinate system). Some image generation conditions are automatically determined by inputting this conversion matrix, and the image generation conditions may be determined by such a method.

<Generation and output of display image>
Generation and output of image data (display image) to be output to the monitor 3 will be described.

  In the endoscope system 1 according to the present embodiment, a portion where the curvature of the surface shape corresponding to the affected part is larger than a predetermined value is detected based on the volume data, and the detected curvature is detected among the color endoscope real images. It is easy to find an affected part by generating a display image in which an image region corresponding to a location larger than a predetermined value is different from the surrounding image region and outputting it on the monitor 3 visually. To do.

  Hereinafter, operations related to generation and output of a display image will be specifically described.

  FIG. 3 is a flowchart showing an operation flow relating to generation and output of a display image. This operation flow is realized by the image processing program PG being read and executed by the control unit 20. First, when the user appropriately operates the operation unit 4 to instruct to start photographing with the endoscope 6 and volume rendering based on the volume data TD, the process proceeds to step S1 in FIG.

  In step S1, the volume data TD stored in the storage unit 22 is read and acquired by the control unit 20, and the process proceeds to step S2.

  In step S2, the storage unit 22 acquires a color endoscope real image of one frame from the endoscope 6, and proceeds to step S3. FIG. 4 is a schematic diagram illustrating an example of a color endoscope real image acquired in step S2. As an object to be imaged by the endoscope 6, the inner wall surface of each part in the body such as the stomach and the inner wall of the intestine can be considered. In FIG. 4 and FIG. It shows how it is.

  In step S3, the position and orientation detector 7 detects the position and orientation of the endoscope 6, and the process proceeds to step S4. Here, the position and posture of the endoscope 6 are detected as the position and posture of the endoscope 6 at the time when the color endoscope real image is acquired in step S2. Information related to the position and orientation of the endoscope 6 (position and orientation information) is output to the control unit 20.

  In step S4, according to the position and orientation information obtained in step S3, that is, the position and orientation of the endoscope 6, the volume rendering conditions (the viewpoint, the position of the projection plane, the line of sight passing through a plurality of points on the projection plane from the viewpoint, etc.) Conditions) is set, and the process proceeds to step S5. Here, a volume rendering condition is set such that a virtual endoscopic image having the same composition as the actual color endoscope image acquired in step S2 is generated. Here, in order to accurately display the state of the inner wall of the intestine and the like, a value very close to 1 is given as the opacity given in general volume rendering.

  In step S5, a virtual endoscopic image is generated based on the volume data acquired in step S1 by a general volume rendering method, and the process proceeds to step S6. The virtual endoscopic image generated here is given a virtual color. FIG. 5 is a schematic diagram illustrating an example of a virtual endoscopic image generated in step S6. As shown in FIG. 5, here, a virtual endoscopic image having the same composition as the actual color endoscope image shown in FIG. 4 acquired in step S2 is generated. Although not shown in FIG. 5, the virtual endoscopic image is given a virtual color. The virtual endoscopic image shown in FIG. 5 shows a state in which a protrusion corresponding to a tumor is formed on the inner wall of the intestine.

  In step S6, a characteristic shape satisfying a condition related to a predetermined shape (predetermined shape condition) is detected from the virtual endoscopic image generated in step S5, and the process proceeds to step S7. Here, for example, edge extraction is performed on the virtual endoscopic image, an edge (contour) having a curvature larger than a predetermined value is detected, and an image region (also referred to as “first image region”) surrounded by the edge is detected. R1 can be detected as a characteristic shape that satisfies a predetermined shape condition. Here, the affected area where the tumor is formed generally uses the fact that the curvature of the surface shape is larger than the surrounding surface shape, and the first image region R1 is an image corresponding to the affected area where the tumor is formed. Detected as a region. That is, it is determined whether or not the image area is the first image area R1 by detecting whether or not the curvature of the contour of the image area to be determined is larger than a predetermined value. In addition, although various things can be considered about the predetermined shape conditions mentioned above, the various methods of detecting an affected part based on a shape are employable in a medical field etc.

  In step S7, the pixel value of the image region corresponding to the first image region R1 corresponding to the characteristic shape detected in step S6 in the color endoscope actual image is changed, and the color endoscope actual image is changed. Image data for display based on the image (also referred to as “real image for display”) is generated, and the process proceeds to step S8. FIG. 6 is a schematic diagram illustrating an example of a real display image. In this step S7, an image area R2 corresponding to the first image area R1 (also referred to as a hatched portion in FIG. 6, “second image area”) R2 of the actual color endoscope image is, for example, green. The pixel value is changed, for example, by increasing the brightness. That is, the pixel value related to the second image region R2 is changed so that the display modes of the respective pixels in the second image region R2 and the surrounding image region are different from each other. Note that the example in which the second image region R2 is green is given here because various organs in the human body usually exhibit a color close to red, and there are few that exhibit green. This is because, by displaying the image region R2 in green, it is possible to obtain a display mode that is distinguishable from other surrounding image regions.

  In step S8, the virtual endoscopic image generated in step S5 relates to an image region (here, the first image region R1 in FIG. 5) corresponding to the characteristic shape detected in step S6. The pixel value is changed to a pixel value indicating a predetermined color (for example, green). Then, surrounding image regions other than the first image region R1 (here, the image region R11 in FIG. 5) correspond to each pixel of the image region (also referred to as “peripheral image region”) R11. A pixel value (color value) relating to each pixel of the actual color endoscope image is assigned. By doing so, display image data (also referred to as “display virtual image”) based on the virtual endoscopic image is generated, and the process proceeds to step S9. FIG. 7 is a schematic diagram illustrating an example of a display virtual image. As shown in FIG. 7, the first image region R1 of the virtual endoscopic image shown in FIG. 5 has a predetermined color (the hatched portion in FIG. 7) and other surrounding second For the image region R11, the pixel values of the color endoscope real image shown in FIG. 4 are assigned (sand hatching portion in FIG. 7), and a display virtual image is generated.

  By doing so, it is possible to generate a display virtual image in which an actual color is given to the virtual endoscopic image.

  In step S9, the display real image and the display virtual image respectively generated in step S7 and step S8 are simultaneously visually output on the monitor 3, and the process returns to step S2. In this step S9, for example, as shown in FIG. 8, a screen in which a display image based on the display actual image and a display image based on the display virtual image are spatially arranged horizontally is displayed on the monitor 3. The 8 illustrates a screen in which the display image based on the display real image is arranged on the left side in FIG. 8 and the display image based on the display virtual image is arranged on the right side in FIG.

  Further, since the process returns from step S9 to step S2, the process from step S2 to step S9 is repeated. Here, assuming that a color endoscope real image of 30 frames per second is obtained, the processing from step S2 to step S9 is performed every 1/30 seconds to obtain a display image (in this case, the display real image and the display real image). A display virtual image) is newly created every 1/30 seconds, and is visually output on the monitor 3.

  As described above, in the endoscope system 1 according to the first embodiment, in the control unit 20, information on the relative position and posture of the endoscope 6 in the three-dimensional region (the three-dimensional body region) related to the volume data. Volume rendering image generation conditions are set according to (position and orientation information). Then, a three-dimensional image (here, a virtual endoscopic image) is generated based on the volume data in accordance with the image generation conditions. Next, an edge is extracted from the virtual endoscopic image, and an image region surrounded by an edge having a curvature larger than a predetermined value is detected as a first image region R1 that satisfies a predetermined shape condition. To do. Then, regarding the color endoscope actual image acquired by the endoscope 6, the display modes of the respective pixels of the second image region R2 corresponding to the first image region R1 and the surrounding image regions are different from each other. An actual display image (that is, a display image) is generated. With such a configuration, since the display mode of the part corresponding to the affected part is different from the surroundings, an area that satisfies a predetermined shape condition corresponding to the affected part can be easily found visually. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  In addition, by changing the pixel value of the second image region R2 to a pixel value related to a color expression such as green, the pixels of the second image region R2 and the surrounding image regions can be expressed in different colors. A displayed color image (that is, a display image) is generated. Such a configuration makes it easier to find a region that satisfies a predetermined shape condition corresponding to the affected area.

  Further, by setting the predetermined shape condition for detecting the first image region R1 corresponding to the affected area as a condition that the curvature of the contour of the image region to be determined is larger than a predetermined value, for example, the curvature is For example, the affected part having a large surface shape can be easily found visually. As a result, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

Second Embodiment
In the endoscope system 1 according to the first embodiment, a virtual endoscopic image is created based on volume data by volume rendering, a first image region corresponding to an affected area is detected, and a color endoscope real image is detected. The pixel value was changed so that the display mode of the second image area corresponding to the detected first image area is different from that of the surrounding image areas. In contrast, in the endoscope system 1A according to the second embodiment, a three-dimensional image composed of polygons based on volume data by surface rendering, that is, a virtual endoscopic image (also referred to as a “three-dimensional surface image”). ), And a display image superimposed with the color endoscope actual image is generated and visually output, thereby making it easier to find the affected area.

  In the endoscope system 1A according to the second embodiment, the contents related to the image processing such as the point that the creation method of the virtual endoscope image based on the volume data is surface rendering, the creation of the display image, and the like. It is only different from the endoscope system 1 according to the first embodiment, and the other parts are the same. Therefore, the same reference numerals are given to the same components and the like, and the description thereof will be omitted. Hereinafter, the generation and output of the display image related to the endoscope system 1A corresponding to the contents of the image processing different from the endoscope system 1 will be described. Will be described.

<Generation and output of display image>
FIG. 9 is a flowchart showing an operation flow relating to generation and output of a display image according to the second embodiment. This operation flow is realized by the image processing program PG being read and executed by the control unit 20. First, when the user appropriately operates the operation unit 4 to instruct to start photographing with the endoscope 6 and volume rendering based on the volume data TD, the process proceeds to step S21 in FIG.

  In step S21, the volume data TD stored in the storage unit 22 is read and acquired by the control unit 20, and the process proceeds to step S22.

  In step S22, the control unit 20 binarizes the voxel value related to the volume data TD, and proceeds to step S23. Here, by binarizing the voxel values related to the volume data TD into those that are included in the predetermined range and those that are not, there is an object that becomes the observation target of the three-dimensional region related to the volume data TD. A distinction is made between two areas: areas and areas that are not. By executing such processing in advance, surface rendering described later can be easily executed.

  In step S23, as in step S2 of FIG. 3, the control unit 20 acquires a color endoscope real image of one frame from the endoscope 6, and proceeds to step S24. Here, for example, a color endoscope real image as shown in FIG. 4 can be acquired.

  In step S24, as in step S3 of FIG. 3, the position and orientation detector 7 detects the position and orientation of the endoscope 6, and the process proceeds to step S25. That is, the control unit 20 acquires position and orientation information related to the endoscope 6.

  In step S25, surface rendering conditions (conditions such as viewpoint, angle of view, and line of sight) are set according to the position and orientation information obtained in step S24, that is, the position and orientation of the endoscope 6, and the process proceeds to step S26. . Here, surface rendering conditions are set such that a virtual endoscopic image having the same composition as the actual color endoscope image acquired in step S23 is generated.

  In step S26, a virtual endoscopic image is generated based on the volume data acquired in step S21 by a general surface rendering method, and the process proceeds to step S27. The virtual endoscopic image generated here is given a virtual color similarly to the virtual endoscopic image shown in FIG. 5 by shading of general surface rendering. Hereinafter, the description will be continued assuming that a virtual endoscopic image as shown in FIG. 5 has been acquired.

  In step S27, the virtual endoscopic image generated in step S26 is mapped to the virtual endoscopic image acquired in step S23 by a general texture mapping method, so that the display virtual image is displayed. Generate and proceed to step S28.

  FIG. 10 is a diagram for explaining the texture mapping in step S27. FIG. 10A illustrates the virtual endoscope image generated in step S26, and FIG. 10B illustrates the color endoscope real image acquired in step S23. Then, as shown in FIG. 10, a color endoscope real image (FIG. 10 (b)) is mapped to each polygon constituting the virtual endoscopic image (FIG. 10 (a)). A virtual image for use (FIG. 10C) can be generated. As the texture mapping method, a known method can be used.

  In step S28, the color endoscope real image acquired in step S23 and the display virtual image generated in step S27 are visually output on the monitor 3, and the process returns to step S23. In step S28, for example, as shown in FIG. 11, a screen on which the display image based on the color endoscope real image and the display image based on the display virtual image are spatially arranged horizontally is displayed on the monitor 3. Is displayed. 11 illustrates a screen in which the display image based on the color endoscope image is arranged on the left side in FIG. 11 and the display image based on the display virtual image is arranged on the right side in FIG.

  Further, since the process returns from step S28 to step S23, the processes from step S23 to step S28 are repeated. If a color endoscope real image of 30 frames per second is obtained here, the processing from step S23 to step S28 is performed every 1/30 seconds to obtain a display image (in this case, a color endoscope real image). An image and a virtual image for display) are newly created every 1/30 seconds and are visually output on the monitor 3.

  As described above, in the endoscope system 1A according to the second embodiment, in the control unit 20, information on the relative position and posture of the endoscope 6 in the three-dimensional region (three-dimensional body region) related to the volume data. A surface rendering image generation condition is set according to (position and orientation information), and a three-dimensional image (here, a virtual endoscopic image) is generated based on the volume data in accordance with the image generation condition. Next, a virtual endoscopic image composed of polygons and accurately representing the unevenness of the internal structure (here, the surface structure of the inner wall of the intestine) is accurately reproduced by a texture mapping technique. The virtual image for display is generated by mapping. Furthermore, a screen in which a display image based on the color endoscope actual image and a display image based on the display actual image are simultaneously arranged horizontally is displayed on the monitor 3. With such a configuration, the display image based on the virtual endoscopic image generated from the volume data is displayed in the same color as the actual image actually acquired by the endoscope, while the unevenness (particularly the surface shape) is displayed. The shape of the part having a large curvature is also clearly shown. Therefore, for example, it becomes easy to compare a display image based on a display virtual image with a display image based on a color endoscope real image. As a result, the image related to the medical endoscope can be displayed in a manner that makes it easy to find the affected area.

<Modification>
As mentioned above, although embodiment of this invention was described, this invention is not limited to the thing of the content demonstrated above.

  For example, in the first embodiment described above, as shown in FIG. 8, a display image based on a real image for display based on a color endoscope real image and a display virtual based on a virtual endoscope image Although a screen in which the display image based on the image is spatially arranged side by side is displayed, the present invention is not limited to this. For example, the display image based on the display actual image as shown in FIG. 6 or shown in FIG. Only a display image based on such a virtual endoscopic image may be displayed on the monitor 3. Even in such a configuration, both display images have an aspect in which it is easy to visually find a region that satisfies a predetermined shape condition corresponding to the affected part, and thus the same effect as that of the first embodiment can be obtained. Can do. Furthermore, since one image having a display mode that clearly uses the actual color and clearly showing the affected part can be displayed on the monitor 3 in a larger manner, the affected part can be found more easily.

  In the first embodiment described above, the display mode of the color endoscope real image is changed by changing the color of the second image area that satisfies the predetermined shape corresponding to the affected area from the surrounding image area. However, the present invention is not limited to this. For example, as shown in FIG. 12, markings (for example, green arrows) indicating the first and second image regions with respect to the virtual endoscope image and the color endoscope real image, respectively. Etc.) A virtual image for display and a real image for display may be generated by adding MG, respectively. That is, for each of the virtual endoscopic image and the color endoscopic image, the first and second image areas may be set to a visual expression format different from the surrounding image areas. Even with such a configuration, similarly to the first embodiment, the presence of marking makes it easy to reliably find a region that satisfies a predetermined shape condition corresponding to the affected part. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  In the first embodiment described above, the display mode of the color endoscopic image and the virtual endoscopic image is changed by changing the color of the image area corresponding to the affected area from the surrounding image area. However, the present invention is not limited to this, and for example, the pixel value may be changed so that the outline of the image area corresponding to the affected part of the color endoscope real image and the virtual endoscope image is emphasized with a predetermined color such as green. That is, for example, an edge (contour) that satisfies a predetermined shape condition in a virtual endoscopic image is set as a first image area, and an image area corresponding to the first image area in a color endoscope real image. May be the second image region. Even with such a configuration, the same effect as in the first embodiment can be obtained.

  In the first embodiment described above, for the surrounding peripheral image areas other than the first image area corresponding to the affected part of the virtual endoscopic image, the color endoscope real image is displayed for each pixel. A pixel value for each corresponding pixel was assigned. However, the present invention is not limited to such a configuration. For example, for the peripheral image region, a virtual image for display is generated while maintaining the color temporarily given when the virtual endoscopic image is generated. As shown, it may be output visually on the monitor 3. Even in such a configuration, in both the color endoscopic image and the display virtual image, the display modes of the respective pixels of the image area corresponding to the affected area and the surrounding image area are different from each other. The display image can be displayed on the monitor 3 in an easy-to-find manner.

  In the second embodiment described above, the display virtual image is generated by mapping the color endoscope real image to the virtual endoscopic image formed by the polygon. However, the present invention is not limited to this. As shown in FIG. 14, the virtual endoscopic image made translucent (FIG. 14 (a)) is aligned with the color endoscope real image (FIG. 14 (b)). The display image (FIG. 14C) may be generated by superimposing the images. Conversely, a display image may be generated by superimposing a semi-transparent color endoscopic image on a virtual endoscopic image. Further, the virtual endoscopic image may not be given a virtual color as a line drawing in which only the outline of each polygon is drawn with a line.

  With such a configuration, a display image in which a virtual endoscopic image with a clear surface unevenness state is superimposed on a color endoscope real image is visually output on the monitor 3, so that the surface structure A display image in which a portion corresponding to an affected part with a large curvature is emphasized can be displayed. That is, the surface shape of each part is easily recognized in the display image. As a result, the image related to the medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  In the first embodiment described above, as shown in FIG. 8, the display real image based on the color endoscope real image and the display virtual image based on the virtual endoscopic image are adjacent to each other. A display image like that arranged in the above was created. However, the present invention is not limited to this configuration. For example, one of a virtual endoscopic image and a color endoscopic image generated by volume rendering is made translucent, and alignment is performed to superimpose both images. A display image may be generated by the above. With this configuration, the monitor 3 visually outputs a display image in which a virtual endoscopic image with a clear surface irregularity state is superimposed on a color endoscope real image. It can be set as the display mode which emphasized the location corresponding to the affected part with a large curvature. That is, an image related to a medical endoscope can be displayed in a manner that makes it easy to find the affected area.

  In the above-described first embodiment, after generating a virtual endoscopic image, the first endoscopic condition is satisfied using a technique such as edge extraction and edge shape recognition for the virtual endoscopic image. An image area was detected. However, the present invention is not limited to this. For example, a region (also referred to as a “first detection region”) that satisfies a predetermined shape condition in the body structure based on a voxel value of the volume data from a three-dimensional region related to the volume data. Detect and generate three-dimensional data (also referred to as “modified three-dimensional data”) in which the data related to the first detection area is changed so that the first detection area can be distinguished from other areas. Then, after reading the changed three-dimensional data, volume rendering is performed, and each pixel of the image area related to the first detection area and the surrounding image area are expressed in different colors, so that it corresponds to the affected area. The virtual endoscope image may be generated by changing the display modes of the respective pixels of the image region related to the first detection region and the surrounding image region. That is, the volume data (in the form in which the predetermined data is given to the area satisfying the predetermined shape (also referred to as “predetermined shape area”) in the three-dimensional area corresponding to the patient's body from the original volume data. (Three-dimensional data) may be changed, temporarily read into the RAM 20b and temporarily stored, and then a virtual endoscopic image may be generated.

  Specifically, based on the voxel value of the volume data, a region having a curvature larger than a predetermined value is detected as a first detection region based on the voxel value of the volume data. By assigning a predetermined color value (for example, a pixel value indicating green) to the voxel related to the detection region, the changed three-dimensional data is generated, and the virtual interior generated by the volume rendering based on the changed three-dimensional data A predetermined color value may be assigned to the image area corresponding to the first detection area so that the endoscopic image has a display mode different from the surrounding image area.

  Then, the first image region (image region corresponding to the predetermined shape region) in the virtual endoscopic image is detected by extracting a portion having a display mode different from the surrounding image region. Can do. The first image area can also be detected by directly detecting predetermined data given to the voxels related to the first detection area.

  Even with such a configuration, the same effect as in the first embodiment can be obtained.

  Further, even when such a configuration is employed, as shown in FIG. 12, the first and second color endoscope real images and virtual endoscopic images corresponding to the predetermined shape regions are of course shown in FIG. You may make it change a pixel value so that it may become the display state to which the marking which each designates an image area | region was added.

  Note that the “patient” in the above-described embodiment includes not only humans but also various animals.

It is a figure which illustrates the outline | summary of the endoscope system which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of an endoscope. It is a flowchart which shows the operation | movement flow which concerns on the production | generation and output of a display image. It is a schematic diagram which shows an example of a color endoscope real image. It is a schematic diagram which shows an example of a virtual endoscopic image. It is a schematic diagram which shows an example of the real image for a display. It is a schematic diagram which shows an example of the virtual image for a display. It is a schematic diagram which shows an example of a display image. It is a flowchart which shows the operation | movement flow which concerns on the production | generation and output of the display image which concerns on 2nd Embodiment. It is a figure explaining a texture mapping. It is a schematic diagram which shows an example of a display image. It is a schematic diagram which shows an example of the display image which concerns on a modification. It is a schematic diagram which shows an example of the display image which concerns on a modification. It is a figure for demonstrating the production | generation of the image for a display which concerns on a modification.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,1A Endoscope system 2 Personal computer 3 Monitor 6 Endoscope 6a Tubing 6b Coupling optical system 6c Scissor part 7 Position and orientation detector 20 Control part 20a ROM
20b RAM
22 storage unit 51 memory card MG marking PG image processing program R1 first image area R2 second image area TD volume data

Claims (19)

A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
3D data related to a 3D area corresponding to the patient's body, an actual image related to the 3D area acquired by a predetermined endoscope, and the 3D area detected by a predetermined position and orientation detection means Reading means for reading position and orientation information related to the position and orientation of the predetermined endoscope into predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Surface image generation means for generating a three-dimensional surface image by surface rendering based on the three-dimensional data in accordance with the image generation conditions;
Display image generation means for generating a display image by mapping the real image to the three-dimensional surface image by texture mapping;
A program comprising: A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
3D data related to a 3D area corresponding to the patient's body, an actual image related to the 3D area acquired by a predetermined endoscope, and the 3D area detected by a predetermined position and orientation detection means Reading means for reading position and orientation information related to the position and orientation of the predetermined endoscope into predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Area detecting means for detecting a first image area satisfying a predetermined shape among the three-dimensional images;
Display image generation means for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image area for the real image;
A program comprising: A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
Of the three-dimensional region corresponding to the patient's body, the three-dimensional data related to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition, and the three-dimensional data acquired by a predetermined endoscope Reading means for reading a real image related to the three-dimensional area and position and orientation information related to the position and orientation of the predetermined endoscope in the three-dimensional area detected by the predetermined position and orientation detection means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
A region detecting means for detecting a first image region corresponding to the predetermined shape region in the three-dimensional image;
Display image generation means for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image area for the real image;
A program comprising: A program according to claim 2 or claim 3, wherein
The display image generating means includes
A program that generates, as the display image, an image in which display modes of pixels of the second image region and the surrounding image region are different from each other. A program according to claim 4, wherein
The display image generating means includes
A program that generates a color image in which the pixels of the second image region and the surrounding image region are displayed in different color representations as the display image. A program according to claim 2 or claim 3, wherein
The display image generating means includes
A program for generating a display image by adding a marking indicating the second image area to the actual image. A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
3D data related to a 3D area corresponding to the patient's body, an actual image related to the 3D area acquired by a predetermined endoscope, and the 3D area detected by a predetermined position and orientation detection means Reading means for reading position and orientation information related to the position and orientation of the predetermined endoscope into predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Area detecting means for detecting a first image area satisfying a predetermined shape among the three-dimensional images;
Based on the three-dimensional image, assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image, the first image region as a surrounding image region Display image generation means for generating a display image by using different visual expression formats;
A program comprising: A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
Of the three-dimensional region corresponding to the patient's body, the three-dimensional data related to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition, and the three-dimensional data acquired by a predetermined endoscope Reading means for reading a real image related to the three-dimensional area and position and orientation information related to the position and orientation of the predetermined endoscope in the three-dimensional area detected by the predetermined position and orientation detection means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
A region detecting means for detecting a first image region corresponding to the predetermined shape region in the three-dimensional image;
Based on the three-dimensional image, assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image, the first image region as a surrounding image region Display image generation means for generating a display image by using different visual expression formats;
A program comprising: A program according to claim 7 or claim 8, wherein
The display image generating means includes
A program that generates, as the display image, an image in which display modes of pixels in the first image region and the surrounding image region are different from each other. A program according to claim 9, wherein
The display image generating means includes
A program that generates a color image in which the pixels of the first image area and the surrounding image area are displayed in different color representations as the display image. A program according to claim 7 or claim 8, wherein
The display image generating means includes
A program for generating a display image by adding a marking indicating the first image area to the three-dimensional image. A program according to any one of claims 2 to 11,
The predetermined shape condition is:
A program characterized in that the condition is that the curvature of the contour of the image area to be determined is larger than a predetermined value. A program that causes a computer to function as an image processing device that generates a display image based on three-dimensional data by being executed in a computer,
The image processing apparatus is
3D data related to a 3D area corresponding to the patient's body, an actual image related to the 3D area acquired by a predetermined endoscope, and the 3D area detected by a predetermined position and orientation detection means Reading means for reading position and orientation information related to the position and orientation of the predetermined endoscope into predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Display image generation means for generating a display image in which the real image and the three-dimensional image are superimposed while aligning the real image and the three-dimensional image;
A program comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Surface image generation means for generating a three-dimensional surface image by surface rendering based on the three-dimensional data in accordance with the image generation conditions;
Display image generation means for generating a display image by mapping the real image to the three-dimensional surface image by texture mapping;
Output means for outputting the display image to the image output device;
An endoscope system comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Area detecting means for detecting a first image area satisfying a predetermined shape among the three-dimensional images;
Display image generation means for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image area for the real image;
Output means for outputting the display image to the image output device;
An endoscope system comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The three-dimensional data is
The three-dimensional data relating to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition in the three-dimensional region,
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
A region detecting means for detecting a first image region corresponding to the predetermined shape region in the three-dimensional image;
Display image generation means for generating a display image by setting a second image area corresponding to the first image area to a visual expression format different from the surrounding image area for the real image;
Output means for outputting the display image to the image output device;
An endoscope system comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Area detecting means for detecting a first image area satisfying a predetermined shape among the three-dimensional images;
Based on the three-dimensional image, assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image, the first image region as a surrounding image region Display image generation means for generating a display image by using different visual expression formats;
Output means for outputting the display image to the image output device;
An endoscope system comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The three-dimensional data is
The three-dimensional data relating to the three-dimensional region in which predetermined data is given to a predetermined shape region that satisfies a predetermined shape condition in the three-dimensional region,
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
A region detecting means for detecting a first image region corresponding to the predetermined shape region in the three-dimensional image;
Based on the three-dimensional image, assigning each pixel value of the real image to each pixel of the three-dimensional image corresponding to each pixel of the real image, the first image region as a surrounding image region Display image generation means for generating a display image by using different visual expression formats;
Output means for outputting the display image to the image output device;
An endoscope system comprising: An endoscope system,
An endoscope for acquiring a real image relating to a three-dimensional region corresponding to a patient's body;
A position and orientation information acquisition device for acquiring position and orientation information related to the position and orientation of the endoscope in the three-dimensional region;
An image processing device for generating a display image based on the three-dimensional data relating to the three-dimensional region;
An image output device for visually outputting the display image;
With
The image processing apparatus is
Reading means for reading the three-dimensional data, the actual image, and the position and orientation information into a predetermined storage means;
Condition setting means for setting image generation conditions according to the position and orientation information;
Three-dimensional image generation means for generating a three-dimensional image based on the three-dimensional data according to the image generation conditions;
Display image generation means for generating a display image in which the real image and the three-dimensional image are superimposed while aligning the real image and the three-dimensional image;
Output means for outputting the display image to the image output device;
An endoscope system comprising:

Images