JP5177668B2 - Stereoscopic image creating apparatus and method, and endoscopy system - Google Patents

Description

  The present invention relates to an apparatus and a method for creating a stereoscopic image that makes a viewer feel a stereoscopic effect using a plurality of images. Furthermore, the present invention relates to a system for creating a stereoscopic image from an endoscopic image.

  Conventionally, a method using a stereo image is well known as a method for giving a stereoscopic effect to an image. In this method, two images (photographs) having parallax are arranged side by side, and one image is viewed with the right eye and the other image is viewed with the left eye (for example, Patent Documents 1 and 2). reference).

  Thus, in order to create a stereo image, two images (photographs) having parallax are required. Usually, in order to obtain two images having parallax, two cameras (imaging devices) arranged at different viewpoint positions are required.

  Various cameras having a special structure provided with a plurality of imaging means so that two cameras having parallax can be captured by one camera have been developed (see, for example, Patent Documents 3 and 4).

Japanese National Patent Publication No. 11-504167 JP 2008-210405 A JP 2001-042464 A JP 2008-167066 A

  When shooting two images with parallax using two cameras, it is necessary to prepare two cameras in advance and place the images at different viewpoint positions before shooting the images. Yes, preparation for shooting is complicated.

  Similarly, when shooting two images with parallax using a single camera having a special structure, it is necessary to prepare a camera having such a special structure in advance. Preparation becomes complicated.

  In the case of the above two examples, two images having parallax need to be captured in advance for a stereo image, and a stereo image cannot be created from an existing image.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a stereoscopic image creation apparatus and creation method that can easily generate a stereoscopic stereo image from an existing image. There is to do.

  The present invention is a shape reconstruction method from shadow (shape from shading technique: Okatani T, Deguchi K. Shape Reconstruction from an Endoscope Image by Shape from Shading Technique for a Point Light Source at the Projection Center.Computer Vision and Image Understanding.Vol. 66; see 119-131, 1997), and extract 3D information with a simple method to create a 3D image. In other words, the present invention converts luminance information of a pixel of a photograph taken by a monocular camera or endoscope having a single optical axis into distance information between the camera and a subject to give unevenness to the photograph, and an appropriate angle on the left and right By displaying the photo with the mark, it gives the original photo a three-dimensional effect. Specifically, it has the following configuration.

  In a first aspect of the present invention, there is provided a stereoscopic image creation device that generates a stereoscopic image including a first image and a second image having parallax and enables stereoscopic viewing from an original image. The stereoscopic image creation apparatus includes image processing means for performing predetermined processing on image data. The image processing means acquires a two-dimensional original image, obtains luminance information of each pixel of the obtained original image, obtains a pseudo depth of each pixel in the depth direction based on the luminance information of each pixel, For each pixel of the original image, pseudo three-dimensional image information including coordinates in the vertical direction, horizontal direction, and depth direction of the image is generated. Thereafter, the image processing means uses the pseudo three-dimensional image information to three-dimensionally rotate the original image rightward by a predetermined angle to generate a first image, and the original image is leftward by a predetermined angle 3 A second image is generated by being rotated dimensionally.

  The image processing means may further comprise display means for displaying the generated first and second images simultaneously or alternately.

In a second aspect of the present invention, there is provided a stereoscopic image creation method for generating, from an original image, a stereo image that includes first and second images having parallax and enables stereoscopic viewing using an information processing device. The The stereoscopic image creation method includes the following steps.
-Obtaining a two-dimensional original image;
-Obtain luminance information of each pixel of the acquired original image, and pseudo-determine the depth in the depth direction of each pixel based on the luminance information of each pixel, and for each pixel of the original image, Generating pseudo three-dimensional image information including coordinates in each of the direction and the depth direction;
-Using the pseudo three-dimensional image information, three-dimensionally rotating the original image rightward by a predetermined angle to generate a first image;
Using the pseudo three-dimensional image information, generating a second image by three-dimensionally rotating the original image leftward by a predetermined angle.

In a third aspect of the present invention, there is provided a program for generating a stereo image including a first image and a second image having parallax and enabling stereoscopic viewing from an original image using an information processing device. . The program causes the control means of the information processing apparatus to execute the following procedure.
-Procedure for acquiring a two-dimensional original image,
-Obtain luminance information of each pixel of the acquired original image, and pseudo-determine the depth in the depth direction of each pixel based on the luminance information of each pixel, and for each pixel of the original image, A procedure for generating pseudo three-dimensional image information including coordinates in the direction and the depth direction,
A procedure for generating the first image by three-dimensionally rotating the original image in the right direction by a predetermined angle using the pseudo three-dimensional image information;
A procedure for generating a second image by three-dimensionally rotating the original image leftward by a predetermined angle using the pseudo three-dimensional image information.

  In a fourth aspect of the present invention, an endoscopic examination system for creating a stereoscopic image from an endoscopic image is provided as an application form of the stereoscopic image creating apparatus. The endoscopy system receives an original image from an endoscope apparatus that captures an image of an inner wall of a tubular structure, and the endoscope apparatus, and generates first and second images from the received original image. And the above-mentioned stereoscopic image creation device. By enabling stereoscopic viewing of the endoscopic image, the state of the inner wall of the tubular structure can be grasped more intuitively, and the inspection accuracy can be improved.

  According to the present invention, luminance information of a pixel of a photograph taken by a monocular camera or endoscope having a single optical axis is converted into distance information between the camera and a subject so that the photograph is uneven, and the right and left are appropriate. Displaying a photograph with an angle gives a three-dimensional effect to the original photograph.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

(Embodiment 1)
1 Configuration of Stereoscopic Image Creating Device FIG. 1 shows a configuration of an embodiment of a stereoscopic image creating device of the present invention. The stereoscopic image creating apparatus 1 is configured by an information processing apparatus such as a personal computer. The stereoscopic image creating apparatus 1 includes a control unit 11 that controls the overall operation, a display unit 17 that performs screen display, an operation unit 19 that is operated by a user, a data storage unit 21 that stores data and programs, and a detachment. And a media drive 23 for reading and writing data from and to a possible recording medium. The display unit 17 is configured by a liquid crystal display, for example, and the operation unit 19 is a keyboard, a mouse, or the like. Further, the stereoscopic image creating apparatus 1 includes an interface 25 for connecting to an external device such as a printer or a network. The removable external recording medium 31 accessed via the media drive 23 includes optical disks such as DVD and CD, memory cards such as SD cards and compact flash (registered trademark), and the like.

  The control unit 11 controls the overall operation of the stereoscopic image creating apparatus 1, and includes a CPU and an MPU, and implements predetermined functions described below by executing predetermined programs. The program executed by the control unit 11 may be provided through a network or from an external recording medium such as a CD-ROM. The control unit 11 includes an image processing unit 13 that performs predetermined image processing to be described later on the image data.

  The data storage unit 21 is means for storing data necessary for processing and a program executed by the control unit 11, and can be configured by a hard disk or a semiconductor memory, for example.

2 Stereo Image Creation Method The stereo image creation device 1 of the present embodiment creates a stereo image (stereoscopic image) used for stereoscopic vision from one original image. Specifically, as shown in FIG. 2, the original image A is three-dimensionally layered, and the original image A is rotated by a predetermined angle α in the left direction and the right direction, respectively. And a stereo image 40 is created by arranging these images side by side as shown in FIG. 3 (a) or (b). By viewing the image B with the right eye and the image C with the left eye in the stereo image 40 created in this way, the viewer can obtain a stereoscopic effect of the image.

3 Operation of Stereoscopic Image Creating Device The specific operation of the stereoscopic image creating device 1 will be described with reference to the flowcharts of FIGS. 4A and 4B. 4A and 4B are executed by the control unit 11.

  When creating a stereoscopic image, image data of the original image A is designated in advance by the user in the stereoscopic image creating apparatus 1. Here, the user designates one image data stored in the data storage unit 21 and the external recording medium 31 via the operation unit 19. As shown in FIG. 4A, the image processing unit 13 of the control unit 11 first reads the image data of the original image A designated by the user via the operation unit 19 into a predetermined work area (S1). The original image data may be acquired via a network.

  Here, in order to create two images B and C rotated from the original image A by a predetermined angle α in the left-right direction, three-dimensional coordinate information is required for the image A. However, the original image A has only two-dimensional information composed of a horizontal direction (x axis) and a vertical direction (y axis) (see FIG. 5). Therefore, in the present embodiment, the image processing unit 13 artificially obtains the coordinates in the depth direction (z-axis direction) of the image from the luminance of the pixel of the original image, and constructs the pseudo three-dimensional image information of the original image A. (S2). The pseudo three-dimensional image information includes coordinates in the horizontal direction (y), vertical direction (x), and depth direction (z) for each pixel. The pseudo three-dimensional information is stored in the data storage unit 21. Details of the pseudo three-dimensional information construction process (S2) will be described later.

  Next, the image processing unit 13 generates images B and C that are three-dimensionally rotated by a predetermined angle α in the left direction and the right direction based on the pseudo three-dimensional image information of the generated original image (S3). ). The generated two images B and C are displayed side by side on the image display unit 17 as a stereo image 40 (S4). Images B and C may be displayed simultaneously or alternately.

  In this way, the stereoscopic image creation device 1 generates a stereo image 40 from one original image A. Further, the image data of the images B, C and / or the stereo image 40 may be stored in the data storage unit 21 so that they can be read and used as necessary.

  Next, with reference to FIG. 4B, the pseudo three-dimensional information construction process in step S2 will be specifically described. This process is executed by the image processing unit 13 of the control unit 11, and in particular, obtains coordinates in the z-axis direction (the depth direction of the image) based on the luminance of each pixel of the image.

  Specifically, in FIG. 4B, the luminance is calculated for each pixel of the original image (S11). At this time, the luminance may be the luminance of any color of red (R), green (G), and blue (B), but basically, the luminance of green, blue, or red depends on the color tone of the subject. It is preferable to properly use luminance information or average luminance information when green, blue, or red is appropriately combined. As a result, a stereo image corresponding to the actual color tone of the subject can be generated.

  Next, the coordinates in the z direction of each pixel are obtained based on the luminance of each pixel (S12). In this case, the z coordinate is determined so that the distance between the objective lens and the portion of the subject corresponding to the pixel is closer as the luminance of the pixel is larger, and the distance is longer as the luminance is smaller (see FIG. 5). ). That is, generally, as the distance between the objective lens of the image pickup apparatus and the subject increases, the amount of light reflected from the subject incident on the objective lens (that is, luminance) decreases. Based on this fact, conversely, it is considered that the distance between the object and the objective lens can be estimated from the luminance of the image. In general, the relationship between the distance from the light source and the luminance (light quantity) has a relationship in which the luminance decreases exponentially according to the distance, as shown in FIG. Therefore, the distance (z coordinate) can be determined from the luminance according to this relationship. The idea of obtaining the z-coordinate based on the luminance as described above is more effective for an image photographed under a condition in which the objective lens of the imaging apparatus and the optical axis of the light source coincide.

  Pseudo three-dimensional image information (pseudo three-dimensional image information) is generated using the z-coordinate obtained for each pixel (S13). Note that the pseudo three-dimensional image information does not necessarily have to be obtained for all the pixels of the image, but may be obtained at least for main observation objects in the image.

  FIG. 7 shows an example of a stereo image created by the stereoscopic image creation device 1 of the present embodiment. FIG. 7A shows an original image. FIG. 7B shows an example of a stereo image created from the original image shown in FIG. 7A with the predetermined angle α = 6 degrees, and FIG. 7C shows the predetermined angle α = 15 degrees. Examples of stereo images created from the original image shown in FIG. The predetermined angle α is preferably 3 to 15 degrees, more preferably 3 to 9 degrees. This is because, when the predetermined angle is too large, the stereoscopic effect is enhanced, but it is far from the image of the actual image, and the image quality is deteriorated.

  As described above, according to the stereoscopic image creating apparatus 1 of the present embodiment, a stereo image can be easily generated from a single original image by simple image processing. For this reason, it is not necessary to prepare two cameras, and it is not necessary to prepare a camera having a complicated structure, and a stereo image can be obtained. In addition, a stereo image can be easily generated using an existing image.

  In the present embodiment, the z coordinate is obtained based on the luminance, and the three-dimensional information is obtained in a pseudo manner rather than exact. For this reason, an art-like image greatly deviating from the actual image may be obtained by appropriately setting the predetermined angle α that is rotated in the left-right direction. Therefore, the stereoscopic image device 1 of the present embodiment may have a secondary effect that an image with a novel design can be generated.

  Further, in the present embodiment, as shown in FIG. 3, the images B and C generated for the right eye and the left eye are arranged side by side, but the images B and C can be seen only by the right eye and the left eye, respectively. When such means (for example, deflection glass) is applied, it is not always necessary to arrange them side by side, and some of them may be displayed in an overlapping manner, or may be displayed with a time difference.

(Embodiment 2)
In the present embodiment, an example of an endoscopic inspection system in which the stereoscopic image creating apparatus is combined with an endoscope used for medical purposes will be described as an example in which the stereoscopic image creating apparatus of Embodiment 1 is applied.

  Conventionally, when using an endoscopic device, an observation region that depends on the field of view of an endoscope inserted into a tubular structure is generally observed and displayed with the naked eye or via a video camera and designated by an observer. The recorded range is recorded by photography or video photography. In this case, the observation and display range is limited to the observation field of the endoscope, and the whole is recorded by repeating local recording. As a result, the entire image cannot be displayed as a single unbroken photograph, so that objectivity is poor in specifying the position of the region of interest. As another method for continuously recording the whole, there is a method of taking a video image, but this method cannot simultaneously display the whole and takes time to browse. Further, the conventional method has a problem that the three-dimensional structure cannot be grasped because only a planar image is recorded.

  The endoscopic inspection system according to the present embodiment can solve the above-described conventional problems by processing a still image obtained from an endoscope with a stereoscopic image generating device to generate a stereo image.

  FIG. 8 shows the system configuration of this embodiment. The endoscopic examination system 100 includes the stereoscopic image creation apparatus 1 described in the first embodiment, and an endoscope device 50 that captures an inner surface of a luminal organ and acquires a video file composed of a plurality of frame images. Prepare. The stereoscopic image creating apparatus 1 and the endoscope device 50 are connected via a cable 51 such as a USB cable.

  Between the endoscope device 50 and the stereoscopic image creating apparatus 1, a video file acquired by the endoscope device 50 is transmitted to the stereoscopic image creating apparatus 1, or a command is sent from the stereoscopic image creating apparatus 1 to the endoscope device 50. Signal can be transmitted. A video capture board may be interposed between the endoscope device 50 and the stereoscopic image creation apparatus 1.

  The endoscope device 50 is, for example, inserted into a luminal organ to control an endoscope scope 42 that images the inner surface of the lumen, and the endoscope scope 42, and is input via the endoscope scope 42. And a control unit 43 for creating a video file based on the signal. The control unit 43 includes a control unit 43a that controls each component in the unit 43, a signal processing unit 43b that creates a video file of the inner surface of the lumen based on a signal input via the endoscope scope 2, And a light source unit 43c which is a light source of illumination light emitted from the distal end portion of the endoscope scope 42 to the observation target. The control unit 43a controls imaging and lighting on / off switching by the endoscope scope 42 and adjusts the amount of light supplied to the endoscope scope 2 from the light source unit 43c according to a user operation.

  The endoscope scope 42 includes an objective lens, a pair of illumination optical fibers, and an operation channel such as a forceps channel and a suction channel, and is provided so as to be exposed at the distal end portion of the endoscope scope 42. It has been. Such a configuration is well known in the art, and the endoscope scope 42 is not limited to this configuration. For example, a configuration in which a single or three or more illumination optical fibers are provided is adopted. May be.

  The stereoscopic image creating apparatus 1 captures a still image in a luminal organ imaged by the endoscope device 50, and creates a stereo image by generating two images obtained by rotating the still image left and right by a predetermined angle. Are displayed on the display unit 17. When the doctor stereoscopically views the stereo image generated in this way, the state of the examination part can be grasped more intuitively, the state of the examination part can be grasped more accurately, and the diagnostic accuracy is improved.

  FIG. 9A is a diagram showing an example of an endoscopic photograph in the large intestine taken by the endoscope device 50, and FIG. 9B is a diagram illustrating a stereoscopic image creating apparatus 1 based on the endoscopic photograph. It is the figure which showed the produced stereo image. When the doctor stereoscopically views such a stereo image, the state of the examination site can be grasped more intuitively, and the state of the tumor or the colon fistula can be grasped more accurately.

(Embodiment 3)
In the present embodiment, a system that incorporates the image construction technique disclosed in International Publication WO2007 / 139187 into the function of the endoscopic inspection system of the second embodiment will be described.

  An image construction device disclosed in International Publication WO2007 / 139187 acquires a moving image or a plurality of still images of an inner wall of a tubular structure photographed by an endoscope. For each image constituting a moving image or a plurality of still images, only pixels on a predetermined test line R set on each image are extracted as shown in FIG. 10A, and the extracted pixels are one-dimensionally extracted. expand. Thereafter, for a plurality of continuous images, as shown in FIG. 10B, the developed images R1R2, R3,... Are combined to generate an image P in which the inner wall of the tubular structure is developed in a plane. FIG. 10C is a diagram showing an example of an image P that is generated from a continuous endoscopic image as shown in FIG. 10A and in which the inner wall of the tubular structure is developed in a plane. Thus, International Publication WO2007 / 139187 discloses a technique for generating an image P in which the inner wall of a tubular structure is developed in a plane.

  The system of the present embodiment generates a stereoscopic image from the image P obtained by developing the inner wall of the tubular structure obtained as described above in a plane using the stereoscopic image device of the first embodiment. The system configuration of the present embodiment is the same as that shown in FIG. 8, but the stereoscopic image creation apparatus 1 further has the above-described image construction function disclosed in International Publication WO2007 / 139187. That is, the stereoscopic image creating apparatus 1 extracts a pixel on a test line from each of a plurality of consecutive images, creates a one-dimensionally developed image, and synthesizes the developed images for the plurality of images, thereby creating a tubular structure It has a function of generating an image in which the inner wall is developed in a plane. Then, the stereoscopic image creating apparatus 1 generates a stereoscopic image from an image in which the inner wall of the tubular structure generated in this way is developed in a plane.

  FIG. 11A is an example of an image taken by the endoscope device 50. FIG. The stereoscopic image creating apparatus 1 extracts pixels on the test line R in the image shown in FIG. 11A and generates a one-dimensionally developed image. A developed image is obtained for a plurality of continuous images, and these are combined to obtain an image in which the inner wall of the tubular structure is developed in a plane. FIG. 11B is an example of an image in which the inner wall of the tubular structure thus obtained is developed in a plane. Further, the stereoscopic image creating apparatus 1 generates left-eye and right-eye stereo images as shown in FIGS. 11C and 11D from the image as shown in FIG. By using the three-dimensional image generated in this way, the state of the examination site can be grasped more intuitively, and the state of the tumor or colon fistula can be grasped more accurately. Note that the images shown in FIGS. 11C and 11D may be displayed by being rotated 90 degrees to 180 degrees.

  In Embodiments 2 and 3, a medical endoscope is used, but an industrial endoscope may be used as long as it can capture an image of the inner wall of the tubular structure. As a result, the state of the examination site can be grasped more intuitively, and the situation of the examination site can be grasped more accurately.

  According to the present invention, a stereo image that enables stereoscopic viewing can be easily generated from one image by image processing. In particular, the present invention is used in a situation where a single light source and an optical axis of the light source are parallel to the optical axis of a camera (objective lens), such as a medical endoscope or an industrial endoscope. Since a stereoscopic image closer to the real object can be obtained, the utility is high.

The figure which shows the structure of the stereo image production apparatus of Embodiment 1 of this invention. The figure explaining the generation process of a stereo image (stereoscopic image) The figure explaining the generated stereo image (stereoscopic image) The flowchart which shows the main process of the stereo image creation operation | movement by the stereo image production apparatus of this embodiment. Flow chart showing pseudo 3D image information construction processing Illustration explaining the x, y, z coordinate axes of the image Diagram showing the relationship between the distance from the light source and the brightness The figure which showed the example of the stereo image produced by the stereo image production apparatus of Embodiment 1 The figure which shows the structure of the endoscopy system of Embodiment 2 of this invention. The figure which showed the example of the stereo image produced by the endoscopy system of Embodiment 2. The figure for demonstrating the production | generation method of the image which expand | deployed planarly the inner wall of the tubular structure in Embodiment 3 of this invention The figure which showed an example of the image which the endoscopy system in Embodiment 3 processes

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Stereoscopic image production apparatus 11 Control part 13 Image processing part 17 Display part 19 Operation part 21 Data storage part 23 Media drive 25 External interface 31 External recording medium 40 Stereo image 50 Endoscopic device 100 Endoscopic inspection system A Original image B Image obtained by rotating the original image leftward by a predetermined angle C Image obtained by rotating the original image rightward by a predetermined angle

Claims (10)

A stereoscopic image creation device that generates a stereoscopic image including a first image and a second image having parallax and enables stereoscopic viewing from an original image,
Image processing means for performing predetermined processing on the image data;
The image processing means includes
Obtain a two-dimensional original image,
The luminance information of each pixel of the acquired original image is obtained, the depth in the depth direction of each pixel is obtained in a pseudo manner based on the luminance information of each pixel, the vertical direction of the image for each pixel of the original image, Generating pseudo three-dimensional image information including coordinates in the horizontal direction and the depth direction,
Using the pseudo three-dimensional image information, the original image is three-dimensionally rotated by a predetermined angle in the right direction to generate the first image,
Using the pseudo three-dimensional image information, the original image is three-dimensionally rotated by a predetermined angle leftward to generate the second image.
A stereoscopic image creation apparatus characterized by the above.   The stereoscopic image creating apparatus according to claim 1, further comprising display means for displaying the first and second images simultaneously or alternately.   2. The stereoscopic image creating apparatus according to claim 1, wherein the image processing unit generates an image in which the first image and the second image are arranged side by side and stores the image in a predetermined storage unit. . A stereoscopic image creation method for generating a stereo image including a first image and a second image having parallax and enabling stereoscopic viewing from an original image using an information processing device,
Obtaining a two-dimensional original image;
The luminance information of each pixel of the acquired original image is obtained, the depth in the depth direction of each pixel is obtained in a pseudo manner based on the luminance information of each pixel, the vertical direction of the image for each pixel of the original image, Generating pseudo three-dimensional image information including coordinates in each of the horizontal direction and the depth direction;
Using the pseudo three-dimensional image information, generating the first image by three-dimensionally rotating the original image by a predetermined angle in the right direction;
Using the pseudo three-dimensional image information, and generating the second image by three-dimensionally rotating the original image leftward by a predetermined angle. Further comprising displaying the first image and the second image simultaneously or alternately,
The stereoscopic image creation method according to claim 4.   5. The stereoscopic image creation method according to claim 4, further comprising the step of generating an image in which the first image and the second image are arranged side by side and storing the image in a predetermined storage unit. A program for generating a stereo image including a first image and a second image having parallax and enabling stereoscopic viewing from an original image using an information processing device,
In the control means of the information processing apparatus,
A procedure for obtaining a two-dimensional original image;
The luminance information of each pixel of the acquired original image is obtained, the depth in the depth direction of each pixel is obtained in a pseudo manner based on the luminance information of each pixel, the vertical direction of the image for each pixel of the original image, A procedure for generating pseudo three-dimensional image information including coordinates in each of the horizontal direction and the depth direction;
Using the pseudo three-dimensional image information, the original image is three-dimensionally rotated by a predetermined angle in the right direction to generate the first image;
Using the pseudo three-dimensional image information, generating the second image by three-dimensionally rotating the original image leftward by a predetermined angle;
A program characterized by having executed.   8. The program according to claim 7, further causing the control means of the information processing apparatus to execute a procedure for displaying the first image and the second image simultaneously or alternately.   The control unit of the information processing apparatus is further caused to execute a procedure of generating an image in which the first image and the second image are arranged side by side and storing the image in a predetermined storage unit. Item 8. The program according to item 7. An endoscope apparatus for capturing an image of the inner wall of the tubular structure;
4. The apparatus according to claim 1, further comprising: an original image received from the endoscope apparatus, and the first and second images generated from the received original image. Endoscopy system characterized by.