JP2009273655A - Image processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing system capable of specifying the shape of an object surface. <P>SOLUTION: This image processing system is provided with a light illumination section for illuminating an illumination light to an object, an image acquisition section for acquiring a surface image, or the image of the surface of the object where the illumination light is illuminated, an illumination direction calculation section for calculating the illumination direction of the illumination light, and a shape calculation section calculating the shape of the surface of the object where the illumination light is illuminated, based on the image content of the surface image and the illumination direction of the illumination light calculated by the illumination direction calculation section. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing system. The present invention particularly relates to an image processing system for processing an image.

A technique has been devised for generating an image signal from which the reflected light component of illumination light is removed by detecting the difference between two video signals due to light of two wavelengths having different light absorption characteristics of the observation site (for example, Patent Documents). 1).
Japanese Patent No. 2643941

  However, in the technique described in the above patent document, information on the shape of the object surface is lost.

  In order to solve the above problem, according to a first embodiment of the present invention, in an image processing system, a light irradiation unit that irradiates an object with irradiation light whose luminance changes spatially with a predetermined luminance change pattern; An irradiation direction calculation unit that calculates the irradiation direction of the irradiation light, an image acquisition unit that acquires a surface image that is an image of the surface of the object irradiated with the irradiation light, a difference in the luminance pattern in the surface image with respect to the luminance change pattern, and A shape calculating unit that calculates the shape of the surface of the object based on the irradiation direction.

  According to a second aspect of the present invention, there is provided an image processing method comprising: a light irradiation stage for irradiating an object with irradiation light whose luminance changes spatially in a predetermined luminance change pattern; and an irradiation direction of the irradiation light. Based on the irradiation direction calculation stage to calculate, the image acquisition stage to acquire the surface image that is the image of the surface of the object irradiated with the irradiation light, the difference of the luminance pattern in the surface image with respect to the luminance change pattern, and the irradiation direction, A shape calculating step of calculating the shape of the surface of the object.

  According to a third aspect of the present invention, there is provided a program for an image processing system, wherein a computer irradiates an object with irradiation light whose luminance changes spatially with a predetermined luminance change pattern, and irradiation An irradiation direction calculation unit that calculates an irradiation direction of light, an image acquisition unit that acquires a surface image that is an image of the surface of an object irradiated with irradiation light, a difference in luminance pattern in a surface image with respect to a luminance change pattern, and an irradiation direction Based on this, it is made to function as a shape calculation unit for calculating the shape of the surface of the object.

  According to the fourth aspect of the present invention, in the image processing system, the irradiation light is irradiated from a light irradiation unit that irradiates the object with irradiation light, and a direction substantially the same as the irradiation direction of the irradiation light irradiated by the light irradiation unit. An imaging unit that captures a surface image that is an image of the surface of the object by imaging the irradiated object, and a positive position that is a position on the surface of the object that regularly reflects the irradiation light based on the luminance value in the surface image. The object at the specular reflection position is obtained by setting the normal direction of the surface in contact with the surface of the object at the specular reflection position to be the irradiation direction of the irradiation light irradiated at the specular reflection position. And a shape calculating unit for calculating the shape of the surface of the surface.

  According to a fifth aspect of the present invention, there is provided an image processing method for imaging an object irradiated with irradiation light from a light irradiation step of irradiating the object with irradiation light and a direction substantially the same as the irradiation direction of the irradiation light. Thus, based on the imaging stage for capturing a surface image, which is an image of the surface of the object, and a normal reflection position that is a position on the surface of the object that regularly reflected the irradiation light based on the luminance value in the surface image, Calculates the shape of the surface of the object at the specular reflection position by calculating the reflection position and setting the normal direction of the surface in contact with the surface of the object at the specular reflection position as the irradiation direction of the irradiation light irradiated at the specular reflection position. A shape calculating step.

  According to the sixth aspect of the present invention, there is provided a program for an image processing stem, wherein the computer has a light irradiation unit that irradiates an object with irradiation light, and a direction substantially the same as the irradiation direction of the irradiation light irradiated by the light irradiation unit From the imaging unit that captures a surface image that is an image of the surface of the object by imaging the object irradiated with the irradiation light, on the surface of the object that regularly reflects the irradiation light based on the luminance value in the surface image Regular reflection position calculation unit for calculating the regular reflection position, which is the position, the normal direction of the surface in contact with the surface of the object at the regular reflection position is set as the irradiation direction of the irradiation light irradiated to the regular reflection position. It functions as a shape calculation unit for calculating the shape of the surface of the object at the position.

  According to a seventh aspect of the present invention, in the image processing system, the first irradiation light that is light in the first wavelength region and the second irradiation light that is light in the second wavelength region shorter than the first wavelength region are provided. A light irradiating unit for irradiating the object, a light reflected from the surface of the object by the first irradiating light irradiated from the light irradiating unit, and a light reflected from the surface of the object by the second irradiating light irradiated from the light irradiating unit The imaging unit that captures the surface image that is the image of the surface of the object, the shadow region specifying unit that specifies the shadow region that is the shadow region from the surface image, and the light that the second irradiation light reflects on the surface of the object The object calculated based on the position and irradiation direction of the shadow area specified from the surface image by the light reflected from the surface of the object based on the position and irradiation direction of the shadow area specified from the surface image by Higher spatial resolution than the surface resolution of the surface , And a shape calculation section that calculates the shape of the surface of the object.

  According to an eighth aspect of the present invention, there is provided an image processing method comprising: first irradiation light that is light in a first wavelength region; and second irradiation light that is light in a second wavelength region shorter than the first wavelength region. The light irradiation stage for irradiating the object, the light reflected from the surface of the object by the first irradiation light irradiated in the light irradiation stage, and the light reflected from the surface of the object by the second irradiation light irradiated in the light irradiation stage The imaging step of capturing a surface image that is an image of the surface of the object, the shadow region specifying step of specifying the shadow region that is the shadow region from the surface image, and the light that the second irradiation light reflects on the surface of the object The object calculated based on the position and irradiation direction of the shadow area specified from the surface image by the light reflected from the surface of the object based on the position and irradiation direction of the shadow area specified from the surface image by Than the spatial resolution of the surface shape In There spatial resolution, and a shape calculation step of calculating the shape of the surface of the object.

  According to a ninth aspect of the present invention, there is provided a program for an image processing system, wherein a computer is irradiated with first irradiation light that is light in a first wavelength region and light in a second wavelength region that is shorter than the first wavelength region. A light irradiation unit that irradiates an object with a certain second irradiation light, a light that is reflected from the surface of the object by the first irradiation light irradiated from the light irradiation unit, and a second irradiation light that is irradiated from the light irradiation unit An imaging unit that captures a surface image that is an image of the surface of the object by light reflected from the surface, a shadow region specifying unit that specifies a shadow region that is a shadow region from the surface image, and the second irradiation light on the surface of the object Based on the position and irradiation direction of the shadow area specified from the surface image by the light reflected from the surface of the object based on the position and irradiation direction of the shadow area specified from the surface image by the reflected light Calculated object surface shape space With high spatial resolution than resolution, to function as a shape calculation section that calculates the shape of the surface of the object.

  According to the tenth aspect of the present invention, the image processing system is a plurality of surface images that are images of the surface of an object captured from different positions, and the peripheral images in the surface image from the central image region. An image acquisition unit that acquires a plurality of surface images with reduced image quality in a region, and an imaging unit that images the surface images when each of the plurality of surface images is captured based on the image contents of the plurality of surface images; A positional relationship identifying unit that identifies a positional relationship with the object; and a shape calculating unit that calculates the shape of the surface based on the image contents of the plurality of surface images and the positional relationship identified by the positional relationship identifying unit. .

  According to an eleventh aspect of the present invention, there is provided an image processing method, which is a plurality of surface images that are images of the surface of an object each captured by an imaging unit from different positions, and in the surface image from a central image region. An image acquisition stage for acquiring a plurality of surface images with reduced image quality in a peripheral image region, and an imaging unit and an object when each of the plurality of surface images is captured based on the image contents of the plurality of surface images A positional relationship specifying step of specifying a positional relationship between the images, and a shape calculating step of calculating the shape of the surface based on the image contents of the plurality of surface images and the positional relationship specified in the positional relationship specifying step.

  According to a twelfth aspect of the present invention, there is provided a program for an image processing system, wherein a computer is a plurality of surface images each of which is an image of a surface of an object imaged from different positions. An image acquisition unit that acquires a plurality of surface images with reduced image quality in a peripheral image region of the surface image, and the surface image when each of the plurality of surface images is captured based on the image contents of the plurality of surface images. Shape calculation that calculates the shape of the surface based on the positional relationship specifying unit that specifies the positional relationship between the imaged imaging unit and the object, the image contents of the plurality of surface images, and the positional relationship specified by the positional relationship specifying unit Function as a part.

  According to a thirteenth aspect of the present invention, in the image processing system, an image acquisition unit that acquires a surface image that is an image of the surface of an object, and a specific subject that is distributed in a predetermined pattern on the surface of the object in the surface image And a shape calculation unit that calculates the shape of the surface of the object based on the distribution of the specific object imaged.

  According to a fourteenth aspect of the present invention, there is provided an image processing method for acquiring a surface image that is an image of the surface of an object, and a specific subject distributed in a predetermined pattern on the surface of the object in the surface image And a shape calculating step of calculating the shape of the surface of the object based on the distribution of the specific object imaged.

  According to a fifteenth aspect of the present invention, there is provided a program for an image processing system, wherein the computer has an image acquisition unit that acquires a surface image that is an image of the surface of the object, and a predetermined pattern on the surface of the object in the surface image Based on the distribution of the specific object obtained by imaging the specific subject distributed in (2), it is made to function as a shape calculation unit that calculates the shape of the surface of the object.

  The above summary of the invention does not enumerate all necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

  Hereinafter, although this invention is demonstrated through embodiment of invention, the following embodiment does not limit the invention concerning a claim. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 shows an example of the configuration of an endoscope system 10 according to an embodiment, together with a living body 190. The endoscope system 10 identifies the shape of the surface from a surface image that is an image of the surface of the living body 190 obtained by imaging the living body 190. Further, the endoscope system 10 generates an output image in which the surface shape is emphasized based on the identified surface shape. Further, the endoscope system 10 specifies a site of an abnormal tissue such as cancer or tumor in the living body 190 based on the specified surface shape.

  The endoscope system 10 includes a scope 100 as a part of an endoscope, forceps 135, an image processing unit 140, a control unit 105, a light irradiation unit 120, an operation unit 160, and an output unit 150. In FIG. 1, an A portion shows an enlarged view of the distal end portion 102 of the scope 100. The endoscope system 10 can function as an imaging system or an image processing system, as will be described below. The living body 190 may be an example of an object in the present invention.

  The scope 100 has an imaging unit 110 and a forceps port 130 and incorporates a light guide 124 as a part of the light irradiation unit 120. The distal end portion 102 of the scope 100 includes a nozzle 138, a lens 112 as a part of the imaging unit 110, and an emission port 128 as a part of the light guide 124. The light guide 124 and the emission port 128 built in the scope 100 function as the light irradiation unit 120 together with the light emitting unit 122 provided outside the scope 100. The scope 100 can also incorporate a light emitting unit 122.

  A forceps 135 is inserted into the forceps port 130, and the forceps port 130 guides the forceps 135 to the distal end portion 102. Note that the forceps 135 may have various tip shapes. In addition to the forceps, various treatment tools for treating the living body 190 may be inserted into the forceps port 130. The nozzle 138 sends water or air to the living body 190.

  The light emitting unit 122 generates light that is emitted from the emission port 128 toward the living body 190. The light emitting unit 122 may generate visible light including light of a red component, a green component, and a blue component. The light emitting unit 122 may be a halogen lamp. In addition, the light emitting unit 122 may generate invisible light such as infrared light. Further, the light emitting unit 122 may generate light including visible light and invisible light.

  The light guide 124 is formed of an optical fiber as an example. The light guide 124 guides the light generated by the light emitting unit 122 to the emission port 128 of the distal end portion 102 of the scope 100. The light generated by the light emitting unit 122 and emitted from the emission port 128 is applied to the living body 190 as irradiation light. In addition, irradiation light can function as light irradiated in order to calculate the shape of the surface of the biological body 190 so that it may demonstrate later. Further, the irradiation light can also function as illumination light for illuminating the living body 190.

  Note that the scope 100 is inserted into an organism as an example. And the imaging part 110 images the biological body 190 inside a living organism with irradiation light. Specifically, the imaging unit 110 captures a surface image that is an image of the surface 192 of the living body 190 with the light from the living body 190 irradiated with the irradiation light. Specifically, the imaging unit 110 receives reflected light that is light reflected by the living body 190 and captures an image of the living body 190. The living body 190 can be exemplified by the digestive mucosa of a living organism such as the mucous membrane of the stomach and the digestive tract of the intestine.

  The image processing unit 140 processes the surface image obtained by the imaging unit 110. For example, the image processing unit 140 analyzes the image content of the surface image and generates an analysis result. For example, the image processing unit 140 analyzes the image content of the surface image and generates an analysis result indicating the surface shape of the living body 190. Specifically, the image processing unit 140 analyzes the surface image to identify the positions of the recesses 196a and the projections 194a and 194b included in the surface 192, and the recesses 196a and 196b included in the surface 192. By doing so, an analysis result indicating the shape of the surface 192 is generated.

  In addition, the convex part 194a and the concave part 196a may be an unevenness of a space scale expected when an abnormal tissue is generated in the living body 190. Further, the convex portion 194b and the concave portion 196b may be irregularities on a space scale that the normal tissue of the living body 190 originally has.

  The image processing unit 140 may generate an output image by performing image processing on the surface image based on the surface shape. For example, the image processing unit 140 may generate an output image in which an image region indicating the convex portion 194a and the concave portion 196a in the surface image is emphasized from an image region indicating the convex portion 194b and the concave portion 196b in the surface image. Then, the image processing unit 140 supplies at least one of the analysis result or the output image to the output unit 150.

  The output unit 150 outputs the analysis result or the output image supplied from the image processing unit 140. For example, the output unit 150 may record the output image on a recording medium such as a nonvolatile memory. The output unit 150 may display an output image. The output unit 150 may display the analysis result or record the analysis result on a recording medium.

  Note that the operation unit 160 acquires an operation instruction from the user. Specifically, the operation unit 160 acquires an instruction including control content for controlling the scope 100 from the user. For example, the operation unit 160 acquires an instruction including a driving amount for driving the distal end portion 102 of the scope 100 from the user. As an example, the operation unit 160 has an angle control knob that is rotated by the user when the angle of the distal end portion 102 of the scope 100 is changed, and an instruction including the amount of rotation that the angle control knob is rotated by the user. Alternatively, an instruction including a drive amount for driving the distal end portion 102 of the scope 100 may be acquired from the user.

  The control unit 105 controls the scope 100 based on the instruction acquired by the operation unit 160 from the user. For example, the control unit 105 may drive the scope 100 based on the instruction. For example, the control unit 105 may change the angle of the distal end portion 102 of the scope 100 by an amount corresponding to the rotation amount acquired by the operation unit 160.

  In addition, the control unit 105 may control the generation of light by the light emitting unit 122. For example, the control unit 105 may control the wavelength of light generated by the light emitting unit 122. Specifically, the control unit 105 may control the timing at which the light emitting unit 122 generates at least one of visible light and invisible light. Note that the control unit 105 may control the generation of light by the light emitting unit 122 based on the instruction acquired by the operation unit 160.

  Further, the control unit 105 may control the contents of image processing performed by the image processing unit 140. For example, the control unit 105 may determine the type of enhancement processing for emphasizing the surface shape based on an instruction from the user acquired by the operation unit 160. Then, the image processing unit 140 may generate an output image by performing the type of enhancement processing determined by the control unit 105 on the surface image.

  Thus, according to the endoscope system 10, the shape of the surface 192 of the living body 190 can be specified. The endoscope system 10 can provide an output image that has been subjected to appropriate image processing according to the shape of the surface 192.

  FIG. 2 shows an example of a block configuration of the image processing unit 140. The image processing unit 140 includes an image acquisition unit 210, a regular reflection position calculation unit 240, a shadow region specification unit 250, a positional relationship specification unit 270, a shape calculation unit 200, and an output image generation unit 290. The positional relationship specifying unit 270 includes a speed calculation unit 242, a control amount acquisition unit 244, and a positional relationship calculation unit 260. Further, the positional relationship calculation unit 260 includes an imaging direction calculation unit 220 and an irradiation direction calculation unit 230.

  The image acquisition unit 210 acquires a surface image that is an image of the surface of the living body 190. Specifically, the image acquisition unit 210 acquires a surface image captured by the imaging unit 110. More specifically, the image acquisition unit 210 acquires a surface image that is an image of the surface of the living body 190 irradiated with the irradiation light.

  Then, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the image content of the surface image. Specifically, the shape calculation unit 200 calculates the shape of the surface of the living body 190 irradiated with the irradiation light based on the image content of the surface image. For example, as will be described later, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the luminance value of the surface image.

  More specifically, the shape calculation unit 200 based on the luminance value of the surface image, at least one of the position of the specific object that the surface image has, the interval between the specific objects, and the size of the specific object And the shape of the surface of the living body 190 may be calculated based on the position of the specified specific object, the interval between the specific objects, and the size of the specific object. As will be described later, the specific object includes a luminance pattern image that appears on the surface 192 when the surface 192 is illuminated with a specific luminance pattern, a pit image that appears on the surface 192, a specular image in the surface image, Includes the shadow image included in the surface image.

  The regular reflection position calculation unit 240 calculates a regular reflection position, which is a position on the surface of the living body 190 that regularly reflected the irradiation light, based on the luminance value in the surface image. And the shape calculation part 200 calculates the shape of the surface in a regular reflection position based on the irradiation direction of irradiation light.

  Specifically, the shape calculation unit 200 sets the normal direction of the surface in contact with the surface of the living body 190 at the specular reflection position as the irradiation direction of the irradiation light irradiated to the specular reflection position, so that the living body at the specular reflection position. The surface shape of 190 is calculated. The shape calculation unit 200 uses the normal direction as the irradiation direction of the irradiation light irradiated to the regular reflection position when the surface image whose regular reflection position is calculated by the regular reflection position calculation unit 240 is captured. Thus, it goes without saying that the shape of the surface of the living body 190 at the regular reflection position is calculated. Further, when the shape calculation unit 200 calculates the shape of the surface based on the regular reflection position, the imaging unit 110 is irradiated with irradiation light from a direction substantially the same as the irradiation direction of the irradiation light irradiated by the light irradiation unit 120. It is assumed that a surface image is captured by capturing the living body 190.

  The light irradiation unit 120 may irradiate the living body 190 with irradiation light from a plurality of different irradiation directions. Then, the imaging unit 110 images the living body 190 irradiated with the irradiation light from the plurality of different irradiation directions by the light irradiation unit 120, respectively, and thereby the plurality of surfaces irradiated with the irradiation light from the plurality of different irradiation directions, respectively. An image may be taken. Then, the regular reflection position calculation unit 240 may calculate a plurality of regular reflection positions, which are positions on the surface of the living body 190 that regularly reflected the irradiation light, based on the luminance value in each of the plurality of surface images. Then, the shape calculation unit 200 captures a surface image in which each normal reflection position of each of the plurality of regular reflection positions is specified with respect to a normal direction of a surface in contact with the surface of the living body 190 at each of the plurality of regular reflection positions. In this case, the shape of the surface of the living body 190 at each of the plurality of regular reflection positions may be calculated by using the irradiation direction of the irradiation light that has been irradiated to each of the plurality of regular reflection positions.

  Thereby, the shape calculation part 200 can calculate the surface shape of a wide area | region instead of a spot-like small area | region. Note that the light irradiation unit 120 has different timings because the direction of the irradiation light irradiated from the exit port 128 changes with time because the angle of the distal end portion 102 of the scope 100 changes with time according to an instruction from the user. The living body 190 may be irradiated with irradiation light from a plurality of irradiation directions. In this case, the specular reflection position calculation unit 240 starts from different directions based on the image contents of each of the plurality of display images captured when the user changes the position / orientation of the scope 100 to observe the surface of the living body. It is possible to calculate the regular reflection position when the irradiation light is irradiated. Thereby, the shape calculation part 200 can calculate a surface shape in the middle of the observation act by a user.

  The shadow area specifying unit 250 specifies a shadow area that is a shadow area from the surface image. Specifically, the shadow area specifying unit 250 may specify the shadow area from the surface image based on the luminance value in the surface image. Then, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the position of the shadow region and the irradiation direction. Specifically, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the position and shape of the shadow region and the irradiation direction of the irradiation light. In addition, the shape calculation unit 200 calculates the shape of the surface of the living body 190 by calculating the height of the surface of the living body 190 based on the position and shape of the shadow region and the irradiation direction of the irradiation light.

  The shape calculation unit 200 is adjacent to the region on the surface of the living body 190 corresponding to the shadow region, and the height of the region close to the light irradiation unit 120 that irradiates the living body 190 with the irradiation light corresponds to the shadow region. Higher than the area on the surface of the living body 190 to be calculated. In addition, when the length of the shadow region in the irradiation direction is longer, the shape calculation unit 200 may calculate the height of the region close to the light irradiation unit 120 higher. In addition, when the imaging unit 110 captures a surface image by capturing the living body 190 from substantially the same direction as the irradiation direction of the irradiation light, the shape calculation unit 200 displays on the surface of the living body 190 indicated by the shadow region. The height of the region adjacent to the shadow region and close to the imaging unit 110 is calculated to be higher than the region on the surface of the living body 190 indicated by the shadow region.

  Note that the output image generation unit 290 generates an output image by performing image processing on the surface image based on the shape of the surface 192 calculated by the shape calculation unit 200. As an example, the output image generation unit 290 may generate an output image by performing shading processing, texture processing, specular processing, gray scale processing, color map processing, and wire frame processing according to the surface shape on the surface image. . In this way, the output image generation unit 290 can generate an output image in which the three-dimensional structure indicated by the shape of the surface 192 calculated by the shape calculation unit 200 is emphasized. The output image generated by the output image generation unit 290 is supplied to the output unit 150, and is recorded or displayed on the output unit 150.

  Note that the output image generation unit 290 may generate an output image obtained by rotating the surface image based on an instruction from the operation unit 160 by the user. For example, the output image generation unit 290 may acquire an instruction including an image rotation amount from the operation unit 160 and generate an output image obtained by rotating the surface image by the rotation amount. Thereby, according to the endoscope system 10, it is possible to provide an image to be obtained when the living body 190 is viewed from the direction desired by the user without actually rotating the scope 100.

  The positional relationship specifying unit 270 specifies the positional relationship between the living body 190 and the imaging unit 110 or the positional relationship between the living body 190 and the light irradiation unit 120. Then, the shape calculating unit 200 may specify the shape of the surface of the living body 190 based on the positional relationship specified by the positional relationship specifying unit 270. Note that the positional relationship between the living body 190 and the imaging unit 110 in the present invention may mean the positional relationship between the living body 190 and the lens 112. In addition, the positional relationship between the living body 190 and the light irradiation unit 120 in the present invention may mean the positional relationship between the living body 190 and the emission port 128.

  Note that, when the lens 112 and the emission port 128 are provided at the distal end portion 102 of the scope 100 as in the present embodiment, the positional relationship between the living body 190 and the imaging unit 110 or the living body 190 and the light irradiation unit 120. The positional relationship between the living body 190 and the distal end portion 102 may mean a positional relationship between the living body 190 and the distal end portion 102. In addition, the positional relationship between the living body 190 and the imaging unit 110 may further include information indicating the imaging direction of the imaging unit 110. Moreover, the positional relationship between the living body 190 and the light irradiation unit 120 may further include information indicating the irradiation direction of the irradiation light. Further, when the imaging direction of the imaging unit 110 and the irradiation direction of the irradiation light are substantially the same, the positional relationship between the living body 190 and the distal end portion 102 further includes information indicating the orientation of the distal end portion 102. Good.

  For example, the positional relationship specifying unit 270 specifies the positional relationship between the imaging unit 110 and the living body 190 when each of the plurality of surface images is captured based on the image contents of the plurality of surface images. Then, the shape calculating unit 200 calculates the surface shape based on the image contents of the plurality of surface images and the positional relationship specified by the positional relationship specifying unit 270.

  Specifically, the imaging unit 110 captures a plurality of images from different positions with respect to the living body 190 by continuously capturing a surface image while the position of the distal end portion 102 of the scope 100 is changing according to an instruction from the user. The surface image of is taken. Then, the image acquisition unit 210 acquires the plurality of surface images. Then, the speed calculation unit 242 calculates the speed at which the imaging unit 110 moves relative to the living body 190 based on the image contents of the plurality of surface images. Note that the speed at which the imaging unit 110 moves relative to the living body 190 may mean the speed at which the lens 112 moves relative to the living body 190. For example, the speed calculation unit 242 specifies the position of the object indicating the same subject from each of the plurality of surface images, and calculates the speed at which the imaging unit 110 moves relative to the living body 190 based on the specified position.

  Then, the positional relationship calculation unit 260 calculates the positional relationship based on the speed calculated by the speed calculation unit 242 and the timing at which the imaging unit 110 images each of the plurality of surface images. Specifically, the positional relationship calculation unit 260 is connected to the imaging unit 110 at another timing based on the positional relationship between the imaging unit 110 and the living body 190 at a specific timing and the speed calculated by the speed calculation unit 242. The positional relationship with the living body 190 is calculated.

  More specifically, the imaging direction calculation unit 220 calculates the imaging direction at another timing based on the imaging direction at a specific timing and the speed calculated by the speed calculation unit 242. Further, the irradiation direction calculation unit 230 calculates the irradiation direction of irradiation light at other timings based on the irradiation direction of irradiation light at a specific timing and the speed calculated by the speed calculation unit 242. When the relationship between the imaging direction and the irradiation direction is determined in advance, the imaging direction calculation unit 220 calculates the imaging direction based on the relationship and the irradiation direction calculated by the irradiation direction calculation unit 230. Can do. Similarly, the irradiation direction calculation unit 230 can calculate the irradiation direction based on the relationship and the imaging direction calculated by the imaging direction calculation unit 220.

  Then, the shape calculation unit 200 calculates the surface shape based on at least one of the irradiation direction calculated by the irradiation direction calculation unit 230 and the imaging direction calculated by the imaging direction calculation unit 220 and the image contents of the plurality of surface images. To do. For example, the image acquisition unit 210 acquires a plurality of surface images captured from different directions by the imaging unit 110 that captures a surface image. Then, the shape calculation unit 200 may calculate the shape of the surface based on the parallax information in the plurality of surface images obtained by imaging from different imaging directions. Further, the shape calculation unit 200 may calculate the shape of the surface based on the luminance difference between the plurality of surface images obtained by imaging when the irradiation light is irradiated from different irradiation directions. .

  As described above, the shape calculation unit 200 calculates the shape of the surface based on the image contents of the plurality of surface images and the positional relationship calculated by the positional relationship calculation unit 260. The shape calculation unit 200 calculates the inclination of the surface of the living body 190 irradiated with the irradiation light with respect to the irradiation direction irradiated with the irradiation light based on the image content of the surface image and the irradiation direction. The surface shape of 190 may be calculated.

  The positional relationship calculation unit 260 can also calculate the positional relationship based on a control amount that controls the position of the distal end portion 102 of the scope 100. Specifically, the positional relationship calculation unit 260 may calculate the positional relationship based on a control value that the control unit 105 controls the position of the distal end portion 102 of the scope 100. For example, the positional relationship calculation unit 260 may calculate the positional relationship based on a control value for controlling a motor that causes the control unit 105 to bend the scope 100. The control amount for controlling the position of the distal end portion 102 of the scope 100 may be an example of the control amount for controlling the irradiation direction of the irradiation light in the present invention.

  And the irradiation direction calculation part 230 calculates the irradiation direction with which irradiation light was irradiated based on the control amount which the control amount acquisition part 244 acquired. For example, the irradiation direction calculation unit 230 stores the amount of change in the irradiation direction in advance in association with the control amount, and the irradiation direction calculation unit 230 stores in association with the control amount acquired by the control amount acquisition unit 244. Based on the amount of change in the irradiation direction and the irradiation direction before the control by the control amount, the irradiation direction after the control by the control amount can be calculated. Similarly, the imaging direction calculation unit 220 can calculate the imaging direction based on the control amount acquired by the control amount acquisition unit 244.

  In the above description, the shape calculation unit 200 is configured to display the shadow image in the surface image, the specular reflection position specified from the surface image, parallax information based on a plurality of surface images captured from different imaging directions, or different irradiation directions. The surface shape was calculated based on luminance information based on a plurality of surface images picked up by light. In addition, the shape calculation unit 200 can extract a specific image pattern from the surface image and calculate the shape of the surface of the living body 190 based on the extracted image pattern. A calculation method in which the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the image pattern will be described later with reference to FIGS.

  As described above, the position of the tip portion 102 is changed by an instruction from the user, whereby the position of the lens 112 or the position of the emission port 128 changes. According to the endoscope system 10, the shape of the surface of the living body 190 can be calculated from the surface image captured while changing the position. In addition, according to the endoscope system 10, an output image that can easily detect abnormal tissue may be provided by performing appropriate image processing according to the shape of the surface 192. Further, according to the endoscope system 10, there is a case where the position of the abnormal tissue or the degree of abnormality thereof can be estimated by extracting the unevenness of the spatial scale peculiar to the abnormal tissue based on the calculated shape. Further, according to the endoscope system 10, the degree of abnormality of the abnormal tissue is determined by specifying the depth of the concave portion of the abnormal tissue, the height of the convex portion of the abnormal tissue, or the shape thereof based on the shape of the surface 192. It may be possible to estimate.

  FIG. 3 shows an example of the configuration of the imaging unit 110. The imaging unit 110 includes a lens 112, a filter unit 310, a light receiving unit 300, and an image generation unit 330. The lens 112 causes the light from the living body 190 to form an image on the light receiving unit 300. The filter unit 310 may be a color filter in which a plurality of color filter elements that selectively transmit light in different wavelength regions are arranged in a matrix on substantially the same plane. The image generation unit 330 generates a surface image based on the amount of light received by the light receiving element included in the light receiving unit 300 described with reference to FIG. The surface image generated by the image generation unit 330 is supplied to the image acquisition unit 210.

  FIG. 4 shows an example of the arrangement of the light receiving elements in the light receiving unit 300. The light receiving unit 300 includes an IR light receiving element 410 that receives infrared component light, an R light receiving element 420 that receives red component light, a G light receiving element 430 that receives green component light, and a blue component light. A light receiving unit 400 having a B light receiving element 440 for receiving light. The light receiving unit 300 is formed by two-dimensionally arranging light receiving units having the same arrangement as the light receiving unit 400. In this figure, the IR light receiving element 410 is formed in the region to which IR is attached, the R light receiving element 420 is formed in the region to which R is attached, and the G light is applied to the region to which G is attached. A light receiving element 430 is formed, and a B light receiving element 440 is formed in a region marked with B.

  As described above, the light receiving unit 300 includes an IR light receiving element 410, an R light receiving element 420, a G light receiving element 430, and a B light receiving element 440 that selectively receive light in a specific wavelength region, respectively, in a predetermined pattern. And arranged in a matrix. The filter unit 310 is provided with color filter elements at positions corresponding to the respective positions of the plurality of light receiving elements of the light receiving unit 300. And each of the some light receiving element which the light-receiving part 300 has receives the light which the color filter element provided in the corresponding position permeate | transmitted.

  Note that the light irradiation unit 120 irradiates light including infrared component light, red component light, green component light, and blue component light. The light receiving unit 300 receives the return light that returns from the living body 190 when irradiated with irradiation light including infrared component light, red component light, green component light, and blue component light. Specifically, the light receiving unit 300 receives the reflected light reflected by the surface 192 of the living body 190 when the irradiation light irradiated from the light irradiation unit 120 is present. For example, the IR light receiving element 410, the R light receiving element 420, the G light receiving element 430, and the B light receiving element 440 respectively include infrared component light, red component light, and green component light included in the reflected light. , And light including blue component light.

  The light irradiation unit 120 may irradiate visible light including red component light, green component light, and blue component light as illumination light for illuminating the living body 190. And the light irradiation part 120 irradiates the light of the infrared component as an example of invisible light as irradiation light for calculating the surface 192 of the biological body 190. Then, the shape calculation unit 200 may calculate the surface 192 of the living body 190 based on the surface image based on the light received by the plurality of IR light receiving elements 410 included in the light receiving unit 300.

  Thus, the light irradiation part 120 irradiates the biological body 190 with the irradiation light which is invisible light as an example. Then, the shape calculation unit 200 determines the shape of the surface of the living body 190 based on the image content of the surface image that is an image of the surface of the living body 190 obtained by imaging the surface of the living body 190 with the irradiation light that is invisible light, and the irradiation direction. Is calculated.

  In addition, the light irradiation part 120 can control the light intensity of invisible light independently of the intensity of illumination light. For this reason, the shape calculation unit 200 may be able to calculate the surface shape with a higher SN ratio. Further, according to the endoscope system 10, a surface image obtained by illumination with visible light and a surface image for shape calculation generated by invisible light can be obtained simultaneously. Therefore, according to the endoscope system 10, even when the distal end portion 102 is moving at high speed, the surface shape is calculated with higher time resolution, and the surface image is displayed based on the calculated shape of the surface 192. Processing can be performed.

  FIG. 5 schematically shows an example of a change in the position of the distal end portion 102. The control unit 105 moves the distal end portion 102 in accordance with an instruction from the user acquired by the operation unit 160. The direction of the optical axis of the lens 112 and the position of the lens 112, the irradiation direction of the irradiation light emitted from the emission port 128, and the position of the emission port 128 change according to the movement of the distal end portion 102.

  The imaging unit 110 generates a plurality of surface images by continuously imaging the surface 192 while the tip portion 102 is moving under the control of the control unit 105 based on an instruction from the user. Then, as described with reference to FIG. 2, the image processing unit 140, in addition to the image contents of the plurality of surface images, the direction of the optical axis of the lens 112 at each of the timings at which the plurality of surface images are captured, The shape of the surface 192 can be calculated based on at least one of the position of the lens 112, the irradiation direction of the irradiation light emitted from the emission port 128, and the position of the emission port 128.

  FIG. 6 shows an example of a method for calculating the surface shape based on the luminance pattern of the surface image. For example, the light irradiation unit 120 irradiates irradiation light whose light amount spatially changes in a predetermined pattern. For example, the light irradiation unit 120 irradiates the living body 190 with irradiation light having a predetermined rectangular pattern on a surface perpendicular to the irradiation direction of the irradiation light. As an example, the rectangular pattern may be a square pattern. Further, the light irradiation unit 120 may irradiate irradiation light that is light that diverges with respect to the irradiation axis 630 of the irradiation light.

  For example, it is assumed that the light irradiation unit 120 emits irradiation light in which a square pattern is drawn on a surface 610 perpendicular to the irradiation direction, as shown in FIG. In this case, a surface image in which squares are arranged at the same interval as in the image 650 is captured by the imaging unit 110. Here, when the surface of the object is inclined in the x direction from a plane perpendicular to the irradiation direction as indicated by a plane 620, a surface image having a deformed rectangular pattern as in the image 660 is captured by the imaging unit 110. Is imaged. For example, in the image 660, since a rectangular pattern deformed in the direction indicated by the direction 670 appears in the drawing, it can be determined that the surface 620 is inclined in the direction indicated by the direction 670 in the image 660. As described above, the shape calculation unit 200 determines the inclination angle of the tangential plane of the surface with respect to the surface perpendicular to the irradiation axis 630 based on the deformation amount of the shape of the luminance pattern extracted from the surface image from the shape of the irradiation pattern. By calculating, the shape of the surface of the living body 190 can be calculated.

  Therefore, if the irradiation direction of the irradiation light is known, the shape calculation unit 200 can calculate the inclination of the object surface based on the deformation amount of the rectangular pattern appearing in the surface image from the square pattern. Thus, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the difference in the luminance pattern in the surface image with respect to the luminance change pattern and the irradiation direction.

  For example, it is assumed that the light irradiation unit 120 emits irradiation light whose luminance changes with a luminance change pattern in which the luminance becomes maximum at predetermined intervals on a predetermined irradiation surface. In this case, the shape calculation unit 200 specifies the corresponding position that is a position on the surface image corresponding to the position where the luminance of the irradiation light is maximized on the surface of the living body 190, the interval between the specified corresponding positions, and the irradiation direction. Based on this, the shape of the surface of the living body 190 is calculated. For example, the shape calculation unit 200 may specify an interval between positions where the luminance is maximum in the surface image, and calculate the shape of the surface of the living body 190 based on the specified interval between corresponding positions and the irradiation direction. it can. As described above, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the difference in the interval between the corresponding positions with respect to the interval between the positions where the luminance of the irradiation light reaches the maximum value on the predetermined irradiation surface and the irradiation direction. can do.

  In addition, the shape calculation unit 200 can calculate a larger inclination angle of the object surface with respect to a plane perpendicular to the irradiation direction when the degree of deformation in the surface image is larger. Further, as shown in the image 660, the length of one side of the rectangular pattern changes in the inclination direction of the surface 620. Therefore, the shape calculation unit 200 can calculate the shape of the object surface by regarding the inclination direction of the object surface from the plane perpendicular to the irradiation direction as the direction in which the length of one side of the rectangle is changing. it can.

  As described above, the light irradiation unit 120 emits irradiation light whose luminance changes spatially in a predetermined luminance change pattern. Then, the shape calculation unit 200 calculates the shape of the surface of the living body 190 based on the luminance pattern in the surface image and the irradiation direction.

  As described with reference to FIG. 5 and the like, the irradiation direction of the irradiation light changes as the tip of the scope 100 moves according to the operation by the user. Therefore, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the luminance pattern in each of the plurality of surface images captured when the irradiation light is irradiated in a plurality of different irradiation directions. it can. For example, the shape calculation unit 200 may average the inclination angle of the surface at the same position calculated from each of the plurality of surface images. Thereby, the calculation accuracy of the surface shape can be further increased.

  As described in connection with this figure, according to the method of calculating the surface shape by irradiating the surface of the living body 190 with the irradiation light of the specific irradiation pattern, the irradiation pattern has a spatial frequency that changes on the surface. The surface shape can be calculated with the corresponding spatial resolution. That is, the shape calculating unit 200 determines the shape of the surface of the living body 190 when irradiated with irradiation light of an irradiation pattern that changes spatially at a higher frequency, for example, irradiation light of an irradiation pattern whose luminance changes at a shorter period. It can be calculated with higher spatial resolution.

  Therefore, the light irradiation unit 120 has the first irradiation light whose luminance changes in a luminance change pattern in which the luminance becomes maximum at a predetermined interval on the predetermined irradiation surface, and the luminance at an interval shorter than the predetermined interval on the predetermined irradiation surface. The living body 190 may be irradiated with the second irradiation light whose luminance changes with the maximum luminance change pattern. Then, the shape calculation unit 200 determines the surface of the living body 190 irradiated with the first irradiation light based on the interval between the corresponding positions and the irradiation direction in the surface image that is an image of the surface of the living body 190 irradiated with the second irradiation light. The shape of the surface of the living body 190 may be calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the living body 190 calculated based on the interval between the corresponding positions and the irradiation direction in the surface image that is an image of For example, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 with higher spatial resolution based on the surface image captured when the second irradiation light having a shorter rectangular pattern is irradiated. .

  Therefore, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 on a global scale based on the surface image irradiated with the irradiation light of the rectangular pattern whose one side is longer than the predetermined length. In addition, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 on a fine scale based on the surface image irradiated with irradiation light of a rectangular pattern whose one side is shorter than a predetermined length. Note that the length of one side of the rectangular pattern may be determined in advance according to the size of a subject to be noted such as a tumor.

  FIG. 7 shows an example of a surface image in which a gland duct opening is imaged. The surface image 700 schematically shows a surface image obtained by photographing a gland duct opening which is an example of the specific subject in the present invention. The surface image 700 includes a plurality of pit images 710 that are examples of the specific object in the present invention.

  By the way, it can be considered that the gland duct openings are distributed substantially uniformly on the surface of the living body 190. Therefore, in the image obtained by photographing the planar surface perpendicular to the imaging direction by the imaging unit 110, the distribution of pit images is substantially uniform. On the other hand, when the inclination of the surface changes spatially, such as when the surface of the living body 190 is curved, the pit image 710 in the image obtained by the imaging unit 110 imaging the surface is displayed. The distribution becomes non-uniform. Therefore, the shape calculation unit 200 can calculate the surface shape based on the distribution of the pit images 710 and the imaging direction of the imaging unit 110.

  An example of the distribution of the pit images 710 is an interval between other pit images 710 located in the vicinity. In the following description, an interval between other pit images 710 located in the vicinity is referred to as a pit interval. Referring to this figure, in surface image 700, the pit interval in the upper left region is shorter than the pit interval in the lower right region. Therefore, the shape calculation unit 200 can determine that the inclination is spatially changed due to the surface being curved from the lower right region toward the upper left region as indicated by the direction 770. As described above, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the position interval between the specific objects in the surface image. The pit interval may be an interval between the centroid positions of the pit image 710.

  As the distribution of the pit images 710, in addition to the interval between the pit images 710, the density of the pit images 710 can be exemplified. That is, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the density of the specific object in the surface image. As described above, the shape calculation unit 200 can calculate the spatial change in the shape of the surface of the living body 190 based on the spatial change in the density of the specific object in the surface image.

  In the above description, the shape calculation unit 200 assumes that the specific subject is distributed substantially uniformly on the surface of the living body 190, but even if the specific subject is non-uniformly distributed on the surface of the living body 190, If the distribution of the specific subject on the surface is known, the surface shape can be calculated based on the distribution of the specific object. As described above, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the distribution of the specific object in which the specific subject distributed in a predetermined pattern on the surface of the living body 190 is captured in the surface image. it can.

  In the above description, the operation of the shape calculation unit 200 has been described by taking as an example a case where the surface shape is calculated based on the distribution of the pit images 710 when the surface of the living body 190 is curved. On the other hand, when the surface of the living body 190 is planar, the pit intervals are uniform. Even in such a case, the shape calculation unit 200 may be able to calculate the shape of the surface of the living body 190 based on the two-dimensional shape of the pit image 710.

  For example, referring to this figure, the shape of the pit image 710 is flatter in the upper left area of the surface image 700 than in the lower right area. Therefore, the shape calculation unit 200 stores in advance a reference pattern indicating the shape of the pit image to be obtained when the pit is imaged from the front, and the reference pattern of the shape pattern indicating the two-dimensional shape of the pit image 710 is stored. The shape of the surface of the living body 190 may be calculated based on the amount of deformation from. An example of the shape pattern is an aspect ratio of pits.

  In addition, if at least the distance in the optical axis direction between the lens 112 and the surface 192 is known, the shape calculation unit 200 calculates the surface shape based on the distance and the density of the pit images 710. Can do. For example, the shape calculation unit 200 may store in advance the interval between pit images to be obtained when a pit is imaged from the front according to the distance. Then, the shape calculating unit 200 determines the living body 190 based on the difference between the pit image interval stored in association with the distance in the optical axis direction between the lens 112 and the surface 192 and the pit interval in the surface image 700. By calculating the inclination of the surface, the shape of the surface of the living body 190 can be calculated. As described above, the shape calculation unit 200 calculates the shape of the surface of the object based on the distance between the surface of the living body 190 and the imaging unit 110 that captures the surface image and the density of the specific object in the surface image. .

  Note that, as described with reference to FIG. 5 and the like, the imaging direction with respect to the surface changes as the distal end of the scope 100 moves in response to an operation by the user. Therefore, the shape calculation unit 200 can calculate the shape of the surface of the living body 190 based on the distribution of the specific object in each of the plurality of surface images obtained by imaging the living body 190 from different directions. For example, the shape calculation unit 200 may average the inclination angle of the surface at the same position calculated from each of the surface images taken from different directions. Thereby, the calculation accuracy of the surface shape can be further increased.

  FIG. 8 shows an example of a surface image including a regular reflection light region. The surface image 800 includes a regular reflection image 810 and a regular reflection image 820 that are images of a region in which the light irradiated from the light irradiation unit 120 is regularly reflected. The regular reflection position calculation unit 240 determines whether the regular reflection image 810 and the regular reflection image 820 are based on the positional relationship identified by the positional relationship identification unit 270 and the positions of the regular reflection image 810 and the regular reflection image 820 on the surface image 800. A specular reflection position, which is a position on the surface of the living body 190 that regularly reflects the irradiation light, can be calculated.

  The regular reflection position calculation unit 240 may calculate the positions of the regular reflection image 810 and the regular reflection image 820 based on the luminance value in the surface image 800. For example, the regular reflection position calculation unit 240 may determine a region having a luminance value higher than a predetermined value in the surface image 800 as a region of the regular reflection image. Further, the regular reflection position calculation unit 240 has a luminance value higher than a predetermined value in the surface image 800, and a region where the difference in luminance from the average luminance value in the peripheral region is larger than a predetermined value. May be determined as a region of a regular reflection image.

  Further, the regular reflection position calculation unit 240 determines a region having a luminance value higher than a predetermined value in the surface image 800 and having a whiteness greater than a predetermined value as a region of the regular reflection image. It's okay. Further, the regular reflection position calculation unit 240 has a luminance value higher than a predetermined value in the surface image 800, the whiteness is larger than the predetermined value, and the average value of the whiteness in the peripheral region is A region where the difference in whiteness is larger than a predetermined value may be determined as the region of the regular reflection image. In this way, the regular reflection position calculation unit 240 can determine the positions of the regular reflection image 810 and the regular reflection image 820 based on the luminance value in the surface image 800.

  When the positions of the specular reflection image 810 and the specular reflection image 820 are determined, the specular reflection position calculation unit 240 captures the positions of the specular reflection image 810 and the specular reflection image 820 and the front surface image 800 when the surface image 800 is captured. The regular reflection position on the surface of the living body 190 can be calculated based on the position of the part. Then, the shape calculation unit 200 calculates the shape of the surface of the living body 190 by assuming that the normal direction of the plane contacting the surface of the living body 190 at the regular reflection position is perpendicular to the imaging direction of the imaging unit 110. be able to.

  Note that, as described with reference to FIG. 5 and the like, the imaging direction with respect to the surface and the irradiation direction of the irradiation light change as the tip of the scope 100 moves according to the operation by the user. The shape calculating unit 200 illuminates when the specular reflection position in each of the plurality of surface images captured by the living body 190 when the irradiation light is irradiated from a plurality of different directions and when each of the plurality of surface images is captured. Based on the light irradiation direction, the shape of the surface of the living body 190 can be calculated in a plane.

  FIG. 9 shows an example of a shadow area in the surface image. The surface image 900 includes a shadow region 920 formed by the convex portion 910 on the surface of the living body 190. The shadow area specifying unit 250 may specify an area whose luminance value is lower than a predetermined value in the surface image 900 as the shadow area 920. Further, the shadow region specifying unit 250 has a luminance value lower than a predetermined value in the surface image 900, and a region where the ratio of the luminance to the average value of the luminance in the peripheral region is smaller than the predetermined value. The shadow area 920 may be specified.

  Then, the shape calculation unit 200 can determine that the convex portion is present in the region closer to the emission port 128 than the region on the surface 192 indicated by the shadow region 920. Thereby, the shape calculation part 200 can calculate the shape of the surface. In addition, in the shadow area 920, the shape calculation unit 200 can calculate the height of the convex portion higher when the length component l in the irradiation direction of the irradiation light is longer. In addition, the shape calculation unit 200 may calculate the shape of the convex portion based on the shape of the shadow region 920.

  Note that, as described with reference to FIG. 5 and the like, the irradiation direction of the irradiation light changes as the tip of the scope 100 moves according to the operation by the user. The shape calculation unit 200 irradiates light when a shadow region in each of a plurality of surface images captured by the living body 190 and each of a plurality of surface images are captured when irradiation light is irradiated from a plurality of different directions. Based on the irradiation direction, the shape of the surface of the living body 190 can be calculated more accurately.

  FIG. 10 shows an example of an image area in which the image quality is reduced in the surface image. The imaging unit 110 supplies the image acquisition unit 210 with the surface images 1000-1 to 1000-8 generated by continuously imaging the surface of the living body 190 from time t1 to time t8. Specifically, the image generation unit 330 generates surface images 1000-1 to 1000-8 using light from the surface of the living body 190 received by the light receiving unit 300, and generates the generated surface images 1000-1 to 1000-1. The surface image 1000-8 is supplied to the image acquisition unit 210.

  Here, the image generation unit 330 has surface images 1000-1 to 1000-3 and surface images 1000-4 to 1000-7 in which the resolution of the peripheral image region 1020 is lower than the resolution of the central image region 1010. Is supplied to the image acquisition unit 210. Note that the image generation unit 330 may generate the surface image 1000-4 and the surface image 1000-8 in which the resolution of the peripheral image region 1020 is the same as the resolution of the central image region 1010 at a predetermined timing. In this manner, the image acquisition unit 210 acquires a plurality of surface images with reduced resolution in the peripheral image region 1020 in the surface image from the central image region 1010 in the surface image.

  Then, the positional relationship specifying unit 270 may specify the positional relationship between the imaging unit 110 and the living body 190 based on the image content of the peripheral area image acquired by the image acquisition unit 210. Then, the shape calculation unit 200 is captured in the central image region 1010 in the plurality of surface images based on the image contents of the central image region 1010 in the plurality of surface images and the positional relationship specified by the positional relationship specifying unit 270. The shape of the surface of the living body 190 is calculated.

  Specifically, the speed calculation unit 242 calculates the speed of the distal end portion 102 with respect to the surface of the living body 190 based on the image contents of the peripheral image region 1020 in the plurality of surface images. Then, the positional relationship calculation unit 260 calculates the positional relationship based on the speed calculated by the speed calculation unit 242 and the timing at which the imaging unit 110 images each of the plurality of surface images. Then, the shape calculation unit 200 is captured in the central image region 1010 in the plurality of surface images based on the image contents of the central image region 1010 in the plurality of surface images and the positional relationship calculated by the positional relationship calculation unit 260. The shape of the surface of the living body 190 is calculated. Note that the speed calculation unit 242 specifies the position of the same object indicating the same subject from the peripheral image area 1020 in each of the surface image 1000-1 to the surface image 1000-8, and based on the specified position of the same object. Thus, the speed of the tip portion 102 with respect to the surface 192 can be calculated.

  In this way, the image acquisition unit 210 acquires a plurality of surface images whose image quality is reduced in the peripheral image region 1020 in the surface image from the central image region 1010 in the surface image. In addition to the above resolution, the index indicating the image quality can be exemplified by the number of colors and the number of gradations of luminance. That is, the image generation unit 330 supplies a surface image having a smaller number of colors than the central image region 1010 in the peripheral image region 1020 to the image acquisition unit 210, or the gradation in the peripheral image region 1020 from the central image region 1010. A small number of surface images can be supplied to the image acquisition unit 210.

  The light receiving element included in the light receiving unit 300 may be an image sensor such as a CMOS. In this case, the image generation unit 330 may generate a surface image with reduced resolution by thinning out and reading out the output of the light receiving element included in the light receiving unit 300.

  Note that the image generation unit 330 may transmit the surface image with reduced image quality in the peripheral image region 1020 at a higher transmission rate than the transmission rate for transmitting the peripheral image region 1020. As described above, the image acquisition unit 210 may acquire the image of the peripheral image region 1020 in the surface image at an acquisition rate higher than the acquisition rate for acquiring the image of the central image region 1010 in the surface image.

  As described above, the image generation unit 330 generates a plurality of surface images in which the amount of image data is reduced in the peripheral image region 1020 in the surface image from the central image region 1010 in the surface image. For this reason, the imaging unit 110 can supply the surface image to the image processing unit 140 at high speed. For this reason, according to the endoscope system 10, even when the distal end portion 102 of the scope 100 moves at a high speed, the positional relationship between the distal end portion 102 and the living body 190 can be calculated with higher time resolution. it can.

  The operation of each part of the endoscope system 10 that calculates the shape of the surface of the living body 190 using light from the living body 190 has been described with reference to FIGS. Here, the light passing through the living body 190 is less likely to be scattered by the tissue of the living body 190 as the wavelength is longer. Therefore, the longer the wavelength region of the irradiated light, the deeper the tissue inside the living body 190 can be reached. For this reason, it can be said that the light in the longer wavelength region out of the light from the living body 190 does not have local information on the surface. Therefore, it is desirable to use a surface image by light in a longer wavelength region in order to calculate a global shape on the surface of the living body 190, and in a shorter wavelength region in order to calculate the fine structure of the surface of the living body 190. It is desirable to use a surface image by light.

  For this reason, the light irradiation unit 120 irradiates the living body 190 with the first irradiation light that is light in the first wavelength region and the second irradiation light that is light in the second wavelength region shorter than the first wavelength region. The imaging unit 110 reflects the light irradiated from the light irradiation unit 120 on the surface of the living body 190 and the second light irradiated from the light irradiation unit 120 on the surface of the living body 190. The surface image is taken with the light.

  For example, when the global shape is calculated as described with reference to FIG. 6, the light irradiation unit 120 emits the first irradiation light as the irradiation light having a larger rectangular pattern. Moreover, when calculating the fine structure in the surface, the light irradiation part 120 irradiates 2nd irradiation light as irradiation light which has a smaller rectangular pattern. Then, the shape calculation unit 200 uses the light with which the first irradiation light is reflected on the surface of the living body 190 based on the interval between the corresponding positions in the surface image and the irradiation direction with the light reflected by the surface of the living body 190 with the second irradiation light. The shape of the surface of the living body 190 may be calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the living body 190 calculated based on the interval between the corresponding positions in the surface image and the irradiation direction.

  In addition, as described with reference to FIG. 9, even when the surface shape is calculated based on the shadow region, the light irradiation unit 120 emits the first irradiation light when calculating the global surface shape. Thus, when calculating the fine structure on the surface, the second irradiation light may be irradiated. The shape calculation unit 200 then reflects the first irradiation light reflected on the surface of the living body 190 based on the shadow region and the irradiation direction identified from the surface image of the second irradiation light reflected on the surface of the living body 190. The shape of the surface of the living body 190 can be calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the living body 190 calculated based on the shadow region and the irradiation direction specified from the surface image obtained by the above.

  In addition, the light irradiation part 120 may irradiate 1st irradiation light and 2nd irradiation light simultaneously. For example, the light irradiation unit 120 uses substantially the same first irradiation light that is light in the first wavelength region and second irradiation light that is light in the second wavelength region that is shorter than the first wavelength region. You may irradiate at the timing. The light receiving unit 300 includes a light receiving element that selectively receives light in a wavelength region including light in the first wavelength region, and a light receiving element that selectively receives light in a wavelength region that includes light in the second wavelength region. Each has a plurality.

  The image generation unit 330 includes a first color component based on light received by a light receiving element that selectively receives light in a wavelength region including light in the first wavelength region, and a wavelength region including light in the second wavelength region. A surface image having a second color component based on the light received by the light receiving element that selectively receives the light is generated. Then, the shape calculation unit 200 determines the interval between the corresponding positions specified based on the first color component in the surface image and the interval between the corresponding positions specified based on the second color component in the surface image and the irradiation direction. The shape of the surface of the living body 190 is calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the living body 190 calculated based on the irradiation direction. As an example, the first irradiation light may be red light, and the light receiving element that selectively receives the light may be the R light receiving element 420. The second irradiation light may be blue light, and the light receiving element that selectively receives the light may be the B light receiving element 440.

  In the above description, an example of the operation of each component of the endoscope system 10 has been described using the living body 190 as an example of a surface shape calculation target. In addition to the living body 190, the present invention can also be used when calculating the surface shape of an article such as an industrially manufactured product or the surface shape of a natural object other than a living body. That is, the object in this invention may be an article or a natural object.

  FIG. 11 shows an exemplary hardware configuration of the image processing unit 140. The image processing unit 140 includes a CPU peripheral unit, an input / output unit, and a legacy input / output unit. The CPU peripheral section includes a CPU 1505, a RAM 1520, a graphic controller 1575, and a display device 1580 that are connected to each other by a host controller 1582. The input / output unit includes a communication interface 1530, a hard disk drive 1540, and a CD-ROM drive 1560 that are connected to the host controller 1582 by the input / output controller 1584. The legacy input / output unit includes a ROM 1510, a flexible disk drive 1550, and an input / output chip 1570 connected to the input / output controller 1584.

  The host controller 1582 connects the RAM 1520, the CPU 1505 that accesses the RAM 1520 at a higher transfer rate, and the graphic controller 1575. The CPU 1505 operates according to the contents of programs stored in the ROM 1510 and the RAM 1520 to control each unit. The graphic controller 1575 acquires image data generated by the CPU 1505 or the like on a frame buffer provided in the RAM 1520 and displays the image data on the display device 1580. Alternatively, the graphic controller 1575 may include a frame buffer that stores image data generated by the CPU 1505 or the like.

  The input / output controller 1584 connects the host controller 1582 to the hard disk drive 1540, the communication interface 1530, and the CD-ROM drive 1560, which are relatively high-speed input / output devices. The hard disk drive 1540 stores programs and data used by the CPU 1505. The communication interface 1530 is connected to the network communication device 1598 to transmit / receive programs or data. The CD-ROM drive 1560 reads a program or data from the CD-ROM 1595 and provides it to the hard disk drive 1540 and the communication interface 1530 via the RAM 1520.

  The input / output controller 1584 is connected to the ROM 1510, the flexible disk drive 1550, and the relatively low-speed input / output device of the input / output chip 1570. The ROM 1510 stores a boot program that is executed when the image processing unit 140 is activated, a program that depends on the hardware of the image processing unit 140, and the like. The flexible disk drive 1550 reads a program or data from the flexible disk 1590 and provides it to the hard disk drive 1540 and the communication interface 1530 via the RAM 1520. The input / output chip 1570 connects various input / output devices via the flexible disk drive 1550 or a parallel port, serial port, keyboard port, mouse port, and the like.

  A program executed by the CPU 1505 is stored in a recording medium such as the flexible disk 1590, the CD-ROM 1595, or an IC card and provided by the user. The program stored in the recording medium may be compressed or uncompressed. The program is installed in the hard disk drive 1540 from the recording medium, read into the RAM 1520, and executed by the CPU 1505. The program executed by the CPU 1505 causes the image processing unit 140 to function as each component included in the image processing unit 140 described with reference to FIGS.

  The program shown above may be stored in an external storage medium. As the storage medium, in addition to the flexible disk 1590 and the CD-ROM 1595, an optical recording medium such as a DVD or PD, a magneto-optical recording medium such as an MD, a tape medium, a semiconductor memory such as an IC card, or the like can be used. Further, a storage device such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium and provided to the image processing unit 140 as a program via the network. As described above, the computer controlled by the program functions as the image processing unit 140. Similarly, it goes without saying that the program can cause the computer to function as each component included in the endoscope system 10.

  Although the present invention has been described using the embodiment, the technical scope of the present invention is not limited to the scope described in the embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

It is a figure which shows an example of a structure of the endoscope system 10 concerning one Embodiment with the biological body 190. FIG. 2 is a diagram illustrating an example of a block configuration of an image processing unit 140. FIG. An example of the configuration of the imaging unit 110 is shown. An example of the arrangement | sequence of the light receiving element in the light-receiving part 300 is shown. It is a figure which shows an example of a position change of the front-end | tip part 102 typically. It is a figure which shows an example of the method of calculating a surface shape based on the luminance pattern of a surface image. It is a figure which shows an example of the surface image by which the duct opening part was imaged. It is a figure which shows an example of the surface image containing a regular reflection light area | region. It is a figure which shows an example of the shadow area | region in a surface image. It is a figure which shows an example of the image area | region where image quality is reduced in a surface image. 2 is a diagram illustrating an example of a hardware configuration of an image processing unit 140. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Endoscope system 100 Scope 102 Tip part 105 Control part 110 Imaging part 112 Lens 120 Light irradiation part 122 Light emission part 124 Light guide 128 Output port 130 Forceps port 135 Forceps 138 Nozzle 140 Image processing part 150 Output part 160 Operation part 190 Living body 192 Surface 194 Convex part 196 Concave part 210 Image acquisition part 200 Shape calculation part 220 Imaging direction calculation part 230 Irradiation direction calculation part 240 Regular reflection position calculation part 242 Speed calculation part 244 Control amount acquisition part 250 Shadow area specifying part 260 Position relation calculation part 270 Position relationship specifying unit 290 Output image generating unit 300 Light receiving unit 310 Filter unit 330 Image generating unit 400 Light receiving unit 410 IR light receiving element 420 R light receiving element 430 G light receiving element 440 B light receiving element 610 Surface 620 Surface 630 Irradiation axis 650 image 660 Image 670 facing 700 surface image 710 pits image 770 facing 800 surface image 810 regular reflection image 820 regular reflection image 900 surface image 910 protrusion 920 shadow area 1000 surface image 1010 image area 1020 image area 1505 CPU
1510 ROM
1520 RAM
1530 Communication interface 1540 Hard disk drive 1550 Flexible disk drive 1560 CD-ROM drive 1570 Input / output chip 1575 Graphic controller 1580 Display device 1582 Host controller 1584 Input / output controller 1590 Flexible disk 1595 CD-ROM
1598 Network communication device

Claims (21)

A light irradiation unit for irradiating an object with irradiation light whose luminance changes spatially in a predetermined luminance change pattern;
An irradiation direction calculation unit for calculating an irradiation direction of the irradiation light;
An image acquisition unit that acquires a surface image that is an image of the surface of the object irradiated with the irradiation light;
An image processing system comprising: a shape calculation unit configured to calculate a shape of the surface of the object based on a difference in luminance pattern in the surface image with respect to the luminance change pattern and the irradiation direction. The light irradiation unit irradiates the irradiation light whose luminance changes with a luminance change pattern in which the luminance is maximized at a predetermined interval on a predetermined irradiation surface,
The shape calculation unit specifies a corresponding position that is a position on the surface image corresponding to a position where the luminance of the irradiation light is maximized on the surface of the object, and determines the interval between the specified corresponding positions and the irradiation direction. The image processing system according to claim 1, wherein the shape of the surface of the object is calculated based on the image. The shape calculation unit calculates the shape of the surface of the object based on a difference in an interval between the corresponding positions with respect to an interval between positions where the luminance of the irradiation light reaches a maximum value and the irradiation direction. Image processing system. The light irradiation unit has a first irradiation light whose luminance changes in a luminance change pattern in which the luminance is maximized at a predetermined interval on a predetermined irradiation surface, and a luminance at an interval shorter than the predetermined interval on the predetermined irradiation surface. Irradiating the object with second irradiation light whose luminance changes in a maximum luminance change pattern;
The shape calculating unit is configured to irradiate the object irradiated with the first irradiation light based on an interval between corresponding positions in the surface image, which is an image of the surface of the object irradiated with the second irradiation light, and the irradiation direction. The shape of the surface of the object is calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the object calculated based on the interval between the corresponding positions in the surface image, which is an image of the surface, and the irradiation direction. The image processing system according to 2. An imaging unit that captures the surface image with light from the object irradiated with the irradiation light;
The light irradiation unit irradiates the object with the first irradiation light, which is light in a first wavelength region, and the second irradiation light, which is light in a second wavelength region shorter than the first wavelength region,
In the imaging unit, the first irradiation light irradiated from the light irradiation unit is reflected by the surface of the object, and the second irradiation light irradiated from the light irradiation unit is reflected by the surface of the object. The surface image is captured by the light,
The shape calculating unit reflects the first irradiation light on the surface of the object based on the interval between the corresponding positions and the irradiation direction in the surface image by the light reflected by the surface of the object. 5. The shape of the surface of the object is calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the object calculated based on the interval between the corresponding positions in the surface image by the light and the irradiation direction. Image processing system. The light irradiation unit irradiates the first irradiation light, which is light in the first wavelength region, and the second irradiation light, which is light in the second wavelength region, at substantially the same timing,
The imaging unit
A first light receiving element that selectively receives light in a wavelength region including light in the first wavelength region;
A second light receiving element that selectively receives light in a wavelength region including light in the second wavelength region;
An image generating unit that generates the surface image having a first color component based on light received by the first light receiving element and a second color component based on light received by the second light receiving element;
The shape calculation unit is specified based on the first color component in the surface image based on the interval between the corresponding positions specified based on the second color component in the surface image and the irradiation direction. The image processing system according to claim 5, wherein the shape of the surface of the object is calculated with a spatial resolution higher than the spatial resolution of the shape of the surface of the object calculated based on the interval between corresponding positions and the irradiation direction. The image processing system according to claim 1, wherein the light irradiation unit irradiates the object with the irradiation light which is invisible light. A control amount acquisition unit for acquiring a control amount for controlling the irradiation direction of the irradiation light emitted by the light irradiation unit;
The image processing system according to claim 1, wherein the irradiation direction calculation unit calculates an irradiation direction irradiated with the irradiation light based on the control amount acquired by the control amount acquisition unit. A light irradiation unit for irradiating an object with irradiation light;
An imaging unit that captures a surface image that is an image of the surface of the object by imaging the object irradiated with the irradiation light from a direction substantially the same as the irradiation direction of the irradiation light irradiated by the light irradiation unit. When,
A regular reflection position calculation unit that calculates a regular reflection position that is a position on the surface of the object that regularly reflected the irradiation light, based on a luminance value in the surface image;
The shape of the surface of the object at the specular reflection position is calculated by setting the normal direction of the surface in contact with the surface of the object at the specular reflection position as the irradiation direction of the irradiation light irradiated to the specular reflection position. An image processing system comprising a shape calculating unit. The light irradiation unit irradiates the object with the irradiation light from different irradiation directions,
The imaging unit captures the plurality of surface images by imaging the objects irradiated with the irradiation light from the different irradiation directions by the light irradiation unit, respectively.
The specular reflection position calculation unit calculates a plurality of specular reflection positions that are positions on the surface of the object that specularly reflected the irradiation light, based on a luminance value in each of the plurality of surface images;
The shape calculation unit captures the surface image in which the specular reflection position of each of the plurality of specular reflection positions is specified with respect to the normal direction of the surface in contact with the surface of the object at each of the plurality of specular reflection positions. The shape of the surface of the object at each of the plurality of regular reflection positions is calculated by setting the irradiation direction of the irradiation light that has been irradiated to each of the plurality of regular reflection positions in the case of Image processing system. A light irradiation unit that irradiates an object with first irradiation light that is light in a first wavelength region and second irradiation light that is light in a second wavelength region shorter than the first wavelength region;
The first irradiation light irradiated from the light irradiation unit is reflected by the surface of the object, and the second irradiation light irradiated from the light irradiation unit is reflected by the surface of the object. An imaging unit that captures a surface image that is an image of the surface of
From the surface image, a shadow area specifying unit that specifies a shadow area that is a shadow area;
Based on the position of the shadow region identified from the surface image by the light reflected by the surface of the object and the irradiation direction, the first irradiation light is reflected by the light reflected by the surface of the object. A shape calculation unit for calculating the shape of the surface of the object with a spatial resolution higher than the spatial resolution of the shape of the surface of the object calculated based on the position of the shadow region specified from the surface image and the irradiation direction; An image processing system provided. Image acquisition that is a plurality of surface images that are images of the surface of an object captured from different positions, and that obtains a plurality of surface images with reduced image quality in a peripheral image region of the surface image from a central image region And
A positional relationship identifying unit that identifies a positional relationship between the imaging unit that captured the surface image and the object when each of the plurality of surface images is captured based on the image contents of the plurality of surface images; ,
An image processing system comprising: a shape calculating unit that calculates the shape of the surface based on the image contents of the plurality of surface images and the positional relationship specified by the positional relationship specifying unit. The image processing system according to claim 12, wherein the image acquisition unit acquires the plurality of surface images with reduced resolution in the peripheral image region of the surface image from the central image region of the surface image. . The image processing according to claim 12, wherein the image acquisition unit acquires an image of the peripheral image region in the surface image at an acquisition rate higher than an acquisition rate for acquiring an image of the central image region in the surface image. system. An image acquisition unit that acquires a surface image that is an image of the surface of the object;
An image processing system comprising: a shape calculation unit configured to calculate a shape of the surface of the object based on a distribution of the specific object in which a specific subject distributed in a predetermined pattern on the surface of the object is captured in the surface image. The shape calculation unit calculates a shape of a surface of the object based on a distribution of the specific object in which a specific subject that is distributed substantially uniformly on the surface of the object is captured in the surface image. The image processing system described. The image processing system according to claim 16, wherein the shape calculation unit calculates the shape of the surface of the object based on the density of the specific object in the surface image. The image processing system according to claim 17, wherein the shape calculation unit calculates a spatial change in the shape of the surface of the object based on a spatial change in the density of the specific object in the surface image. The shape calculation unit calculates the shape of the surface of the object based on a distance between the surface of the object and an imaging unit that captures the surface image, and a density of the specific object in the surface image. The image processing system according to 17. The image processing system according to claim 16, wherein the shape calculation unit calculates the shape of the surface of the object based on a position interval between the different specific objects in the surface image. The shape calculation unit according to claim 16, wherein the shape calculation unit calculates the shape of the surface of the object based on a distribution of the specific object in each of the plurality of surface images obtained by imaging the object from a plurality of different directions. Image processing system.

Images