JP6706026B2 - Endoscope system and operating method of endoscope apparatus - Google Patents

Description

The present invention relates to actuation how the endoscope system and endoscope apparatus, in particular, on how to set measurement points.

BACKGROUND ART Conventionally, in an industrial field or a medical field, an endoscope apparatus that inserts a long insertion portion into an object to be inspected and takes an image of the inside of the object to be inspected by an imaging element provided in a distal end portion located at a tip of the insertion portion. Has been put to practical use. Further, in recent years, in order to grasp the shape of a photographed subject, a system (endoscope system) of an endoscope device equipped with a three-dimensional measurement technique has been commercialized.

The three-dimensional measurement technology is roughly classified into a passive method and an active method. The passive method is, for example, a method typified by so-called stereo measurement, in which measurement is performed based on a three-dimensional image generated from an image corresponding to the right eye and an image corresponding to the left eye of a subject. Further, the active method is, for example, so-called fringe projection measurement, in which light of a structured pattern in a stripe shape is applied to a subject and measurement is performed by using information on changes in stripe intervals included in a captured image. The method is represented by.

The passive type measurement includes an image sensor corresponding to the right eye and an image sensor corresponding to the left eye. In stereo measurement, it is common to search for corresponding points from the image corresponding to the left eye and the image corresponding to the right eye, but if there are few textures (texture, touch, etc.) according to the unevenness of the surface of the subject, it will be supported. It is difficult to find the point accurately. On the other hand, the active measurement can be realized by providing a structure for irradiating light of a structured pattern in the distal end portion of the endoscope device. The active measurement has the advantage that the measurement can be performed without depending on the texture and the processing time is short when the measurement is performed on the entire screen of the captured image.

Therefore, for example, as in Patent Document 1 and Patent Document 2, as a configuration for performing three-dimensional measurement, a technique of an endoscope system equipped with an active measurement, that is, a configuration for performing fringe projection measurement is proposed. ing. In the techniques disclosed in Patent Document 1 and Patent Document 2, a three-dimensional image is obtained from a plurality of images captured by irradiating a subject a plurality of times by performing a phase shift that imparts a phase difference to structured light. Is generated and three-dimensional measurement is performed.

Further, for example, Patent Document 3 discloses a control method and a processing procedure (processing flow) at the time of photographing an object in an endoscope system that performs three-dimensional measurement. In the technique disclosed in Patent Document 3, first, in the inspection mode, normal light that does not include a striped pattern is radiated to photograph a subject, and an image of the subject is captured by the endoscope system. A user (user) sets the position of a measurement point at which three-dimensional measurement is desired. After that, a structured pattern of light is irradiated in the measurement mode to photograph a subject, a three-dimensional image is generated from the image photographed at this time, and a three-dimensional image at a position corresponding to the set measurement point is generated. Take measurements. For example, in the case of measuring the distance between two points, the technique disclosed in Patent Document 3 sets two measurement points for measuring the length of an image captured in the inspection mode and sets the measurement mode. The distance between the two points is measured based on the positions (for example, the coordinates) corresponding to the two measurement points included in the three-dimensional image generated from the image captured in.

JP-A-5-087543 U.S. Pat. No. 7,821,649 U.S. Pat. No. 8,107,083

However, in the technique disclosed in Patent Document 3, as described above, after setting the measurement points for the image captured in the inspection mode, a three-dimensional image is generated from the image captured in the measurement mode. The measurement is performed based on the generated three-dimensional image. That is, in the technique disclosed in Patent Document 3, the image at which the measurement point is set and the image at which the measurement is performed are different images, and there is a time difference between the timings at which the images are captured. Therefore, if there is a blur or a difference in appearance of the subject between the image in which the measurement point is set and the image in which the measurement is performed, the measurement performance deteriorates due to these factors.

More specifically, for example, when shooting a subject by irradiating a structured pattern of light in the measurement mode, if the tip moves to a position different from that in the inspection mode, motion blur occurs. To do. In other words, the position of the subject included in the image captured in the inspection mode for setting the measurement point and the position of the subject included in the image captured in the measurement mode for performing the measurement are displaced. .. When this displacement of the position of the subject occurs, measurement is performed at a position desired by the user, that is, a position different from the position where the measurement point is set, and a correct measurement value cannot be obtained. When the position of the subject included in the image to be measured is different from the position of the subject included in the image in which the measurement points are set, for example, the position of the subject included in the image to be measured is corrected by image processing, It is also conceivable to adopt a method in which measurement is performed after the measurement point is aligned with the position of the subject included in the set image. However, in the processing for correcting the position of the subject by image processing, for example, when the surface of the subject has few irregularities (the texture is few), the combined position may include an error. Accurate alignment cannot be achieved, and the effect of the alignment correction process cannot be obtained sufficiently.

In addition, for example, if there is unevenness on the surface of the subject, whiteout due to halation or blackout due to the shadow of unevenness will occur in the image to be measured. Correct measurement values cannot be obtained from images in which halation or the like has occurred. For example, when two measurement points are set in the image for setting the measurement points and the distance between the two points along the unevenness of the surface of the object to be measured is measured, halation or the like occurs in the image to be measured. However, it is not possible to determine whether it is the surface of the test object or the shadow due to the unevenness of the surface, and the width of the unevenness on the surface of the test object cannot be accurately measured.

The present invention has been made based on recognition of the above problems, in the endoscope system with 3-D measurement technique, an endoscope system and endoscope capable of obtaining an accurate measurement result and its object is to provide an actuation how the device.

According to the first aspect of the present invention, the endoscope system illuminates the test object with observation light for observing the test object, and measures the test object three-dimensionally. A pattern projection unit that irradiates the test object with measurement light in a predetermined wavelength band of a structured pattern, irradiation of the observation light by the illumination unit to the test object, and the pattern projection unit an irradiation light control unit for controlling the irradiation to the specimen of the measuring light, before SL on the basis of the measurement image of the object, the three-dimensional measurement with respect to the test object corresponding to the specified measurement point An endoscope system including a three-dimensional measurement unit that performs the setting, and a display unit that displays the setting image and the measurement point , wherein the three-dimensional measurement unit simultaneously emits the observation light and the measurement light. generating said set titration, images based on the measurement image of the test object was obtained, wherein the test object based on the information of the measurement image corresponding to the position of the set the measuring point on the setting image The above-mentioned three-dimensional measurement is performed.

According to a second aspect of the present invention, in the endoscope system according to the first aspect, the three-dimensional measurement unit includes the measurement light included in the measurement image when performing the three-dimensional measurement. Three-dimensional pixel information used for the three-dimensional measurement is generated based on a first wavelength band measurement image which is an image of the object by reflection of light in the same wavelength band, and the generated three-dimensional pixel information is stored. Based on the three-dimensional shape restoration processing for generating the included three-dimensional measurement image and the three-dimensional pixel information corresponding to the position of the measurement point set in the setting image included in the three-dimensional measurement image, A three-dimensional measurement process for performing the three-dimensional measurement may be executed.

According to a third aspect of the present invention, in the endoscope system of the second aspect, the irradiation light control unit simultaneously performs the observation light and the measurement light when performing the three-dimensional measurement. The three-dimensional measurement unit irradiates the measurement object, and the three-dimensional measurement unit performs the measurement image on the first wavelength band measurement image and the measurement image included in the measurement image before performing the three-dimensional shape restoration process. Image separation processing is performed to separate the light into a second wavelength band measurement image, which is an image of the test object due to reflection of light in a wavelength band different from that of the light, and is parallel to the three-dimensional shape restoration processing. Structured pattern removal processing for generating a measurement light removal image by removing the component of the measurement light contained in the wavelength band measurement image, the measurement light removal image, and a composite image obtained by combining the second wavelength band measurement image The composite image generation processing for generating the above may be sequentially executed.

According to a fourth aspect of the present invention, in the endoscope system of the third aspect, the irradiation light control unit sets the light amount of the observation light to be irradiated to the test object at the same time as the measurement light. The light may be adjusted according to the amount of the measurement light.

According to the fifth aspect of the present invention, in the endoscope system according to the third aspect or the fourth aspect, the three-dimensional measurement unit may use the synthetic image as the setting image.

According to a sixth aspect of the present invention, in the endoscope system according to the third aspect or the fourth aspect, the three-dimensional measurement unit displays an image in which the composite image is superimposed on the three-dimensional measurement image. It may be the setting image.

According to a seventh aspect of the present invention, in the endoscope system of the second aspect, the three-dimensional measurement unit includes the first wavelength band measurement image in parallel with the three-dimensional shape restoration processing. The structured light pattern removal processing for generating a measurement light removal image from which the component of the measurement light is removed may be executed, and the measurement light removal image may be used as the setting image.

According to an eighth aspect of the present invention, in the endoscope system of the second aspect, the three-dimensional measurement unit may use the three-dimensional measurement image as the setting image.

According to a ninth aspect of the present invention, in the endoscope system according to the second aspect, the three-dimensional measurement unit adds an image obtained by previously irradiating the three-dimensional measurement image with the observation light. An image obtained by superimposing a corresponding observed image of the test object may be used as the setting image.

According to a tenth aspect of the present invention, in the endoscope system according to any one of the first to ninth aspects, the measurement light has a structured pattern, and the intensity of the light is sinusoidal. It may be a striped pattern that changes in a wavy manner.

According to an eleventh aspect of the present invention, in the endoscope system according to any one of the first aspect to the tenth aspect, the pattern projection unit includes the structured pattern of the measurement light. The phase is shifted by a predetermined shift amount to irradiate the test object a plurality of times, and the three-dimensional measurement unit responds to each image obtained by the irradiation of each of the phase-shifted measurement lights. The three-dimensional measurement may be performed based on a plurality of the measurement images.

According to a twelfth aspect of the present invention, in the endoscope system according to any one of the first to eleventh aspects, the measurement light may be laser light.

According to a thirteenth aspect of the present invention, in the endoscope system according to any one of the first to twelfth aspects, the measurement light is in the wavelength band of visible light. It may be.

According to a 14th aspect of the present invention, in the endoscope system according to the 13th aspect, the wavelength band of the measurement light may be a red wavelength band.

According to a fifteenth aspect of the present invention, the third aspect to the sixth aspect, or the third aspect to the sixth aspect cited above In the endoscope system according to any one of the tenth to fourteenth aspects, the observation light is a wavelength band of visible light, and the three-dimensional measurement unit, in the image separation processing, the measurement image , The measurement image by reflection of light in the red wavelength band, the measurement image by reflection of light in the green wavelength band, and the measurement image by reflection of light in the blue wavelength band, the measurement light and The measurement image by reflection of light in the same wavelength band may be the first wavelength band measurement image, and each of the measurement images by reflection of light in the wavelength band different from the measurement light may be the second wavelength band measurement image. Good.

According to a sixteenth aspect of the present invention, any one of the eleventh aspect, which cites any one of the third aspect to the seventh aspect, or any one of the third aspect to the seventh aspect. In the endoscope system according to any one of the twelfth aspect to the fifteenth aspect, which cites the eleventh aspect, which cites one aspect, the three-dimensional measurement unit includes the structured pattern removal process. In, the value of each pixel at the same position included in the plurality of first wavelength band measurement images corresponding to each of the plurality of measurement images is arithmetically averaged to generate one measurement light removal image. May be.

According to a seventeenth aspect of the present invention, any one of the sixth aspect, the eighth aspect and the ninth aspect, or the sixth aspect, the eighth aspect and the ninth aspect. In the endoscope system of any one of the tenth aspect to the sixteenth aspect, which cites any one of the three aspects, the three-dimensional measurement unit is generated in the three-dimensional shape restoration process. A depth image representing the depth of the object may be generated based on the three-dimensional pixel information, and the generated depth image may be the three-dimensional measurement image.

According to an eighteenth aspect of the present invention, a method of operating the endoscope apparatus includes an illumination unit you emits observation light for observing the specimen in the specimen, three-dimensional the test object in the pattern projection unit a predetermined wavelength band of the measurement light you exit the specimen of the structured pattern in order to measure the emission into the test object of the observation light by the illumination unit, the and outgoing light control unit that controls the emission into the test object of the measurement light by the pattern projection unit, on the basis of the measurement image of the test object in response to the video obtained by emitting at least the measuring light And a three-dimensional measurement unit that performs three-dimensional measurement on the object according to a designated measurement point , wherein the three-dimensional measurement unit performs the measurement based on the measurement image. Generating a setting image for setting a point and displaying the setting image on a display unit; and the three-dimensional measurement unit, information of the measurement image corresponding to the position of the measurement point set in the setting image. wherein the step of performing the three-dimensional measurement with respect to the test object, when performing measurement of the three-dimensional, pre SL emitted light control unit, the measuring light and at the same time the test object and the observation light based on the And the three-dimensional measurement unit is an image of the object by reflection of light in the same wavelength band as the measurement light included in the measurement image when performing the three-dimensional measurement. A three-dimensional shape restoration process for generating three-dimensional pixel information used for the three-dimensional measurement based on one wavelength band measurement image, and for generating a three-dimensional measurement image including the generated three-dimensional pixel information; And executing a three-dimensional measurement process for performing the three-dimensional measurement based on the three-dimensional pixel information included in the three-dimensional measurement image and corresponding to the position of the measurement point set in the setting image. And the three-dimensional measurement unit, before performing the three-dimensional shape restoration process, the measurement image, a wavelength different from the first wavelength band measurement image and the measurement light included in the measurement image. parallel a step of executing an image separation processing for separating the second wavelength band measurement image is an image of the test object from the reflection of light in a band, the three-dimensional measuring unit, and the 3D reconstruction process A structured pattern removal process for generating a measurement light removal image by removing the component of the measurement light included in the first wavelength band measurement image, the measurement light removal image, and the second wavelength band measurement image. And a step of sequentially executing a combined image generation process for generating a combined image obtained by combining and .
According to the nineteenth aspect of the present invention, the endoscope system illuminates the test object with observation light for observing the test object, and measures the test object three-dimensionally. A pattern projection unit that irradiates the test object with measurement light in a predetermined wavelength band of a structured pattern, irradiation of the observation light by the illumination unit to the test object, and the pattern projection unit An irradiation light control unit that controls irradiation of measurement light onto the test object, and at least a designated measurement point based on a measurement image of the test object according to an image obtained by irradiating the measurement light. And a three-dimensional measurement unit that performs three-dimensional measurement on the object according to the three-dimensional measurement unit, and the three-dimensional measurement unit generates a setting image for setting the measurement point based on the measurement image. Is displayed on the display unit, the three-dimensional measurement is performed on the object based on the information of the measurement image corresponding to the position of the measurement point set in the setting image, and the three-dimensional measurement is performed. When performing, the irradiation light control unit causes the observation light and the measurement light to irradiate the test object at the same time, and when performing the three-dimensional measurement, the three-dimensional measurement unit displays the measurement image on the measurement image. Three-dimensional pixel information used for the three-dimensional measurement is generated and generated based on a first wavelength band measurement image which is an image of the object to be measured by reflection of light in the same wavelength band as the included measurement light. 3D shape restoration processing for generating a 3D measurement image including the 3D pixel information, and 3D corresponding to the position of the measurement point set in the setting image included in the 3D measurement image A three-dimensional measurement process for performing the three-dimensional measurement based on pixel information; and the three-dimensional measurement unit performs the three-dimensional measurement process on the measurement image before performing the three-dimensional shape restoration process. Image separation processing for separating the second wavelength band measurement image, which is an image of the object to be measured by reflection of light in a wavelength band different from the measurement light included in the measurement image, And a structured pattern removal process for generating a measurement light removal image in which the components of the measurement light included in the first wavelength band measurement image are removed in parallel with the three-dimensional shape restoration process. Then, a synthetic image generation process for generating a synthetic image by synthesizing the measurement light removal image and the second wavelength band measurement image is sequentially executed.
According to a twentieth aspect of the present invention, in the endoscope system according to the nineteenth aspect, the irradiation light control unit sets the light amount of the observation light to be irradiated to the test object at the same time as the measurement light. The light may be adjusted according to the amount of the measurement light.
According to a twenty-first aspect of the present invention, in the endoscope system according to the nineteenth aspect or the twentieth aspect, the three-dimensional measurement unit may use the synthetic image as the setting image.
According to a twenty-second aspect of the present invention, in the endoscope system according to the nineteenth aspect or the twentieth aspect, the three-dimensional measurement unit is configured to display an image obtained by superimposing the composite image on the three-dimensional measurement image. It may be an image for setting.
According to a twenty-third aspect of the present invention, in the endoscope system according to any one of the nineteenth aspect to the twenty-second aspect, the observation light is a wavelength band of visible light, and the three-dimensional measurement is performed. The section, in the image separation processing, the measurement image, the measurement image by reflection of light in the red wavelength band, the measurement image by reflection of light in the green wavelength band, the reflection of light in the blue wavelength band And the measurement image by reflection of light in the same wavelength band as the measurement light is used as the first wavelength band measurement image, and the measurement image by reflection of light in a wavelength band different from the measurement light May be used as the second wavelength band measurement image.
According to a twenty-fourth aspect of the present invention, in the endoscope system according to any one of the nineteenth aspect to the twenty-second aspect, in the structured pattern removing process, Values of respective pixels at the same positions included in the plurality of first wavelength band measurement images corresponding to the plurality of measurement images may be arithmetically averaged to generate one measurement light removal image. ..
According to a twenty-fifth aspect of the present invention, in the endoscope system according to the twenty-second aspect, the three-dimensional measurement unit is configured to perform the target operation based on the three-dimensional pixel information generated in the three-dimensional shape restoration process. A depth image representing the depth of the inspection object may be generated, and the generated depth image may be the three-dimensional measurement image.

According to the present invention, it is possible to obtain an effect that an accurate measurement result can be obtained in an endoscope system equipped with a three-dimensional measurement technique.

It is the figure which showed an example of schematic structure of the endoscope system by the 1st Embodiment of this invention. 6 is a timing chart showing an example of light irradiation timing in the endoscope system of the first embodiment. It is a figure showing an outline of processing in an endoscope system of a 1st embodiment of the present invention. 6 is a flowchart showing a processing procedure in the endoscope system of the first embodiment. It is a flow chart which showed a processing procedure in an endoscope system by a 2nd embodiment of the present invention. It is a flow chart which showed a processing procedure in an endoscope system by a 3rd embodiment of the present invention. It is the flowchart which showed the processing procedure in the endoscope system by the 4th Embodiment of this invention. It is the flowchart which showed the processing procedure in the endoscope system by the 5th Embodiment of this invention. It is the flowchart which showed the processing procedure in the endoscope system by the 6th Embodiment of this invention.

<First Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in the following description, the case where the endoscope system of the present invention is an industrial endoscope apparatus will be described. FIG. 1 is a diagram showing an example of a schematic configuration of an industrial endoscope device (hereinafter, simply referred to as “endoscope device”) which is an endoscope system according to a first embodiment of the present invention. The endoscope apparatus 1 shown in FIG. 1 is used for observing the inside of an object to be inspected, observing an object to be inspected at a position where a normal observation apparatus is difficult to access, and the like.

The endoscope device 1 includes a long and flexible insertion portion 3 and a main body portion 2 to which a proximal end of the insertion portion 3 is connected. Further, the distal end portion 4 located at the distal end of the insertion portion 3 on the side opposite to the base end connected to the main body portion 2 has an image capturing unit 5 for capturing an image of an object to be inspected, and an image capturing unit 5 for capturing light within a range captured by the image capturing unit. Illumination unit 6 for irradiating light and a pattern projection unit 7 for projecting light of a light and dark structured pattern onto a test object are partially arranged. On the end surface 40 of the tip portion 4, an opening 45 for allowing light from the outside to enter the imaging unit 5, an illumination window 46 for irradiating the front surface with light from the illumination unit 6, and a pattern projection unit 7 are provided. A projection window 47 for irradiating the front surface with a structured pattern of light is provided.

As shown in FIG. 1, the insertion portion 3 is formed in a tubular shape, and is inserted into the inside of the test object or in the access path to the test object, and the tip part 4 is near the test object. Lead. In the following description, the pattern for structuring the light projected by the pattern projection unit 7 onto the test object is a striped pattern in which the intensity of the light changes, that is, the intensity of the light changes in a horizontal sinusoidal shape ( The vertical stripe pattern will be described below.

The image pickup section 5 is configured to include an image pickup control section 51 arranged in the main body section 2, and an image pickup element 52 and an objective optical system 53 arranged in the tip section 4. The objective optical system 53 is an optical lens group having a predetermined angle of view that directly faces the opening 45 provided in the end surface 40 of the tip portion 4. Note that the configuration of the endoscope apparatus 1 shown in FIG. 1 shows a case where the objective optical system 53 is configured by one optical lens, but the objective optical system 53 is configured by a plurality of optical lenses. May be. The objective optical system 53 causes the reflected light from the test object within the observation range defined by the predetermined angle of view, which is incident from the opening 45, to be incident on the image sensor 52, and the image of the test object is captured by the image sensor 52. An image is formed on the imaging surface. Note that the configuration of the endoscope apparatus 1 shown in FIG. 1 shows a case where the objective optical system 53 is provided with a cover member 54 having a light transmitting property for closing the opening of the opening 45.

The image sensor 52 has a plurality of pixels arranged in a two-dimensional matrix, captures an image of a subject formed by the objective optical system 53, and outputs a pixel signal corresponding to the captured image, for example, a CCD. An image sensor represented by a (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. The image pickup element 52 is driven by a drive signal output from the image pickup control section 51 in the main body section 2, which is transmitted via a signal line 55 provided in the insertion section 3. Further, the image sensor 52 outputs each pixel signal corresponding to the captured video as a video signal to the imaging control unit 51 via the signal line 55. The image sensor 52 can be configured by appropriately selecting a known technique including an image sensor.

The imaging control unit 51 performs various controls such as driving of the imaging element 52 and settings when acquiring a video signal according to the control from the main control unit 81. More specifically, the imaging control unit 51 drives the imaging element 52 arranged in the distal end portion 4 via the signal line 55 in the insertion unit 3. Further, the imaging control unit 51 controls the output of a video signal corresponding to the video imaged by the imaging device 52 to the signal line 55, that is, the transmission of the video signal via the signal line 55. Further, the imaging control unit 51 outputs the video signal acquired from the imaging device 52 via the signal line 55 to the video processing unit 86 arranged in the main body unit 2.

The illumination section 6 includes an illumination light source 61 and a coupling lens 62 arranged in the main body section 2, a fiber bundle 63 passing through the insertion section 3, and an illumination optical system 64 arranged in the tip section 4. Composed. The illumination light source 61 is a general white light source. As the light source of the illumination light source 61, a light emitting element such as a white LED or a laser, or an electric lamp such as a halogen lamp can be used.

The coupling lens 62 is an optical lens that converges the light emitted from the illumination light source 61 to the same extent as the diameter of the fiber bundle 63. The coupling lens 62 introduces the converged light into the fiber bundle 63. The fiber bundle 63 is an optical fiber that guides the light introduced from the coupling lens 62 to the illumination optical system 64 through the insertion section 3. In the present invention, the structure and type of the fiber bundle 63 are not particularly limited, and for example, a general light guide or the like can be used as the fiber bundle 63.

The illumination optical system 64 is an optical lens group that faces the illumination window 46 provided on the end surface 40 of the tip portion 4. Note that the configuration of the endoscope apparatus 1 shown in FIG. 1 shows the case where the illumination optical system 64 is configured by one optical lens, but the illumination optical system 64 is configured by a plurality of optical lenses. May be. The illumination optical system 64 spreads the light emitted from the illumination light source 61 and guided by the fiber bundle 63 to an observation range defined by a predetermined angle of view in the objective optical system 53 and emits the light from the illumination window 46. As a result, the light emitted from the illumination light source 61 is evenly applied to the front surface of the distal end portion 4, that is, the observation range of the test object in the endoscope apparatus 1. The illumination optical system 64 may include a cover member (not shown) having a light transmitting property so as to close the opening of the illumination window 46.

The pattern projection unit 7 is arranged in the pattern projection light source 71, the coupling lens 72, and the pattern control unit 76 arranged in the main body unit 2, the optical fiber 73 passing through the insertion unit 3, and the distal end unit 4. It is configured to include a fringe generation unit 74 and a projection optical system 75. The pattern projection light source 71 is a light source in a predetermined specific wavelength band. The pattern projection light source 71 can use a light emitting element such as a laser or an LED of a predetermined color as a light source. The wavelength band of the light emitted by the pattern projection light source 71 may be the wavelength band of visible light or the wavelength band of infrared light.

The coupling lens 72 is an optical lens that converges the laser light of the specific wavelength band emitted from the pattern projection light source 71 to the same extent as the diameter of the optical fiber 73. The coupling lens 72 introduces the converged laser light into the optical fiber 73. The optical fiber 73 is an optical fiber that guides the laser light introduced from the coupling lens 72 to the stripe generation unit 74 through the insertion unit 3. In addition, in the present invention, the configuration and type of the optical fiber 73 are not particularly limited as in the fiber bundle 63.

The stripe generation unit 74 is emitted from the pattern projection unit 7 in response to the control signal output from the pattern control unit 76 in the main body 2 transmitted via the signal line 77 provided in the insertion unit 3. From the laser light guided by the optical fiber 73, light (hereinafter, referred to as “pattern light”) that is converted into a structured pattern for projecting on a test object is generated. Further, the stripe generation unit 74 causes the generated pattern light to have a predetermined phase difference according to the control signal output from the pattern control unit 76. Then, the fringe generation unit 74 emits the pattern light having a phase difference, that is, the phase shift, to the projection optical system 75. In the endoscope apparatus 1, the structured pattern converted by the pattern projection unit 7 for projecting on the test object is a vertical stripe pattern. Therefore, the stripe generation unit 74 converts the laser light into vertical stripe pattern light, and further emits pattern light having a predetermined phase shift to the projection optical system 75.

The pattern control unit 76 controls the structured pattern and the phase difference of the pattern light emitted by the fringe generation unit 74 to the projection optical system 75 under the control of the main control unit 81. More specifically, the pattern control unit 76 uses the control signal output to the stripe generation unit 74 via the signal line 77 to determine the intensity of the stripes in the vertical stripe pattern generated by the stripe generation unit 74, that is, the intensity of the laser light. Controls the phase when changing to a sine wave in the horizontal direction. In addition, the pattern control unit 76 uses the control signal output to the fringe generation unit 74 via the signal line 77 to provide a phase difference to the pattern light generated by the fringe generation unit 74, that is, the pattern light projected onto the test object. Controls the amount of phase shift.

The projection optical system 75 is an optical lens group that directly faces the projection window 47 provided on the end surface 40 of the tip portion 4. In the configuration of the endoscope apparatus 1 shown in FIG. 1, the case where the projection optical system 75 is composed of one optical lens is shown, but the projection optical system 75 is composed of a plurality of optical lenses. May be. The projection optical system 75 spreads the pattern light emitted after the fringe generation unit 74 has converted the laser light and phase-shifted it to the observation range defined by the predetermined angle of view in the objective optical system 53, and then from the projection window 47. Let out. As a result, the phase-shifted pattern light is evenly applied to the observation area of the subject on the front surface of the distal end portion 4 of the endoscope apparatus 1. Note that the projection optical system 75 may also have a cover member (not shown) having a light transmitting property in order to close the opening of the projection window 47, as in the illumination optical system 64.

In addition, in the endoscope device 1, the light source control unit 80, the image processing unit 86, the operation unit 82, the monitor 87, the main control unit 81, and the main storage device (RAM) 83 are provided in the main body unit 2. , An auxiliary storage unit 84 and a ROM 85.

The light source control unit 80 controls light emission, that is, irradiation of light to a test object by controlling on/off of the illumination light source 61 and the pattern projection light source 71 according to the control from the main control unit 81. To do. A detailed description of a method of controlling the light emission of the illumination light source 61 and the pattern projection light source 71 in the light source controller 80 will be described later.

The video processing unit 86 performs various processes on the video signal output from the imaging control unit 51 under the control of the main control unit 81, and a video (image) based on the video signal, that is, the imaging unit. An image of the object taken by the image pickup device 52 provided in No. 5 is generated. Then, the video processing unit 86 outputs the generated image of the test object to the main control unit 81.

The operation unit 82 is a user interface or a controller operated by a user (user) to control the operation of the endoscope apparatus 1. The user operates the operation unit 82 to cause the endoscope apparatus 1 to execute a desired operation or process. For example, by operating the operation unit 82, the user gives an instruction to change the direction in which the distal end portion 4 faces when inserting the insertion unit 3 into the test object, and to image the test object. Further, by operating the operation unit 82, the user sets the position of the measurement point when performing three-dimensional measurement to a desired position. The operation unit 82 outputs information indicating the content operated by the user to the main control unit 81.

The monitor 87 is a display unit that displays the operation and operation content of the endoscope apparatus 1 and the image of the photographed subject. The monitor 87 displays the image input from the main control unit 81. The user observes the image displayed on the monitor 87 and operates the operation unit 82 to observe the test object using the endoscope apparatus 1.

The main controller 81 controls the components of the endoscope device 1, that is, the entire endoscope device 1. For example, each component in the endoscope apparatus 1 is selected according to a photographing operation by the image pickup unit 5 when the endoscope apparatus 1 observes a subject, a display operation of an image of the subject on the monitor 87, and the like. Control the behavior of the element. The main control unit 81 is connected to an external storage unit such as a RAM 83, an auxiliary storage unit 84, and a ROM 85. The main control unit 81 performs various processes based on the data stored in the connected RAM 83, auxiliary storage unit 84, or ROM 85.

The RAM 83 is, for example, a rewritable volatile memory such as a DRAM, and temporarily stores data used by the main control unit 81 to execute a process, data generated during the process, and the like. The RAM 83 temporarily stores the image of the test object generated by the video processing unit 86 and output to the main control unit 81. Further, the image of the inspection object stored in the RAM 83 is read by the main control unit 81 and output to the monitor 87.

The auxiliary storage unit 84 is a storage device such as a storage device or a magnetic storage device having a rewritable non-volatile memory, and stores data of a final result of the processing executed by the main control unit 81 and the like. The ROM 85 is, for example, a rewritable non-volatile memory such as an EPROM or an EEPROM, and stores programs such as firmware for starting the endoscope apparatus 1 (main control unit 81) and processing executed by the main control unit 81. Stores programs, etc.

In addition, the main control unit 81 performs a three-dimensional measurement process in the endoscope device 1 based on the image of the test object generated by the video processing unit 86. At this time, the main control unit 81 causes the monitor 87 to display an image of the test object for setting the measurement points when performing three-dimensional measurement. Then, the main control unit 81 performs three-dimensional measurement desired by the user, based on the information on the position of the measurement point designated by the user, which is included in the information indicating the content of the operation input from the operation unit 82. , The measurement result is displayed on the monitor 87. The detailed description of the processing procedure of the three-dimensional measurement in the main controller 81 will be described later.

With such a configuration, in the endoscopic device 1, the pattern light is phase-shifted and radiated to the test object a plurality of times, and a three-dimensional image is generated from a plurality of images captured in each phase to perform a three-dimensional measurement. I do. In the endoscope device 1, the user of the endoscope device 1 operates the operation unit 82 while checking the image displayed on the monitor 87 to guide the distal end portion 4 to the vicinity of the object to be observed. There are two operation modes: an observation mode and a measurement mode in which the operation section 82 is operated while checking the image displayed on the monitor 87 to set a desired measurement point and three-dimensional measurement is performed. The switching of the operation mode is performed, for example, by the user operating the operation unit 82.

When the operation mode of the endoscope apparatus 1 is the observation mode, the illumination unit 6 emits white illumination light (hereinafter, referred to as “observation light”), and the imaging unit 5 sequentially inspects the object at a predetermined cycle. Take a picture. Then, the video processing unit 86 sequentially generates a video image (image) of the test object (hereinafter, referred to as “observation mode image”) based on the video signals sequentially output from the imaging unit 5. Then, the main control unit 81 temporarily stores the observation mode image output from the video processing unit 86 in the RAM 83, then reads it again and displays it on the monitor 87. Thereby, the user of the endoscope apparatus 1 operates the operation unit 82 while confirming the observation mode image displayed on the monitor 87 in the observation mode to guide the distal end portion 4 to the vicinity of the object to be observed. You can

On the other hand, when the operation mode of the endoscope apparatus 1 is the measurement mode, the pattern projection unit 7 irradiates the pattern light (hereinafter, referred to as “measurement light”), and the imaging unit 5 causes a predetermined number of the test objects. Take a video. Then, the video processing unit 86 generates a video image (image) of the test object (hereinafter, referred to as “measurement mode image”) based on a predetermined number of video signals output from the imaging unit 5. Then, the main control unit 81 temporarily stores the measurement mode image output from the video processing unit 86 in the RAM 83 and then reads it again to perform a three-dimensional measurement process. In this three-dimensional measurement process, as described above, the image of the test object for setting the measurement point, which is generated based on the measurement mode image, is displayed on the monitor 87. Thereby, the user of the endoscope apparatus 1 can operate the operation unit 82 and set a desired measurement point while checking the image displayed on the monitor 87 in the measurement mode. That is, the endoscope apparatus 1 can perform three-dimensional measurement at the position of the measurement point set in the measurement mode image.

In the configuration of the endoscope apparatus 1 shown in FIG. 1, some components of the imaging unit 5, the illumination unit 6, and the pattern projection unit 7 in the distal end portion 4 located at the distal end of the insertion unit 3. Shows the configuration in which is arranged. However, the endoscope device 1 may have a configuration in which the constituent elements provided in the distal end portion 4 are replaceable. For example, a part of the objective optical system 53 that forms the image pickup unit 5, a part of the illumination optical system 64 that forms the illumination unit 6, and one of the stripe generation unit 74 and the projection optical system 75 that form the pattern projection unit 7. And the part are formed as an optical adapter. At this time, at the tip portion 4, another part of the image pickup device 52 and the objective optical system 53 that form the image pickup part 5, another part of the illumination optical system 64 that forms the illumination part 6, and the pattern projection part. 7 and the other part of the projection optical system 75 that constitutes the optical system 7. Then, the distal end portion 4 and the optical adapter are made detachable. With such a configuration, the optical adapter can be replaced with an appropriate one according to the object to be observed.

Next, the operation of the endoscope device 1 will be described. First, control of light emitted by the illumination light source 61 and the pattern projection light source 71 will be described. FIG. 2 is a timing chart showing an example of the irradiation timing of light in the endoscope system (endoscope device 1) of the first embodiment. In the timing chart shown in FIG. 2, in the measurement mode, the pattern light (measurement light) phase-shifted by 90° (π/2) is applied to the test object four times to capture four measurement mode images. In this case, that is, an example in which the endoscope apparatus 1 performs an operation of generating a three-dimensional image from four measurement mode images.

When the endoscope apparatus 1 is operating in the observation mode, the light source control unit 80 controls only the illumination light source 61 to be in the ON state, and controls so that only the observation light is emitted to the test object. At this time, the light source control unit 80 adjusts the light amount of the observation light to a light amount suitable for the imaging unit 5 to photograph the test object. For example, when the image of the test object is dark or the test object is distant, the amount of observation light is increased, and when the image of the test object is bright or when the test object is near, the observation light is increased. Dimming to reduce the amount of light. Since the existing method based on the amount of light and the exposure time can be used as the method of adjusting the observation light in the light source control unit 80, detailed description thereof will be omitted. As a result, the imaging unit 5 outputs to the video processing unit 86 a video signal obtained by shooting at each timing when the observation light is irradiated to the test object, and the monitor 87 displays the observation mode image generated by the video processing unit 86. To display.

Further, when the endoscope apparatus 1 is operating in the measurement mode, the light source control unit 80 controls both the illumination light source 61 and the pattern projection light source 71 to be in the ON state, so that both the observation light and the measurement light are emitted. It is controlled to irradiate the test object. At this time, the light source control unit 80 adjusts the light amount of the observation light to a light amount according to the light amount of the measurement light. However, the endoscope apparatus 1 irradiates the observation light even in the measurement mode. That is, in the endoscope device 1, the illumination light source 61 is not turned off even in the measurement mode. More specifically, the light source control unit 80 sets the light amount of the observation light in the observation mode so that the structured pattern (vertical stripe pattern) of the measurement light emitted by the pattern projection unit 7 can be determined. The light amount is adjusted so that the combined light amount of the observation light and the measurement light becomes a light amount suitable for the imaging unit 5 to photograph the object. In the timing chart shown in FIG. 2, for example, the time during which the pattern projection light source 71 emits light, that is, the irradiation time of the measurement light is lengthened by PWM (Pulse Width Modulation) control, and the illumination light source 61 is controlled by voltage control, for example. This shows a case where the intensity of the emitted light is low, that is, the brightness of the observation light is darkened, and the measurement light and the observation light are dimmed. As a result, the image pickup unit 5 outputs to the image processing unit 86 the respective image signals obtained by taking images at the respective timings when the combined light of the observation light and the measurement light is applied to the test object, and the image processing unit 86. Based on the measurement mode image generated by 86, measurement points are set and three-dimensional measurement is performed.

It should be noted that, as a method for making it possible to determine the structured pattern of the measurement light emitted by the pattern projection unit 7 in the measurement mode, a method of dimming only the light in the same wavelength band as the wavelength band of the measurement light is used. It may be. For example, when the pattern projection light source 71 is a light source in the red wavelength band, an optical filter for attenuating light in the red wavelength band and transmitting light in the wavelength band other than red is used as an optical path of the observation light in the measurement mode. The light adjustment may be performed so that the structured pattern of the red measuring light can be discriminated by adopting the configuration of being inserted into. Further, for example, when the illumination light source 61 is a white light source configured by red, green and blue LEDs and the pattern projection light source 71 is a light source in the red wavelength band, only the light amount of the red LED is emitted. Is less than in the observation mode, and the light amounts of the green LED and the blue LED are the same as in the observation mode, so that the structured pattern of the red measuring light can be discriminated. You may.

As described above, in the endoscope device 1, the observation light is irradiated to the test object without being turned off even in the measurement mode. As a result, in the endoscope device 1, it is possible to obtain a plurality of measurement mode images photographed with the combined light of the observation light and the measurement light. Then, the endoscope apparatus 1 performs various processes on the plurality of measurement mode images to restore the three-dimensional shape of the test object and reproduce the color of the test object.

Next, processing performed on the measurement mode image in the endoscope apparatus 1 will be described. FIG. 3 is a diagram showing an outline of processing in the endoscope system (endoscope device 1) of the first embodiment. The process performed on the measurement mode image in the endoscope apparatus 1 is a process of restoring the three-dimensional shape of the test object (hereinafter referred to as “three-dimensional shape restoration process”) and a process of reproducing the color of the test object. (Hereinafter, referred to as “color reproduction processing”). The schematic diagram shown in FIG. 3 shows an outline of the three-dimensional shape restoration processing and the color reproduction processing performed based on the four measurement mode images. In the following description, it is assumed that the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser light source in the red wavelength band.

In the measurement mode, the endoscope apparatus 1 irradiates the test object four times with light in which the observation light with a reduced light amount and the measurement light with the phase shifted by 90° (π/2) are mixed, and the light is mixed four times. The four measurement mode images as shown in FIG. 3 are acquired by photographing the test object during light irradiation. After acquiring the four measurement mode images, the main control unit 81 first separates each pixel signal included in each acquired measurement mode image for each color. As a result, the main control unit 81 causes the red measurement mode image including only the pixel signals corresponding to the red (R) wavelength band and the green measurement mode including only the pixel signals corresponding to the green (G) wavelength band. An image and a blue measurement mode image including only pixel signals corresponding to the blue (B) wavelength band are generated. Then, the main control unit 81 performs the three-dimensional shape restoration process and the color reproduction process in parallel based on the red measurement mode image, the green measurement mode image, and the blue measurement mode image, respectively.

The three-dimensional shape restoration process is performed based on the red measurement mode image. The red measurement mode image is an image of a test object that reflects light in the red wavelength band, and includes a structured pattern (vertical stripe pattern) irradiated as measurement light, as shown in FIG. It is an image. The main control unit 81 performs an operation of an existing three-dimensional image, for example, as shown in the following formula (1) on the four red measurement mode images to obtain the phase φ of each pixel included in the red measurement mode image. Is calculated at high speed.

φ(x,y)=tan ⁻¹ ((I ₃ −I ₁ )/(I ₀ −I ₂ )) (1)

In the above formula (1), φ(x, y) is a horizontal direction (direction in which columns are arranged: column direction) x of pixels arranged in a two-dimensional matrix in the image sensor 52, and a vertical direction (direction in which rows are arranged). Represents the phase of the pixel (x, y) located at the coordinate position of y). Further, in the above formula (1), I ₀ represents the luminance of the pixel (x, y) included in the first red measurement mode image, and I ₁ is the pixel ((1) included in the second red measurement mode image ( x, y) represents the luminance, I ₂ represents the luminance of the pixel (x, y) included in the third red measurement mode image, and I ₃ represents the pixel included in the fourth red measurement mode image ( x, y) represents the brightness.

Then, the main control unit 81 obtains a height having a relative relationship with the calculated phase φ for each pixel included in the red measurement mode image. Accordingly, the main control unit 81 causes the pixel group data (hereinafter, “three-dimensional point group data”) in which all the pixels included in the red measurement mode image are represented by the coordinates of the horizontal direction, the vertical direction, and the height. Is generated).

In this way, the main control unit 81 performs the three-dimensional shape restoration process based on the four measurement mode images. Then, the main control unit 81 performs a three-dimensional measurement process based on the three-dimensional point cloud data generated by the three-dimensional shape restoration process. Note that, in the schematic diagram shown in FIG. 3, the case where the three-dimensional shape restoration process is performed based on the four measurement mode images is shown. However, in general, the measurement mode image necessary for performing the three-dimensional shape restoration process is shown. The number may be 3 or more. Therefore, the number of measurement mode images (red measurement mode images in FIG. 3) used for performing the three-dimensional shape restoration process in the endoscope apparatus 1, that is, for generating three-dimensional point cloud data is not particularly specified.

On the other hand, the color reproduction processing is performed based on each of the red measurement mode image, the green measurement mode image, and the blue measurement mode image. The green measurement mode image is an image of the test object that reflects light in the green wavelength band, and does not include the structured pattern (vertical stripe pattern) irradiated as the measurement light. In addition, the blue measurement mode image is an image of the test object that reflects light in the blue wavelength band, and includes a structured pattern (vertical stripe pattern) irradiated as measurement light, like the green measurement mode image. Not an image.

In the color reproduction processing, the main control unit 81 performs existing image processing such as noise removal processing on each of the four green measurement mode images and the blue measurement mode image to perform S/N (Signal/Noise). ) Is high, one green image (hereinafter referred to as “green image”) and one blue image (hereinafter referred to as “blue image”) are generated. In the following description, when a green image and a blue image are represented without being distinguished from each other, they are referred to as “stripeless images”. Note that the main control unit 81 may add and average the values of the pixels at the same coordinate positions included in the four green measurement mode images when generating the green image. Similarly, when generating the blue image, the main control unit 81 may average the values of the pixels at the same coordinate positions included in the four blue measurement mode images.

Further, in the color reproduction process, the main control unit 81 generates one red image (hereinafter, referred to as “red image”) from which the structured pattern (vertical stripe pattern) is removed, based on the four red measurement mode images. To do. More specifically, the values of the pixels at the same coordinate positions included in the four red measurement mode images are added and averaged. As a result, the phase component of the structured pattern (vertical stripe pattern) included in each red measurement mode image is removed (cancelled). In the following description, the red image is also referred to as “stripe erased image”. When generating the red image, the main control unit 81 sets the value of each pixel included in the red image, that is, the red level to be equal to the green level in the green image or the blue level in the blue image. You may perform the process which corrects so that.

In this way, the main control unit 81 generates images for each color (red image, green image, and blue image) based on the four measurement mode images in the color reproduction process. Then, the main control unit 81 combines the generated stripe-free image (green image and blue image) and the stripe-erased image (red image) to generate red (R), green (G), and blue (B). Generate a reproduced color reproduction image.

The main control unit 81 causes the monitor 87 to display the color reproduction image generated by the color reproduction processing as an image of the test object for setting a measurement point when performing three-dimensional measurement. Then, after the position (coordinates) of the measurement points are designated by the user, the main control unit 81 performs three-dimensional measurement desired by the user based on the coordinates of the three-dimensional point group data corresponding to the designated measurement points. I do.

Next, a processing procedure of three-dimensional measurement in the endoscope device 1 will be described. FIG. 4 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) of the first embodiment. FIG. 4 shows a processing procedure in the measurement mode.

When the operation mode of the endoscope apparatus 1 becomes the measurement mode, the main control unit 81 first controls the light source control unit 80 to cause the illumination unit 6 and the pattern projection unit 7 to emit light in the measurement mode. Let Then, the main control unit 81 controls the imaging unit 5 to photograph a plurality of objects to be inspected, causes the video processing unit 86 to generate a plurality of measurement mode images, and causes the RAM 83 to store the measurement mode images. It is acquired (step S01).

Subsequently, the main control unit 81 reads out each measurement mode image stored in the RAM 83, and separates the read measurement mode images for each color (step S02). As a result, each measurement mode image is separated into a red measurement mode image, a green measurement mode image, and a blue measurement mode image. Then, the main controller 81 performs the three-dimensional shape restoration process and the color reproduction process in parallel. The main control unit 81 temporarily stores again each of the red measurement mode image, the green measurement mode image, and the blue measurement mode image separated in step S02 in the RAM 83, and performs measurement required for the next process. The mode image may be read again from the RAM 83 and the subsequent processing may be performed.

In the three-dimensional shape restoration processing, the main control unit 81 first performs noise removal processing on each of the plurality of red measurement mode images (step S11). Subsequently, the main control unit 81 performs computations on existing 3-dimensional image, such as the above equation (1), it calculates the phase φ of the respective pixels included in the red measurement mode image (step S12). Subsequently, the main control unit 81 performs a noise removal process of removing a noise component included in the calculated phase φ of each pixel (step S13). Then, the main control unit 81 calculates the spatial coordinates (coordinates indicating the height) of each pixel included in the red measurement mode image based on the calculated relative relationship between the phase φ and the height, and the horizontal direction x and It is three-dimensional pixel group data (x, y, z) in which coordinates in the vertical direction y and coordinates representing the calculated height (hereinafter, referred to as “height direction z”) are associated with each pixel. Three-dimensional point cloud data is generated (step S14). Then, the main control unit 81 stores the generated three-dimensional point cloud data in the RAM 83.

On the other hand, in the color reproduction process, the main control unit 81 first performs the stripe erasing process for removing the structured pattern (vertical stripe pattern) based on the plurality of red measurement mode images to generate one stripe erasing image. (Step S21). Then, the main control unit 81 performs noise removal processing on each of the plurality of green measurement mode images and each of the blue measurement mode images to generate one stripe-free image (step S22). Subsequently, the main control unit 81 combines the generated striped image and the striped image to generate a color reproduction image (step S23). Then, the main control unit 81 stores the generated color reproduction image in the RAM 83. The main control unit 81 temporarily stores the striped image generated in step S21 or the striped image generated in step S22 in the RAM 83, and stores the striped image or the stored striped image when generating the color reproduction image. The image without stripes may be read again from the RAM 83 and the process of step S23 may be performed.

Then, the main controller 81 reads out the color reproduction image generated in step S23 from the RAM 83 and outputs it to the monitor 87 (step S24). As a result, the monitor 87 displays the color reproduction image input from the main control unit 81 as an image of the test object for setting measurement points when performing three-dimensional measurement. Then, the user of the endoscope apparatus 1 operates the operation unit 82 while confirming the color reproduction image displayed on the monitor 87, and sets measurement points for performing desired three-dimensional measurement. After that, when the measurement point is set by the user, the main control unit 81 acquires information indicating the position of the measurement point set by the operation of the user from the operation unit 82 (step S25).

Subsequently, when the main control unit 81 acquires the information on the position of the measurement point set by the user in step S25, the position (coordinate (x, y)) of the acquired measurement point and the pixels in the three-dimensional point cloud data are acquired. And the pixel data (x, y, z) of the three-dimensional point group data corresponding to the position (coordinate (x, y)) of the measurement point is acquired (step S31). The pixel data of the three-dimensional point cloud data acquired by the main control unit 81 is not limited to the pixel data of the pixel located at the position of the measurement point, and all the pixel data used in the three-dimensional measurement to be performed. To get. For example, when performing three-dimensional measurement for measuring the distance between two set measurement points (distance between two points), all pixel data located between the two measurement points is acquired.

Next, the main control unit 81 executes three-dimensional measurement based on each of the acquired pixel data (x, y, z) of the three-dimensional point cloud data (step S32). Then, the main control unit 81 performs a process of superimposing OSD (On-Screen Display) data for indicating the measurement result on an image representing the executed three-dimensional measurement result, for example, the displayed color reproduction image. image Ru is displayed and output to a monitor 87 and performing (step S33). Then, the main control unit 81 ends the process in the measurement mode.

This allows the user to confirm the desired three-dimensional measurement result. The main control unit 81 not only displays the three-dimensional measurement result on the monitor 87 and presents it to the user, but also, for example, in the auxiliary storage unit 84, the information indicating the position of the set measurement point and the three-dimensional measurement result. The measurement result may be stored in association with each other. As a result, the user can confirm the result of the three-dimensional measurement performed previously.

According to the first embodiment, the illumination unit (illumination unit 6) that irradiates the object with the observation light (white illumination light) for observing the object, and the object is three-dimensionally measured. A pattern projection unit (pattern projection unit 7) for irradiating the test object with measurement light in a predetermined wavelength band of a structured pattern (pattern light of a pattern of vertical stripes) and an observation light from the illumination unit 6. An irradiation light control unit (light source control unit 80) that controls irradiation of the inspection object and irradiation of the measurement light by the pattern projection unit 7 to the inspection object, and at least an image obtained by irradiation of the measurement light. An internal view including a three-dimensional measurement unit (main control unit 81) that performs three-dimensional measurement on the inspection object according to the designated measurement point based on the measurement image (measurement mode image) of the inspection object. In the endoscope system (endoscope device 1), the main control unit 81 generates a setting image (image of a test object: color reproduction image) for setting a measurement point based on the measurement mode image. Three-dimensional measurement is performed on the test object based on the information (three-dimensional point cloud data) of the measurement mode image corresponding to the position of the measurement point set on the image of the test object displayed on the display unit (monitor 87). An endoscope system (endoscope device 1) for performing is configured.

Further, according to the first embodiment, the main control unit 81 reflects the light in the same wavelength band (red wavelength band) as the measurement light included in the measurement mode image when performing three-dimensional measurement. 3D pixel information (pixel data of 3D point cloud data) used for 3D measurement is generated and generated based on the first wavelength band measurement image (red measurement mode image) that is an image of the test object. A three-dimensional shape restoration process (steps S11 to S14) for generating a three-dimensional measurement image (three-dimensional point cloud data) including pixel data, and an image of the object included in the three-dimensional point cloud data are set. The endoscope apparatus 1 that executes the three-dimensional measurement processing (steps S31 to S33) that performs three-dimensional measurement based on the pixel data corresponding to the position of the measured point is configured.

Further, according to the first embodiment, the light source control unit 80 causes the observation light and the measurement light to irradiate the test object at the same time when performing the three-dimensional measurement, and the main control unit 81 causes the three-dimensional measurement. Before the shape restoration process is performed, the measurement mode image is a red measurement mode image and a second wavelength band measurement which is an image of the test object due to reflection of light in a wavelength band different from the measurement light included in the measurement mode image. Image separation processing (step S02) for separating the image (green measurement mode image and blue measurement mode image) is performed, and the measurement light component (vertical stripes) included in the red measurement mode image is parallel to the three-dimensional shape restoration processing. Structured pattern removal processing (stripe removal processing: step S21) for generating a measurement light removal image (red image: striped image) from which a pattern is removed, a striped image, and a second wavelength band measurement image (noise removal) The endoscope apparatus 1 that sequentially executes a combined image generation process (steps S22 to S23) that generates a combined image (color reproduction image) by combining the processed green image and blue image: image without stripes. Composed.

Further, according to the first embodiment, the light source control unit 80 controls the light amount of the observation light to be irradiated to the object simultaneously with the measurement light according to the light amount of the measurement light. Composed.

Further, according to the first embodiment, the main controller 81 configures the endoscope apparatus 1 in which the color reproduction image is the image of the test object (steps S24 to S25).

Further, according to the first embodiment, the measurement light is an endoscope in which the structured pattern is a striped pattern (vertical striped pattern) in which the intensity of the light changes in a sinusoidal shape (in the horizontal direction). The device 1 is configured.

In addition, according to the first embodiment, the pattern projection unit 7 irradiates the test object multiple times by shifting the phase of the structured pattern (vertical stripe pattern) in the measurement light by a predetermined shift amount. Then, the main controller 81 configures the endoscope apparatus 1 that performs three-dimensional measurement based on a plurality of measurement mode images corresponding to respective images obtained by the irradiation of the respective phase-shifted measurement lights. It

In addition, according to the first embodiment, the endoscope apparatus 1 that is the laser light is used as the measurement light.

Further, according to the first embodiment, the endoscope apparatus 1 in which the wavelength band of the measurement light is the wavelength band of visible light is configured.

Further, according to the first embodiment, the endoscope apparatus 1 in which the wavelength band of the measurement light is the red wavelength band is configured.

Further, according to the first embodiment, the observation light is in the wavelength band of visible light, and the main control unit 81 causes the measurement mode image to be a measurement image by reflection of light in the red wavelength band in the image separation processing. (Red measurement mode image), measurement image by reflection of light in green wavelength band (green measurement mode image), and measurement image by reflection of light in blue wavelength band (blue measurement mode image) The measurement image obtained by reflection of light in the same wavelength band as the light (red wavelength band) is used as the red measurement mode image, and the measurement images obtained by reflection of light in the wavelength band different from the measurement light are respectively measured in green and blue. The endoscope apparatus 1 is configured as follows.

Further, according to the first embodiment, the main control unit 81, in the structured pattern removal process, the same position (coordinate position) included in the plurality of red measurement mode images corresponding to each of the plurality of measurement mode images. The endoscope apparatus 1 configured to add and average the values of the respective pixels to generate one striped image is configured.

Further, according to the first embodiment, the illumination unit (illumination unit 6) that irradiates the test object with the observation light (white illumination light) for observing the test object, and the test object in three dimensions. The pattern projection unit (pattern projection unit 7) for irradiating the object with the measurement light in the predetermined wavelength band of the structured pattern for measurement in (the pattern light of the vertical stripe pattern), and the observation light by the illumination unit 6. The irradiation light control unit (light source control unit 80) for controlling the irradiation of the object to be inspected and the irradiation of the measuring light to the object by the pattern projection unit 7, and at least the image obtained by irradiating the measuring light. And a three-dimensional measurement unit (main control unit 81) that performs three-dimensional measurement on the test object corresponding to the designated measurement point based on the measured image (measurement mode image) of the corresponding test object. A measurement method in an endoscope system (endoscope device 1), in which a main control unit 81 sets an image for setting (measurement object image: color reproduction image) for setting a measurement point based on a measurement mode image. ) Is generated and displayed on the display unit (monitor 87), and the object is measured based on the information (three-dimensional point cloud data) of the measurement mode image corresponding to the position of the measuring point set in the image of the object. A measurement method for performing three-dimensional measurement is configured.

As described above, in the endoscope apparatus 1, the color reproduction image generated from the measurement mode image acquired in the measurement mode is used as the image of the object for setting the measurement points when performing the three-dimensional measurement. Display as. As a result, in the endoscope device 1, while checking the displayed image of the test object, the measurement point is located at a position corresponding to the measurement mode image acquired in the measurement mode in order to perform three-dimensional measurement. Can be set. That is, in the conventional endoscope apparatus, the image for setting the measurement point and the image for performing the three-dimensional measurement are different images taken at different timings, whereas the endoscope apparatus 1 has the three-dimensional image. Measurement points can be set on an image generated from the same image as the image on which measurement is performed. As a result, the endoscope device 1 can obtain a measurement result obtained by accurately performing three-dimensional measurement.

For example, in the endoscope apparatus 1, even when the tip portion 4 moves when the operation mode is switched from the observation mode to the measurement mode, the user can change the angle of view from the color reproduction image displayed on the monitor 87. Can be confirmed. As a result, the endoscope apparatus 1 can obtain a correct measurement result based on the position of the set measurement point without measuring different positions due to motion blur. In addition, for example, in the endoscope apparatus 1, even when halation or the like occurs in the measurement mode image, the user sets the observation point of the halation or the like occurring from the color reproduction image displayed on the monitor 87. You can check it before you do. Then, the user can set a desired measurement point according to the state of halation that is occurring. In other words, if the position where the user wants to set the measurement point is not related to the halation that is occurring, set the measurement point in the current color reproduction image, and set the position where the user wants to set the measurement point. If halation or the like has occurred, the object to be photographed in the measurement mode can be redone and the measurement point can be set in the state in which halation or the like has not occurred. As a result, the endoscope device 1 can obtain a correct measurement result based on the position of the set measurement point without performing an incorrect measurement due to halation or the like.

<Second Embodiment>
Next, an endoscope system according to the second embodiment of the present invention will be described. In the endoscope system (endoscope device 1) of the first embodiment, a color reproduction image is displayed on the monitor 87 as an image of a test object for setting a measurement point when performing three-dimensional measurement. The case was explained. That is, in the endoscope system (endoscope device 1) of the first embodiment, the image of the test object for setting the measurement point when performing the three-dimensional measurement is a two-dimensional image. The case where the three-dimensional point cloud data generated by the three-dimensional shape restoration process is used in the three-dimensional measurement has been described. However, the three-dimensional point cloud data is included in the image of the test object for setting the measurement points when performing three-dimensional measurement, that is, the user sets the measurement points for the three-dimensional image. You can also The endoscope system of the second embodiment is an endoscope system that allows a user to set measurement points on a three-dimensional image. The case where the endoscope system of the second embodiment is also an industrial endoscope apparatus will be described.

The configuration of the endoscope device of the second embodiment is similar to the configuration of the endoscope device 1 of the first embodiment shown in FIG. Therefore, detailed description of the configuration of the endoscope apparatus of the second embodiment will be omitted, and in the following description, the three-dimensional measurement processing in the endoscope apparatus of the second embodiment will be described. In the following description, when the constituent elements of the endoscope apparatus of the second embodiment are represented, the same reference numerals as the constituent elements of the endoscope apparatus 1 of the first embodiment shown in FIG. 1 are used. explain.

FIG. 5 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) according to the second embodiment of the present invention. FIG. 5 shows a processing procedure in the measurement mode, similar to the processing procedure in the endoscope apparatus 1 of the first embodiment shown in FIG. In the following description, the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser light source in the red wavelength band, as in the endoscope device 1 of the first embodiment. To do.

When the operation mode of the endoscope apparatus 1 becomes the measurement mode, the main control unit 81, in steps S01 to S02, similarly to the processing procedure in the endoscope apparatus 1 of the first embodiment, a plurality of measurement mode images. Is acquired, and the acquired measurement mode image is separated for each color. Then, the main control unit 81 performs each of the three-dimensional shape restoration processing and the color reproduction processing in parallel, as in the endoscope device 1 of the first embodiment.

In the three-dimensional shape restoration processing, the main control unit 81 performs the three-dimensional shape restoration processing similar to the processing procedure in the endoscope apparatus 1 of the first embodiment in steps S11 to S14 to measure a plurality of red colors. Three-dimensional point cloud data is generated from the mode image, and the generated three-dimensional point cloud data is stored in the RAM 83. On the other hand, in the color reproduction processing, in step S21 to step S23, the main control unit 81 performs the color reproduction processing similar to the processing procedure in the endoscope apparatus 1 of the first embodiment, and performs the striping processing. One striped image and one noise-free striped image are generated, and a color reproduction image in which the striped image and the striped image are combined is generated and stored in the RAM 83.

Subsequently, the main control unit 81 reads each of the three-dimensional point cloud data generated in step S14 and the color reproduction image generated in step S23 from the RAM 83, and sets them as pixel data included in the three-dimensional point cloud data. A three-dimensional image in which color information included in the color reproduction image is superimposed is generated, and the generated three-dimensional image is stored again in the RAM 83 (step S311). More specifically, the main controller 81 controls the horizontal x and vertical y coordinates of each pixel data included in the three-dimensional point cloud data, and the horizontal x and vertical coordinates of each pixel included in the color reproduction image. By collating with the coordinate in the vertical direction y, the corresponding color information (r, g, b) is added to the pixel data (x, y, z) of the three-dimensional point group data located at the same coordinate (x, y). Add. Thereby, the three-dimensional point cloud data is a color three-dimensional image including red information r, green information g, and blue information b in addition to the horizontal direction x, the vertical direction y, and the height direction z. Pixel data.

In addition to color information, for example, visibility information may be added to the pixel data of the three-dimensional point cloud data. This visibility information is information indicating whether each pixel in each image or each predetermined rectangular range is transparent (translucent in some cases), and It is information indicating a display method when a shadowed portion is represented by unevenness. Accordingly, the three-dimensional point cloud data is pixel data including the horizontal direction x, the vertical direction y, the height direction z, the red information r, the green information g, and the blue information b, and the visibility a. Become.

Then, the main controller 81 reads the three-dimensional point cloud data (x, y, z, r, g, b) stored in the RAM 83 in step S311, and outputs it to the monitor 87 (step S312). As a result, the monitor 87 displays the image of the three-dimensional point cloud data input from the main control unit 81 as an image of the test object for setting the measurement points when performing three-dimensional measurement. Then, the user of the endoscope device 1 operates the operation unit 82 while confirming the image of the three-dimensional point cloud data displayed on the monitor 87, and sets the measurement points for performing the desired three-dimensional measurement. To do. The image of the three-dimensional point cloud data displayed on the monitor 87 in step S312 is a three-dimensional image. Therefore, the user operates the operation unit 82 to set the measurement point at a desired position while confirming the shape of the test object such as rotating the three-dimensional image of the test object displayed on the monitor 87. be able to. After that, when the measurement point is set by the user, the main control unit 81 sets the measurement point from the operation unit 82 by the user's operation, similarly to step S25 in the processing procedure in the endoscope apparatus 1 of the first embodiment. Information representing the position of the measured point is acquired (step S313).

Subsequently, when the main control unit 81 acquires the information on the position of the measurement point set by the user in step S313, in step S31 to step S33, it is the same as the processing procedure in the endoscope apparatus 1 of the first embodiment. Then, the pixel data (x, y, z) of the three-dimensional point group data corresponding to the position of the measurement point (coordinates (x, y)) is acquired, three-dimensional measurement is executed, and the executed three-dimensional measurement is performed. The result is output to the monitor 87, and the processing in the measurement mode ends. As a result, the user can confirm the desired three-dimensional measurement result as in the processing procedure in the endoscope apparatus 1 according to the first embodiment.

According to the second embodiment, the three-dimensional measurement unit (main control unit 81) is an image (three-dimensional image) in which the composite image (color reproduction image) is superimposed on the three-dimensional measurement image (three-dimensional point cloud data). An endoscope system (endoscope device 1) having a setting image (image of the object to be inspected) is configured.

As described above, in the endoscope apparatus 1 according to the second embodiment, the 3D point cloud data generated by the 3D shape restoration process for the measurement mode image acquired in the measurement mode is generated from the same measurement mode image. The three-dimensional image is generated by superimposing the color information included in the color reproduction image. Then, in the endoscope apparatus 1 of the second embodiment, the generated three-dimensional image of the test object is displayed as an image of the test object for setting a measurement point when performing three-dimensional measurement. That is, also in the endoscope apparatus 1 of the second embodiment, as in the endoscope apparatus 1 of the first embodiment, an image generated from the same image as the image for performing three-dimensional measurement is set as the measurement point. The image is displayed as an image of the test object. Then, in the endoscope apparatus 1 of the second embodiment, while confirming the shape of the test object by a function of assisting the setting of the measurement point by the user, such as rotating the displayed three-dimensional image of the test object. , Measurement points can be set at more detailed positions. As a result, also in the endoscope apparatus 1 of the second embodiment, it is possible to obtain the measurement result obtained by accurately performing the three-dimensional measurement, as in the endoscope apparatus 1 of the first embodiment.

<Third Embodiment>
Next, an endoscope system according to the third embodiment of the present invention will be described. In the endoscope system of the third embodiment as well, like the endoscope system (endoscope device 1) of the second embodiment, the user can set the measurement points on the three-dimensional image. It is an endoscope system. However, the endoscope system of the third embodiment differs from the endoscope system (endoscope device 1) of the second embodiment in the three-dimensional image of the test object to be generated. The case where the endoscope system of the third embodiment is also an industrial endoscope apparatus will be described.

The configuration of the endoscope device of the third embodiment is similar to the configuration of the endoscope device 1 of the first embodiment shown in FIG. Therefore, detailed description of the configuration of the endoscope apparatus of the third embodiment is omitted, and in the following description, the three-dimensional measurement processing in the endoscope apparatus of the third embodiment will be described. Note that, also in the following description, the same reference numerals as those of the constituent elements of the endoscope apparatus 1 of the first embodiment shown in FIG. 1 are used to represent the constituent elements of the endoscope apparatus of the third embodiment. Explain.

FIG. 6 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) according to the third embodiment of the present invention. FIG. 6 shows the measurement procedure similar to the processing procedure in the endoscope apparatus 1 of the first embodiment shown in FIG. 4 and the processing procedure in the endoscope apparatus 1 of the second embodiment shown in FIG. The processing procedure in mode is shown. In the following description, the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser in the red wavelength band, as in the endoscope device 1 of the first and second embodiments. Description will be given assuming that the light source is a light source.

When the operation mode of the endoscope apparatus 1 becomes the measurement mode, the main control unit 81, in steps S01 to S02, similarly to the processing procedure in the endoscope apparatus 1 of the first embodiment and the second embodiment. , A plurality of measurement mode images are acquired, and the acquired measurement mode images are separated for each color. Then, the main control unit 81 performs each of the three-dimensional shape restoration processing and the color reproduction processing in parallel, as in the endoscope device 1 of the first and second embodiments.

In the three-dimensional shape restoration process, the main control unit 81 performs the same three-dimensional shape restoration process as the processing procedure in the endoscope device 1 of the first embodiment and the second embodiment in steps S11 to S14. Then, three-dimensional point cloud data is generated from the plurality of red measurement mode images, and the generated three-dimensional point cloud data is stored in the RAM 83. Further, the main control unit 81 performs an operation of the existing image processing technology based on the generated three-dimensional point cloud data to visualize the depth information of the object to be examined and display the depth image (depth). Map) and stores the generated depth map in the RAM 83 (step S151).

On the other hand, in the color reproduction process, in step S21 to step S23, the main control unit 81 performs the same color reproduction process as the processing procedure in the endoscope device 1 according to the first and second embodiments, A color reproduction image is generated by combining the striped image generated by the striping process and the non-striped image generated by the noise removal process, and is stored in the RAM 83.

Subsequently, the main control unit 81 reads each of the depth map generated in step S151 and the color reproduction image generated in step S23 from the RAM 83, and pastes the color reproduction image on the depth map as a texture. A depth map including color information is generated, and the generated depth map is stored again in the RAM 83 (step S321). More specifically, the main control unit 81 calculates the existing image processing technology and includes each pixel corresponding to the surface of the monochrome depth map generated based on the three-dimensional point cloud data in the color reproduction image. The color depth map representing the texture of the test object is generated by superimposing the information of the colors to be displayed.

Subsequently, the main control unit 81 reads out the color depth map stored in the RAM 83 in step S321 and outputs it to the monitor 87 (step S322). As a result, the monitor 87 displays the image of the color depth map input from the main control unit 81 as an image of the test object for setting measurement points when performing three-dimensional measurement. Then, the user of the endoscope apparatus 1 operates the operation unit 82 while checking the image of the color depth map displayed on the monitor 87, and sets the measurement points for performing the desired three-dimensional measurement. .. The image of the color depth map displayed on the monitor 87 in step S312 is also a three-dimensional image, as in the endoscope apparatus 1 of the second embodiment. Therefore, the user can set the measurement point at a desired position while confirming the shape of the test object displayed on the monitor 87, as in the endoscopic device 1 of the second embodiment. After that, when the measurement point is set by the user, the main control unit 81 performs step S25 in the processing procedure of the endoscope apparatus 1 of the first embodiment or the endoscope apparatus 1 of the second embodiment. Similar to step S313 in the processing procedure, the information indicating the position of the measurement point set by the user operation is acquired from the operation unit 82 (step S323).

Subsequently, in steps S31 to S33, the main control unit 81, similar to the processing procedure in the endoscope apparatus 1 of the first and second embodiments, the position of the measurement point set by the user ( The pixel data (x, y, z) of the three-dimensional point group data corresponding to the coordinates (x, y) is acquired, three-dimensional measurement is executed, and the executed three-dimensional measurement result is output to the monitor 87. Then, the process in the measurement mode ends. Accordingly, the user can confirm the desired three-dimensional measurement result, as in the processing procedure in the endoscope apparatus 1 according to the first and second embodiments.

According to the third embodiment, the three-dimensional measurement unit (main control unit 81) performs the inspection based on the generated three-dimensional pixel information (pixel data of three-dimensional point cloud data) in the three-dimensional shape restoration process. An endoscope system (endoscope device 1) is configured that generates a depth image (depth map) representing the depth of an object and uses the generated depth map as a three-dimensional measurement image.

As described above, in the endoscope apparatus 1 of the third embodiment, the depth map is generated based on the three-dimensional point cloud data generated by the three-dimensional shape restoration process for the measurement mode image acquired in the measurement mode. To do. Furthermore, in the endoscope apparatus 1 of the third embodiment, the generated depth map is superimposed with the color information included in the color reproduction image generated from the measurement mode image that is the same as the measurement mode image that generated this depth map. To generate a color depth map (three-dimensional image). Then, in the endoscope apparatus 1 according to the third embodiment, the generated color depth map image of the test object is used as an image of the test object for setting a measurement point when performing three-dimensional measurement. indicate. That is, also in the endoscope apparatus 1 of the third embodiment, an image generated from the same image as the image for performing three-dimensional measurement, as in the endoscope apparatus 1 of the first embodiment and the second embodiment. Is displayed as an image of a test object for setting a measurement point. Then, also in the endoscope apparatus 1 of the third embodiment, similar to the endoscope apparatus 1 of the second embodiment, the measurement point by the user such as rotating the three-dimensional image of the displayed object is measured. It is possible to set measurement points at detailed positions while confirming the shape of the object to be inspected by the function of assisting the setting. As a result, even in the endoscope apparatus 1 of the third embodiment, the measurement result obtained by accurately performing the three-dimensional measurement is obtained as in the endoscope apparatus 1 of the first embodiment and the second embodiment. Obtainable.

<Fourth Embodiment>
Next, an endoscope system according to the fourth embodiment of the present invention will be described. In the endoscope system (endoscope device 1) of the first to third embodiments, the image of the object to be displayed on the monitor 87 for setting the measurement points when performing three-dimensional measurement. However, the case where the image is a color image has been described. However, the image of the test object for setting the measurement points when performing three-dimensional measurement is not limited to a color image, and may be a monochrome image, that is, a grayscale image. .. The endoscope system of the fourth embodiment is an endoscope system in which a user sets measurement points on a monochrome image. The case where the endoscope system of the fourth embodiment is also an industrial endoscope apparatus will be described.

The configuration of the endoscope apparatus of the fourth embodiment is similar to that of the endoscope apparatus 1 of the first embodiment shown in FIG. Therefore, detailed description of the configuration of the endoscope apparatus of the fourth embodiment will be omitted, and in the following description, a three-dimensional measurement process in the endoscope apparatus of the fourth embodiment will be described. Note that, also in the following description, when the constituent elements of the endoscope apparatus of the fourth embodiment are represented, the same reference numerals as those of the constituent elements of the endoscope apparatus 1 of the first embodiment shown in FIG. 1 are used. Explain.

FIG. 7 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) according to the fourth embodiment of the present invention. FIG. 7 shows a processing procedure in the measurement mode, similar to the processing procedure in the endoscope apparatus 1 of the first to third embodiments shown in FIGS. 4 to 6. In the following description, the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser in the red wavelength band, as in the endoscope devices 1 of the first to third embodiments. Description will be given assuming that the light source is a light source.

When the operation mode of the endoscope apparatus 1 becomes the measurement mode, the main control unit 81, in steps S01 to S02, similarly to the processing procedure in the endoscope apparatus 1 of the first to third embodiments. , Multiple measurement mode images are acquired, and each measurement mode image is separated for each color. Then, the main control unit 81 performs each of the three-dimensional shape restoration processing and the color reproduction processing in parallel, as in the endoscope device 1 of the first to third embodiments.

In the three-dimensional shape restoration process, the main control unit 81 performs the same three-dimensional shape restoration process in steps S11 to S14 as the processing procedure in the endoscope device 1 of the first to third embodiments. Then, three-dimensional point cloud data is generated from the plurality of red measurement mode images, and the generated three-dimensional point cloud data is stored in the RAM 83.

On the other hand, in the color reproduction process, in step S21, the main control unit 81 performs a striping process similar to the processing procedure in the endoscope device 1 of the first to third embodiments, and a plurality of red colors are obtained. One striped image (monochrome image) from which the structured pattern (vertical stripe pattern) has been removed is generated based on the measurement mode image, and the generated striped image is stored in the RAM 83. In the endoscopic device 1 of the fourth embodiment, since the monochrome image is the image of the test object for setting the measurement point when performing the three-dimensional measurement, the main control unit 81 The processing of step S22 and the processing of step S23, which are performed in the processing procedure in the endoscope apparatus 1 of the first to third embodiments, are not performed in the color reproduction processing. In other words, in the endoscope apparatus 1 of the fourth embodiment, the main control unit 81 generates the stripe-free image for which noise removal processing has been performed in the color reproduction processing, and the striped image and the stripe-free image, respectively. Is not performed.

Subsequently, the main control unit 81 reads the striped image generated in step S21 from the RAM 83 and outputs it to the monitor 87 (step S241). As a result, the monitor 87 displays the monochrome striped image input from the main control unit 81 as an image of the test object for setting measurement points when performing three-dimensional measurement. Then, the user of the endoscope apparatus 1 operates the operation unit 82 while confirming the monochrome striped image displayed on the monitor 87, and sets a measurement point for performing desired three-dimensional measurement. After that, when the measurement point is set by the user, the main control unit 81 operates the operation unit 82 to operate the operation unit 82 in step S25 in the same manner as the processing procedure in the endoscope apparatus 1 of the first embodiment. Acquires information indicating the position of the set measurement point.

Subsequently, when the main control unit 81 acquires the information on the position of the measurement point set by the user in step S25, in steps S31 to S33, the endoscope apparatus according to the first to third embodiments. Similar to the processing procedure in 1, the pixel data (x, y, z) of the three-dimensional point group data corresponding to the position of the measurement point (coordinates (x, y)) is acquired, and three-dimensional measurement is executed. The executed three-dimensional measurement result is output to the monitor 87, and the processing in the measurement mode ends. Thereby, the user can confirm the desired three-dimensional measurement result, as in the processing procedure in the endoscope apparatus 1 of the first to third embodiments.

According to the fourth embodiment, the three-dimensional measurement unit (main control unit 81) performs the measurement light included in the first wavelength band measurement image (red measurement mode image) in parallel with the three-dimensional shape restoration process. Structured pattern removal processing (stripe removal processing: step S21) that generates a measurement light removal image (red image: striped image) in which components have been removed is executed, and the striped image is set as an image for setting (image of the object to be inspected). The endoscope system (endoscope device 1) is defined as follows.

As described above, although the endoscope device 1 of the fourth embodiment is a monochrome image, it is three-dimensional like the endoscope device 1 of the first to third embodiments. An image generated from the same image as the measurement mode image for performing measurement (stripe erased image) is displayed as an image of the test object for setting a measurement point when performing three-dimensional measurement. Thereby, also in the endoscope apparatus 1 of the fourth embodiment, as in the endoscope apparatus 1 of the first to third embodiments, while confirming the image of the object to be inspected, the desired position can be obtained. The measurement point can be set to. As a result, even in the endoscope device 1 of the fourth embodiment, the measurement result obtained by accurately performing the three-dimensional measurement is obtained as in the endoscope device 1 of the first to third embodiments. Obtainable.

Furthermore, in the endoscope apparatus 1 of the fourth embodiment, since the image of the object to be inspected for setting the measurement point is a monochrome striped image, one of the first to third embodiments It is possible to reduce the calculation load of color reproduction processing for generating a color image as in the endoscope apparatus 1. As a result, in the endoscope apparatus 1 of the fourth embodiment, the object to be measured for setting the measurement point is earlier than the endoscope apparatus 1 of the first to third embodiments. Images can be displayed.

In addition, in the endoscope apparatus 1 of the fourth embodiment, as described above, in the color reproduction processing, the image without stripes and the color reproduction image are not generated. That is, the endoscope apparatus 1 of the fourth embodiment does not perform the processing based on each of the green measurement mode image and the blue measurement mode image. For this reason, in the endoscope apparatus 1 of the fourth embodiment, the observation light from the illumination unit 6 does not have to be applied to the test object at the same time as the measurement light in the measurement mode. That is, in the endoscope apparatus 1 of the fourth embodiment, when operating in the measurement mode, the light source control unit 80 controls the illumination light source 61 to be in the off state (turns off) and is not exposed to the observation light. Irradiation to the specimen may not be performed. In this case, when the endoscope apparatus 1 according to the fourth embodiment is operating in the measurement mode, the light to be inspected is only the measurement light, and the three-dimensional image for the measurement mode image acquired in the measurement mode is obtained. In the shape restoration process, the change in the gradation of the measurement light included in each red measurement mode image can be discriminated with higher accuracy, and the spatial coordinates (height) can be calculated when generating the three-dimensional point cloud data. Can be performed with higher accuracy. Further, in this case, since the measurement mode image acquired in the measurement mode of the endoscope apparatus 1 of the fourth embodiment is the red measurement mode image, the processing of step S02 in the processing procedure shown in FIG. 7 is performed. It may not be necessary, and the processing procedure can be facilitated.

In the endoscopic device 1 according to the fourth embodiment, a case where a monochrome striped image is displayed on the monitor 87 as an image of a test object for setting measurement points when performing three-dimensional measurement explained. That is, in the endoscopic device 1 of the fourth embodiment, the case where the image of the test object for setting the measurement point when performing the three-dimensional measurement is a two-dimensional image has been described. However, in the endoscopic device 1 of the fourth embodiment, although the image is a monochrome image, the user does not see the three-dimensional image as in the endoscopic device 1 of the second embodiment or the third embodiment. It is also possible to set the measurement point by setting. The processing procedure in this case is such that the main control unit 81 does not perform all the processes in the color reproduction process and the colors included in the color reproduction image for the three-dimensional point cloud data and the depth map generated by the three-dimensional shape restoration process. Is a process of displaying an image on the monitor 87 without superimposing the information of No. 1, and the other processing procedure is the processing procedure of the endoscope apparatus 1 of the second embodiment shown in FIG. 5 or the processing procedure of FIG. It is the same as the processing procedure of the endoscope apparatus 1 of the third embodiment. In the measurement mode, when irradiating only the measurement light on the object to be inspected, the processing procedure does not include the processing of step S02 as described above. Therefore, in the endoscope apparatus 1 according to the fourth embodiment, a process of displaying a monochrome three-dimensional image on the monitor 87 as an image of a test object for setting a measurement point when performing three-dimensional measurement. Detailed description of the procedure is omitted.

According to the fourth embodiment, the main controller 81 configures the endoscope apparatus 1 in which the three-dimensional measurement image (three-dimensional image: three-dimensional point cloud data or depth map) is used as the image of the subject. It

<Fifth Embodiment>
Next, an endoscope system according to the fifth embodiment of the present invention will be described. In the endoscope system (endoscope device 1) of the first to third embodiments, the image of the object to be displayed on the monitor 87 for setting the measurement points when performing three-dimensional measurement. The case where the color information included in the color reproduction image generated from the same image as the image on which the three-dimensional measurement is performed has been described as the color information of. However, the color information included in the observation mode image captured in the observation mode can also be used as the color information in the image of the test object for setting the measurement points when performing three-dimensional measurement. The endoscope system of the fifth embodiment uses the color information included in the observation mode image as the color information of the image of the object to be measured for setting the measurement point when performing the three-dimensional measurement. It is an endoscope system. The case where the endoscope system of the fifth embodiment is also an industrial endoscope apparatus will be described.

The configuration of the endoscope apparatus of the fifth embodiment is similar to that of the endoscope apparatus 1 of the first embodiment shown in FIG. Therefore, a detailed description of the configuration of the endoscope apparatus of the fifth embodiment will be omitted, and in the following description, a three-dimensional measurement process in the endoscope apparatus of the fifth embodiment will be described. Note that, also in the following description, when the constituent elements of the endoscope apparatus of the fifth embodiment are represented, the same reference numerals as those of the constituent elements of the endoscope apparatus 1 of the first embodiment shown in FIG. 1 are used. Explain.

FIG. 8 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) according to the fifth embodiment of the present invention. FIG. 8 shows a processing procedure in the measurement mode, similar to the processing procedure in the endoscope apparatus 1 of the first to fourth embodiments shown in FIGS. 4 to 7. In the following description, the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser in the red wavelength band, as in the endoscope devices 1 of the first to fourth embodiments. Description will be given assuming that the light source is a light source.

When the operation mode of the endoscope apparatus 1 is changed to the measurement mode, the main control unit 81 determines in step S01 the same as the processing procedure in the endoscope apparatus 1 of the first to fourth embodiments. Acquire the measurement mode image. Subsequently, the main control unit 81 reads out (acquires) an observation mode image acquired in advance, for example, an observation mode image last captured and stored in the RAM 83 in the observation mode period before shifting to the measurement mode. (Step S011). Subsequently, in step S02, the main control unit 81 reads out each measurement mode image stored in the RAM 83, similarly to the processing procedure in the endoscope apparatus 1 of the first to fourth embodiments, Each measurement mode image is separated for each color. Then, the main control unit 81 performs the three-dimensional shape restoration process and the color reproduction process in parallel, as in the endoscope device 1 according to the first to fourth embodiments.

In the three-dimensional shape restoration process, the main control unit 81 performs the three-dimensional shape restoration process similar to the processing procedure in the endoscope device 1 of the first to fourth embodiments in steps S11 to S14. Then, three-dimensional point cloud data is generated from the plurality of red measurement mode images, and the generated three-dimensional point cloud data is stored in the RAM 83.

On the other hand, in the color reproduction process, the main control unit 81 extracts color information of each pixel (hereinafter, referred to as “color information”) included in the acquired observation mode image, and the extracted color information of each pixel is stored in the RAM 83. (Step S261).

Subsequently, the main control unit 81 reads each of the three-dimensional point cloud data generated in step S14 and the color information extracted in step S261 from the RAM 83, and each pixel data included in the three-dimensional point cloud data, The displacement of the color information is detected. Then, the main control unit 81 performs a correction to match the position of the color information with the position of each pixel data included in the three-dimensional point group data based on the detected position shift, and the corrected color information is again stored in the RAM 83. (Step S331). For example, in the process of step S331, the main control unit 81, in the shift detection process, the position (coordinates) of the pixel representing the characteristic shape of the object included in the three-dimensional point group data, and the color information. The difference between the position (coordinates) of the pixel representing the characteristic shape of the object included in the extracted observation mode image is detected as the positional shift between the pixel data and the color information. Then, the main control unit 81 changes the positions (coordinates) of all the color information to the positions (coordinates) in which the deviation has been eliminated in the deviation correction processing, and stores the positions in the RAM 83 again.

Subsequently, the main control unit 81 extracts a region such as halation included in the three-dimensional point cloud data (hereinafter, referred to as “halation region”), and stores information on the extracted halation region in the RAM 83 (step S332). ..

Subsequently, the main control unit 81 reads the color information in which the misregistration is corrected in step S331 and the halation area information extracted in step S332 from the RAM 83, and positions the color information in the halation area based on the read halation area information. Correct the color information. Then, the main controller 81 stores the corrected color information in the RAM 83 again (step S333). More specifically, the main control unit 81 sets the color information located within the range of the halation region (within the range represented by the coordinates) to the color information indicating that halation or the like is occurring, for example, white color. Alternatively, it is changed to black and stored again in the RAM 83. This allows the user of the endoscope apparatus 1 to check the image displayed on the monitor 87 for whiteout or blackout due to halation.

Subsequently, the main control unit 81 reads out the three-dimensional point group data generated in step S14 and the color information corrected in step S333 from the RAM 83, and colors the pixel data included in the three-dimensional point group data. A three-dimensional image on which information is superimposed is generated, and the generated three-dimensional image is stored again in the RAM 83 (step S334). More specifically, the main controller 81 controls the horizontal x and vertical y coordinates of each pixel data included in the three-dimensional point cloud data and the horizontal x and vertical y coordinates of each color information. And the corresponding color information (r, g, b) is added to the pixel data (x, y, z) of the three-dimensional point group data located at the same coordinates (x, y). As a result, the three-dimensional point cloud data has a red color in addition to the horizontal direction x, the vertical direction y, and the height direction z, similarly to the three-dimensional point cloud data in the endoscope apparatus 1 of the second embodiment. It becomes pixel data of a color three-dimensional image including information r, green information g, and blue information b. Note that the pixel data of the three-dimensional point cloud data is added with information other than color, such as visibility information, for example, as in the three-dimensional point cloud data in the endoscope apparatus 1 according to the second embodiment. Good.

Subsequently, the main control unit 81 reads the three-dimensional image stored in the RAM 83 in step S334 and outputs it to the monitor 87 (step S312). As a result, the monitor 87 displays the three-dimensional image input from the main control unit 81 as an image of the test object for setting measurement points when performing three-dimensional measurement. Then, the user of the endoscope apparatus 1 operates the operation unit 82 while confirming the three-dimensional image displayed on the monitor 87, and sets a measurement point for performing desired three-dimensional measurement. It should be noted that, similarly to the endoscope apparatus 1 according to the second embodiment, the user measures the desired position while confirming the shape of the test object by rotating the displayed three-dimensional image of the test object. You can set points. After that, when the measurement point is set by the user, the main control unit 81 acquires information indicating the position of the measurement point set by the operation of the user from the operation unit 82 (step S313).

Subsequently, in step S31 to step S33, the main control unit 81, similarly to the processing procedure in the endoscope apparatus 1 of the first to fourth embodiments, the position of the measurement point set by the user ( The pixel data (x, y, z) of the three-dimensional point group data corresponding to the coordinates (x, y) is acquired, three-dimensional measurement is executed, and the executed three-dimensional measurement result is output to the monitor 87. Then, the processing in the measurement mode ends. Thus, the user can confirm the desired three-dimensional measurement result, as in the processing procedure in the endoscope device 1 according to the first to fourth embodiments.

According to the fifth embodiment, the three-dimensional measurement unit (main control unit 81) causes the three-dimensional measurement image (three-dimensional point cloud data) to be irradiated with the observation light in advance, and the object corresponding to the image obtained. An endoscope system (endoscope device 1) is configured in which an image (three-dimensional image) on which an observation image (observation mode image) of an inspection object is superimposed is used as a setting image (image of the inspection object).

As described above, in the endoscope apparatus 1 according to the fifth embodiment, the color information included in the observation mode image acquired in advance is generated by the three-dimensional shape restoration process for the measurement mode image acquired in the measurement mode. A three-dimensional image is generated by superimposing it on the three-dimensional point cloud data. Then, in the endoscope apparatus 1 of the fifth embodiment, the generated three-dimensional image of the test object is displayed as an image of the test object for setting measurement points when performing three-dimensional measurement. That is, also in the endoscope device 1 of the fifth embodiment, an image generated from the same image as the image for performing three-dimensional measurement, as in the endoscope devices 1 of the first to fourth embodiments. Is displayed as an image of a test object for setting a measurement point. Then, in the endoscope apparatus 1 of the fifth embodiment, while confirming the shape of the test object by a function of assisting the setting of the measurement point by the user, such as rotating the displayed three-dimensional image of the test object. In order to perform three-dimensional measurement, it is possible to set measurement points at positions corresponding to the measurement mode image acquired in the measurement mode on a one-to-one basis. As a result, even in the endoscope apparatus 1 of the fifth embodiment, the measurement result obtained by accurately performing the three-dimensional measurement is obtained as in the endoscope apparatus 1 of the first to fourth embodiments. Obtainable.

Moreover, in the endoscope apparatus 1 according to the fifth embodiment, since the color information of the image of the object for setting the measurement point is extracted from the observation mode image acquired in advance in the observation mode, the red (R ), green (G), and blue (B) are the color levels when the object reflects white illumination light (observation light). Therefore, the endoscope apparatus 1 according to the fifth embodiment can display the image of the test object for setting the measurement point at the same color level as the observation mode image.

In the endoscope apparatus 1 of the fifth embodiment, as described above, the color information extracted from the observation mode image acquired in advance in the observation mode is used to restore the three-dimensional shape of the measurement mode image acquired in the measurement mode. It is displayed by being superimposed on the three-dimensional point cloud data generated by the processing. That is, the endoscope apparatus 1 according to the fifth embodiment does not perform the color reproduction process based on the measurement mode image acquired in the measurement mode. Therefore, in the endoscope apparatus 1 of the fifth embodiment, in the measurement mode, it is not necessary to irradiate the observation light from the illumination unit 6 to the test object at the same time as the measurement light, and the operation in the measurement mode is performed. The light source control unit 80 may control the illumination light source 61 to be in the off state (turn off the light) so as not to irradiate the test light with the observation light. That is, it is possible to make only the measurement light the light with which the test object is irradiated when the endoscope apparatus 1 of the fifth embodiment is operating in the measurement mode. As a result, in the endoscope apparatus 1 of the fifth embodiment, when the three-dimensional shape restoration process is performed on the measurement mode image acquired in the measurement mode, the measurement light included in each red measurement mode image The change in gradation can be determined with higher accuracy, and the calculation of spatial coordinates (height) when generating the three-dimensional point cloud data can be performed with higher accuracy. Further, the processing of step S02 in the processing procedure of the endoscope apparatus 1 of the fifth embodiment shown in FIG. 8 does not have to be performed, and the processing procedure can be simplified.

In the endoscopic device 1 of the fifth embodiment, the observation mode image acquired in step S011 is the last observation image captured in the observation mode image before shifting to the measurement mode and stored in the RAM 83. The case has been described. However, the observation mode image acquired in step S011 is not limited to the observation mode image stored in the RAM 83 in the observation mode. For example, an image similar to the observation mode image acquired in the observation mode period by first irradiating the test object with the observation light after shifting to the measurement mode may be the observation mode image acquired in step S011. .. In this case, the order of step S01 and step S011 in the flowchart shown in FIG. 8 may be exchanged.

<Sixth Embodiment>
Next, an endoscope system according to the sixth embodiment of the present invention will be described. Similarly to the endoscope system (endoscope device 1) of the fifth embodiment, the endoscope system of the sixth embodiment also performs three-dimensional measurement of the color information included in the observation mode image. The endoscope system is used as color information of an image of an object to be measured for setting the measurement points. However, in the endoscope system of the sixth embodiment, the format of using the observation mode image as the color information of the image of the subject is the endoscope system of the fifth embodiment (endoscope device 1). Different from The endoscopic system according to the sixth embodiment is also described as being an industrial endoscopic device.

The configuration of the endoscope apparatus of the sixth embodiment is similar to the configuration of the endoscope apparatus 1 of the first embodiment shown in FIG. Therefore, a detailed description of the configuration of the endoscope apparatus of the sixth embodiment will be omitted, and in the following description, a three-dimensional measurement process in the endoscope apparatus of the sixth embodiment will be described. Note that, also in the following description, when the constituent elements of the endoscope apparatus of the sixth embodiment are represented, the same reference numerals as those of the constituent elements of the endoscope apparatus 1 of the first embodiment shown in FIG. 1 are used. Explain.

FIG. 9 is a flowchart showing a processing procedure in the endoscope system (endoscope device 1) according to the sixth embodiment of the present invention. FIG. 9 shows a processing procedure in the measurement mode, similar to the processing procedure in the endoscope device 1 of the first to fifth embodiments shown in FIGS. 4 to 8. In the following description, the illumination light source 61 is a white LED light source and the pattern projection light source 71 is a laser in the red wavelength band, as in the endoscope devices 1 of the first to fifth embodiments. Description will be given assuming that the light source is a light source.

When the operation mode of the endoscope apparatus 1 becomes the measurement mode, the main control unit 81, in steps S01 to S02, similarly to the processing procedure in the endoscope apparatus 1 of the fifth embodiment shown in FIG. Each of a plurality of measurement mode images and an observation mode image acquired in advance is acquired, and the acquired measurement mode images are separated for each color. Then, the main control unit 81 performs each of the three-dimensional shape restoration processing and the color reproduction processing in parallel, as in the endoscope device 1 of the fifth embodiment.

In the three-dimensional shape restoration processing, the main control unit 81 performs the same three-dimensional shape restoration processing in steps S11 to S14 as the processing procedure in the endoscope device 1 of the first to fifth embodiments. Then, three-dimensional point cloud data is generated from the plurality of red measurement mode images, and the generated three-dimensional point cloud data is stored in the RAM 83.

On the other hand, in the color reproduction processing, the main control unit 81 performs existing image processing such as noise removal processing on the acquired observation mode image and stores the processed observation mode image in the RAM 83 (step S271).

Subsequently, the main control unit 81 reads each of the three-dimensional point cloud data generated in step S14 and the observation mode image processed in step S261 from the RAM 83, and the respective pixel data included in the three-dimensional point cloud data. , Detecting the positional deviation from the observation mode image. Then, the main control unit 81 performs a correction to match the position of the observation mode image with the position of each pixel data included in the three-dimensional point cloud data based on the detected positional deviation, and the corrected observation mode image is displayed. It is again stored in the RAM 83 (step S341). For example, in the process of step S341, the main control unit 81 processes the position (coordinates) of the pixels representing the characteristic shape of the object included in the three-dimensional point group data in the shift detection process. The difference between the position (coordinates) of the pixel representing the characteristic shape of the test object included in the observation mode image is detected as the positional deviation between the pixel data and the observation mode image. Then, in the shift correction process, the main control unit 81 changes the positions (coordinates) of all the pixels in the observation mode image to the positions (coordinates) where the shift has been eliminated, and stores them in the RAM 83 again.

Subsequently, in step S332, the main control unit 81 extracts the halation region included in the three-dimensional point cloud data, similarly to the processing procedure in the endoscope apparatus 1 of the fifth embodiment shown in FIG. Information on the extracted halation area is stored in the RAM 83.

Subsequently, the main control unit 81 reads the observation mode image in which the shift is corrected in step S341 and the halation region information extracted in step S332 from the RAM 83, and based on the read halation region information, the observation mode image is displayed. Display the halation area on. Then, the main control unit 81 stores the observation mode image in which the halation region is displayed in the RAM 83 again (step S343). More specifically, the main control unit 81 determines, for each pixel included in the observation mode image, the value of the luminance of the pixel located within the range of the halation region (within the range represented by the coordinates), such as halation. Is changed to a value of the brightness indicating that the occurrence of the occurrence of, for example, a value of a whiteout or a blackout level, and is stored in the RAM 83 again. This allows the user of the endoscope apparatus 1 to check the image displayed on the monitor 87 for whiteout or blackout due to halation.

Subsequently, the main control unit 81 reads each of the three-dimensional point cloud data generated in step S14 and the observation mode image displaying the halation area in step S343 from the RAM 83 and sets the observation mode image in the three-dimensional point cloud data. Three-dimensional point cloud data attached as a texture is generated, and the generated three-dimensional point cloud data is stored again in the RAM 83 (step S344). More specifically, the main control unit 81 calculates the color of each pixel included in the observation mode image by performing the calculation of the existing image processing technology and adding each pixel data included in the monochrome three-dimensional point cloud data. By superimposing the above information, color three-dimensional point cloud data representing the texture of the object is generated.

Subsequently, the main control unit 81 reads the three-dimensional point cloud data stored in the RAM 83 in step S334 and outputs it to the monitor 87 (step S345). As a result, the monitor 87 displays the image of the three-dimensional point cloud data input from the main control unit 81 as an image of the test object for setting the measurement points when performing three-dimensional measurement. Then, the user of the endoscope device 1 operates the operation unit 82 while confirming the image of the three-dimensional point cloud data displayed on the monitor 87, and sets the measurement points for performing the desired three-dimensional measurement. To do. The image of the three-dimensional point cloud data displayed on the monitor 87 in step S345 is also a three-dimensional image of the object to be inspected, as in the endoscope apparatus 1 of the fifth embodiment. Therefore, the user can set the measurement point at a desired position while confirming the shape of the test object displayed on the monitor 87, as in the endoscope apparatus 1 of the fifth embodiment. After that, when the measurement point is set by the user, the main control section 81 acquires information indicating the position of the measurement point set by the operation of the user from the operation section 82 (step S346).

Subsequently, in step S31 to step S33, the main control unit 81, similar to the processing procedure in the endoscope device 1 of the first to fifth embodiments, the position of the measurement point set by the user ( The pixel data (x, y, z) of the three-dimensional point group data corresponding to the coordinates (x, y) is acquired, three-dimensional measurement is executed, and the executed three-dimensional measurement result is output to the monitor 87. Then, the process in the measurement mode ends. This allows the user to confirm the desired three-dimensional measurement result, as in the processing procedure in the endoscope apparatus 1 according to the first to fifth embodiments.

As described above, in the endoscope apparatus 1 according to the sixth embodiment, the three-dimensional point group generated by the three-dimensional shape restoration process for the measurement mode image acquired in the measurement mode from the observation mode image acquired in advance. Color 3D point cloud data (3D image) is generated by pasting as texture onto the data. Then, in the endoscope apparatus 1 of the sixth embodiment, the image of the generated three-dimensional point cloud data is displayed as the image of the test object for setting the measurement points when performing three-dimensional measurement. That is, also in the endoscope device 1 of the sixth embodiment, an image generated from the same image as the image for performing three-dimensional measurement, as in the endoscope devices 1 of the first to fifth embodiments. Is displayed as an image of a test object for setting a measurement point. Moreover, also in the endoscope apparatus 1 of the sixth embodiment, the image of the three-dimensional point cloud data to be displayed has the same color level as that of the observation mode image, as in the endoscope apparatus 1 of the fifth embodiment. Is an image of. Then, also in the endoscope apparatus 1 of the sixth embodiment, similar to the endoscope apparatus 1 of the fifth embodiment, the measurement point by the user is rotated by rotating the three-dimensional image of the displayed test object. It is possible to set the measurement point at a desired position while confirming the shape of the test object by the function of assisting the setting. As a result, even in the endoscope device 1 of the sixth embodiment, the measurement result obtained by accurately performing the three-dimensional measurement is obtained as in the endoscope device 1 of the first to fifth embodiments. Obtainable.

Note that, also in the endoscope apparatus 1 of the sixth embodiment, as described above, the observation mode image acquired in advance in the observation mode is pasted and displayed on the three-dimensional point cloud data. That is, also in the endoscope apparatus 1 of the sixth embodiment, similar to the endoscope apparatus 1 of the fifth embodiment, color reproduction processing based on the measurement mode image acquired in the measurement mode is not performed. Therefore, even in the endoscope apparatus 1 according to the sixth embodiment, it is possible to irradiate the test object with only the measurement light without irradiating the test object with the observation light in the measurement mode. As a result, also in the endoscope apparatus 1 of the sixth embodiment, the three-dimensional shape restoration process is performed on the measurement mode image acquired in the measurement mode, as in the endoscope apparatus 1 of the fifth embodiment. At this time, the change in the gradation of the measurement light can be determined with higher accuracy, and the calculation of the spatial coordinates (height) when generating the three-dimensional point cloud data can be performed with higher accuracy. Then, the processing procedure can be simplified without performing the processing of step S02 in the processing procedure of the endoscope apparatus 1 of the sixth embodiment shown in FIG.

In addition, in the endoscope apparatus 1 of the sixth embodiment, the observation mode image is attached to the three-dimensional point cloud data and used as an image of the object to be measured for setting the measurement point when performing three-dimensional measurement. The case of displaying on the monitor 87 has been described. However, in the endoscopic device 1 of the sixth embodiment, as in the endoscopic device 1 of the third embodiment shown in FIG. 6, each of the surfaces that hit the surface of the depth map generated by the three-dimensional shape restoration process. It is also possible to attach an observation mode image to the pixel as a texture and display it on the monitor 87. In the processing procedure in this case, the main control unit 81 generates the depth map in the three-dimensional shape restoration processing, as in the endoscope apparatus 1 of the third embodiment shown in FIG. 6, and in the processing of step S344. Only the process of pasting the observation mode image to the depth map is performed, and the other processing procedure can be considered similar to the processing procedure of the endoscope apparatus 1 of the sixth embodiment described above. Therefore, in the endoscope apparatus 1 of the sixth embodiment, when the depth map to which the observation mode image is attached is displayed as the image of the object to be measured for setting the measurement point when performing the three-dimensional measurement. Detailed description regarding the processing procedure is omitted.

As described above, according to the embodiment for carrying out the present invention, three-dimensional measurement is performed based on the measurement mode image acquired in the measurement mode. At this time, in the embodiment for carrying out the present invention, an image of a test object for setting measurement points for performing three-dimensional measurement is generated based on a measurement mode image for performing three-dimensional measurement. indicate. Thereby, in the embodiment for carrying out the present invention, it is possible to set the measurement point at a position corresponding to the position of each pixel included in the image for which the three-dimensional measurement is performed, on a one-to-one basis. That is, in the embodiment for carrying out the present invention, the images (three-dimensional observation images in the embodiment) taken at different timings are used for images for performing three-dimensional measurement as in the conventional endoscope apparatus. Instead of setting the measurement points, the measurement points can be set on an image generated based on the same image as the image on which the three-dimensional measurement is performed. As a result, in the embodiment for carrying out the present invention, it is possible to obtain a measurement result obtained by accurately performing three-dimensional measurement.

Further, in the embodiment for carrying out the present invention, an image generated based on the same image as the image on which three-dimensional measurement is performed, that is, an image on which three-dimensional measurement is performed is a measurement point when performing three-dimensional measurement. Since it is displayed as an image of the test object for setting, the image itself for performing three-dimensional measurement can be confirmed before setting the measurement point. Therefore, in the embodiment for carrying out the present invention, the angle of view of the measurement mode image changes from the angle of view of the observation mode image due to the movement of the tip when the operation mode is switched from the observation mode to the measurement mode. Even if the motion blur has occurred, there is no need to perform three-dimensional measurement at different measurement point positions due to motion blur, and the correct measurement point position is obtained. The measurement result can be obtained. Further, in the embodiment for carrying out the present invention, even if halation or the like occurs in the measurement mode image, it is necessary to confirm that halation or the like occurs in the measurement mode image before setting the measurement point. You can Accordingly, in the embodiment for carrying out the present invention, the measurement point is set at a position avoiding the position where re-imaging of the measurement mode image or the halation occurs in accordance with the state of the halation that occurs. It is possible to obtain the correct measurement result at the position of the desired measurement point without erroneously performing three-dimensional measurement due to halation or the like.

In the present embodiment, the pattern projection light source 71 included in the endoscope system (endoscope device 1) is a laser light source in the red wavelength band, that is, the pattern light (measurement light) is red. I explained. However, the type of light source and the color of the pattern light (wavelength band) are not limited to those shown in the embodiment for carrying out the present invention, and as described above, the color (wavelength band) of the predetermined color It may be an LED or infrared light.

Further, in the present embodiment, the case has been described in which the pattern light is phase-shifted and a plurality of measurement mode images are acquired to generate three-dimensional point cloud data for performing three-dimensional measurement. However, the method of generating data for performing three-dimensional measurement is not limited to the method of phase shifting the pattern light shown in the embodiment for carrying out the present invention, and the light of some structured pattern is used. Similarly, the idea of the present invention can be applied to any method of generating data for performing three-dimensional measurement by irradiating a subject with.

Further, in the present embodiment, the case where the structured pattern in the patterned light is a vertical striped pattern in which the light intensity changes in the shape of a sine wave in the horizontal direction has been described. However, the structured pattern in the patterned light is not limited to the structured pattern shown in the mode for carrying out the invention. For example, when the structured light pattern is a horizontal stripe pattern in which the light intensity changes in a vertical sinusoidal pattern, or in a lattice pattern in which the light intensity changes in a horizontal and vertical sinusoidal pattern. Even in the case of the pattern, the concept of the present invention can be similarly applied.

Further, in the present embodiment, the case where the main control unit 81 executes the processing according to the processing procedure shown in FIGS. 4 to 9 has been described. However, the constituent elements that execute each processing procedure are not limited to the constituent elements shown in the embodiment for carrying out the present invention. For example, even if the video processing unit 86 or a component (not shown) dedicated for performing three-dimensional measurement is configured to execute each processing procedure under the control of the main control unit 81. Good. In this case, the storage unit that temporarily stores the data used to execute each processing is not the RAM 83 shown in the embodiment for carrying out the present invention, but the video processing unit 86 or a component (not shown). Alternatively, it may be another connected storage unit.

Moreover, in this embodiment, the case where the endoscope system (endoscope device) of the present invention is an industrial endoscope device has been described. However, the idea of the present invention is not limited to application to an industrial endoscope device, and can also be applied to, for example, a medical endoscope device.

Further, in the present embodiment, the configuration in which the operation unit 82 and the monitor 87 are provided in the main body 2 has been described. However, the operation unit 82 and the monitor 87 are not limited to the configuration provided inside the main body unit 2, and may be configured, for example, arranged outside the main body unit 2 and connected to the main body unit 2.

Although the embodiment of the present invention has been described above with reference to the drawings, the specific configuration is not limited to this embodiment and includes various modifications without departing from the spirit of the present invention. Be done.

1. Endoscope device (endoscope system)
2... Body part 3... Insertion part 4... Tip part 40... End face 45... Opening part 46... Illumination window (illumination part)
47... Projection window (pattern projection unit)
5... Imaging unit 51... Imaging control unit 52... Imaging device 53... Objective optical system 54... Cover member 55... Signal line 6... Illumination unit 61... Illumination light source (Illumination part)
62... Coupling lens (illumination part)
63... Fiber bundle (illumination part)
64... Illumination optical system (illumination section)
7: pattern projection unit 71: pattern projection light source (pattern projection unit)
72... Coupling lens (pattern projection unit)
73... Optical fiber (pattern projection unit)
74: Stripe generation unit (pattern projection unit)
75... Projection optical system (pattern projection unit)
76... Pattern control unit (pattern projection unit)
77... Signal line (pattern projection unit)
80... Light source control unit (irradiation light control unit)
81...Main control unit (three-dimensional measurement unit)
82: operation unit 83: RAM (three-dimensional measurement unit)
84... Auxiliary storage unit (three-dimensional measurement unit)
85...ROM (three-dimensional measurement unit)
86... Image processing section 87... Monitor (display section)

Claims (25)

An illumination unit that irradiates the test object with observation light for observing the test object,
A pattern projection unit that irradiates the test object with measurement light in a predetermined wavelength band of a structured pattern for measuring the test object in three dimensions.
An irradiation light control unit that controls irradiation of the observation light by the illumination unit to the test object, and irradiation of the measurement light by the pattern projection unit to the test object ,
Based on the measurement image before Symbol test object, the three-dimensional measurement unit that performs three-dimensional measurement with respect to the test object corresponding to the specified measurement point,
A display unit that displays the setting image and the measurement points,
Bei to give a,
The three-dimensional measurement unit, the observation light and the obtained by irradiating the measurement light at the same time on the basis of the measurement image of the object to generate the set titration, image, set the measurement to the setting image The three-dimensional measurement is performed on the object based on the information of the measurement image corresponding to the position of a point,
An endoscope system characterized in that. The three-dimensional measurement unit,
When performing the three-dimensional measurement,
Three-dimensional pixel information used for the three-dimensional measurement is generated based on a first wavelength band measurement image which is an image of the test object due to reflection of light in the same wavelength band as the measurement light included in the measurement image. And a three-dimensional shape restoration process for generating a three-dimensional measurement image including the generated three-dimensional pixel information,
A three-dimensional measurement process for performing the three-dimensional measurement based on the three-dimensional pixel information included in the three-dimensional measurement image and corresponding to the position of the measurement point set in the setting image,
To run the
The endoscope system according to claim 1, wherein: The irradiation light control unit,
When performing the three-dimensional measurement, the observation light and the measurement light are simultaneously irradiated to the test object,
The three-dimensional measurement unit,
Before performing the three-dimensional shape restoration process,
The measurement image is a first wavelength band measurement image and a second wavelength band measurement image which is an image of the object to be measured by reflection of light in a wavelength band different from the measurement light included in the measurement image. Perform image separation processing to separate,
In parallel with the three-dimensional shape restoration processing,
A structured pattern removal process for generating a measurement light removal image in which the components of the measurement light included in the first wavelength band measurement image are removed,
A synthetic image generation process for generating a synthetic image by synthesizing the measurement light removal image and the second wavelength band measurement image,
Are executed sequentially,
The endoscope system according to claim 2, wherein: The irradiation light control unit,
The light amount of the observation light to be irradiated to the test object at the same time as the measurement light is adjusted according to the light amount of the measurement light,
The endoscope system according to claim 3, wherein: The three-dimensional measurement unit,
The composite image as the setting image,
The endoscope system according to claim 3 or 4, wherein. The three-dimensional measurement unit,
An image obtained by superimposing the composite image on the three-dimensional measurement image is used as the setting image,
The endoscope system according to claim 3 or 4, wherein. The three-dimensional measurement unit,
In parallel with the three-dimensional shape restoration processing,
A structured pattern removal process for generating a measurement light removal image in which the component of the measurement light included in the first wavelength band measurement image is removed,
Run
The measurement light removal image as the setting image,
The endoscope system according to claim 2, wherein: The three-dimensional measurement unit,
The three-dimensional measurement image is used as the setting image,
The endoscope system according to claim 2, wherein: The three-dimensional measurement unit,
An image obtained by superimposing an observation image of the test object according to an image obtained by previously irradiating the observation light on the three-dimensional measurement image is used as the setting image.
The endoscope system according to claim 2, wherein: The measurement light is
The structured pattern is a striped pattern in which the light intensity changes sinusoidally.
The endoscope system according to any one of claims 1 to 9, wherein: The pattern projection unit,
The phase of the structured pattern in the measurement light is phase-shifted by a predetermined shift amount, and the test object is irradiated a plurality of times,
The three-dimensional measurement unit,
The three-dimensional measurement is performed on the basis of a plurality of the measurement images corresponding to the respective images obtained by the irradiation of the respective measurement lights with the phase shifted,
The endoscope system according to any one of claims 1 to 10, wherein: The measurement light is
Laser light,
The endoscope system according to any one of claims 1 to 11, wherein: The measurement light is
The wavelength band is a wavelength band of visible light,
The endoscope system according to any one of claims 1 to 12, wherein: The measurement light is
The wavelength band is a red wavelength band,
The endoscope system according to claim 13, wherein: The observation light is a wavelength band of visible light,
The three-dimensional measurement unit,
In the image separation processing, the measurement image, the measurement image by reflection of light in the red wavelength band, the measurement image by reflection of light in the green wavelength band, the measurement by reflection of light in the blue wavelength band An image, and the measurement image obtained by reflection of light in the same wavelength band as the measurement light is used as the first wavelength band measurement image, and each of the measurement images obtained by reflection of light in a wavelength band different from the measurement light is obtained. The second wavelength band measurement image,
In any one of Claims 3 to 6, or in any one of Claims 10 to 14, which refers to any one of Claims 3 to 6, The described endoscope system. The three-dimensional measurement unit,
In the structured pattern removal process, the values of respective pixels at the same positions included in the plurality of first wavelength band measurement images corresponding to the plurality of measurement images are arithmetically averaged to obtain one measurement. Generate a light removal image,
Claim 11 that cites any one of claims 3 to 7, or claim 11 that cites any one of claims 3 to 7. The endoscope system according to any one of claims 12 to 15. The three-dimensional measurement unit,
In the three-dimensional shape restoration process, a depth image representing the depth of the test object is generated based on the generated three-dimensional pixel information, and the generated depth image is used as the three-dimensional measurement image.
Claims 6, 8 or 9, wherein any one of claims 6, 8 or 9 is cited. 16. The endoscope system according to any one of 16 items. A method of operating an endoscopic device, comprising:
An illumination unit you emits observation light for observing the specimen in the specimen,
Wherein a pattern projection unit a predetermined wavelength band of the measurement light you exit the specimen of the structured pattern to be measured with the specimen three-dimensional,
An exit to the test object of the observation light by the illumination unit, and the emitted light controller that controls the emission into the test object of the measurement light by the pattern projection unit,
A three-dimensional measurement unit that performs three-dimensional measurement on the object to be measured according to a designated measurement point, based on at least a measurement image of the object to be measured according to an image obtained by emitting the measurement light.
In the endoscopic device provided with ,
A step in which the three-dimensional measurement unit generates a setting image for setting the measurement point based on the measurement image and causes the display unit to display the setting image ;
A step in which the three-dimensional measurement unit performs the three-dimensional measurement on the object based on the information of the measurement image corresponding to the position of the measurement point set in the setting image ,
When performing the three-dimensional measurement, pre SL emitted light control unit, a step of emitting on the said observation light and the measurement light at the same time the test object,
When performing the three-dimensional measurement, the three-dimensional measurement unit is a first wavelength band measurement image which is an image of the object to be measured by reflection of light in the same wavelength band as the measurement light included in the measurement image. A three-dimensional shape restoration process for generating three-dimensional pixel information used for the three-dimensional measurement and a three-dimensional measurement image including the generated three-dimensional pixel information, and the three-dimensional measurement image. A three-dimensional measurement process for performing the three-dimensional measurement based on the three-dimensional pixel information corresponding to the position of the measurement point set in the setting image included in
The three-dimensional measurement unit, before performing the three-dimensional shape restoration processing, the measurement image of the first wavelength band measurement image and the light of the wavelength band different from the measurement light included in the measurement image. a step of executing an image separation processing for separating the second wavelength band measurement image is an image of the test object by reflection,
The three-dimensional measurement unit, in parallel with the three-dimensional shape restoration process, a structured pattern removal process for generating a measurement light removal image in which the component of the measurement light included in the first wavelength band measurement image is removed. A step of sequentially executing a synthetic image generation process for generating a synthetic image by synthesizing the measurement light removed image and the second wavelength band measurement image .
Method of operating the endoscope apparatus you wherein a. An illumination unit that irradiates the test object with observation light for observing the test object,
A pattern projection unit that irradiates the test object with measurement light in a predetermined wavelength band of a structured pattern for measuring the test object in three dimensions.
An irradiation light control unit that controls irradiation of the observation light by the illumination unit to the test object, and irradiation of the measurement light by the pattern projection unit to the test object,
A three-dimensional measurement unit that performs three-dimensional measurement on the object to be measured according to a designated measurement point, based on at least a measurement image of the object to be measured that is obtained by irradiating the measurement light.
Equipped with
The three-dimensional measurement unit generates a setting image for setting the measurement point based on the measurement image and causes the display unit to display the setting image, which corresponds to the position of the measurement point set in the setting image. The three-dimensional measurement is performed on the object based on the information of the measurement image,
When performing the three-dimensional measurement, the irradiation light control unit causes the observation light and the measurement light to be simultaneously irradiated to the test object,
When performing the three-dimensional measurement, the three-dimensional measurement unit is a first wavelength band measurement image which is an image of the object to be measured by reflection of light in the same wavelength band as the measurement light included in the measurement image. A three-dimensional shape restoration process for generating three-dimensional pixel information used for the three-dimensional measurement based on the three-dimensional measurement information, and a three-dimensional measurement image including the generated three-dimensional pixel information; And a three-dimensional measurement process of performing the three-dimensional measurement based on the three-dimensional pixel information corresponding to the position of the measurement point set in the setting image,
The three-dimensional measurement unit, before performing the three-dimensional shape restoration processing, the measurement image of the first wavelength band measurement image and the light of the wavelength band different from the measurement light included in the measurement image. Image separation processing is performed to separate the second wavelength band measurement image, which is an image of the test object due to reflection,
The three-dimensional measurement unit, in parallel with the three-dimensional shape restoration process, a structured pattern removal process for generating a measurement light removal image in which the component of the measurement light included in the first wavelength band measurement image is removed. A synthetic image generation process for generating a synthetic image by synthesizing the measurement light removal image and the second wavelength band measurement image is sequentially executed.
An endoscope system characterized in that. The irradiation light control unit,
The light amount of the observation light to be irradiated to the test object at the same time as the measurement light is adjusted according to the light amount of the measurement light,
The endoscope system according to claim 19, wherein: The three-dimensional measurement unit,
The composite image as the setting image,
21. The endoscope system according to claim 19 or 20, wherein: The three-dimensional measurement unit,
An image obtained by superimposing the composite image on the three-dimensional measurement image is used as the setting image,
21. The endoscope system according to claim 19 or 20, wherein: The observation light is a wavelength band of visible light,
The three-dimensional measurement unit,
In the image separation processing, the measurement image, the measurement image by reflection of light in the red wavelength band, the measurement image by reflection of light in the green wavelength band, the measurement by reflection of light in the blue wavelength band An image, and the measurement image obtained by reflection of light in the same wavelength band as the measurement light is used as the first wavelength band measurement image, and each of the measurement images obtained by reflection of light in a wavelength band different from the measurement light is obtained. The second wavelength band measurement image,
The endoscope system according to any one of claims 19 to 22, wherein: The three-dimensional measurement unit,
In the structured pattern removal process, the values of respective pixels at the same positions included in the plurality of first wavelength band measurement images corresponding to the plurality of measurement images are arithmetically averaged to obtain one measurement. Generate a light removal image,
The endoscope system according to any one of claims 19 to 22, wherein: The three-dimensional measurement unit,
In the three-dimensional shape restoration process, a depth image representing the depth of the test object is generated based on the generated three-dimensional pixel information, and the generated depth image is used as the three-dimensional measurement image.
23. The endoscope system according to claim 22, wherein: