WO2022049806A1 - Calibration device and method - Google Patents

Abstract

Provided are a calibration device and method whereby calibration relating to a virtual scale is performed in consideration of the effect of distortion, etc., caused by an imaging optical system.　In the present invention, a chart (101) is provided with an index figure plane (104) formed from three index figures (104a, 104b, 104c). The three index figures (104a, 104b, 104c) are separated by two or more colors on the basis of a specific color evaluation standard or on the basis of the transmittance of a color filter of an imaging element (32). A scale parameter for display of a virtual scale by an extended display (18) is acquired from a calibration image obtained by imaging of the chart (101) by an endoscope (12).

Description

Calibration device and method

The present invention relates to a calibration device and a method for calibrating a virtual scale for measuring the size of a subject.

In the endoscope device, the distance to the subject or the size of the subject is acquired. For example, in Patent Document 1, an illumination light and a spot light for measurement are illuminated on a subject, and a virtual scale for measuring the size of the subject is formed on the captured image in correspondence with the position of the spot light illuminated on the subject. It is displayed in. Further, Patent Document 1 describes to calibrate the relationship between the position of the spot light and the size of the subject. In Patent Document 1, the spot light for measurement is illuminated on a graph paper-shaped chart, and the calibration image obtained by imaging each distance from the chart shows the irradiation position of the measurement light and the size of a virtual scale of a specific size. It defines the relationship with.

International Publication No. 2018/051680

It is known that the peripheral portion of the captured image obtained by the endoscope device is distorted by the imaging optical system. On the other hand, in Patent Document 1, the virtual scale is displayed without considering the influence of distortion and the like due to the imaging optical system. Therefore, when the subject is located in the peripheral portion of the captured image, it may not be possible to accurately measure the size of the subject by the virtual scale.

An object of the present invention is to provide a calibration device and a method capable of calibrating a virtual scale for measuring the size of a subject in consideration of the influence of distortion and the like due to an imaging optical system.

The present invention is a calibration device that calibrates a virtual scale for displaying on a display and measuring the size of a subject, and is a chart provided with an index graphic plane composed of a plurality of index figures. The index figure of is a chart in which two or more colors are separated based on a specific color evaluation standard or a transmission based on the transmission rate of the color filter of the image pickup element provided in the endoscope, and the internal vision. The distance changing mechanism that changes the distance between the charts between the tip of the mirror and the chart, the reference position of the index graphic plane, and the irradiation position of the measurement light emitted from the tip of the endoscope toward the chart. It is equipped with a position adjustment mechanism that moves the tip of the endoscope or the chart to adjust the positional relationship, and an image control processor. The image control processor is a reference position on an index graphic plane and an irradiation position of measurement light. A calibration image obtained by imaging a chart in a state of matching with an endoscope is acquired, and a scale parameter for displaying a virtual scale on a display is acquired from the calibration image.

The specific color evaluation criterion is the hue circle, and the color of the index figure shall include white, black, or two or more of a plurality of colors corresponding to a plurality of hues separated by 100 ° or more in the hue circle. Is preferable. It is preferable that the background portion of the chart other than the index graphic plane is black. The chart is preferably a light transmissive type.

During the distance setting period in which the distance between charts is set by the distance changing mechanism, only the irradiation position of the measured light is displayed and / or is displayed in the calibration image by turning off the backlight provided on the light transmission type chart. It is preferable to store the irradiation position of the measurement light and hide the index graphic plane. Calibration is performed by moving the tip of the endoscope or the chart by the position adjustment mechanism and turning on the backlight provided on the light transmissive chart during the calibration image acquisition period for acquiring the calibration image. It is preferable to display the irradiation position of the measurement light and to display the index graphic plane in the measurement image. In the calibration image acquisition period, it is preferable to display the alignment figure corresponding to the index figure plane on the calibration image in order to make the tip of the endoscope face the chart.

The distance between the charts is set to the set distance by sandwiching it between the endoscope holding part that holds the tip of the endoscope, the chart holding part on which the chart is placed, and the endoscope holding part and the chart holding part. It is preferable to provide a matching distance setting jig. The display preferably displays the work of operations related to the calibration mode, including at least one of a distance changing mechanism, a position adjusting mechanism, a calibration image acquisition, or a scaling parameter acquisition. The display preferably performs an operation display related to the calibration mode including at least one of a distance changing mechanism, a position adjusting mechanism, a calibration image acquisition, or a scale parameter acquisition.

The present invention is a calibration method for calibrating a virtual scale for displaying on a display and measuring the size of a subject, and is a chart provided with an index graphic plane composed of a plurality of index figures by a distance changing mechanism. A distance change step that changes the distance between the chart and the tip of the endoscope, and a position adjustment mechanism that measures the reference position of the index graphic plane and the measurement emitted from the tip of the endoscope toward the chart. In order to adjust the positional relationship with the light irradiation position, the movement step to move the tip of the endoscope or the chart, and each time the distance between the charts is changed, the reference position on the index graphic plane and the irradiation position of the measured light Acquire the calibration image acquisition step to acquire the calibration image obtained by imaging the chart in the state of matching with the endoscope, and acquire the scale parameters for displaying the virtual scale on the display from the calibration image. It has a parameter acquisition step, and in the chart, the plurality of index figures are separated according to two or more colors based on a specific color evaluation standard, or the transmission rate of the color filter of the image pickup element provided in the endoscope. The separation is based on.

In the distance changing step, by turning off the backlight provided on the light-transmitting chart, only the irradiation position of the measured light is displayed in the calibration image, and the index graphic plane is not displayed. It is preferable to display it.

In the calibration image acquisition step, by turning on the backlight provided in the light transmission type chart, the irradiation position of the measurement light can be displayed and the index graphic plane can be displayed in the calibration image. preferable. The index figure is a circle, and in the parameter acquisition step, the circumscribed rectangle extraction process for extracting the circumscribed rectangle inscribed in the index figure image in the calibration image and the fitting parameter of the ellipse inscribed in the circumscribed rectangle are calculated as scale parameters. It is preferable to perform the inscribed ellipse parameter calculation process.

The scale parameters include a scale parameter having a first distance between charts and a scale parameter having a second distance between charts. In the parameter acquisition step, the distance between charts is changed by a distance changing mechanism. Each time, the parameter for the scale of the first distance is acquired, and after the parameter acquisition step, the interpolation process for calculating the parameter for the scale of the second distance is performed by performing the interpolation process based on the parameter for the scale of the first distance. It is preferable to perform the steps.

According to the present invention, it is possible to calibrate the virtual scale for measuring the size of the subject in consideration of the influence of distortion and the like due to the imaging optical system.

It is a schematic diagram of an endoscope system. It is a top view which shows the tip part of an endoscope. It is a block diagram which shows the function of an endoscope apparatus. It is a block diagram which shows the measurement light emission part. It is explanatory drawing which shows the spot SP formed on the subject by the measurement light. It is explanatory drawing which shows the relationship | It is a block diagram which shows the function of a signal processing part. It is an image diagram which shows the spot and the 1st virtual scale when the observation distance is a near-end Px. It is an image diagram which shows the spot and the 1st virtual scale when the observation distance is Py near the center. It is an image diagram which shows the spot and the 1st virtual scale when the observation distance is a far end Pz. It is explanatory drawing which shows the 1st virtual scale of the cross type, the graduated cross type, the distorted cross type, the circle and the cross type, and the point cloud type for measurement. FIG. 3 is an image diagram showing three concentric markers having the same color. It is an image diagram which shows three concentric markers of different colors. It is an image diagram which shows the distortion concentric marker. It is an image diagram which shows the intersection line and the scale. It is a schematic diagram of a calibration device. It is a plan view of a chart. It is explanatory drawing which shows the color wheel. It is a graph which shows the transmittance distribution of the color filter of an image sensor. It is a schematic diagram of the calibration mechanism provided with the distance change mechanism and the position adjustment mechanism. It is an image diagram of the calibration image when the backlight is turned off. It is an image diagram of the calibration image when the backlight is turned on. It is explanatory drawing which shows that the tip part of an endoscope can rotate in an XZ plane or a YZ plane. It is a block diagram which shows the function of the signal processing part corresponding to a calibration mode. It is explanatory drawing which shows the binarization processing or the circumscribed rectangle extraction processing. It is explanatory drawing which shows the inscribed ellipse parameter calculation processing. It is an image diagram which shows the scale confirmation screen. It is explanatory drawing which shows the interpolation processing. It is a flowchart which shows a series flow of a calibration mode. It is an image diagram which shows the setting completion operation icon. It is explanatory drawing which shows the rectangle for confirmation of roundness. It is an image diagram of a chart image in a moving step. It is an image diagram which shows the position adjustment completion icon. It is an image diagram which shows the scale confirmation icon. It is an image diagram which displays the work content. It is an image diagram which shows the image for alignment.

As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a user interface 16, an extended processor device 17, and an extended display 18. Have. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The endoscope 12 has an insertion portion 12a to be inserted into the body to be observed, an operation portion 12b provided at the base end portion of the insertion portion 12a, and a curved portion 12c and a tip provided on the tip end side of the insertion portion 12a. It has a portion 12d. The curved portion 12c bends by operating the operating portion 12b. The tip portion 12d is directed in a desired direction by the bending motion of the bending portion 12c.

Further, the operation unit 12b includes a mode changeover switch 12f used for mode switching operation, a still image acquisition instruction switch 12g used for instructing acquisition of a still image to be observed, and a zoom operation unit used for operating the zoom lens 21b. 12h is provided.

The processor device 14 is electrically connected to the display 15 and the user interface 16. The display 15 outputs and displays an image or information of an observation target processed by the processor device 14. The user interface 16 has a keyboard, a mouse, a touch pad, a microphone, and the like, and has a function of accepting input operations such as function settings. The expansion processor device 17 is electrically connected to the processor device 14. The expansion display 18 outputs and displays an image, information, or the like processed by the expansion processor device 17.

The endoscope 12 has a normal observation mode, a special observation mode, a length measurement mode, and a calibration mode, and can be switched by the mode changeover switch 12f. The normal observation mode is a mode in which the observation target is illuminated by the illumination light. The special observation mode is a mode in which the observation target is illuminated with special light different from the illumination light. In the length measurement mode, the illumination light or the measurement light is illuminated on the observation target, and a virtual scale used for measuring the size of the observation target or the like is displayed on the captured image obtained by imaging the observation target. The captured image on which the virtual scale is not superimposed is displayed on the display 15, while the captured image on which the virtual scale is superimposed is displayed on the extended display 18. In the calibration mode, the virtual scale is calibrated in order to adjust the virtual scale, which may vary in size or the like depending on the image pickup optical system 21 of the endoscope 12, for each endoscope 12.

The illumination light is light used to give brightness to the entire observation target and observe the entire observation target. Special light is light used to emphasize a specific area of an observation target. The measurement light is the light used for displaying the virtual scale. Further, in the present embodiment, the virtual scale displayed on the image will be described, but the actual scale may be provided in the actual lumen so that the actual scale can be confirmed through the image. In this case, it is conceivable that the actual scale is inserted through the forceps channel of the endoscope 12 and the actual scale is projected from the tip portion 12d.

When the user operates the still image acquisition instruction switch 12g, the screen of the display 15 freezes and also emits an alert sound (for example, "pee") to the effect that the still image is acquired. Then, the still image of the captured image obtained before and after the operation timing of the still image acquisition instruction switch 12g is stored in the still image storage unit 42 (see FIG. 3) in the processor device 14. The still image storage unit 42 is a storage unit such as a hard disk or a USB (Universal Serial Bus) memory. When the processor device 14 can be connected to the network, the still image of the captured image is stored in the still image storage server (not shown) connected to the network in place of or in addition to the still image storage unit 42. You may.

It should be noted that the still image acquisition instruction may be given by using an operation device other than the still image acquisition instruction switch 12g. For example, a foot pedal may be connected to the processor device 14, and a still image acquisition instruction may be given when the user operates the foot pedal (not shown) with his / her foot. You may use the foot pedal for mode switching. Further, a gesture recognition unit (not shown) that recognizes a user's gesture is connected to the processor device 14, and when the gesture recognition unit recognizes a specific gesture performed by the user, a still image acquisition instruction is given. You may do it. The mode switching may also be performed using the gesture recognition unit.

Further, a line-of-sight input unit (not shown) provided near the display 15 is connected to the processor device 14, and the line-of-sight input unit recognizes that the user's line of sight is within a predetermined area of the display 15 for a certain period of time or longer. If this is the case, a still image acquisition instruction may be given. Further, a voice recognition unit (not shown) may be connected to the processor device 14, and when the voice recognition unit recognizes a specific voice emitted by the user, a still image acquisition instruction may be given. The mode switching may also be performed using the voice recognition unit. Further, an operation panel (not shown) such as a touch panel may be connected to the processor device 14, and a still image acquisition instruction may be given when the user performs a specific operation on the operation panel. The mode switching may also be performed using the operation panel.

As shown in FIG. 2, the tip of the endoscope 12 has a substantially circular shape, and the image pickup optical system 21 located closest to the subject side among the optical members constituting the image pickup optical system of the endoscope 12 and the subject. An illumination optical system 22 for irradiating the subject with illumination light, a measurement light emission unit 23 for illuminating the subject with the measurement light, an opening 24 for projecting the treatment tool toward the subject, and air supply / water supply. Is provided with an air supply / water supply nozzle 25 for performing the above.

The optical axis Ax of the imaging optical system 21 extends in a direction perpendicular to the paper surface. The vertical first direction D1 is orthogonal to the optical axis Ax, and the horizontal second direction D2 is orthogonal to the optical axis Ax and the first direction D1. The imaging optical system 21 and the measurement light emitting unit 23 are arranged along the first direction D1.

As shown in FIG. 3, the light source device 13 includes a light source unit 30 and a light source processor 31. The light source unit 30 generates illumination light for illuminating the subject. The illumination light emitted from the light source unit 30 is incident on the light guide LG and is applied to the subject through the illumination lens 22a provided in the illumination optical system 22. The light source unit 30 includes, as a light source of illumination light, a white light source that emits white light, or a plurality of light sources including a white light source and a light source that emits light of other colors (for example, a blue light source that emits blue light). Is used. The light source processor 31 is connected to the system control unit 41 of the processor device 14. The illumination light may be a white mixed color light in which blue light, green light, and red light are combined. In this case, it is preferable to design the illumination lens 22a so that the irradiation range of the green light is larger than the irradiation range of the red light.

The light source processor 31 controls the light source unit 30 based on an instruction from the system control unit 41. The system control unit 41 gives an instruction regarding the light source control to the light source processor 31, and also controls the light source 23a (see FIG. 4) of the measurement light emission unit 23. In the case of the normal observation mode or the special observation mode, the system control unit 41 controls to turn on the illumination light or the special light and turn off the measurement light. In the case of the length measurement mode, the system control unit 41 turns on the illumination light and controls to turn on the measurement light. In the calibration mode, the system control unit 41 controls to turn off the illumination light and turn on the measurement light.

The tip portion 12d of the endoscope 12 is provided with an illumination optical system 22, an image pickup optical system 21, and a measurement light emission unit 23. The illumination optical system 22 has an illumination lens 22aa, and the light from the light guide LG is irradiated to the observation target through the illumination lens 22a. The image pickup optical system 21 includes an image pickup optical system 21a, a zoom lens 21b, and an image pickup element 32. The reflected light from the observation target is incident on the image pickup device 32 via the image pickup optical system 21a. As a result, a reflected image to be observed is formed on the image pickup device 32.

The image sensor 32 is a color image sensor, which captures a reflected image of a subject and outputs an image signal. The image pickup device 32 is preferably a CCD (Charge Coupled Device) image pickup sensor, a CMOS (Complementary Metal-Oxide Semiconductor) image pickup sensor, or the like. The image pickup device 32 used in the present invention is a color image pickup sensor for obtaining a red image, a green image, and a red image of three colors of R (red), G (green), and B (blue). The red image is an image output from a red pixel provided with a red color filter RF (see FIG. 19) in the image sensor 32. The green image is an image output from a green pixel provided with a green color filter GF (see FIG. 19) in the image sensor 32. The blue image is an image output from a blue pixel provided with a blue color filter BF (see FIG. 19) in the image sensor 32. The image pickup device 32 is controlled by the image pickup control unit 33.

The image signal output from the image sensor 32 is transmitted to the CDS / AGC circuit 34. The CDS / AGC circuit 34 performs correlated double sampling (CDS (Correlated Double Sampling)) and automatic gain control (AGC (Auto Gain Control)) on an image signal which is an analog signal. The image signal that has passed through the CDS / AGC circuit 34 is converted into a digital image signal by the A / D converter (A / D (Analog / Digital) converter) 35. The A / D converted digital image signal is input to the processor device 14 via the communication I / F (Interface) 36 of the endoscope 12 and the communication I / F (Interface) 37 of the light source device 13.

In the processor device 14, programs related to various processes or controls are incorporated in a program storage memory (not shown). The system control unit 41 configured by the image control processor operates a program embedded in the program storage memory to connect to the communication I / F (Interface) 37 of the light source device 13 and to receive a signal. The functions of the processing unit 39 and the display control unit 40 are realized.

The receiving unit 38 receives the image signal transmitted from the communication I / F 37 and transmits it to the signal processing unit 39. The signal processing unit 39 has a built-in memory for temporarily storing an image signal received from the receiving unit 38, and processes an image signal group which is a set of image signals stored in the memory to generate an captured image. The receiving unit 38 may directly send the control signal related to the light source processor 31 to the system control unit 41.

In the signal processing unit 39, when the normal observation mode is set, the blue image of the captured image is on the B channel of the display 15, the green image of the captured image is on the G channel of the display 15, and the red image of the captured image is on the G channel. By performing signal allocation processing assigned to each R channel of the display 15, a color captured image is displayed on the display 15. Also in the length measurement mode, the same signal allocation processing as in the normal observation mode is performed.

On the other hand, in the signal processing unit 39, when the special observation mode is set, the red image of the captured image is not used for the display of the display 15, and the blue image of the captured image is used for the B channel and the G channel of the display 15. By assigning the green image of the captured image to the R channel of the display 15, the captured image of pseudo color is displayed on the display 15. Further, when the signal processing unit 39 is set to the length measurement mode, the signal processing unit 39 transmits the captured image including the irradiation position of the measurement light to the data transmission / reception unit 43. The data transmission / reception unit 43 transmits data related to the captured image to the expansion processor device 17. The data transmission / reception unit 43 can receive data or the like from the expansion processor device 17. The received data can be processed by the signal processing unit 39 or the system control unit 41.

The display control unit 40 displays the captured image generated by the signal processing unit 39 on the display 15. The system control unit 41 performs various controls on the endoscope 12, the light source device 13, the processor device 14, and the extended processor device 17. The image pickup device 32 is controlled via the image pickup control unit 33 provided in the endoscope 12. The image pickup control unit 33 also controls the CDS / AGC34 and the A / D35 in accordance with the control of the image pickup element 32.

The expansion processor device 17 receives the data transmitted from the processor device 14 by the data transmission / reception unit 44. The signal processing unit 45 performs processing related to the length measurement mode based on the data received by the data transmission / reception unit 44. Specifically, a virtual scale display pattern is determined from the captured image including the irradiation position of the measurement light, and the virtual scale based on the determined display pattern is superimposed and displayed on the captured image. The display control unit 46 displays the captured image on which the virtual scale is superimposed and displayed on the extended display 18. The data transmission / reception unit 44 can transmit data or the like to the processor device 14.

As shown in FIG. 4, the measurement light emitting unit 23 includes a light source 23a, a diffractive optical element DOE23b (Diffractive Optical Element), a prism 23c, and a measurement light lens 23d. The light source 23a emits light of a color that can be detected by the pixels of the image pickup element 32 (specifically, visible light), and is a light emitting element such as a laser light source LD (Laser Diode) or an LED (Light Emitting Diode). , Includes a condenser lens that collects the light emitted from this light emitting element.

In the present embodiment, the wavelength of the light emitted by the light source 23a is, for example, a red laser light of 600 nm or more and 650 nm or less, but light in another wavelength band, for example, green light of 495 nm or more and 570 nm or less is used. May be good. The light source 23a is controlled by the system control unit 41, and emits light based on an instruction from the system control unit 41. The DOE23b converts the light emitted from the light source into the measurement light for obtaining the measurement information.

The prism 23c is an optical member for changing the traveling direction of the measured light after conversion by the DOE23b. The prism 23c changes the traveling direction of the measurement light so as to intersect the field of view of the imaging optical system 21. The details of the traveling direction of the measurement light will also be described later. The measurement light Lm emitted from the prism 23c passes through the measurement light lens 23d and irradiates the subject.

By irradiating the subject with the measured light, as shown in FIG. 5, a spot SP as a circular region is formed in the subject. The position of the spot SP (irradiation position of the measured light) is specified by the irradiation position detection unit 54 (see FIG. 7), and a virtual scale representing the actual size is set according to the position of the spot SP. The set virtual scale is displayed on the captured image. As will be described later, the virtual scale includes a plurality of types such as a first virtual scale and a second virtual scale, and the user determines which type of virtual scale is displayed on the captured image. Selection is possible by instructions. As the user's instruction, for example, the user interface 16 is used.

Instead of the measurement light lens 23d, it may be a measurement assist slit formed in the tip portion 12d of the endoscope. Further, it is preferable to apply an antireflection coating (AR (Anti-Reflection) coating) (antireflection portion) to the measurement light lens 23d. In this way, the antireflection coat is provided by the irradiation position detection unit 54, which will be described later, when the measurement light is reflected without passing through the measurement light lens 23d and the ratio of the measurement light irradiated to the subject decreases. This is because it becomes difficult to recognize the position of the spot SP formed on the subject by the measurement light.

The measurement light emitting unit 23 may be any as long as it can emit the measurement light toward the field of view of the image pickup optical system 21. For example, the light source 23a may be provided in the light source device 13, and the light emitted from the light source 23a may be guided to the DOE 23b by an optical fiber. Further, the measurement light Lm may be emitted in a direction crossing the field of view of the imaging optical system 21 by installing the light sources 23a and DOE23b at an angle with respect to the optical axis Ax without using the prism 23c.

As for the traveling direction of the measurement light, as shown in FIG. 6, the measurement light Lm is emitted in a state where the optical axis Lm of the measurement light Lm intersects the optical axis Ax of the imaging optical system 21. Assuming that observation is possible in the range Rx of the observation distance, in the near-end Px, near-center Py, and far-end Pz of the range Rx, the measurement light Lm in the imaging range (indicated by arrows Qx, Qy, Qz) at each point. It can be seen that the positions of the spots SP formed on the subject (points where the arrows Qx, Qy, and Qz intersect with the optical axis Ax) are different. The shooting angle of view of the imaging optical system 29b is represented in the region sandwiched between the two solid lines 48, and the measurement is performed in the central region (the region sandwiched by the two dotted lines 49 ") of the shooting angle of view where aberration is small. I am doing it.

As described above, by emitting the measurement light Lm in a state where the optical axis Lm of the measurement light intersects the optical axis Ax, the sensitivity of the movement of the spot position to the change in the observation distance is high, so that the size of the subject is large. Can be measured with high accuracy. Then, by taking an image of the subject illuminated by the measurement light with the image pickup device 32, an image pickup image including the spot SP can be obtained. In the captured image, the position of the spot SP differs depending on the relationship between the optical axis Ax of the imaging optical system 21 and the optical axis Lm of the measured light Lm and the observation distance, but if the observation distance is short, the same actual size ( For example, the number of pixels indicating 5 mm) increases, and the number of pixels decreases as the observation distance increases. The third direction D3 is a method orthogonal to the first direction D1 and the second direction.

As shown in FIG. 7, the signal processing unit 45 of the expansion processor device 17 detects the position of the spot SP in the captured image in order to recognize the position of the spot SP and set the virtual scale. And a second signal processing unit 52 that sets a virtual scale according to the position of the spot SP.

The first signal processing unit 50 includes an irradiation position detection unit 54 that detects the irradiation position of the spot SP from the captured image. It is preferable that the irradiation position detection unit 54 acquires the coordinates of the center of gravity of the spot SP as the irradiation position of the spot SP.

The second signal processing unit 52 sets the first virtual scale as a virtual scale for measuring the size of the subject based on the irradiation position of the spot SP, and sets the scale display position of the first virtual scale. .. The second signal processing unit 52 refers to the scale table 55 that stores the virtual scale image whose display pattern changes depending on the irradiation position of the spot SP and the irradiation position of the spot in association with the irradiation position of the spot, and irradiates the spot SP. Set the virtual scale corresponding to the position.

The virtual scale differs in size or shape, for example, depending on the irradiation position and scale display position of the spot SP. The display of the virtual scale image will be described later. Further, the stored contents of the scale table 55 are maintained even when the power of the expansion processor device 17 is turned off. The scale table 55 stores the scale parameter and the irradiation position in association with each other, but the distance to the subject corresponding to the irradiation position (distance between the tip portion 12d of the endoscope 12 and the subject) and the scale parameter. You may memorize it in association with.

It should be noted that the scale parameters are required for each irradiation position, and the data capacity becomes large. Therefore, considering the viewpoints such as the capacity of the memory that can be stored in the endoscope 12, the startup, and the processing time, the endoscope is used for endoscopy. It is preferable to hold it in the extended processor device 17 (or processor device 14) rather than holding it in a memory (not shown) in the mirror 12. Also, as will be described later, the virtual scale image is created from the scale parameters obtained by calibration, but if a virtual scale image is created from the scale parameters in the length measurement mode, loss time will occur and processing will occur. The real-time property of is impaired.

Therefore, after connecting the endoscope 12 to the endoscope connection portion, once creating a virtual scale image from the scale parameters and updating the scale table 55, the virtual scale is not created from the representative points. , The virtual scale image is displayed using the updated scale table 55. Further, in the second signal processing unit 52, in an emergency when it becomes difficult to superimpose and display the image, the irradiation position of the spot SP and the actual size of the subject correspond to the irradiation position of the spot SP instead of the virtual scale image to be superimposed and displayed on the length measurement image. A reference scale that determines the size of the scale is displayed on the reference scale image in relation to the number of pixels. The length measurement image is an image including an irradiation position of measurement light such as a spot SP and / or a virtual scale image, and is an image displayed on the extended display 18. The reference scale image is an image including the irradiation position of the measurement light and / or the reference scale image, and is an image displayed on the extended display 18.

Further, the second signal processing unit 52 provides a table update unit 56 for updating the scale table 55 when the endoscope 12 is connected to the endoscope connection unit (not shown) of the light source device 13. I have. The reason why the scale table 55 can be updated in this way is that the endoscope 12 has a different positional relationship between the optical axis Lm of the measurement light and the image pickup optical system 21 depending on the model and serial number, and accordingly. This is because the display pattern of the virtual scale image also changes. In the table update unit 56, a parameter table 57 that stores the scale parameters obtained in the calibration mode in association with the irradiation position is used. Details of the table update unit 56 and the parameter table 57 will be described later. The parameter table 57 may be stored in association with the distance to the subject corresponding to the irradiation position (distance between the tip portion 12d of the endoscope 12 and the subject) and the representative point data.

The display control unit 46 controls the display mode of the virtual scale to be different depending on the irradiation position and the scale display position of the spot SP when the length measurement image in which the virtual scale is superimposed on the captured image is displayed on the extended display 18. .. Specifically, the display control unit 46 displays the length measurement image on which the first virtual scale is superimposed centering on the spot SP on the extended display 18. As the first virtual scale, for example, a circular measurement marker is used. In this case, as shown in FIG. 8, when the observation distance is close to the near end Px (see FIG. 6), the actual size is 5 mm (captured image) in line with the center of the spot SP1 formed on the tumor tm1 of the subject. A virtual scale M1 indicating (horizontal and vertical directions) is displayed.

Since the marker display position of the virtual scale M1 is located in the peripheral portion of the captured image affected by the distortion caused by the imaging optical system 21, the virtual scale M1 has an elliptical shape according to the influence of the distortion and the like. .. Since the above marker M1 substantially coincides with the range of the tumor tm1, the tumor tm1 can be measured to be about 5 mm. It should be noted that the spot may not be displayed on the captured image, and only the first virtual scale may be displayed.

Further, as shown in FIG. 9, when the observation distance is close to Py near the center (see FIG. 6), the actual size is 5 mm (horizontal direction of the captured image) in accordance with the center of the spot SP2 formed on the tumor tm2 of the subject. And the virtual scale M2 indicating the vertical direction) is displayed. Since the scale display position of the virtual scale M2 is located in the center of the captured image that is not easily affected by distortion by the imaging optical system 21, the virtual scale M2 is circular without being affected by distortion or the like. ing.

Further, as shown in FIG. 10, when the observation distance is close to the far end Pz (see FIG. 6), the actual size is 5 mm (horizontal direction of the captured image) so as to be aligned with the center of the spot SP3 formed on the tumor tm3 of the subject. And the virtual scale M3 indicating the vertical direction) is displayed. Since the scale display position of the virtual scale M3 is located in the peripheral portion of the captured image affected by the distortion caused by the imaging optical system 21, the virtual scale M3 has an elliptical shape according to the influence of the distortion and the like. .. As shown in FIGS. 8 to 10 above, the size of the first virtual scale corresponding to the same actual size of 5 mm becomes smaller as the observation distance becomes longer. Further, the shape of the first virtual scale differs depending on the marker display position according to the influence of the distortion caused by the imaging optical system 21.

In FIGS. 8 to 10, the center of the spot SP and the center of the marker are displayed so as to coincide with each other. However, if there is no problem in terms of measurement accuracy, the first virtual scale is placed at a position away from the spot SP. It may be displayed. However, even in this case as well, it is preferable to display the first virtual scale in the vicinity of the spot. Further, instead of deforming and displaying the first virtual scale, the first virtual scale in a state where the distortion aberration of the captured image is corrected and not deformed may be displayed on the corrected captured image.

Further, in FIGS. 8 to 10, the first virtual scale corresponding to the actual size of the subject of 5 mm is displayed, but the actual size of the subject is an arbitrary value (for example, 2 mm, depending on the observation target and the observation purpose). 3 mm, 10 mm, etc.) may be set. Further, in FIGS. 8 to 10, the first virtual scale has a substantially circular shape, but as shown in FIG. 11, a cross shape in which vertical lines and horizontal lines intersect may be used. Further, a graduated cross shape in which a scale Mx is added to at least one of the vertical line and the horizontal line of the cross shape may be used. Further, as the first virtual scale, a distorted cross shape in which at least one of a vertical line and a horizontal line is tilted may be used. Further, the first virtual scale may be a circle in which a cross shape and a circle are combined and a cross shape. In addition, the first virtual scale may be a measurement point cloud type in which a plurality of measurement point EPs corresponding to the actual size from the spot are combined. Further, the number of the first virtual scale may be one or a plurality, and the color of the first virtual scale may be changed according to the actual size.

As the first virtual scale, as shown in FIG. 12, three concentric virtual scales M4A, M4B, and M4C (the sizes are 2 mm, 5 mm, and 10 mm in diameter, respectively) having different sizes are placed on the tumor tm4. The spot SP4 formed in the center may be displayed on the captured image. Since these three concentric virtual scales display a plurality of virtual scales, the trouble of switching can be saved, and measurement is possible even when the subject has a non-linear shape. When displaying multiple concentric virtual scales centered on the spot, instead of specifying the size and color for each virtual scale, prepare a combination of multiple conditions in advance and select from the combinations. You may be able to do it.

In FIG. 12, all three concentric virtual scales are displayed in the same color (black), but when displaying a plurality of concentric markers, a plurality of colored concentric markers whose colors are changed by the virtual scales are displayed. May be. As shown in FIG. 13, the virtual scale M5A is represented by a dotted line representing red, the virtual scale M5B is represented by a solid line representing blue, and the virtual scale M5C is represented by a alternate long and short dash line representing white. By changing the color of the virtual scale in this way, the distinctiveness is improved and measurement can be easily performed.

Further, as the first virtual scale, in addition to a plurality of concentric virtual scales, as shown in FIG. 14, a plurality of distorted concentric virtual scales in which each concentric circle is distorted may be used. In this case, the distorted concentric virtual scales M6A, virtual scales M6B, and virtual scales M6C are displayed in the captured image centering on the spot SP5 formed on the tumor tm5.

As the measurement light, the light formed as a spot when the subject is irradiated is used, but other light may be used. For example, when the subject is irradiated, as shown in FIG. 15, a planar measurement light formed as an intersecting line 58 on the subject may be used. In this case, as a virtual scale, a second virtual scale consisting of a scale 59 as an index of the size of the subject (for example, polyp P) is generated on the intersection line 58 and the intersection line 58. When a planar measurement light is used, the irradiation position detection unit 54 detects the position of the intersection line 58 (irradiation position of the measurement light). The lower the crossing line 58 is, the closer the observation distance is, and the higher the crossing line 58 is, the farther the observation distance is. Accordingly, the lower the crossing line 58 is, the larger the spacing between the scales 59 is, and the higher the crossing line 58 is located, the smaller the spacing between the scales 59 is.

The calibration mode is performed using the calibration device 100 shown in FIG. The calibration device 100 is a device for calibrating the virtual scale for displaying on the extended display 18 and measuring the size of the subject. The calibration device 100 includes a chart 101, a distance changing mechanism 102, a position adjusting mechanism 103, an expansion processor device 17, and an expansion display 18. The expansion processor device 17 and the expansion display 18 are also used as the endoscope system 10. Further, the expansion processor device 17 only needs to have at least the functions of the calibration image acquisition unit 121 and the scale parameter acquisition unit 122 required for the calibration mode.

As shown in FIG. 17, the chart 101 is provided with an index graphic plane 104 composed of three index figures 104a, 104b, and 104c. The index figure 104a is a circle having a diameter of 5 mm, the index figure 104b is a circle having a diameter of 10 mm, and the index figure 104c is a circle having a diameter of 20 mm. As will be described later, these index figures 104a, 104b, and 104c are affected by the imaging optical system 21 in the calibration image and have a shape such as an ellipse in which the circle is deformed. By providing a plurality of index figures on the chart 101, it is possible to simultaneously acquire scale parameters of a virtual scale having a size corresponding to each index figure. As a result, the calibration time can be shortened as compared with the case of acquiring the scale parameter of the virtual scale with one index figure.

The index figure plane 104 is a plane in which the index figures 104a, 104b, 104c are in contact with each other at the reference position ST at one point, but may be a concentric plane having the same center of the index figures 104a, 104b, 104c. Further, although the index figure is circular, it may be a regular polygon. Also in this case, it may be a plane in which a plurality of regular polygons are in contact with each other at one reference position ST, or a concentric plane having the same center of gravity of the plurality of regular polygons. Further, the index figure may be a cross, a line segment, or the like.

The three index figures 104a, 104b, 104c are separated by two or more colors based on a specific color evaluation standard. As a specific color evaluation standard, it is preferable to use the hue circle 105 shown in FIG. The color of the index figure is preferably white, black, or includes two or more of a plurality of colors corresponding to hues separated by 100 ° or more on the hue circle 105.

Here, as a combination of a plurality of colors corresponding to hues separated by 100 ° or more in the color wheel, B (blue), G (green), R (red) (each separated by 120 ° in the color wheel), or , C (cyan), M (magenta), Y (yellow) (each separated by 120 ° in the color wheel). For example, white, green, and red are preferable as the color combination of the three index figures 104a, 104b, and 104c.

Further, it is preferable that the colors of the index figures are separated based on the transmittance of the color filter of the image sensor 32. For example, in the image pickup element 32, when the four colors of the red color filter RF, the green color filter GF, the blue color filter BF, and the blue-green color filter BGF have the transmission transmission distribution shown in FIG. 19, each color. It is preferable that the color corresponding to the wavelength range in which the transmittance of the color filter is equal to or higher than a certain level is the color of the index figure. In the case of the color filter of FIG. 19, the color near 450 nm, the color near 500 nm, the color near 550 nm, and the color near 630 nm, which are the colors near the wavelength where the transmittance is the highest in each color filter, are used as index figures. It is preferable to use the color of.

When white light is used as the backlight 107, the colors of the index figure n (n is a natural number) are the red signal value SRn of the index figure n, the green signal value SGn of the index figure n, and the blue signal of the index figure n. It is preferably determined based on the value SBn. The light spectrum of the backlight 107 is L (λ), the spectral sensitivity of the red pixel (transmission of the red color filter RF) is R (λ), and the spectral sensitivity of the green pixel (transmission of the green color filter GF) is G (). λ), when the spectral sensitivity of the blue pixel (transparency of the blue color filter BF) is B (λ), the red signal value SRn of the index figure n, the green signal value SGn of the index figure n, and the index figure n The blue signal value SBn is as shown in [Equation 1] below.

Here, as the color of the index figure n (n is a natural number), when the three components SR1, SG1, and SB1 are represented by S1, etc., a combination of colors in which S1, S2, ..., Sn can be easily separated is used. It is preferable to choose. Since Sn has three components, the number of index figures is three (n = 3), and the closer S1, S2, and S3 are to orthogonality, the easier it is to separate, which is particularly preferable.

The index figure is preferably filled with a specific color in order to make it easier to recognize the index figure in the circumscribed rectangle extraction process or the inscribed ellipse parameter calculation process described later. In this case, all of the index figures 104a, 104b, and 104c may be filled, or only a part thereof may be filled. Further, it is preferable that the background portion 106 of the chart 101 other than the index graphic plane 104 is black. Further, as will be described later, the chart 101 is preferably a light transmission type that transmits light in order to display or hide the index graphic plane in the calibration image by the backlight 107. By making the chart 101 a light transmissive type, it is possible to increase the contrast of the index figure plane 104 and improve the recognition accuracy of the index figure.

The base material of the chart 101 is preferably a light-transmitting base material, and includes a glass plate, a ceramic plate, a metal plate, a film, a reversal film, and the like. As the material for developing the color of the index graphic plane, a sensitive material, an inkjet, a color resist or the like is preferable. For example, when the color of the index figure is two or more, it is preferable that the base material of the chart 101 is a glass plate (reversal film) and the material that develops the color of the index figure is a color resist.

The chart 101 is provided with an alignment mark 108 used to face the chart 101 and the tip portion 12d of the endoscope. The alignment mark 108 is wedge-shaped, and the chart 101 or the tip portion 12d is moved so that the alignment line AL (see FIG. 32) comes to the connecting portion 108a on the calibration image.

As shown in FIG. 20, the distance changing mechanism 102 changes the inter-chart distance CD between the tip portion 12d of the endoscope and the chart 101. The distance changing mechanism 102 mounts the chart 101. The distance changing mechanism 102 is attached to the base 111 so as to be movable in the Z direction, which is the vertical direction. A chart holding portion 109 on which the chart 101 is placed is attached to the distance changing mechanism 102. By moving the chart holding unit 109 in the Z direction, the inter-chart distance CD is changed.

When adjusting the inter-chart distance CD to the set distance, the distance setting jig 112 corresponding to the set distance is sandwiched between the chart holding unit 109 and the endoscope holding unit 110, and the inter-chart distance CD is set to the set distance. Is preferable. For example, when the set distance is set to 40 mm, the distance between the charts can be set to 40 mm by sandwiching the distance setting jig 112 having a length of 40 mm between the chart holding portion 109 and the endoscope holding portion 110. can. It is preferable to use a distance setting jig whose length can be adjusted.

As shown in FIG. 20, the position adjusting mechanism 103 adjusts the positional relationship between the reference position ST of the index graphic plane 104 of the chart 101 and the irradiation position (spot SP) of the measured light, so that the tip portion of the endoscope is adjusted. Move 12d. The position adjusting mechanism 103 is attached to the base 111 so as to be movable in the X direction or the Y direction orthogonal to the Z direction. The X direction is orthogonal to the Y direction. The position adjusting mechanism 103 is attached with an endoscope holding portion 110 that holds the tip portion 12d of the endoscope. By moving the endoscope holding portion 110 in the X direction or the Y direction, the positional relationship between the reference position ST of the index graphic plane 104 on the XY plane and the irradiation position of the measurement light is adjusted.

The chart holding unit 109 is provided with a backlight 107 on the surface on which the chart is placed and around the surface thereof. The backlight 107 can be turned on or off by the user. During the distance setting period in which the distance CD between charts is set by the distance changing mechanism 102, the backlight 107 is turned off. When the backlight 107 is turned off, only the spot SP (irradiation position of the measured light) is displayed in the calibration image obtained by the endoscope 12 imaging the light transmission type chart 101 as shown in FIG. 21. The index figure plane 104 is hidden. By hiding the index graphic plane 104, the irradiation position detection unit 54 can easily recognize the irradiation position of the measurement light. Further, in the distance setting period, the irradiation position (spot SP) of the measurement light may be stored every time the distance between charts is set to the set distance. The irradiation position of the measurement light is preferably stored in a temporary storage memory (not shown) for storing the irradiation position of the expansion processor device 17.

On the other hand, after the distance setting period, in the calibration image acquisition period in which the tip portion 12d of the endoscope is moved in the X direction or the Y direction by the position adjusting mechanism 103 and the calibration image is acquired, the backlight is backed up. Turn on the light 107. When the backlight 107 is lit, the calibration image displays the index graphic plane 104 and the alignment mark 108 in addition to the spot SP, as shown in FIG. 22. As a result, the positions of the reference position ST and the spot SP of the index graphic plane can be adjusted, and the index graphic plane can be recognized by the extrinsic rectangle extraction process or the inscribed ellipse parameter calculation process.

In order to face the chart 101 and the tip portion 12d of the endoscope, the endoscope holding portion 110 is rotatable in an XZ plane at an angle ψ in a specific range, or can be rotated at an angle ψ in a specific range, as shown in FIG. , It is preferable that the YZ plane can be rotated at an angle φ in a specific range. The tip portion 12d of the endoscope held by the endoscope holding portion 110 is rotated at angles ψ and φ so that the optical axis Ax of the imaging optical system and the index graphic plane 104 of the chart are perpendicular to each other. This state is defined as a state in which the chart 101 and the tip portion 12d of the endoscope face each other. The endoscope holding portion 110 may be rotatable at an angle θ in a specific range on the XY plane. Depending on the shape of the endoscope insertion portion 12a held by the endoscope holding portion 110, the endoscope holding portion 110 is rotated at an angle θ so that the chart 101 and the tip portion 12d of the endoscope are positive. It is preferable to deal with it.

In the expansion processor device 17, a third signal processing unit 120 is provided for the calibration mode. In the expansion processor device 17, programs related to various processes are incorporated in a program memory (not shown). The expansion processor device 17 is provided with a central control unit (not shown) configured by an image control processor. When the central control unit executes the program in the program memory, the functions of the first signal processing unit 50, the second signal processing unit 52, and the third signal processing unit 120 are realized as shown in FIG. 24.

The third signal processing unit 120 includes a calibration image acquisition unit 121, a scale parameter acquisition unit 122, and an interpolation processing unit 123. The extended display 18 performs an operation display related to the calibration mode including at least one of the distance changing mechanism 102, the position adjusting mechanism 103, the acquisition of the calibration image, and the acquisition of the scale parameters. In the present embodiment, the setting completion operation icon 132, the position adjustment completion icon 134, or the scale confirmation icon 136, which will be described later, are included in the operation display related to the calibration mode. The "display" in the claims includes the extended display 18.

The calibration image acquisition unit 121 is in a state where the reference position ST of the index graphic plane 104 of the chart 101 and the irradiation position (spot SP) of the measurement light match when the calibration image acquisition instruction is given by the user interface 16. The calibration image 124 obtained by imaging the chart 101 of the above with the endoscope 12 is acquired. As shown in FIG. 25, in the calibration image 124, the index figures 104a, 104b, 104c, which are circular in the chart 101, are elliptical index figure images 126a, which are deformed by the optical characteristics of the imaging optical system 21 and the like. It is 126b and 126c. The degree of deformation increases toward the periphery of the screen.

The scale parameter acquisition unit 122 acquires scale parameters for displaying the virtual scale on the extended display 18 from the calibration image 124. The calibration image 124 includes a blue calibration image 124b output from the blue pixel of the image sensor 32, a green calibration image 124 g output from the green pixel of the image sensor 32, and a red calibration output from the red pixel of the image sensor 32. It consists of an image sensor 124r. The blue calibration image 124b, the green calibration image 124g, and the red calibration image 124r of these three colors are subjected to binarization processing, extrinsic rectangle extraction processing, and inscribed ellipse parameter calculation processing, respectively.

It is preferable that the blue calibration image 124b, the green calibration image 124g, and the red calibration image 124r are weighted and added according to the color of the index figure. For example, when the index figure 104a is white, the index figure 104b is green, and the index figure 104c is blue as the color of the index figure, the blue calibration image 124b becomes the green calibration image 124g or the red calibration image 124r. It is preferable to add and add a specific weighting coefficient to make the blue index graphic image 126a (image corresponding to the index graphic 104c) easy to recognize. Further, in the green calibration image 124g, a green index graphic image 126b (an image corresponding to the index graphic 104b) is obtained by adding a specific weighting coefficient to the blue calibration image 124b or the red calibration image 124r and adding them. It is preferable to make it easy to recognize. Further, in the red calibration image 124r, a white index graphic image 126c (an image corresponding to the index graphic 104a) is obtained by adding a specific weighting coefficient to the blue calibration image 124b or the green calibration image 124g and adding them. It is preferable to make it easy to recognize.

The index graphic image 126a and the index graphic image 126c are extracted by performing the binarization process for blue on the blue calibration image 124b. Further, the index graphic image 126b is extracted by performing the binarization process for green on the green calibration image 124g. The index graphic image 126a is extracted by performing the binarization process for red on the red calibration image 124r. The circumscribed rectangle extraction process is performed on the image after the binarization process. When performing the binarization process, it is preferable to perform processing such as shading correction and masking on the calibrated image of each color.

By performing the circumscribed rectangle process on the blue calibration image 124b after the binarization process, the circumscribed rectangle 128c circumscribed on the index graphic image 126c can be obtained. Further, by performing the circumscribed rectangle process on the green calibration image 124 g after the binarization process, the circumscribed rectangle 128b circumscribed on the index graphic image 126b can be obtained. Further, by performing the circumscribed rectangle process on the red calibration image 124r after the binarization process, the circumscribed rectangle 128a circumscribed on the index graphic image 126a can be obtained. The inscribed ellipse parameter calculation process is performed on the image after extracting these circumscribed rectangles. For the circumscribed rectangle processing or the inscribed ellipse parameter calculation processing, it is preferable to use, for example, OpenCV's findContours, boundingRect, minAreaRect method, and the like. Further, it is preferable to extract the circumscribing rectangle by pattern matching, template matching, use of a learning model that has been machine-learned (for example, CNN (Convolutional Neural Network)), or the like.

As shown in FIG. 26, the fitting parameter of the ellipse 130c inscribed in the circumscribed rectangle 128c is calculated by performing the inscribed ellipse parameter calculation process on the circumscribed rectangle 128c. The calculated fitting parameter of the ellipse 130c becomes a scale parameter for displaying a virtual scale having a diameter of 20 mm. The fitting parameter of the ellipse is composed of a plurality of parameters such as the center of the ellipse (X coordinate, Y coordinate), the length in the X axis direction, the length in the Y axis direction, and the inclination of the ellipse.

Similarly, by performing the inscribed ellipse parameter calculation process on the circumscribed rectangle 128b, the fitting parameter of the ellipse 130b inscribed on the circumscribed rectangle 128b is calculated. The calculated fitting parameter of the ellipse 130b becomes a scale parameter for displaying a virtual scale having a diameter of 10 mm. Further, by performing the inscribed ellipse parameter calculation process on the circumscribed rectangle 128a, the fitting parameter of the ellipse 130a inscribed in the circumscribed rectangle 128a is calculated. The calculated fitting parameter of the ellipse 130a becomes a scale parameter for displaying a virtual scale having a diameter of 5 mm. The fitting parameter of the inscribed ellipse is relatively excellent as a parameter when displaying a circle in an image of an endoscope.

When the fitting parameter of the inscribed ellipse is calculated, an ellipse is generated from the fitting parameter of the inscribed ellipse. Then, as shown in FIG. 27, a scale confirmation screen is displayed in which the generated ellipse is superimposed on the calibration image as a virtual scale. In the case of FIG. 27, on the scale confirmation screen, the ellipses 130a, 130b, 130c generated from the fitting parameters of the inscribed ellipse are superimposed and displayed on the corresponding index graphic images 126a, 126b, 126c as virtual scales, respectively. Will be done. By using the fitting parameter of the inscribed ellipse, it is possible not only to generate an ellipse but also to convert from a circle to a line segment, from a line segment to a circle, and from a cross to a grid.

The user who performs the calibration confirms whether the positions, sizes, etc. of the ellipses 130a, 130b, 130c match the index graphic images 126a, 126b, 126c. If it is determined that the user is matching, the user operates the scale confirmation icon 136 (see FIG. 34) using the user interface 16. When the scale confirmation icon 136 is operated, the third signal processing unit 120 associates the irradiation position of the measurement light with the scale parameter and stores it in the parameter table 57. On the other hand, if it is determined that the user does not match, the operation is restarted from the inter-chart distance CD.

The interpolation processing unit 123 performs interpolation processing based on the first distance scale parameter P2 obtained by setting the inter-chart distance CD by the distance change mechanism 102, so that the second distance scale parameter that is not in the first distance can be used. Calculate P2. As shown in FIG. 28, when the inter-chart distance CD is changed at 1 mm intervals by the distance changing mechanism 102 and the scale parameter P1 of the first distance is acquired for each change, the inter-chart distance CD is 2 mm and the diameter is 5 mm. The scale parameter (5 mm) of the virtual scale is PM5_2, the distance between charts is 3 mm, and the scale parameter (5 mm) of the virtual scale having a diameter of 5 mm is PM5_3. A scale parameter having a distance CD between the charts of 2 mm and 3 mm, which is 2.5 mm, is calculated as a second distance scale parameter P2 by interpolation processing. Specifically, the scale parameter PM5_2.5 having a chart-to-chart distance CD of 2.5 mm is calculated as the second distance scale parameter P2 by the calculation for interpolation processing of (PM5_2 + PM5_3) / 2.

Similarly, the 3.5 mm scale parameter PM5_3.5 whose inter-chart distance is between 3 mm and 4 mm is calculated by interpolation processing, and the 4.5 mm scale parameter PM5_4 where the inter-chart distance is between 4 mm and 5 mm. 5 is calculated by interpolation processing. The scale parameter obtained by the interpolation process is stored in the parameter table 57 in association with the irradiation position of the measurement light corresponding to the distance between the charts. As described above, in the actual calibration mode, even if the scale parameters can be obtained only for the discrete inter-chart distances, by using the interpolation processing, the scale parameters can be obtained for all the inter-chart distances. Obtainable.

Next, a series of calibration mode flows will be described with reference to the flowchart of FIG. 29. The calibration mode is carried out in a dark room. The user who calibrates the virtual scale sets the tip portion 12d of the endoscope on the endoscope holding portion 110, and places the chart 101 on the chart holding portion 109. Next, the user operates the mode changeover switch 12f to switch to the calibration mode. As a result, the measurement light is applied to the chart 101. In this state, the distance change step is executed. In the following steps, it is preferable to display the work contents of each step on the extended display 18 so that the work contents of each step can be understood as the work contents of the operation related to the calibration mode.

In the distance changing step, the distance changing mechanism 102 creates an inter-chart distance CD between the chart 101 provided with the index graphic plane 104 consisting of the three index figures 104a, 104b, and 104c and the tip portion 12d of the endoscope. change. In the distance change step, the inter-chart distance CD for acquiring the scale parameter is predetermined as the set distance. For example, the set distance is preferably in the range of 1 mm to 50 mm and at intervals of 1 mm.

In the distance change step, by turning off the backlight 107 provided on the light transmission type chart 101, only the irradiation position of the measurement light is displayed in the calibration image, and the index graphic plane 104 is hidden. There is. This makes it easier for the irradiation position detection unit 54 to detect the irradiation position of the measurement light.

In the distance change step, as shown in FIG. 30, on the extended display 18, a setting completion operation icon 132 for instructing that the setting of the set distance is completed is displayed. When the inter-chart distance CD is set to the set distance, the user operates the setting completion operation icon 132 using the user interface 16. According to this, the irradiation position detection unit 54 detects the irradiation position (position of the spot SP) of the measurement light. In the distance changing step, the distance CD between charts may be set by the distance setting jig 112. After the distance change step, the measurement light may be turned off. In this case, the movement step is performed using only the alignment mark 108.

In the distance changing step, the distance changing step may be provided with a face-to-face confirmation step step for confirming whether or not the chart 101 and the tip portion 12d of the endoscope are facing each other. In the face-to-face confirmation step, as shown in FIG. 31, in the calibration image 124, confirmation is performed with the roundness of the largest circular index graphic image 126c among the index graphic plane images 125. When the index graphic image 126c is inscribed in the rectangle 140 for checking the roundness, it is assumed that the chart 101 and the tip portion 12d of the endoscope face each other, assuming that the index graphic image 126c has the highest roundness. .. On the other hand, when the index graphic image 126c does not inscribe the rectangle 140 for checking the roundness and a part of the index graphic image 126c protrudes from the rectangle 140, it is assumed that they do not face each other. In this case, it is preferable to adjust the angles ψ and φ of the tip portion 12d of the endoscope so as to face each other. In the step of determining whether or not to face each other, as the work content 142 to be displayed on the extended display 18, as shown in FIG. 35, "Move the left end of the white circle to the spot SP. Completed. Then press the OK button. "

If the shape of the virtual scale is a cross, it is judged whether or not they are facing each other based on whether or not the line thickness is the same on the top and bottom or left and right. When the shape of the virtual scale is a triangle, it is determined whether or not they face each other based on whether or not they have the same size on the top, bottom, left, and right.

After the distance change step, a movement step is performed. As shown in FIG. 32, in the moving step, the tip of the endoscope is adjusted by the position adjusting mechanism 103 in order to adjust the positional relationship between the reference position ST of the index graphic plane 104 of the chart and the irradiation position SP of the measurement light. Move 12d. The movement of the tip portion 12d of the endoscope is performed in a plane parallel to the chart 101, that is, in the XY plane.

In the movement step, the index graphic plane 104 is displayed in the calibration image by turning on the backlight 107 provided on the light transmission type chart 101. This makes it easier for the scale parameter acquisition unit 122 to recognize the index graphic images 126a, 126b, 126c included in the index graphic plane image 125.

In the movement step, the alignment line AL is displayed in the vertical direction and the horizontal direction in the calibration image 124. The tip portion 12d of the endoscope is moved in the X direction or the Y direction so that the alignment line AL comes to the connecting portion 108a of the alignment mark. In this case, the vertical and horizontal alignment lines AL are cross-shaped lines, and the intersection of the vertical and horizontal alignment lines AL is located in the center of the screen. Therefore, by superimposing the intersection of the alignment line AL on the center of the index graphic plane image 125 of the chart 101, the alignment can be facilitated.

In the movement step, as shown in FIG. 33, on the extended display 18, a position adjustment completion icon 134 for instructing that the reference position ST of the index graphic plane 104 and the irradiation position SP of the measurement light match is displayed. ing. When the reference position ST of the index graphic plane 104 and the irradiation position SP of the measurement light are in the same state, the user operates the position adjustment completion icon 134 using the user interface 16.

When the position adjustment completion icon 134 is operated, the calibration image acquisition step is performed. In the calibration image acquisition step, the endoscope 12 captures a chart 101 in a state where the reference position ST of the index graphic plane 104 and the irradiation position SP of the measurement light match with the endoscope 12 to acquire a calibration image. do. The acquired calibration image is sent to the extended processor device 17 via the processor device 14. The calibration image acquisition unit 121 acquires the calibration image obtained by the endoscope 12 when the position adjustment completion icon 134 is operated as the calibration image for acquiring the scale parameter. As a result, the calibration image acquisition step is completed, and then the parameter acquisition step is performed.

In the parameter acquisition step, the scale parameters for displaying the virtual scale on the extended display 18 are acquired from the calibration image 124. Binarization processing, inscribed rectangle extraction processing, and inscribed ellipse parameter calculation processing are performed on the three colors of the blue calibration image 124b, the green calibration image 124g, and the red calibration image 124r included in the calibration image 124, respectively. ..

In the binarization process, the blue calibration image 124b, the green calibration image 124g, and the red calibration image 124r are binarized for blue, binarized for green, and binarized for red, respectively. Perform the process. In the circumscribed rectangle extraction process, the circumscribed rectangles 128a, 128b, 128c that circumscribe the index graphic images 126a, 126b, 126c in the calibration image 124 are extracted. In the inscribed ellipse parameter calculation process, the fitting parameter of the ellipse inscribed in the circumscribed rectangles 128a, 128b, 128c is calculated as a scale parameter. As a result, the parameter acquisition step is completed, and then the virtual scale confirmation step is performed.

In the virtual scale confirmation step, an ellipse is generated from the fitting parameter of the inscribed ellipse, which is a parameter for scale. Then, in order to confirm whether or not the user is appropriate as a virtual scale, the ellipses 130a, 130b, and 130c generated from the fitting parameters of the inscribed ellipse are used as virtual scales on the scale confirmation screen, and the corresponding index graphic images 126a are used. , 126b, 126c are superimposed and displayed (see FIG. 27). In the virtual scale confirmation step, when the user looks at the scale confirmation screen and determines that the positions, sizes, etc. of the ellipses 130a, 130b, 130c match the index graphic images 126a, 126b, 126c, the user Uses the user interface 16 to operate the scale confirmation icon 136 provided on the extended display 18, as shown in FIG. 34.

When the scale confirmation icon 136 is operated, the third signal processing unit 120 associates the irradiation position of the measurement light with the scale parameter and stores it in the parameter table 57. On the other hand, if it is determined that the user does not match, the operation is restarted from the inter-chart distance CD.

After the virtual scale confirmation step is completed, change the distance CD between charts to the next set distance in the distance change step, and perform the same operation. Then, the same operation or the like is performed for all the set distances until the acquisition of the scale parameters is completed.

In the calibration mode, the interpolation processing step may be performed. In the interpolation processing step, by performing interpolation processing based on the first distance scale parameter P2 obtained by setting the interchart distance CD by the distance change mechanism 102, the second distance scale parameter P2 that is not in the first distance is performed. Is calculated. The second distance scale parameter obtained by the interpolation process is stored in the parameter table 57 in association with the irradiation position of the measurement light corresponding to the distance between the charts.

In the moving step, as shown in FIG. 36, in order to make the tip of the endoscope face the chart in the calibration image 124, the alignment figure 150 corresponding to the index figure plane image 125 is provided. It may be displayed. Similarly, in the calibration image 124, the alignment figure 151 corresponding to the alignment image 148 may be displayed in order to face the calibration image 124. When the position of the index figure plane image 125 and the alignment figure 150 are aligned and / or the positions of the alignment image 148 and the alignment image 151 are aligned, it is preferable that they face each other. After the movement step is completed, the alignment figures 150 and 151 are preferably hidden.

In the above embodiment, the signal processing unit 39, the display control unit 40, the system control unit 41, the first signal processing unit 50, the second signal processing unit 52, the irradiation position detection unit 54, the scale table 55, the table update unit 56, The hardware structure of the processing unit that executes various processes such as the parameter table 57, the third signal processing unit 120, the calibration image acquisition unit 121, the scale parameter acquisition unit 122, and the interpolation processing unit 123 is Various processors as shown below. For various processors, the circuit configuration is changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various processing units. It includes a programmable logic device (PLD), which is a possible processor, a dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing various processes, and the like.

One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). May be done. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units. Second, as typified by System On Chip (SoC), there is a form that uses a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Furthermore, the hardware-like structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined. The hardware structure of the storage unit is a storage device such as an HDD (hard disk drive) or SSD (solid state drive).

10 Endoscopic system 12 Endoscope 12a Insertion part 12b Operation part 12c Curved part 12d Tip part 12f Mode changeover switch 12g Still image acquisition instruction switch 12h Zoom operation part 13 Light source device 14 Processor device 15 Display 16 User interface 17 Expansion processor device 18 Extended display 21 Imaging optical system 22 Illumination optical system 22a Illumination lens 23 Measurement light emission unit 23a Light source 23b DOE
23c Prism 23d Measurement light lens 24 Aperture 25 Air supply / water supply nozzle 30 Light source 32 Image sensor 34 CDS / AGC circuit 35 A / D converter 36 Communication I / F
37 Communication I / F
38 Reception unit 39 Signal processing unit 40 Display control unit 41 System control unit 42 Still image storage unit 43 Data transmission / reception unit 44 Data transmission / reception unit 45 Signal processing unit 46 Display control unit 48 Solid line 49 Dotted line 50 First signal processing unit 52 Second signal Processing unit 54 Irradiation position detection unit 55 Scale table 56 Table update unit 57 Parameter table 58 Crossing line 59 Scale 100 Calibration device 101 Chart 102 Distance change mechanism 103 Position adjustment mechanism 104 Index graphic plane 104a, 104b, 104c Index graphic 105 hue Ring 106 Background part 107 Backlight 108 Alignment mark 108a Connecting part 109 Chart holding part 110 Endoscope holding part 111 Base 112 Distance setting jig 121 Calibration image acquisition part 122 Scale parameter acquisition part 123 Interpolation processing part 124 Calibration Image 124b Blue calibration image 124g Green calibration image 124r Red calibration image 125 Index graphic plane image 126a, 126b, 126c Index graphic image 128a, 128b, 128c External rectangle 130a, 130b, 130c Elliptical 132 Setting complete operation icon 134 Position adjustment Completion icon 136 Scale confirmation icon 140 Roundness confirmation rectangle 142 Work content 148 Alignment image 151, 152 Alignment figure AL Alignment line BF Blue color filter BGF Blue-green color filter CD Chart distance D1 1st direction D2 2nd direction D3 3rd direction EP measurement point GF Green color filter RF Red color filter tm1, tm2, tm3, tm4, tm5 Tumor SP Spot SP1, SP2, SP3, SP4, SP5 Spot ST Reference position M1, M2, M3 Virtual scale M4A, M4B , M4C, M5A, M5B, M5C Concentric markers M6A, M6B, M6C Distorted concentric markers Mx Scale P Polyp P1 Parameter for 1st distance scale P2 Parameter for 2nd distance scale

Claims (15)

1. It is a calibration device that calibrates the virtual scale for displaying on the display and measuring the size of the subject.
   It is a chart provided with an index figure plane composed of a plurality of index figures, and the plurality of index figures are separated by two or more colors based on a specific color evaluation standard, or provided in an endoscope. A chart that is separated based on the transmittance of the color filter of the image pickup element, and
   A distance changing mechanism that changes the distance between charts between the tip of the endoscope and the chart,
   In order to adjust the positional relationship between the reference position of the index graphic plane and the irradiation position of the measurement light emitted from the tip of the endoscope toward the chart, the tip of the endoscope or the chart The position adjustment mechanism to move the
   Equipped with an image control processor
   The image control processor is
   A calibration image obtained by imaging the chart in a state where the reference position of the index graphic plane and the irradiation position of the measurement light match with the endoscope is acquired.
   A calibration device that acquires scale parameters for displaying the virtual scale on the display from the calibration image.
2. The particular color evaluation criterion is the color wheel.
   The calibration device according to claim 1, wherein the color of the index figure includes white, black, or two or more of a plurality of colors corresponding to a plurality of hues separated by 100 ° or more in the color wheel.
3. The calibration device according to claim 1 or 2, wherein the background portion of the chart other than the index graphic plane is black.
4. The calibration device according to any one of claims 1 to 3, wherein the chart is a light transmission type.
5. During the distance setting period in which the distance between the charts is set by the distance changing mechanism, by turning off the backlight provided on the light transmission type chart, only the irradiation position of the measured light is displayed in the calibration image. The calibration device according to claim 4, wherein the display and / or the irradiation position of the measurement light is stored and the index graphic plane is hidden.
6. During the calibration image acquisition period in which the tip of the endoscope or the chart is moved by the position adjustment mechanism and the calibration image is acquired, the backlight provided on the light transmissive chart is turned on. The calibration device according to claim 4, wherein the calibration image displays the irradiation position of the measurement light and displays the index graphic plane.
7. The calibration device according to claim 6, which displays an alignment figure corresponding to the index figure plane on the calibration image in order to make the tip of the endoscope face the chart during the calibration image acquisition period.
8. An endoscope holding portion that holds the tip of the endoscope,
   The chart holding unit on which the chart is placed and the chart holding unit
   The calibration device according to any one of claims 1 to 7, further comprising a distance setting jig that is sandwiched between the endoscope holding portion and the chart holding portion and that adjusts the distance between charts to the set distance.
9. The display is claimed to display the work contents of an operation related to a calibration mode including at least one of the distance changing mechanism, the position adjusting mechanism, the acquisition of the calibration image, or the acquisition of the parameters for scale. The calibration device according to any one of 1 to 8.
10. Claims 1 to 9 indicate that the display performs an operation display related to a calibration mode including at least one of the distance changing mechanism, the position adjusting mechanism, the acquisition of the calibration image, and the acquisition of the parameters for scale. The calibration device according to any one of the items.
11. It is a calibration method that calibrates the virtual scale for displaying on the display and measuring the size of the subject.
    A distance changing step for changing the distance between charts between a chart provided with an index figure plane composed of a plurality of index figures and the tip of the endoscope by a distance changing mechanism, and
    In order to adjust the positional relationship between the reference position of the index graphic plane and the irradiation position of the measurement light emitted from the tip of the endoscope toward the chart by the position adjustment mechanism, the endoscope is used. A moving step for moving the tip or the chart,
    Every time the distance between the charts is changed, the chart in a state where the reference position of the index graphic plane and the irradiation position of the measurement light match is imaged with the endoscope, and the calibration image obtained is acquired. Image acquisition step and
    It has a parameter acquisition step of acquiring a scale parameter for displaying the virtual scale on the display from the calibration image.
    In the chart, the plurality of index figures are separated by two or more colors based on a specific color evaluation standard or based on the transmittance of a color filter of an image pickup device provided in the endoscope. Calibration method.
12. In the distance changing step, by turning off the backlight provided on the light-transmitting chart, only the irradiation position of the measured light is displayed in the calibration image, and the index graphic plane is not displayed. The calibration method according to claim 11, which is displayed.
13. In the calibration image acquisition step, by turning on the backlight provided in the light transmission type chart, the irradiation position of the measurement light is displayed in the calibration image, and the index graphic plane is displayed. The calibration method according to claim 11 to be displayed.
14. The index figure is a circle.
    In the parameter acquisition step, the circumscribed rectangle extraction process for extracting the circumscribed rectangle inscribed in the index graphic image in the calibration image and the fitting parameter of the ellipse inscribed in the circumscribed rectangle are calculated as the scale parameters. The calibration method according to any one of claims 11 to 13, wherein the parameter calculation process is performed.
15. The scale parameters include a scale parameter having a chart-to-chart distance of a first distance and a scale parameter having a chart-to-chart distance of a second distance.
    In the parameter acquisition step, every time the distance between the charts is changed by the distance changing mechanism, the scale parameter of the first distance is acquired.
    Any one of claims 11 to 14 that performs an interpolation processing step of calculating the scale parameter of the second distance by performing an interpolation process based on the scale parameter of the first distance after the parameter acquisition step. The calibration method described.

Images