JP2023007331A - Endoscope system, medical image processing device, and operation method therefor - Google Patents

Abstract

To provide an endoscope system, a medical image processing device and an operation method therefor which allow a user to recognize a correct boundary line in real time.SOLUTION: A medical image processing device acquires a reference image that is a medical image with which boundary line information related to a boundary line that is a boundary between an abnormal region and a normal region and landmark information related to a landmark and a captured image that is the medical image captured in real time, detects the landmark from the captured image, calculates a ratio of match between the landmark included in the reference image and the landmark included in the captured image, estimates a correspondence relationship between the reference image and the captured image on the basis of the ratio of match and information regarding the landmarks included in the reference image and the captured image, and generates a superimposition image in which the boundary line associated with the reference image is superimposed on the captured image on the basis of the correspondence relationship.SELECTED DRAWING: Figure 11

Description

TECHNICAL FIELD The present invention relates to an endoscope system, a medical image processing apparatus, and an operation method thereof for assisting surgery such as endoscopic submucosal dissection.

Endoscopic submucosal dissection (ESD) enables resection of tumors that are too large for endoscopic mucosal resection (EMR), making it a highly invasive surgical procedure. is no longer required. Since ESD is performed under an endoscope, it has the advantage of being minimally invasive. On the other hand, the thickness of the gastrointestinal tract is as thin as 5 to 7 mm in the stomach and 3 to 5 mm in the large intestine.

A technique capable of indicating a lesion area is known in order to assist an operator during surgery. For example, a region of interest including a lesion is detected, and the coordinates of the boundary between the region of interest and a non-region of interest, the coordinates of the position inside the region of interest along the boundary, or the coordinates of the center of gravity of the region of interest are is known (Patent Document 1).

WO2020/090731

Advances in image processing have made it possible to show the areas and boundaries of differentiated lesions on images. The range also changes. In other words, since images captured in real time change from moment to moment, it is difficult to accurately determine the boundary between abnormal and normal regions in real time, and depending on the imaging conditions, the displayed boundary may be inaccurate. may be. In addition, when ESD is applied, it is difficult to distinguish the boundary line by low-magnification observation (distant view observation) that can overlook the entire lesion, and it is necessary to perform high-magnification observation (near view observation) by enlarging a part of the lesion. In view of the above situation, there is a demand for a technology capable of recognizing, in real time, accurate boundary lines determined from images captured under optimal conditions. Furthermore, there is a need for a technology that allows confirmation of how the previously diagnosed borderline has changed on current real-time images.

SUMMARY OF THE INVENTION An object of the present invention is to provide an endoscope system, a medical image processing apparatus, and an operating method thereof that can recognize an accurate boundary line in real time.

A medical image processing apparatus according to the present invention is a medical image processing apparatus comprising a processor. Acquire a reference image, which is a medical image that associates boundary line information and landmark information associated with landmarks, which are characteristic structures of the subject, and acquire a captured image, which is a medical image captured in real time. Then, the landmarks are detected from the captured image, and the degree of matching between the landmarks included in the reference image and the landmarks included in the captured image is calculated. , and generates a superimposed image by superimposing the boundary associated with the reference image on the captured image based on the correspondence.

It is preferable that the captured image and the reference image are medical images captured at the same magnification. Preferably, the captured image is a medical image captured in a distant view, and the reference image is a medical image captured in a close view.

The captured image is a medical image captured so as to include a portion of the abnormal region, and the reference image is a medical image including the entire abnormal region, and a portion of the boundary associated with the reference image is the captured image. It is preferable to generate a superimposed image superimposed on the .

The reference image is preferably generated by joining together a plurality of magnified medical images, which are medical images in which a part of the abnormal region is photographed in the foreground.

When the reference image consists of a first magnified medical image and a second magnified medical image taken at a position different from the first magnified medical image, the processor calculates landmark information and boundaries associated with the first magnified medical image. Preferably, the first magnified medical image and the second magnified medical image are spliced together to generate a reference image based on a common relationship between the information and the landmark information and boundary line information associated with the second magnified medical image. .

The processor preferably identifies and demarcates abnormal and normal regions. The boundary line is preferably set by user operation.

Preferably, the reference image is a medical image captured by illuminating the subject with special light, and the captured image is a medical image captured by illuminating the subject with normal light.

It is preferable that the photographed image for detecting the landmark and the photographed image for superimposing the boundary line are photographed images photographed at the same time.

It is preferable that the captured image for detecting the landmark and the captured image for superimposing the boundary line are captured images captured at different times. The calculation of the degree of matching is preferably continued until an end instruction is given.

The processor generates a display image and displays the superimposed image in a first display section of the display image and displays the reference image in a second display section of the display image that is different from the first display section. Control is preferred.

The processor acquires an enlarged photographed image as a photographed image, identifies an abnormal area and a normal area included in the enlarged photographed image, sets a boundary line, detects landmarks from the enlarged photographed image, and adds boundaries to the enlarged photographed image. It is preferable to associate the boundary line information attached to the line with the landmark information attached to the landmark and use it as a reference image.

The processor generates a superimposed image, identifies the abnormal region and the normal region included in the photographed image related to the superimposed image on which the boundary line is superimposed, when there is an update instruction, and identifies the boundary between the abnormal region and the normal region as the boundary line , and when there is an update determination instruction, it is preferable to update the boundary superimposed on the captured image by using the update boundary as the determined update boundary.

A method for operating a medical image processing apparatus of the present invention comprises steps of acquiring a medical image of a subject photographed by an endoscope, boundary line information associated with a boundary line that is a boundary between an abnormal region and a normal region, and Acquiring a reference image, which is a medical image associated with landmark information associated with a landmark, which is a characteristic structure; Acquiring a captured image, which is a medical image captured in real time; a step of detecting landmarks; a step of calculating a degree of matching between the landmarks included in the reference image and the landmarks included in the captured image; estimating a correspondence between the reference image and the captured image based on the correspondence; and generating a superimposed image in which the boundary associated with the reference image is superimposed on the captured image based on the correspondence.

An endoscope system of the present invention includes the medical image processing apparatus described above and an endoscope.

According to the present invention, an accurate boundary line can be recognized in real time.

1 is an explanatory diagram of a configuration of an endoscope system; FIG. 1 is a block diagram showing functions of an endoscope system; FIG. 4 is a graph showing spectra of violet light V, blue light B, green light G, and red light R in normal light. 5 is a graph showing spectra of violet light V, blue light B, green light G, and red light R in special light. FIG. 10 is an explanatory diagram showing a first light emission pattern in image analysis mode; FIG. 11 is an explanatory diagram showing a second light emission pattern in image analysis mode; FIG. 10 is an image diagram showing an example of a photographed image in which landmarks are detected; FIG. 10 is an image diagram showing an example of a photographed image in which landmarks are connected by link lines; FIG. 10 is an image diagram showing an example of a reference image in which a boundary line is shown on the boundary between an abnormal region and a normal region; FIG. 10 is an explanatory diagram showing an example of calculating the degree of matching between a reference image and a captured image; FIG. 10 is an explanatory diagram showing generation of a superimposed image; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a captured image and a reference image are captured at the same magnification; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a photographed image is observed in a distant view and a reference image is photographed in a near view; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a captured image is captured so as to include a portion of an abnormal region and a reference image includes the entire abnormal region; FIG. 4 is an explanatory diagram showing that a reference image is generated by joining enlarged medical images; FIG. 10 is a block diagram showing functions of a reference image generator in a boundary line display mode; FIG. 10 is an explanatory diagram showing generation of a reference image by joining enlarged medical images using a common relationship; FIG. 10 is an explanatory diagram showing identification of an abnormal region, setting of boundaries, and detection of landmarks in an enlarged medical image when automatically setting boundaries. FIG. 11 is a block diagram showing functions of a reference image generation unit when a boundary line is set by a user operation; FIG. 10 is an explanatory diagram of setting a boundary line by user operation; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a reference image is captured using special light and a captured image is captured using normal light; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a captured image for detecting a landmark and a captured image for superimposing a boundary line are captured images captured at the same time; FIG. 10 is an explanatory diagram showing generation of a superimposed image when a captured image for detecting a landmark and a captured image for superimposing a boundary line are captured images captured at different times; 4 is a flow chart showing a series of flows in a boundary line display mode; FIG. 4 is an image diagram showing a display image; 3 is a block diagram showing functions of a captured image input unit and a reference image generation unit in a reference image generation mode; FIG. FIG. 10 is an explanatory diagram showing generation of a reference image in a reference image generation mode; FIG. 4 is a block diagram showing functions of a captured image input unit and a reference image generation unit in a boundary line update mode; FIG. 10 is an explanatory diagram showing generation of a superimposed image in the boundary line update mode; FIG. 9 is an explanatory diagram showing generation of an updated boundary line superimposed image; FIG. 11 is an explanatory diagram showing generation of a determined update boundary line superimposed image; FIG. 4 is an explanatory diagram of the configuration of an endoscope system when using a rigid endoscope; FIG. 4 is an explanatory diagram showing the configuration of a light receiving section;

As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, a medical image processing device 11, a display 17 and a user interface 19. A medical image processing device 11 is connected to the endoscope system 10 via a processor device 15 . The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15 .

The endoscope 12 includes an insertion portion 12a to be inserted into the body of an observation target, an operation portion 12b provided at the proximal end portion of the insertion portion 12a, a bending portion 12c provided at the distal end side of the insertion portion 12a, and a distal end portion. and a portion 12d. The bending portion 12c is bent by operating the angle knob 12e of the operation portion 12b. The distal end portion 12d is directed in a desired direction by the bending motion of the bending portion 12c. A forceps channel (not shown) for inserting a treatment tool or the like is provided from the insertion portion 12a to the distal end portion 12d. The treatment instrument is inserted into the forceps channel from the forceps port 12j.

An optical system for forming a subject image and an optical system for illuminating the subject with illumination light are provided inside the endoscope 12 . The operation unit 12b is provided with an angle knob 12e, a mode changeover switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The mode changeover switch 12f is used for an observation mode changeover operation. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image. A zoom operation unit 12 i is used for operating the zoom lens 42 .

The light source device 14 generates illumination light. The display 17 displays medical images. The medical image includes a photographed image that is a medical image photographed in real time by the endoscope 12, information on a boundary line that is the boundary between an abnormal region and a normal region, and landmarks that are the characteristic structures of the subject, which will be described later. A reference image, which is a medical image associated with information, and a superimposed image in which a boundary line is superimposed on a captured image are included. Note that real time does not mean exactly one time or exactly the same time, but means the time including the latest fixed period in one endoscopy. In addition, an abnormal region refers to a region in which an abnormality is observed, such as a region in which a tumor exists or a region in which inflammation is observed, among observation targets. A normal region refers to a region other than an abnormal region in which no abnormality is observed. Furthermore, the abnormal region may be defined as a "tumor region" and the normal region as a "non-tumor region". good.

The user interface 19 has a keyboard, a mouse, a touch pad, a microphone, a tablet 241, a touch pen 242, and the like, and has a function of receiving input operations such as function settings. The processor device 15 performs system control of the endoscope system 10 and image processing and the like on image signals transmitted from the endoscope 12 .

In FIG. 2 , the light source device 14 includes a light source section 20 and a light source processor 21 that controls the light source section 20 . The light source unit 20 has, for example, a plurality of semiconductor light sources, which are turned on or off. When turned on, the light emission amount of each semiconductor light source is controlled to emit illumination light for illuminating the observation target. The light source unit 20 includes a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light Emitting Diode) 20d. It has four color LEDs. The light source unit 20 may be built in the endoscope 12 , the light source control unit may be built in the endoscope 12 , or may be built in the processor device 15 .

The endoscope system 10 has a mono light emission mode, a multi light emission mode, a boundary line display mode, a reference image generation mode, and a boundary line update mode, which are switched by the mode changeover switch 12f. The mono light emission mode is a mode in which illumination light of the same spectrum is continuously emitted to illuminate an observation target. The multi-emission mode is a mode in which a plurality of illumination lights with different spectra are emitted while being switched according to a specific pattern to illuminate an observation target. The illumination light includes normal light (broadband light such as white light) used to give brightness to the entire observation object and to observe the entire observation object, or light for emphasizing a specific region of the observation object. Special lights used for Also, in the mono light emission mode, the illumination light of a different spectrum may be switched by operating the mode switch 12f. For example, the normal light may be used as the first illumination light and the special light may be used as the second illumination light, or the special light may be used as the first illumination light and the normal light may be used as the second illumination light.

As shown in FIG. 3, when emitting normal light, the V-LED 20a emits violet light V with a central wavelength of 405±10 nm and a wavelength range of 380-420 nm. The B-LED 20b generates blue light B with a central wavelength of 450±10 nm and a wavelength range of 420-500 nm. The G-LED 20c generates green light G with a wavelength range of 480-600 nm. The R-LED 20d emits red light R with a central wavelength of 620-630 nm and a wavelength range of 600-650 nm. As shown in FIG. 4, special light in which the amount of violet light V is greater than the amount of other blue light B, green light G, and red light R may be used.

In the border display mode, when a tumor is found in the subject, a superimposed image is obtained by superimposing the border line of the reference image, in which the border line between the abnormal region and the normal region is defined in advance, on the photographed image captured in real time. is generated and displayed on the display 17 for the user to assist mucosa incision in ESD. The reference image generation mode is a mode for generating a reference image using a captured image. The boundary line update mode is a mode for updating the boundary line while displaying a superimposed image in which the boundary line of the reference image is superimposed on the captured image.

The light source processor 21 controls the light amounts of the four colors of violet light V, blue light B, green light G, and red light R independently. In the case of the mono light emission mode, illumination light of the same spectrum is continuously emitted for each frame. For example, by illuminating the observation target with normal light (first illumination light) and capturing an image, the display 17 displays a first illumination light image with natural colors. In mono light emission mode, the first illumination light and the second illumination light are switched to illuminate the observation object with the special light (the second illumination light), and an image of the second illumination light emphasizing a specific structure is displayed. 17 may be displayed. The first illumination light image and the second illumination light image are types of medical images.

The light used when performing ESD is usually the first illumination light. The second illumination light may be used when it is desired to check the infiltration range of the lesion before performing ESD. In the boundary line display mode, a medical image is obtained with the light emission pattern in either the mono light emission mode or the multi light emission mode, or the first illumination light image or the second illumination light image is obtained with the light emission pattern in the mono light emission mode. You can choose which one to obtain as a medical image.

On the other hand, in the case of the multi-emission mode, control is performed to change the light amounts of the violet light V, blue light B, green light G, and red light R according to a specific pattern. For example, as the light emission pattern, as shown in FIG. As shown, the number of frames in the first illumination period switches between the first illumination light and the second illumination light as in the second emission pattern, which is different in each first illumination period. 5 and 6, arrows indicate the direction of time passage. A frame means that an imaging sensor (not shown) provided at the distal end portion 12d of the endoscope starts receiving return light from the observation target, and the output of the accumulated charge signal is completed based on the received light. refers to the time between Also, the second illumination period is a period during which the subject is illuminated with the second illumination light.

Light emitted from each of the LEDs 20a to 20d (see FIG. 2) enters a light guide 23 via an optical path coupling portion 22 composed of mirrors, lenses, and the like. The light guide 23 propagates the light from the optical path coupling portion 22 to the distal end portion 12 d of the endoscope 12 .

A distal end portion 12d of the endoscope 12 is provided with an illumination optical system 30a and an imaging optical system 30b. The illumination optical system 30 a has an illumination lens 31 , and the illumination light propagated by the light guide 23 is applied to the observation target via the illumination lens 31 . When the light source unit 20 is incorporated in the distal end portion 12d of the endoscope 12, the light is emitted toward the object through the illumination lens of the illumination optical system without passing through the light guide. The imaging optical system 30 b has an objective lens 41 and an imaging sensor 43 . Light emitted from the object to be observed by illumination light enters the imaging sensor 43 via the objective lens 41 and the zoom lens 42 . As a result, an image of the observation target is formed on the imaging sensor 43 . The zoom lens 42 is a lens for enlarging an observation target, and is moved between the tele end and the wide end by operating the zoom operation section 12i.

The imaging sensor 43 is a primary color sensor, and includes B pixels (blue pixels) having blue color filters, G pixels (green pixels) having green color filters, and R pixels (red pixels) having red color filters. and three types of pixels.

Also, the imaging sensor 43 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging processor 44 controls the imaging sensor 43 . Specifically, an image signal is output from the imaging sensor 43 by reading the signal of the imaging sensor 43 by the imaging processor 44 . The output image signal is transmitted to the captured image acquisition section 60 of the processor device 15 .

The captured image acquisition unit 60 performs various signal processing such as defect correction processing, offset processing, demosaicing processing, matrix processing, white balance adjustment, gamma conversion processing, and YC conversion processing on the received image signal. Next, image processing including 3×3 matrix processing, gradation conversion processing, color conversion processing such as three-dimensional LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement is performed.

When the user wants to acquire a captured image as a still image, by operating the still image acquisition instruction switch 12h, a signal regarding a still image acquisition instruction is sent to the endoscope 12, the light source device 14, and the processor device 15, and the still image is captured. The image is acquired and stored in a still image storage (not shown).

In the processor device 15, the functions of the photographed image acquisition section 60 are realized by operating the program in the program memory by the first central control section 55, which is an image control processor.

The captured image generated by the captured image acquisition unit 60 is transmitted to the medical image processing apparatus 11 . The medical image processing apparatus 11 includes a photographed image input unit 100, a reference image recording unit 110, a first landmark detection unit 120, a match calculation unit 130, a boundary line position estimation unit 140, a display control unit 150, and a reference image generation unit 200. , and a second central control unit 101 (see FIG. 2).

In the medical image processing apparatus 11, the program in the program memory is operated by the second central control unit 101 constituted by an image analysis processor, thereby causing the photographed image input unit 100, the reference image recording unit 110, the first The functions of the landmark detection unit 120, the matching degree calculation unit 130, the boundary line position estimation unit 140, the display control unit 150, and the reference image generation unit 200 are realized.

In the boundary line display mode, the captured image is sent to the captured image input unit 100 of the medical image processing apparatus 11 (see FIG. 2). In the boundary line display mode, the captured image input unit 100 transmits the captured image to the first landmark detection unit 120 . As shown in FIG. 7, the first landmark detection unit 120 detects landmarks that are characteristic structures of a subject from a captured image 121 captured in real time (in FIG. 7, the landmarks are surrounded by circles 122). ). Landmarks include folds 123 of the gastrointestinal tract, blood vessels 124, structures of glandular structures, as well as lesions 125 (indicated by hatching in FIG. 7), structures highlighted with a fluorescent agent, and the like. The fluorescent agent is, for example, ICG (Indocyanine Green). Hereinafter, in order to avoid complication of the drawing, any one of the circles 122 indicating landmarks will be given a lead line and a symbol unless otherwise specified.

After detecting the landmark, the first landmark detection unit 120 further acquires the position information of the landmark. As shown in FIG. 7, when a plurality of landmarks are detected from the captured image 121 by the detection process, it is preferable that the landmarks can be distinguished from each other. For example, each landmark can be given a distinguishing number for distinction. Also, as shown in FIG. 8, landmarks (represented by circles 122) are connected by link lines 126, the positional information of the landmarks are associated with each other, and the positional relationship of the landmarks in the captured image 121 is recorded. is preferred.

The photographed image in which the landmark is detected is transmitted to the degree-of-match calculation unit 130 . In addition to the photographed image in which the landmark is detected, the coincidence calculation unit 130 receives the boundary line 114 (FIG. 9) that is the boundary between the abnormal region 112 and the normal region 113 as shown in FIG. A reference image 111 is transmitted in which boundary line information associated with a boundary line (indicated by a dashed line in the figure) and landmark information associated with a landmark are associated with each other. The reference image is a medical image recorded in advance by the user in the reference image recording unit 110 and in which an accurate boundary line 114 is set in advance on the boundary between the abnormal region 112 and the normal region 113 . The boundary information includes boundary position information associated with the boundary. The landmark information includes landmark position information associated with the reference image and the positional relationship between the landmarks connected by the link line 126 (not shown in FIG. 9). The boundary line information and the landmark information are preferably associated by connecting the boundary line and each landmark with a link line 126 .

As shown in FIG. 10, the degree-of-match calculation unit 130 compares the reference image 111 and the captured image 121 to calculate the degree of match. The degree of matching is a value indicating how much the landmarks associated with the reference image 111 are included in the landmarks detected from the captured image 121 . In the example of FIG. 10, landmarks included in the captured image 121 (circles 122a, 122b, 122c, 122d, 122e, and 122f indicate locations) and landmarks included in the reference image 111 (circles 122a, 122b, 122c, 122d, 122f and 122g) are calculated. In this case, since the photographed image 121 includes landmarks other than one landmark (circle 122g) included in the reference image 111, the photographed image 121 includes five of the six landmarks included in the reference image. included. Therefore, in the example of FIG. 10, for example, the degree of matching is 83% (rounded to the nearest whole number).

As shown in FIG. 11 , when the degree of matching is equal to or greater than the threshold, the boundary line position estimation unit 140 calculates the positional information of the landmarks detected from the captured image, the positional relationship of each landmark, and the landmarks associated with the reference image. A correspondence relationship between the reference image and the captured image is estimated from the position information of the landmarks, the positional relationship of each landmark, and the positional information of the boundary line. The correspondence is transmitted to display control section 150 . Note that the threshold value of the degree of matching can be arbitrarily set. The correspondence relationship between the reference image and the captured image is defined by recognizing which landmark on the captured image each landmark on the reference image matches from the position information of the landmarks on the reference image and the captured image. From the positional relationship between the landmarks connected by the link lines 126 on the image and the positional relationship between the landmarks connected by the link lines 126 on the captured image, it is possible to determine which part of the captured image the boundary corresponds to. It is an estimated relationship.

Based on the correspondence relationship between the reference image 111 and the captured image 121, the display control unit 150 generates a superimposed image 151 in which the boundary line 114 associated with the reference image 111 is superimposed on the captured image as shown in FIG. The superimposed image 151 is a medical image obtained by superimposing the boundary line 114 associated with the reference image 111 on the captured image 121 captured in real time.

With the above configuration, the most probable boundary line can be confirmed in real time by superimposing the accurate boundary line associated with the reference image on the captured image captured in real time using the positional relationship of the landmarks. can be done. In addition, since it is possible to visually recognize how much the boundary line of the reference image differs from the real-time image, it is possible to compare the reference image, which is a past medical image, with the captured image, which is a current medical image. It is possible to confirm the expansion and contraction of the invasion range of the tumor that changed during the period.

The superimposed image is preferably created when the captured image 121 and the reference image 111 are captured at the same magnification as shown in FIG. When the photographed image 121 and the reference image 111 are obtained at the same magnification, the boundary line 114 can be superimposed accurately. In addition, even when the scene of the captured image changes due to ESD treatment, such as marking on mucous membranes or local injection of fluid, accurate boundary lines are displayed using the positional relationship of landmarks. be able to.

Furthermore, as shown in FIG. 13, it is preferable to create a superimposed image 151 when the captured image 121 is a medical image captured in a distant view and the reference image 111 is a medical image captured in a close view. In this case, the captured image 121 is captured at a lower magnification than the reference image 111 . By estimating the position where the boundary line is superimposed based on the positional relationship of the landmarks (indicated by circles 122) included in the reference image 111 that match the landmarks of the captured image 121, , the boundary 114 can be superimposed. When defining the boundary between the abnormal region and the normal region, it may be difficult to set the boundary line 114 unless the mucosal structure and microvessels 127 are observed in detail using a medical image photographed in the foreground. In the case where close-up photography is required to set the boundary line, it becomes more difficult to accurately determine the boundary line 114 in distant-view photography that allows a bird's-eye view of the entire lesion. In such a case, if there are landmarks that can be identified in both the medical image taken in the foreground and the medical image taken in the background, it is possible to acquire images in real time by taking a distant image at a distance where the boundary line cannot be distinguished. Also, it is possible to estimate the boundary line 114 by using the correspondence relationship between the boundary line and the landmark, and display the boundary line in the photographed image 121 photographed in the background.

Furthermore, as shown in FIG. 14, the captured image 121 is a medical image captured so as to include a part of the abnormal region 112 and structures such as microvessels 127, and the reference image 111 is the entire abnormal region 112. , it is preferable to create a superimposed image 151 in which a portion of the boundary associated with the reference image 111 is superimposed on the captured image 121 . In this case, the captured image 121 is captured at a higher magnification than the reference image 111 . In close-up photography, even a slight blur greatly affects the captured image 121 . By generating the superimposed image 151 when performing real-time close-up photography in which the influence of blurring is large, it is possible to present an accurate boundary line to a user such as an operator. In addition, it is possible to easily check the range of the tumor that has changed during the period from acquisition of the reference image 111 to acquisition of the photographed image 121 .

As shown in FIG. 15, the reference image is preferably generated as one reference image 111 by joining enlarged medical images 202 and 203, which are medical images in which a part of the abnormal region is photographed in the foreground. The magnified medical image includes a boundary line 114 and information associated with the boundary line 114, and landmarks (indicated by circles 122h, 122i, 122j, 122k, 122l, 122m and 122n) and information associated with the landmarks, respectively, They are associated by the associating unit 260 by a method to be described later. A magnified medical image is a still image captured under optimal conditions, such as a close-up shot at a magnification that allows the boundary between an abnormal area and a normal area to be distinguished, and no out-of-focus or halation. By generating a reference image by joining magnified medical images, a high-definition and accurate boundary line obtained in advance can be superimposed on the captured image.

An example of generating a reference image by joining enlarged medical images will be described. The magnified medical images 202 and 203 are joined together by inputting the magnified medical images 202 and 203 into the magnified medical image input unit 201 of the reference image generation unit 200 shown in FIG. Sent and done. The magnified medical image may be a magnified photographed image, which will be described later, or a medical image input from outside the endoscope system 10 . The reference image generation unit 200 includes an enlarged medical image input unit 201, an enlarged medical image combining unit 210, an abnormality identifying unit 220 (to be described later), a boundary line setting unit 230, a second landmark detecting unit 250, and an associating unit 260.

As shown in FIG. 17, the case where the reference image 111 is composed of a first enlarged medical image 204 and a second enlarged medical image 205 captured at a position different from the first enlarged medical image 204 is illustrated. In this case, first, the enlarged medical image input unit 201 transmits the first enlarged medical image 204 and the second enlarged medical image 205 to the enlarged medical image combining unit 210 . The magnified medical image combiner 210 combines landmark information (landmarks are indicated by circles 122h, 122i, 122j, 122k, and 122l) associated with the first magnified medical image 204 and boundary line information (the boundary line 114 is indicated by a dashed line). ) and landmark information associated with the second magnified medical image (landmarks are indicated by circles 122j, 122j, 122k, 122m and 122n) and boundary information (boundary 114 is indicated by a dashed line), A common relationship is recognized, and based on the common relationship, the first enlarged medical image 204 and the second enlarged medical image 205 are joined together to generate a reference image.

The common relationship refers to whether or not the landmark information (landmark) and boundary line information (boundary line) associated with the first enlarged medical image 204 and the second enlarged medical image 205 are common (coincident) with each other. Refers to a determined relationship. Landmark information includes landmark position information and landmark positional relationships. The boundary line information includes boundary position information.

In the specific example shown in FIG. 17, the reference image 111 is generated based on the common relationship between the landmark information related to the two landmarks indicated by circles 112i and 112j and the boundary line information. In order to combine the magnified medical images, it is preferable that at least two or more landmarks are common among the magnified medical images.

In the above specific example, the case where the reference image consists of the first enlarged medical image 204 and the second enlarged medical image 205 was illustrated. It can also be applied to the case of joining together the magnified medical images from 1 to N by recognizing the common relationship between each magnified medical image. With the above configuration, a reference image can be generated by joining a plurality of enlarged medical images associated with landmark information and boundary line information. As a result, it is possible to improve the accuracy of the boundary lines superimposed on the captured image based on the reference image in which the boundary lines are precisely set. Also, since the number of landmarks associated with one reference image increases, the correspondence relationship between the reference image and the captured image can be estimated with higher accuracy from the positional relationship of the landmarks.

Boundaries associated with the reference image are preferably set by automatically identifying abnormal and normal regions. In this case, the enlarged medical image is transmitted from the enlarged medical image input unit 201 to the abnormality identifying unit 220 (see FIG. 16). As shown in FIG. 18, the abnormality identification unit 220 identifies the abnormal region 112 and the normal region 113 in the enlarged medical image 206, the boundary line setting unit 230 (see FIG. 16) sets the boundary line 114, and Get the location information of the boundary. Next, the second landmark detection unit 250 (see FIG. 16) detects landmarks (indicated by circles 122) from the magnified medical image 206, and further acquires positional information of the landmarks and the positional relationship of the landmarks. The associating unit 260 (see FIG. 16) receives boundary line information (boundary line position information) from the boundary line setting unit 230, and further receives landmark information (landmark position information and positional relationship of landmarks). The association unit 260 associates the magnified medical image, boundary information, and landmark information, and uses the magnified medical image as a reference image. The reference image is transmitted from the reference image generation unit 200 to the reference image recording unit 110 and recorded. The magnified medical image associated with the boundary line information and the landmark information may be sent to the magnified medical image combiner 210 and used for merging with other magnified medical images. With the above configuration, the automatically set boundary line information can be used to superimpose the captured image.

The abnormality identifying unit 220 is preferably a trained model for identifying abnormal regions and normal regions that has been trained using medical image data for teachers in which abnormal regions and normal regions have been previously identified using machine learning. Information on abnormal regions and normal regions in the teacher medical image data may be added by a skilled doctor or automatically by a device other than the medical image processing device 11 . It is preferable to use deep learning for machine learning to generate the trained model, for example, it is preferable to use a multi-layer convolutional neural network. In addition to deep learning, machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using neural networks, Includes generative adversarial networks and the like.

Also, it is preferable that the boundary associated with the reference image is set by a user operation. In this case, as shown in FIG. 19 , the abnormality identification section 220 and the boundary line setting section 230 of the reference image generation section 200 may be used as the boundary line input section 240 . When the boundary line is set by user operation, a user such as a doctor determines the boundary between the abnormal region 112 and the normal region 113 and sets the boundary line. For example, as shown in FIG. 20, a boundary line 114 is drawn with a touch pen 242 on an enlarged medical image 206 displayed on a tablet 241, and a boundary line input instruction is given via a boundary line input button (not shown) or voice. Boundaries may be set by doing. When a boundary line input instruction is given, the position information of the boundary line is automatically obtained. With the above configuration, the boundary line identified by the doctor can be used to superimpose the captured image. Also in this case, the second landmark detection unit 250 detects landmarks in the enlarged medical image to acquire landmark information, and the association unit 260 receives the boundary line information from the boundary line input unit 240, It is preferable to receive the landmark information from the second landmark detection unit 250, associate the enlarged medical image, the boundary line information, and the landmark information, and use the enlarged medical image as the reference image.

As shown in FIG. 21, the reference image 111 is a medical image captured by illuminating the subject with special light, and the captured image 121 is a medical image captured by illuminating the subject with normal light. preferable. The special light is illumination light that makes blood vessels and glandular structures easier to see, and makes it easier to recognize the characteristic blood vessel arrangement and pit pattern that tumors have. Therefore, it is easy to determine the boundary between the abnormal region and the normal region in the medical image acquired using the special light. For the above reasons, it is preferable to use a medical image acquired using special light as a reference image. On the other hand, the normal light has a natural color tone, and the normal light is used to illuminate the subject when performing ESD treatment. Therefore, the captured image is preferably a medical image acquired using normal light. With the above configuration, the boundary line of the reference image in which the boundary between the abnormal region and the normal region is accurately determined using special light can be superimposed on the photographed image obtained using normal light.

When superimposing the boundary associated with the reference image on the captured image, the first landmark detection unit 120 superimposes the boundary on the captured image in which the landmark is detected according to the processing speed for detecting landmarks from the captured image. The captured images may or may not be the same. It is preferable that a processor having a high processing speed for detecting landmarks is used, and the photographed image for detecting the landmarks and the photographed image for superimposing the boundary line are photographed images photographed at the same time. A specific description will be given below. As illustrated in FIG. 22, when the group of captured images 160 acquired in chronological order is acquired in order of time t−2, time t−1, and time t, the captured image acquired at time t is taken as a photographed image A161. When the landmark detection processing speed is high, both the captured image 121 in which the landmark is detected and the captured image on which the boundary associated with the reference image 111 is superimposed become the captured image A161. In this case, a superimposed image in which accurate boundary lines are superimposed in real time can be obtained with high accuracy.

On the other hand, a processor whose processing speed for detecting landmarks is slow may be used, and the captured image for detecting landmarks and the captured image for superimposing boundaries may be captured images captured at different times. Specifically, as illustrated in FIG. 23, a captured image group 160 acquired in chronological order is acquired in order of time t−2, time t−1, time t, and time t+1. If so, the captured image acquired at time t is defined as captured image A161, and the captured image acquired at time t+1 is defined as captured image B162. When the landmark detection processing speed is low, the captured image 121 in which the landmark is detected becomes the captured image A161, and the captured image on which the boundary associated with the reference image 111 is superimposed becomes the captured image B162. . In this case, even if a low-speed processor is used, it is possible to obtain a superimposed image in which accurate boundary lines are superimposed almost in real time while ensuring a certain degree of accuracy.

Note that in FIGS. 21 and 22, one frame of captured image is acquired at time t−2, time t−1, time t, and time t+1, but multiple frames of captured image are acquired between each time. may have been For example, there may be a difference of several frames between time t and time t+1.

In the boundary line display mode, the calculation of the matching degree of the landmarks in the captured image and the reference image is continued until an end instruction is given. In the boundary line display mode, landmarks are sequentially detected and the degree of matching with the reference image is calculated for the photographed image acquired in real time. be. A series of operations for displaying the boundary line ends, for example, when marking or incising the mucous membrane in ESD is completed, or when an end instruction is given via the user interface 19 . In addition, even when the mode is changed by the mode changeover switch 12f, it is regarded as an end instruction, and the calculation of the degree of coincidence is ended.

A series of flow of the border display mode is shown in the flow chart of FIG. First, the captured image acquired by the captured image acquisition section 60 is input to the captured image input section 100 (ST101). Next, the first landmark detection unit 120 detects landmarks in the captured image (ST102), and the degree-of-match calculation unit 130 determines how the landmarks associated with the reference image match the landmarks detected from the captured image. A degree of coincidence, which is a value indicating how much is included, is calculated (ST103). Here, if the degree of matching is less than the threshold, the process shifts to the selection of whether or not to issue an end instruction. On the other hand, if the degree of matching is equal to or greater than the threshold (ST104), a correspondence relationship is estimated to which position in the captured image the boundary associated with the reference image corresponds to (ST105), and the boundary is captured. A superimposed image superimposed on the image is generated (ST106). When the generation of the superimposed image on which the boundary line is superimposed is to be terminated, the boundary line display mode is terminated by issuing an end instruction (ST07).

The display control unit 150 generates a display image 170 as shown in FIG. 25, displays the superimposed image 151 in the first display section 171 of the display image 170, and displays the reference image 111 in the second display section 172. It is preferable to Note that the display method of the first display section 171 and the second display section 172 is not limited to this. For example, they may be arranged vertically instead of horizontally. Also, the endoscope system 10 includes a first display (not shown) and a second display (not shown) different from the first display, so that the superimposed image 151 and the reference image 111 are displayed on different displays. may

A reference image generation mode for generating a reference image from a captured image will be described below. The method of generating a reference image from a magnified medical image has been described above (see FIG. 18). A reference image is generated from the magnified medical image as the magnified photographed image.

In the reference image generation mode, as shown in FIG. 26 , the captured image input unit 100 transmits the input medical image (captured image) to the enlarged medical image input unit 201 of the reference image generation unit 200 . The enlarged photographed image is preferably a still image acquired using the still image acquisition instruction switch 12h. The subsequent steps are the same as the method for generating a reference image from an enlarged medical image (see FIG. 18). This is explained below. The magnified medical image input unit 201 transmits the magnified captured image to the abnormality identification unit 220 . As shown in FIG. 27, the abnormality identification unit 220 identifies the abnormal region 112 and the normal region 113 in the enlarged photographed image 208, the boundary line setting unit 230 sets the boundary line 114, and acquires the position information of the boundary line. do. Next, the second landmark detection unit 250 detects landmarks (indicated by circles 122) from the magnified photographed image 208, and acquires the positional information of the landmarks and the positional relationship of the landmarks. The association unit 260 receives boundary line information (boundary line position information) from the boundary line setting unit 230, and further receives landmark information (landmark position information and landmark positional relationship) from the second landmark detection unit 250. ). The associating unit 260 associates the magnified captured image, the boundary line information, and the landmark information, and uses the magnified captured image 208 as a reference image. A reference image generated from the enlarged photographed image 208 is transmitted from the reference image generation unit 200 to the reference image recording unit 110 and recorded.

In addition, the enlarged photographed image 208 associated with the boundary line information and the landmark information is transmitted to the enlarged medical image combining unit 210, and another enlarged photographed image associated with the boundary line information and the landmark information is sent as a new reference image. May be used to combine images. With the above configuration, it is possible to generate a new reference image from the captured image, and use the boundary line of the newly generated reference image to superimpose on the captured image.

Hereinafter, a description will be given of a boundary line update mode in which a boundary line is set from a captured image and the boundary line is updated while a superimposed image in which a boundary line associated with a reference image is superimposed on a captured image is displayed. The boundary line update mode is a mode in which when the boundary line displayed in the superimposed image differs from the boundary line that the user who observes the superimposed image thinks, the boundary line can be updated according to the user's instruction. In this mode, for example, after performing an endoscopic examination once and obtaining a reference image, another endoscopic examination is performed, and when an image is obtained at the same site as the reference image, the tumor infiltration range is expanded and it is effective to obtain a new update boundary line.

In the boundary line update mode, first, in the same flow as in the boundary line display mode (see the flowchart in FIG. 23), a superimposed image is generated by superimposing the boundary line associated with the reference image on the captured image. At this time, an update instruction is transmitted to the medical image processing apparatus 11 via the user interface 19 at an arbitrary timing when the boundary line is to be updated. FIG. 28 is a block diagram showing functions within the medical image processing apparatus 11, and FIGS. 29 and 30 are explanatory diagrams of the boundary update mode using image diagrams. When an update instruction is input, as shown in FIG. 28, the photographed image input unit 100 changes the photographed image 121 (in FIG. 29, land The photographed image in which the mark is detected and the photographed image in which the boundary line is superimposed are assumed to be the same photographed image.) are transmitted to the enlarged medical image input unit 201 of the reference image generation unit 200 as the enlarged photographed image 209 . Next, as shown in FIG. 30, the abnormality identification unit 220 identifies the abnormal area 112 and the normal area 113 in the enlarged photographed image 209, and the boundary line setting unit 230 sets the updated boundary line 115 (indicated by a two-dot chain line). to get the location of the update boundary. Next, the display control unit 150 receives the position information of the updated boundary line from the boundary line setting unit 230, and in addition to the boundary line 114, superimposes the updated boundary line 115 on the captured image to generate an updated boundary line superimposed image 270. and display the updated boundary line superimposed image 270 on the display 17 .

When the user sees the updated boundary line superimposed image 270 and determines that the updated boundary line may be determined as a new boundary line to be associated with the captured image, the user issues an update determination instruction via the user interface 19 to medical image processing. Send to device 11 . When an update decision instruction is issued, the display control unit 150 superimposes the update boundary line 115 (indicated by a two-dot chain line) on the captured image 121 as a determined update boundary line 116 (indicated by a dotted line) as shown in FIG. The determined boundary line 114 is updated, and a determined updated boundary line superimposed image 271 is generated and displayed on the display 17 .

Further, when an update determination instruction is issued, an enlarged photographed image in which an update boundary line is set may be used as a reference image. When an update decision instruction is issued, the second landmark detection unit 250 may detect the landmark code from the magnified photographed image code, and acquire the positional information of the landmark and the positional relationship of the landmark. The association unit 260 receives the updated boundary line position information from the boundary line setting unit 230 and further receives the landmark information from the second landmark detection unit 250 . The associating unit 260 associates the magnified captured image, the updated boundary line information, and the landmark information, and uses the magnified captured image as a reference image. The reference image generated from the enlarged photographed image is transmitted from the reference image generation unit 200 to the reference image recording unit 110 and recorded.

Alternatively, the magnified photographed image associated with the boundary line information and the landmark information may be transmitted to the magnified medical image combining unit 210 and used for merging with other magnified photographed images. With the above configuration, a reference image can be newly generated from a captured image, and the updated boundary line of the newly generated reference image can be used as a boundary line to be superimposed on the captured image.

Although a flexible endoscope is used as the endoscope 12 in the above-described embodiment, the present invention is also suitable when a rigid endoscope (laparoscope/rigid scope) used for surgical operations or the like is used. When a flexible endoscope is used, a photographed image of the superficial mucosa of the observation target viewed from the lumen side of the hollow organ is acquired. When using a rigid endoscope, a photographed image of the observation target viewed from the serosal side is acquired.

When a rigid endoscope is used as an endoscope, as shown in FIG. A user interface 319 is provided. Illumination light from the light source device 314 enters the endoscope 312 through the light guide 323 and illuminates the observation target inside the abdominal cavity.

As shown in FIG. 33, the light receiving unit 330 includes a spectroscopic element 331 such as a dichroic mirror that disperses the light from the endoscope 312, and an imaging element 332 such as a CMOS image sensor that senses and images the divided light. etc., and outputs an image signal based on the reflected light from the observation target. The spectroscopic element 331 may be plural or singular, and may not be provided. At least one imaging element 332 is provided, and a plurality of imaging elements 332 may be provided.

In surgical operations using a rigid scope, the ICG fluorescence method is used to identify sentinel lymph nodes, evaluate blood flow, and determine the extent of resection. In the ICG fluorescence method, since the biological half-life of ICG is about 3 minutes, the period during which ICG can be visualized and observed after intravenous administration of ICG is limited, and repeated administration of ICG is required. rice field. On the other hand, when the present invention is used in a surgical operation using the endoscope 12 as a rigid scope, ICG is administered once, and the boundary line setting unit 230 sets the boundary of the observation target during a period during which the observation target can be evaluated, The second landmark detection unit 250 acquires the landmark information and generates the reference image, so that the boundary line can be reproduced in the captured image captured in real time without administering ICG thereafter.

In the present embodiment, an example in which the medical image processing apparatus 11 is connected to the endoscope system 10 has been described, but the present invention is not limited to this, and other devices such as an ultrasonic imaging apparatus, a radiographic apparatus, and the like have been described. A medical device may be used. In addition, part or all of the captured image acquisition unit 60 and/or the first central control unit 55 of the endoscope system 10 is, for example, an image processing device that communicates with the processor device 15 and cooperates with the endoscope system 10. can be provided. For example, it can be provided in a diagnosis support device that acquires an image captured by the endoscope 12 directly from the endoscope system 10 or indirectly from the PACS. In addition, the endoscope system 10 is connected to various inspection devices such as the first inspection device, the second inspection device, . Of these, part or all of the captured image acquisition unit 60 and/or the first central control unit 55 can be provided.

In this embodiment, a captured image acquisition unit 60, a captured image input unit 100, a reference image recording unit 110, a first landmark detection unit 120, a match calculation unit 130, a boundary line position estimation unit 140, a display control unit 150, and The hardware structure of a processing unit (processing unit) that executes various processes such as the reference image generation unit 200 is various processors as described below. Various processors include CPUs (Central Processing Units), FPGAs (Field Programmable Gate Arrays), etc., which are general-purpose processors that function as various processing units by executing software (programs). Programmable Logic Devices (PLDs), which are processors, dedicated electric circuits, which are processors having circuit configurations specifically designed to perform various types of processing, and the like.

One processing unit may be configured by one of these various processors, and may be configured by a combination of two or more processors of the same type or different types (for example, multiple FPGAs or a combination of a CPU and an FPGA). may Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units in one processor, first, as represented by computers such as clients and servers, one processor is configured by combining one or more CPUs and software, There is a form in which this processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined. The hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

10, 300 endoscope system 11, 311 medical image processing apparatus 12 endoscope (flexible endoscope)
12a insertion portion 12b operation portion 12c bending portion 12d tip portion 12e angle knob 12f observation mode changeover switch 12h still image acquisition instruction switch 12i zoom operation portion 12j forceps port 14, 314 light source device 15, 315 processor device 17, 317 display 19, 319 User interface 20 Light source 20a V-LED
20b B-LED
20c G-LED
20d R-LED
21 light source processor 22 optical path coupling section 23 light guide 30a illumination optical system 30b imaging optical system 31 illumination lens 41 objective lens 42 zoom lens 43 imaging sensor 44 imaging processor 55 first central control section 60 captured image acquisition section 100 captured image input Section 101 Second Central Control Section 110 Reference Image Recording Section 111 Reference Image 112 Abnormal Region 113 Normal Region 114 Boundary Line 115 Update Boundary Line 116 Update Decision Boundary Line 120 First Landmark Detection Section 121 Photographed Image 122, 122a, 122b, 122c , 122d, 122e, 122f, 122g, 122h, 122i, 122j, 122k, 122l, 122m, 122n Circle 123 Gastrointestinal folds 124, 127 Blood vessel 125 Lesion 126 Link line 130 Coincidence calculator 140 Boundary position estimator 150 Display Control unit 151 Superimposed image 160 Photographed image group 161 Photographed image A
162 Photographed image B
170 display image 171 first display section 172 second display section 200 reference image generator 201 enlarged medical image input section 202, 203, 206 enlarged medical image 204 first enlarged medical image 205 second enlarged medical image 208, 209 Enlarged photographed image 210 Enlarged medical image combining unit 220 Abnormality identification unit 230 Boundary line setting unit 240 Boundary line input unit 241 Tablet 242 Touch pen 250 Second landmark detection unit 260 Association unit 270 Updated boundary superimposed image 271 Determined updated boundary superimposed image 312 Endoscope (Rigid Endoscope)
330 light receiving unit 331 spectral element 332 imaging element

Claims (17)

A medical imaging device comprising a processor,
The processor
Acquire medical images of the subject with an endoscope,
Obtaining a reference image that is the medical image in which boundary line information associated with the boundary line that is the boundary between the abnormal region and the normal region is associated with landmark information that is associated with landmarks that are characteristic structures of the subject. death,
Acquiring a captured image that is the medical image captured in real time,
detecting the landmark from the captured image;
calculating a degree of matching between the landmarks included in the reference image and the landmarks included in the captured image;
estimating a correspondence relationship between the reference image and the captured image based on the degree of matching and the landmark information included in the reference image and the captured image;
A medical image processing apparatus that generates a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship. 2. The medical image processing apparatus according to claim 1, wherein the captured image and the reference image are the medical images captured at the same magnification. 2. The medical image processing apparatus according to claim 1, wherein the captured image is the medical image captured in a distant view, and the reference image is the medical image captured in a near view. The captured image is the medical image captured so as to include a portion of the abnormal region,
The reference image is the medical image including the entire abnormal region,
2. The medical image processing apparatus according to claim 1, wherein the superimposed image is generated by superimposing a portion of the boundary line associated with the reference image on the captured image. 5. The medical image processing apparatus according to any one of claims 1 to 4, wherein the reference image is generated by joining enlarged medical images, which are the medical images in which a part of the abnormal region is photographed in the foreground. The processor
When the reference image consists of a first magnified medical image and a second magnified medical image captured at a position different from the first magnified medical image,
Based on a common relationship between the landmark information and the boundary information associated with the first enlarged medical image and the landmark information and the boundary information associated with the second enlarged medical image, the first 6. The medical image processing apparatus according to claim 5, wherein the magnified medical image and the second magnified medical image are joined together to generate the reference image. The processor
identifying the abnormal region and the normal region;
The medical image processing apparatus according to any one of claims 1 to 6, wherein the boundary line is set. The medical image processing apparatus according to any one of claims 1 to 6, wherein the boundary line is set by a user's operation. the reference image is the medical image captured by illuminating the subject with special light;
The medical image processing apparatus according to any one of claims 1 to 8, wherein the captured image is the medical image captured by illuminating the subject with normal light. The medical image according to any one of claims 1 to 9, wherein the captured image for detecting the landmark and the captured image for superimposing the boundary line are the captured images captured at the same time. processing equipment. 10. The medical image processing according to any one of claims 1 to 9, wherein the captured image for detecting the landmark and the captured image for superimposing the boundary line are the captured images captured at different times. Device. 12. The medical image processing apparatus according to any one of claims 1 to 11, wherein the calculation of the matching degree is continued until an end instruction is given. The processor generates an image for display,
displaying the superimposed image in a first display section of the display image;
The medical image processing apparatus according to any one of claims 1 to 12, wherein control is performed to display the reference image in a second display section different from the first display section of the display image. The processor
Acquiring an enlarged photographed image as the photographed image,
identifying the abnormal region and the normal region included in the enlarged photographed image and setting the boundary line;
detecting the landmark from the enlarged photographed image;
14. The reference image according to any one of claims 1 to 13, wherein the magnified photographed image is associated with the boundary line information associated with the boundary line and the landmark information associated with the landmark, and is used as the reference image. Medical imaging equipment. The processor
generating the superimposed image;
Identifying the abnormal region and the normal region included in the captured image related to the superimposed image with the boundary line when there is an update instruction, and using the boundary line between the abnormal region and the normal region as the boundary line If you set an update boundary and
If there is an update decision instruction,
14. The medical image processing apparatus according to any one of claims 1 to 13, wherein the boundary superimposed on the captured image is updated using the update boundary as a determined update boundary. a step of acquiring a medical image of a subject photographed by an endoscope;
Obtaining a reference image that is the medical image in which boundary line information associated with the boundary line that is the boundary between the abnormal region and the normal region is associated with landmark information that is associated with landmarks that are characteristic structures of the subject. and
obtaining a captured image, which is the medical image captured in real time;
detecting the landmark from the captured image;
calculating a degree of matching between the landmarks included in the reference image and the landmarks included in the captured image;
estimating a correspondence relationship between the reference image and the captured image based on the degree of matching and the landmark information included in the reference image and the captured image;
and generating a superimposed image in which the boundary line associated with the reference image is superimposed on the captured image based on the correspondence relationship. An endoscope system comprising the medical image processing apparatus according to any one of claims 1 to 15 and an endoscope.

Images