JP5530406B2 - Method of operating endoscope insertion direction detection device and endoscope insertion direction detection device - Google Patents

Description

The present invention relates to a method for detecting an insertion direction of an endoscope, and more particularly to an operation method of an endoscope insertion direction detection device for observation and inspection of an object having a lumen structure such as a large intestine in the medical field and a pipe pipe in the industrial field. The present invention relates to an endoscope insertion direction detection device .

  In recent years, medical endoscopes that allow observation of internal organs of the body cavity by inserting a flexible elongated insertion part into the body cavity, and can perform various therapeutic treatments using a treatment tool inserted into the channel as necessary. Mirrors are widely used. Also in the industrial field, industrial endoscopes are used for observing the internal corrosion state of building piping and the like.

  In insertion of an endoscope in these various examinations, a doctor or a technician who is an operator determines an advancing direction while observing an endoscopic image.

  On the other hand, for example, the insertion procedure in the large intestine examination has a high degree of difficulty and requires a lot of skill. The causes include the complexity of the shape of the large intestine, the narrowness of the lumen, and their individual differences. can give. Moreover, since it is necessary to perform insertion reliably and carefully, it is very burdensome for an inexperienced doctor.

  Endoscope insertion is basically performed in the direction in which the lumen extends, but the lumen direction does not always exist in the field of view of the endoscopic device, and the bent part of the large intestine (sigmoid) In the case of proximity to the intestinal wall / fold (fold, etc.), it is necessary to determine the insertion direction based on the experience and knowledge of the doctor who is the operator.

  Under such conditions, doctors gain a lot of examination experience and determine the insertion direction using various judgment factors.

Japanese Patent No. 2710384 Japanese Patent No. 2680111

  However, unskilled doctors have little knowledge and experience about what information to use and how to make judgments, and it is necessary to pull back the endoscope to capture the lumen in the field of view again. This was a cause of delays in time and patient distress.

  Also, in the piping inspection in the industrial field, the burden on the operator during insertion has been increased due to complicated bending of the pipes constituting the piping.

  On the other hand, Japanese Patent No. 2710384 and Japanese Patent No. 2680111 have been disclosed as insertion direction detection methods, but these are mainly intended to detect a lumen existing in the image field of view, and the tube is viewed from within the field of view. The effect of detecting the insertion direction, for example, when the cavity was removed was not obtained.

The present invention relates to an operating method of an endoscope insertion direction detecting device capable of prompting an operator to perform an appropriate operation when detection of the insertion direction becomes difficult, such as when the mucosal surface is too close in a colonoscopy, and An object of the present invention is to provide an endoscope insertion direction detecting device .

An operation method of an endoscope insertion direction detection device according to one aspect of the present invention is an endoscope insertion direction detection device to which an endoscope image obtained by observing a subject having a lumen structure with an endoscope is input. a method of operating, the second processing unit of the endoscope insertion direction detecting device, determining whether the endoscope from a first endoscopic image is to observation target in excessively close state of the step, when it is determined that there is no the over-approaching state in the second step, the processing unit of the endoscope insertion direction detecting device, the movement of the lumen from the first endoscopic image Detecting a movement vector to be tracked .
An endoscope insertion direction detecting device according to an aspect of the present invention includes an input unit to which a first endoscope image obtained by observing a subject having a lumen structure with an endoscope is input, and the first From the first endoscopic image, it is determined whether or not the endoscope is in an over-proximity state with respect to the subject. And a processing unit that detects a movement vector that tracks the movement of the lumen.

According to the present invention, the operation of the endoscope insertion direction detection device that can prompt the operator to perform an appropriate operation when detection of the insertion direction becomes difficult, such as when the mucosal surface is too close in a colonoscopy A method and an endoscope insertion direction detection device can be provided.

The block diagram which shows the whole structure of the endoscope apparatus which concerns on the 1st Embodiment of this invention. Explanatory drawing for demonstrating insertion of the endoscope of FIG. The block diagram which shows the structure of the insertion direction detection apparatus of FIG. The figure explaining the display of the insertion direction in the display apparatus of the insertion direction detection apparatus of FIG. 1st explanatory drawing which shows the example of the endoscopic image used as the insertion direction judgment object in the insertion direction detection apparatus of FIG. 2nd explanatory drawing which shows the example of the endoscopic image used as the insertion direction judgment object in the insertion direction detection apparatus of FIG. The figure explaining the insertion state of the endoscope of FIG. The flowchart for demonstrating the insertion direction detection process in the insertion direction detection apparatus of FIG. Explanatory drawing for demonstrating the setting of the sampling pixel in the lumen direction detection in the process of FIG. The figure explaining the spatial differentiation process for the gradient vector calculation in the process of FIG. First explanatory view for explaining lumen direction detection based on the gradient vector in the process of FIG. FIG. 8 is a second explanatory diagram for explaining lumen direction detection based on the gradient vector in the process of FIG. Explanatory drawing for demonstrating the other setting of the sampling pixel in the process of FIG. The figure explaining the process using the area division in the process of FIG. Explanatory drawing for demonstrating the division | segmentation for associating the outermost periphery of an image with a luminal direction in the process of FIG. Explanatory drawing about vector projection in the processing of FIG. The figure explaining the halation which has the circular arc shape based on the 2nd Embodiment of this invention The flowchart for demonstrating the insertion direction detection process corresponding to the halation of FIG. The figure explaining the thinning process in the process of FIG. The figure explaining the sampling in the process of FIG. Explanatory drawing for demonstrating circular arc approximation with respect to the shape of halation in the process of FIG. Explanatory drawing for demonstrating the search range of the circular arc center for the approximation in the process of FIG. Explanatory drawing for demonstrating the determination method of the luminal direction in the process of FIG. The figure explaining the expansion process in the process of FIG. Explanatory drawing for demonstrating the processing content in case there exist two or more halations in the process of FIG. The flowchart for demonstrating the insertion direction detection process which concerns on the 3rd Embodiment of this invention. Explanatory drawing for demonstrating the determination method of the luminal direction in the process of FIG. Explanatory drawing for demonstrating circular arc shape determination in the process of FIG. The flowchart for demonstrating the insertion direction detection process which concerns on the 4th Embodiment of this invention. FIG. 29 is a first explanatory diagram for explaining a method for determining a lumen direction in the processing of FIG. FIG. 29 is a second explanatory diagram for explaining the method for determining the lumen direction in the process of FIG. 3rd explanatory drawing for demonstrating the determination method of the luminal direction in the process of FIG. 4th explanatory drawing for demonstrating the determination method of the luminal direction in the process of FIG. The flowchart for demonstrating the insertion direction detection process which concerns on the 5th Embodiment of this invention. Explanatory drawing for demonstrating the case where it determines with the lumen direction detection based on the movement vector in the process of FIG. 34 being unsuitable Explanatory drawing for demonstrating a result display when the insertion direction detection in the process of FIG. 34 is inappropriate or impossible.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following, the detection of the insertion direction with respect to the insertion of the endoscope into the large intestine will be described, but the same processing can be applied to a tubular insertion target such as a pipe in the industrial field. In general, a pipe or the like is a rigid body compared to a large intestine that is an organ in a body cavity, and since there is no movement of a target due to pulsation or the like, conditions for detecting an insertion direction are loose, and thus a good effect can be obtained.

  In the series of embodiments of the present invention, the main purpose is to present endoscope insertion auxiliary information to the operator based on the detection of the insertion direction. However, automatic insertion by combination with automation of endoscope operation is performed. Of course, the application of is possible. In this case, the endoscope automatic operation is controlled so as to proceed in the detected insertion direction.

First embodiment:
FIGS. 1 to 16 relate to the first embodiment of the present invention, FIG. 1 is a block diagram showing the overall configuration of the endoscope apparatus, and FIG. 2 is an explanation for explaining insertion of the endoscope of FIG. 3 is a configuration diagram showing the configuration of the insertion direction detection device of FIG. 1, FIG. 4 is a diagram for explaining the display of the insertion direction in the display device of the insertion direction detection device of FIG. 3, and FIG. 5 is the insertion direction of FIG. FIG. 6 is a first explanatory diagram illustrating an example of an endoscopic image that is an insertion direction determination target in the detection device, and FIG. 6 is a second description that illustrates an example of an endoscopic image that is an insertion direction determination target in the insertion direction detection device of FIG. 7 is a diagram for explaining the insertion state of the endoscope of FIG. 1, FIG. 8 is a flowchart for explaining the insertion direction detection processing in the insertion direction detection device of FIG. 3, and FIG. 9 is the processing in FIG. FIG. 10 is an explanatory diagram for explaining the setting of the sampling pixel in the lumen direction detection. FIG. 11 is a diagram for explaining the spatial differentiation process for calculating the gradient vector in the processing, FIG. 11 is a first explanatory diagram for explaining the lumen direction detection based on the gradient vector in the processing of FIG. 8, and FIG. FIG. 13 is an explanatory diagram for explaining another setting of the sampling pixel in the processing of FIG. 8, and FIG. 14 is an explanatory diagram for explaining other settings of the sampling pixel in the processing of FIG. FIG. 15 is a diagram for explaining processing using region division in the processing of FIG. 8, FIG. 15 is an explanatory diagram for explaining division for associating the outermost image periphery with the lumen direction in the processing of FIG. 8, and FIG. It is explanatory drawing regarding the projection of the vector in a process.

  In the first embodiment of the present invention, an insertion direction detection method for determining a lumen direction based on the gradient direction of light and dark when the lumen deviates from the image field of view, and an operation based on the processing result of the method An insertion direction detecting device that can realize a smooth endoscopic examination by presenting insertion direction information to a person will be described.

  First, the endoscope apparatus according to the present embodiment will be described with reference to FIG. In FIG. 1, an endoscope apparatus according to the present embodiment includes an endoscope 1, a control device 6, an observation monitor 9, an insertion direction detection device 12 that detects and displays an insertion direction, and an A / D conversion. A container 11 is provided.

  The endoscope 1 includes an operation unit 2 and a flexible insertion unit 3, and includes a solid-state imaging device (CCD) (not shown) and a light guide end that emits irradiation light at the distal end of the insertion unit 3. It is connected to the guide control device 8 via the connector 5.

  The endoscope 1 also includes a fiber for transmitting the irradiation light and a universal cord 4 for transmitting and receiving video signals and various control information. By inserting the insertion portion 3 into the large intestine 13, an image inside the body cavity is provided. get information.

  Further, the control device 6 includes a light source unit 7 for generating irradiation light for the endoscope 1 and a video signal processing circuit 8 for performing signal processing on a video signal input from the endoscope 1. Yes.

  The image information in the body cavity visualized by the control device 6 is input as an analog RGB signal to the observation monitor 9, digitized via the A / D converter 11, and input to the insertion direction detection device 12. .

  Next, insertion of an endoscope will be described with reference to FIG. In the large intestine examination by the endoscope 1, as shown in FIG. 2, by inserting the elongated and flexible insertion portion 3 of the endoscope 1 into the large intestine 13, the inside of the body cavity is observed. Yes. In insertion, a doctor who is an operator uses an angle knob (not shown) provided in the operation unit 2 (an operation unit for bending the endoscope tip up and down, left and right via a wire or the like built in the insertion unit 3). Using the technique of bending the tip, pushing and pulling the insertion part 3 and twisting, observation from the anus to the deepest part of the ileocecum (the part where the small and large intestines are connected) is performed.

  Next, the configuration of the insertion direction detection device 12 in the present embodiment will be described with reference to FIG.

  As shown in FIG. 3, the insertion direction detection device 12 includes a computer 20 for executing a series of processes related to insertion direction detection on the RGB image signal input from the A / D converter 11, and an insertion direction detection result. And a display device 21 for displaying.

  Further, the computer 20 stores a storage device 25 that stores a main program 26 for detecting an insertion direction, a CPU (central processing unit) 23 and a main memory 24 for executing an insertion direction detection process using the main program 26. , An A / D converter 11, a storage device 25, and an I / O control circuit 22 for controlling each input / output between the display device 21 and the CPU 23.

  The main program 26 is a program for executing a series of processing accompanying detection of the insertion direction in the present embodiment, an acquisition request for an image signal from the A / D converter 11 to the I / O control circuit 22, and the display device 21. Insertion direction detection processing result display request and the like are made.

  In the present embodiment, the endoscope image is quantized by the A / D converter 11 into 8 bits in which each RGB plane takes a value from 0 to 255, and the image size is respectively horizontal and vertical. Assume ISX and ISY. In the following description, the positions of pixels constituting the endoscopic image are displayed based on a coordinate system in which the upper left end point of the image is (0, 0) and the lower right end point is (ISX-1, ISY-1). .

  Next, a display example of the insertion direction detection result on the display device 21 of the insertion direction detection device 12 will be described.

  For a hollow organ such as the large intestine, an endoscope is inserted in the direction in which the lumen extends. In such a case, it is necessary to insert it securely and safely while always capturing the lumen in the field of view of the image. However, there are many cases where the lumen is out of view for various reasons described later, and it is often difficult for inexperienced doctors to determine in which direction the lumen exists. Occur. In such a case, it is necessary to retract the endoscope 1 once so that the lumen enters the visual field (extraction operation: hereinafter referred to as “Pull Back operation”), which causes a delay in examination time.

  The insertion direction detection device 12 in the present embodiment detects the insertion direction by applying an image processing method when there is no clear lumen in the endoscopic image, and provides the insertion direction information to the doctor performing the examination. By presenting, smooth endoscopy is realized.

  As specific insertion direction information, as shown in FIG. 4, any arrow in 8 directions at 45 ° intervals is superimposed and displayed on the endoscopic image. Doctors who perform endoscopy can grasp the lumen in the visual field by performing operations such as angle operation, push-pull, and twisting so that the endoscope advances in the direction of the superimposed arrow. Become.

  In the present embodiment, for convenience of explanation, the insertion direction is any one of the eight directions, but it is possible to display it in more detail than 16 directions at 22.5 °, etc. It can be set as appropriate depending on the skill level and necessity.

  Next, an insertion direction detection method in the present embodiment will be described.

  The insertion procedure in the large intestine examination is very difficult and requires a lot of skill. The causes include the complexity of the shape of the large intestine, the narrowness of the lumen, and the individual differences between them. Moreover, since it is necessary to perform insertion reliably and carefully, it is very burdensome for an inexperienced doctor. Endoscope insertion is basically performed in the direction in which the lumen extends, but the lumen direction does not always exist in the field of view of the endoscopic device, and the bent part of the large intestine (sigmoid) When the endoscope is too close to the intestinal wall / fold (fold, etc.), it is necessary to determine the insertion direction based on the experience and knowledge of the doctor who is the operator. Under such conditions, doctors gain a lot of examination experience and determine the insertion direction using various judgment factors.

  Specifically, for example, in the endoscopic image shown in FIG. 5, it can be determined that the insertion is continued in the straight direction from the clear lumen existing in the visual field. On the other hand, in the endoscopic image shown in FIG. 6, since there is no lumen in the field of view, the insertion direction, that is, the direction in which the lumen exists is determined based on some information.

  In the present embodiment, a direction detection method using a light / dark gradient (referred to as a gradient) direction will be described as a technique for determining an insertion direction when a lumen does not exist in the visual field.

  One of the factors for determining the insertion direction when there is no lumen in the field of view is the light / dark change direction in the image. For example, when the positional relationship between the large intestine 13 and the distal end of the insertion portion 3 of the endoscope is as shown in FIG. 7, a global change in brightness occurs from a position closer to the distal end of the endoscope to a position farther away. Since the insertion direction is a direction far from the distal end of the endoscope, it is possible to detect the insertion direction based on detection of the change direction from the bright part to the dark part in the image.

  FIG. 8 shows a flow of a series of processes in the insertion direction detection of the present embodiment. Here, processing is applied to each frame of the endoscopic image signal input via the A / D converter 11.

  In step S1 of FIG. 8, an R image is acquired from among the RGB images of the input endoscopic image. In this embodiment, processing using an R image will be described. However, similar processing can be applied using a G, B image, a luminance image (= 0.3R + 0.6G + 0.1B), or the like.

  In subsequent step S2, M sampling pixels (M is an integer of 1 or more) are set on the R image. Sampling pixels are defined to sample the entire image. An example of setting sampling pixels is shown in FIG.

  In step S3, a gradient vector for obtaining the light / dark gradient direction of each sampling pixel is calculated. In the present embodiment, spatial differentiation processing is used as a gradient vector calculation method. FIG. 10 is an explanatory diagram for explaining a gradient vector calculation method using spatial differentiation processing.

  First, as shown in FIG. 10, neighboring pixels of size N × N (N = 5 in FIG. 10) centering on the sampling pixel P are extracted, and pixels located at the horizontal, vertical, and diagonal ends are A , B, C, D, E, F, G, H. From these pixel values, horizontal and horizontal spatial differential values SX and SY are obtained as follows.

[Equation 1]
SX = (C + E + H)-(A + D + F) (1)
[Equation 2]
SY = (F + G + H)-(A + B + C) (2)
In equations (1) and (2), A, B, C, D, E, F, G, and H represent the density values of each pixel.

Using the obtained SX and SY, the gradient vector V (the underline indicates that V is a vector) is
V = (SX, SY) (3)
It is expressed.

  Further, the magnitude | V | of V and the gradient direction Vθ can be calculated as follows.

[Equation 4]
[Equation 5]
Vθ = tan-1 (SY / SX) (5)
The gradient vector V described above is calculated as Vi (i = 1, 2,..., M) for M sampling pixels.

  Subsequently, the lumen direction is detected in step S4. The processing content of the lumen direction detection in the present embodiment will be described with reference to FIGS.

The points on the outer periphery of the image corresponding to the eight directions described with reference to FIG. Assuming that the coordinates of Qk are (qx, qy) and the coordinates of the sampling pixel Pi are (px, py), the vector Qk connecting Qk and Pi is [Equation 6]
Qk = (qx-px, qy-py) (6)
Is obtained.

  Since the lumen exists in a direction that changes from light to dark in the image, the reverse direction vector V′θi of the light / dark gradient Vi of the sampling pixel Pi (the gradient direction calculated by the equation (5) indicates the direction from dark to bright). Qk closest to (inverted) is the lumen direction.

  Specifically, φik shown in FIG. 12 is obtained for each Qk from Equation (7). Here, φik is an angle formed by V′θi and Qk.

[Equation 7]
φik = cos−1 {(Qk · V′θi) / (| Qk | × | V′θi |)} (7)
However, “·” indicates an inner product of vectors, and “||” indicates the size of the vector.

  φik takes a value of −180 <φik ≦ + 180, and approaches 0 when the lumen exists in the direction of the point Qk, that is, when the gradient direction is close to the direction of Qk (the unit is “degree”).

  Therefore, by calculating φik for all sampling pixels in the lumen direction and obtaining k that minimizes the following error evaluation value, the most reliable direction in which the light / dark gradient changes from light to dark can be obtained.

[Equation 8]
In step S5, the direction corresponding to the obtained lumen direction is set as the insertion direction, and the arrow information shown in FIG. 4 is superimposed on the image, displayed on the display device 21, and then returned to step S1 to the next frame. Repeat a series of processes.

  In the present embodiment, N = 5 is determined for N that determines the size of the neighborhood shown in FIG. 10, but a global light / dark gradient is detected using a larger value (for example, N = 17). It is also possible to do. Furthermore, prior to the calculation of the gradient vector from each sample pixel, by applying reduced pass filtering (so-called blur mask) as pre-processing, the influence of mucous membrane structures such as noise and blood vessels is eliminated, and the insertion direction detection accuracy is improved. Can be planned.

  In the error evaluation, the total sum of the angles φik shown in the equation (8) is used. However, it is possible to replace the function with another function related to φik.

  Further, threshold processing for removing the influence of halation and preventing erroneous detection can be performed on the sample pixel P and the neighboring pixels A to H for calculating the gradient vector in FIG. Specifically, for each pixel included in a local region of size N × N including A to H and P, a comparison is made with a threshold value THL (for example, THL = 250), and any pixel is If it exceeds THL, the position of the sample pixel P is changed, or it is not used for insertion direction detection.

  When the magnitude | Vi | = 0, it indicates that there is no light / dark gradient, so it is not used for detecting the insertion direction. Specifically, for the corresponding Vi, instead of the equation (7), φik may be set to φi1 = φi2 =... = ΦiK = 0 for all k (sampling pixels of size | Vi | = 0) Does not contribute to the result of equation (8)).

In the lumen direction detection in step S4, the direction (angle) Vθ of the gradient vector V of the light / dark change is used, but in the endoscopic image, there is a probability that the lumen exists in the direction in which the light / dark change is large. Since it is high, it is possible to use the size | V |. Specifically, the maximum value max | Vi | is obtained for Vi (1 ≦ i ≦ M) calculated for M sampling images. Then, for all Vis, [Equation 9]
αi = | Vi | / max | Vi | (9)
The normalization represented by

Αi in equation (9) is αi = 1 for the i-th gradient vector Vi giving max | Vi |, and takes the value of 0 <αi ≦ 1 in the other cases (where | Vi | ≠ 0). In order to use αi as a weighting coefficient, Equation (8) is expressed as [Equation 10].
Replace.

  Further, with respect to the sampling pixels, the example in which the entire image is set equally as shown in FIG. 9 has been described. However, as shown in FIG. 13, for example, as shown in FIG. You may set only to the image peripheral part which contains more.

  Further, as shown in FIG. 14, the image may be divided into regions, and the insertion direction may be detected based on the following processing for the direction corresponding to each region. In FIG. 14, sampling pixels are set in the divided areas at the periphery of the image. In these divided areas, one area corresponds to one direction based on the settings shown in FIGS. 11 and 15 (except for the central part of the image).

  In FIG. 15, U1 to U8 are points on the outermost periphery of the image set based on three equal divisions of the lengths in the horizontal and vertical directions, and U1-U3, U1-U2, U2-U4, U3-U5, U4. The sections on the outermost periphery determined by the combination of -U6, U5-U7, U7-U8 and U6-U8 correspond to the directions indicated by Q1, Q2, ..., Q7 and Q8 in Fig. 11, respectively.

In each region, a region having the maximum magnitude of change in brightness in the corresponding direction is detected, and the corresponding direction is determined as the insertion direction. Specifically, for Mk sampling pixels in the region k (1 ≦ k ≦ K), the size of projection in the direction toward the point Qk corresponding to each region k of the direction vector (an example is shown in FIG. 16). Sum of [Equation 11]
Is calculated.

  Ξ (k) is obtained for all regions, and the direction corresponding to k giving the maximum ξ (k) is set as the insertion direction.

  As described above, according to the insertion direction detection method in the first embodiment of the present invention and the insertion direction detection device that presents the insertion direction information to the operator based on the processing result of the method, the endoscopy is performed. Therefore, it is possible to provide insertion assistance information to an operator who is not skilled in the field, and to realize a smooth endoscopy.

Second embodiment:
FIGS. 17 to 25 relate to the second embodiment of the present invention, FIG. 17 is a diagram for explaining halation having an arc shape, and FIG. 18 is a diagram for explaining insertion direction detection processing corresponding to the halation of FIG. FIG. 19 is a diagram for explaining thinning processing in the processing of FIG. 18, FIG. 20 is a diagram for explaining sampling in the processing of FIG. 18, and FIG. 21 is an arc approximation for the shape of halation in the processing of FIG. FIG. 22 is an explanatory diagram for explaining the search range of the arc center for approximation in the processing of FIG. 18, and FIG. 23 is a method for determining the lumen direction in the processing of FIG. FIG. 24 is a diagram for explaining the expansion processing in the processing of FIG. 18, and FIG. 25 is a diagram for explaining the processing contents when there are a plurality of halation in the processing of FIG.

  Since the second embodiment is almost the same as the first embodiment, only different points will be described, and the same components are denoted by the same reference numerals and description thereof will be omitted.

  In the second embodiment of the present invention, an insertion direction detecting method for determining a lumen direction using a shape feature of halation generated by reflection of a mucosal surface or the like when the lumen deviates from the image visual field. An insertion direction detection device that can realize a smooth endoscopic examination by presenting insertion direction information to the operator based on the processing result of the technique will be described.

  In endoscopy, due to strong specular reflection of the mucosal surface facing the tip of the endoscope, a phenomenon occurs in which the output signal from the CCD becomes saturated or becomes significantly larger than the surroundings. Are known. In an endoscopic observation image on a large intestine or the like having a lumen shape, such a halation may form an arc shape.

  For example, in the inserted state shown in FIG. 7, arc-shaped halation as shown in FIG. 17 occurs. A skilled doctor uses arc-shaped halation as auxiliary information for insertion because the center direction of the arc coincides with the lumen direction, and in the example of FIG. 17, the insertion proceeds in the right direction of the image field.

  In the present embodiment, by applying the image processing method, it is determined whether or not the halation generated in the image forms an arc shape, and if it is formed, the center direction of the arc is estimated, This is insertion direction information.

  Since the configurations of the endoscope apparatus and the insertion direction detecting apparatus in the present embodiment are the same as those in the first embodiment and the contents of the main program 26 are different, the image processing contents executed in the main program 26 are the same. Details will be described.

  FIG. 18 is a flowchart for explaining the insertion direction detection processing in the main program 26 of the present embodiment. In step S11, as in step S1 shown in FIG. 8 in the description of the first embodiment, an R image is acquired from the input RGB images of the endoscopic image. Similar processing can be applied using G, B images, luminance images, or the like.

In subsequent step S12, halation pixels are extracted by a binarization process using a threshold value for each surface element of the input R image. Specifically, the binary image H is created based on the pixel value r (x, y) at the coordinates (x, y) (0 ≦ x <ISX, 0 ≦ y <ISY). The value of each surface element h (x, y) in H is [Equation 12].
h (x, y) = 1 if r (x, y) ≧ THL
h (x, y) = 0 if r (x, y) <THL (12)
Determined by

  Here, the setting of the threshold THL is appropriately changed such that THL = 255 or THL = 240 with a slight margin. This corresponds to the fact that even if the pixel value does not reach the maximum value of 255, it may be visually recognized as halation.

  Next, in step S13, since the halation does not always form an arc shape, it is necessary to determine whether the halation can be used for detecting the insertion direction. Therefore, the halation arc using the binary image H is used. Perform shape determination processing.

  In the present embodiment, a circle for approximating the extracted halation is determined exploratively by changing the center and radius as parameters, and the shape of the halation to be processed and the arc defined by the equation are determined. Evaluate the error.

  First, as a pre-processing for the halation image H, a known thinning process (addition of a degeneration process when a small branch or the like occurs) is applied. FIG. 19 shows an example of a thinning process result for halation (black corresponds to pixel value 1 in FIG. 19).

  The degeneration processing is also disclosed in, for example, “Introduction to Computer Image Processing P.75-P83, Supervised by Hideyuki Tamura, edited by Japan Industrial Technology Center, published by Soken Publishing Co., Ltd., released by Nebula.”

  Subsequently, sampling is performed for h (x, y) constituting the halation after thinning. Sampling is performed by determining NS h (x, y) so as to be substantially equidistant after determining both pixels of the halation and the middle pixel in the number of extended pixels. Hereinafter, the extracted Ns h (x, y) will be described as new hj (xj, yj), 1 ≦ j ≦ Ns. FIG. 20 shows an example of sampling. This sampling is for shortening the calculation time, and can be omitted if the performance of the CPU 23 is sufficient.

  Next, for the pixel hj (xj, yj) extracted by thinning and sampling, it is determined whether or not the halation forms an arc shape by the following series of processing.

First, the search range of the center and radius of the circle for approximation is set. As shown in FIG. 21, a triangle PQR composed of h1 (x1, y1), hNs / 2 (xNs / 2, yNs / 2) and hNs (xNs, yNs) is defined, and a half orthogonal from Q to the side PR Find the straight line τ. Next, as shown in FIG. 22, points S1 and S2 where PS1 = QS1 and RS2 = QS2 on τ are obtained. A center point S (coordinates are (xs, ys)) of S1 and S2, and a circle within a radius r = s / 2 (where s is the length of S1S2) is defined as a center search range. K center candidate points Ck are set (1 ≦ K, 1 ≦ k ≦ K). The radius rk of the circle centered on the center candidate point Ck is determined based on Ck, and rk = CkQ. If the coordinates of Ck are (cxk, cyk), the error evaluation value of Ns hj (xj, yj) for the circle Ck is
[Equation 13]
Can be obtained.

K pieces of ε (k) are calculated based on the equation (13), and the minimum value min (ε (k)) is compared with a threshold value THE for determining whether or not the arc shape can be recognized. ,
[Formula 14]
min (ε (k)) <THE (14)
If it is, it will determine with it being a circular arc shape.

  In step S14, if the expression (14) is satisfied, the process proceeds to step S15. If not satisfied, the process returns to step S11 to perform branch determination for executing processing for the next frame.

  In step S15, the lumen direction is determined. Since the lumen direction exists in the center direction of the arc approximating the halation shape, a half line τk connecting the center Ck of the circle Ck that gives min (ε (k)) and the point Q in FIG. 21 is as shown in FIG. (Since the straight line equation can be easily obtained from the coordinates of two points, detailed description is omitted), and the intersection point T with the outermost periphery of the image is obtained. For the four outermost sides of the image, the horizontal and vertical lengths are respectively divided into three equal parts (ISX / 3, ISY / 3) as shown in FIG. 15, and are associated with the eight directions shown in FIG. The insertion direction is determined based on where the intersection point T is located. In FIG. 23, the intersection T is in the lower left direction (the direction of Q6 in FIG. 11).

  In step S16, as in step S5 in the first embodiment, the obtained arrow information indicating the entry direction is superimposed on the image, displayed on the display device 21, and then returned to step S11 to return to the next frame. Repeat a series of processes.

  As described above, according to the insertion direction detection method in the second embodiment of the present invention and the insertion direction detection device that presents the insertion direction information to the operator based on the processing result of the method, the endoscopy is performed. Therefore, it is possible to provide insertion assistance information to an operator who is not skilled in the field, and to realize a smooth endoscopy.

  Further, since it is determined whether or not the shape of the halation is appropriate for detecting the lumen direction, erroneous insertion direction auxiliary information is not displayed.

    In addition, as shown in FIG. 24A, the binarized image H of halation may have a complicated outer peripheral shape, which causes many unnecessary branches and the like after the thinning process is applied. In such a case, by applying a known expansion process, the thinning process is applied after smoothing the outer peripheral shape as shown in FIG. 24B (for the expansion process and the thinning process, for example, “Computer (Introduction to image processing, P.75-P83, supervised by Hideyuki Tamura, edited by Japan Industrial Technology Center, published by Soken Publishing Co., Ltd.)

  Further, in the present embodiment, the arc center Ck is determined in an exploratory manner, but it is also possible to apply a shape extraction method such as Hough transformation.

  Also, as shown in FIG. 25A, when there are a plurality of halation in the image, it is assumed that the end points hk and hk + 1 of each halation after thinning are connected as shown in FIG. 25B. A series of processing shown in this embodiment mode may be applied. For determining whether or not connection is possible, for example, based on the coordinates of the end point hk, whether hk + 1 exists within a circular region having a radius rhk (rhk is appropriately determined, for example, rhk = 20) centered on hk. Can be determined.

Third embodiment:
26 to 28 relate to the third embodiment of the present invention, FIG. 26 is a flowchart for explaining the insertion direction detecting process, and FIG. 27 is a method for determining the lumen direction in the process of FIG. FIG. 28 is an explanatory diagram for explaining arc shape determination in the process of FIG.

  Since the third embodiment is almost the same as the first embodiment, only different points will be described, and the same components are denoted by the same reference numerals and description thereof will be omitted.

  In the third embodiment of the present invention, when the lumen deviates from the image field of view, etc., the insertion direction detection using the shape feature of the halation shown in the second embodiment of the present invention is different. A method and an insertion direction detection device capable of realizing a smooth endoscopic examination by presenting insertion direction information to an operator based on the processing result of the method will be described.

  Since the configurations of the endoscope apparatus and the insertion direction detecting apparatus in the present embodiment are the same as those in the first embodiment and the contents of the main program 26 are different, the image processing contents executed in the main program 26 are the same. Details will be described.

  FIG. 26 is a flowchart for explaining the insertion direction detection processing in the main program 26 of the present embodiment. In step S21, as in step S1 shown in FIG. 8 in the description of the first embodiment, an R image is acquired from the input RGB images of the endoscopic image. Similar processing can be applied using G, B images, luminance images, or the like.

  In step S22, a halation pixel is extracted and a binary image H is created by applying the same processing as in step S12 shown in the second embodiment.

  In the subsequent step S23, similarly to step S13 in the second embodiment, h (x, y) is generated and sampled by applying the thinning process to the halation image H described with reference to FIGS. Do. For sampling, Ns h (x, y) are extracted so that the end points of the halation and the substantially equal intervals are obtained. Hereinafter, the extracted Ns h (x, y) will be described as hj (xj, yj), 1 ≦ j ≦ Ns.

  In the present embodiment, Ns-1 pixel groups (h1 (x1, y1), h2 (x2, y2)), (h2 (x2, y2), h3) from the extracted hj (xj, yj). (X3, y3)), ..., (hNs-2 (xNs-2, yNs-2), hNs-1 (xNs-1, yNs-1)), (hNs-1 (xNs-1, yNs-1) and hNs (xNs, yNs)) are determined to obtain a perpendicular bisector L (k) with respect to the line segment connecting each of them. Here, 1 ≦ k ≦ Ns−1.

FIG. 27 shows an example of calculating L (k) and lumen direction candidates. The obtained L (k) intersects at the center C of the arc when hj (xj, yj) is correctly present on the arc. Therefore, a lumen direction candidate can be determined by associating the position of C with the eight directions shown in FIG. Further, when L (k) does not intersect at one point, the center of gravity CR of the pixel in the closed region R constituted by a plurality of intersections is obtained as shown in FIG. The coordinates (Xc, Yc) of the center of gravity CR are [Equation 14].
It becomes.

  Here, Nr is the number of pixels in the region R, and xri and yri are the coordinates in the horizontal and vertical directions of the i-th pixel in R.

  In a succeeding step S24, it is determined whether or not the obtained lumen direction candidate is reliable, that is, whether or not the halation forms an arc shape. Specifically, the area (number of pixels) Ra of the closed region R shown in FIG. 28 is compared with a threshold value THR, and if Ra ≦ THR, it is determined that the arc shape is reliable. If Ra> THR, it is determined that the arc shape is not reliable because it is not an arc shape.

  In step S25, if the lumen direction candidate is reliable as a result of the determination in step S24, the process proceeds to step S26, and if not, the process returns to step S21 to perform branch determination for executing processing for the next frame.

  In step S26, as in step S5 in the first embodiment, the obtained arrow information indicating the entry direction is superimposed on the image, displayed on the display device 21, and then returned to step S21 to return to the next frame. Repeat a series of processes.

  As described above, according to the insertion direction detection method in the third embodiment of the present invention and the insertion direction detection device that presents the insertion direction information to the operator based on the processing result of the method, the endoscopy is performed. Therefore, it is possible to provide insertion assistance information to an operator who is not skilled in the field, and to realize a smooth endoscopy. Further, since it is determined whether or not the shape of the halation is appropriate for detecting the lumen direction, erroneous insertion direction auxiliary information is not displayed.

  In the above description, the processing based on the intersection coordinates of L (k) is applied to the estimation of the lumen direction candidate in step S23. However, as shown in FIG. ), And the direction that intersects with the most L (k) can be determined as the lumen direction candidate.

Fourth embodiment:
FIGS. 29 to 33 relate to the fourth embodiment of the present invention, FIG. 29 is a flowchart for explaining the insertion direction detecting process, and FIG. 30 is a method for determining the lumen direction in the process of FIG. FIG. 31 is a second explanatory diagram for explaining the method for determining the lumen direction in the process of FIG. 29, and FIG. 32 is a method for determining the lumen direction in the process of FIG. FIG. 33 is a fourth explanatory diagram for explaining the method for determining the lumen direction in the process of FIG. 29. FIG.

  Since the fourth embodiment is almost the same as the first embodiment, only different points will be described, and the same components are denoted by the same reference numerals and description thereof will be omitted.

  In the fourth embodiment of the present invention, the lumen is based on the time-series movement state of the visual field from the state in which the lumen is present in the visual field, such as when the lumen is out of the image visual field. An insertion direction detection method for determining a direction and an insertion direction detection device that can realize a smooth endoscopic examination by presenting insertion direction information to an operator based on the processing result of the method will be described.

FIG. 29 is a flowchart for explaining the insertion direction detection processing in the main program 26 of the present embodiment. In step S31, as in step S1 shown in FIG. 8 in the description of the first embodiment, an R image is acquired from the input RGB images of the endoscopic image. Similar processing can be applied using G, B images, luminance images, or the like. In step S32, in order to detect a lumen existing in the field of view, dark pixel extraction is performed by binarization using threshold processing THD. Specifically, the binary image D is created based on the pixel value r (x, y) at the coordinates (x, y) (0 ≦ x <ISX, 0 ≦ y <ISY). The value of each pixel d (x, y) in D is [Equation 15]
d (x, y) = 1 if r (x, y) ≦ THD d (x, y) = 0 if r (x, y)> THD (15)
Determined by Here, the threshold value THD is set to THD = 20, for example.

  In the subsequent step S33, the number Nd of pixels having the value 1 in the binary image D, that is, the extracted dark part pixels is calculated.

In step S34, a comparison is made between Nd obtained in step S33 and the threshold value THC in order to determine whether or not a sufficient number of dark pixels is obtained to determine that a lumen is present in the visual field. Do. In this embodiment, THC is 10% of the total number of pixels [Equation 16]
THC = (ISX × ISY) / 10 (16)
It is determined. If Nd ≧ THC, the process proceeds to step S35, and if not, the process proceeds to step S37.

  In step S35, the center of gravity R of Nd dark pixels extracted by the lumen existing in the field of view is obtained. FIG. 30 shows an example of the center of gravity R. For the original image (R image) shown in FIG. 30 (a), dark pixel extraction is performed by threshold processing to obtain the luminal region in the shaded area in FIG. 30 (b), and the center of gravity R is obtained.

In step S36, the movement vector of R is estimated based on the position change with respect to the center of gravity R with respect to the previous frame. Hereinafter, a description will be given with reference to FIG. In FIG. 31, F1, F2 and F3 indicate frames of images input in time series. The center of gravity of the luminal dark part region obtained in the frame F1 is R1, and its coordinates are (xr1, yr1). Similarly, assuming that the center of gravity obtained in the frame F2 is R2 and its coordinates are (xr2, yr2), the movement vector v1 for tracking the movement of the lumen in the field of view is [Equation 17].
v1 = (xr2-xr1, yr2-yr1) (17)
Can be calculated.

  Similarly, a movement vector v2 for the frames F2 to F3 is obtained. Thereafter, by storing the center of gravity position Ri and the movement vector vi (i is an integer of 1 or more), the center of gravity position of the lumen can be tracked as shown in FIG.

  If there is no lumen in the image field, the process proceeds from step S34 to step S37 to estimate the lumen direction.

  If a lumen exists in the frame Fi and deviates from the field of view at Fi + 1, the lumen exists in the direction of the immediately preceding movement vector vi. Therefore, by setting the direction closest to the direction of the movement vector vi as the insertion direction, it is possible to put the lumen into the visual field. For example, in the state indicated by F5 in FIG. 32, the lower right direction corresponding to Q8 in FIG. In order to determine the direction closest to ri, an angle formed by the expression (7) shown in the first embodiment is evaluated.

  In step S38, as in step S5 in the first embodiment, the obtained arrow information indicating the insertion direction is superimposed on the image, displayed on the display device 21, and then returned to step S21 to return to the next frame. Repeat a series of processes.

  As described above, according to the insertion direction detection method in the fourth embodiment of the present invention and the insertion direction detection device that presents the insertion direction information to the operator based on the processing result of the method, the time series of the visual field Because it can detect the direction of the lumen when the lumen deviates from the image field of view and presents the insertion direction information to the operator based on the state of movement, it is not skilled in endoscopy It is possible to provide insertion assistance information to the operator, and it is possible to realize a smooth endoscopy.

  It should be noted that in the counting of the number of extracted pixels Nd in step S33, labeling and contraction / expansion processing are applied as preprocessing for excluding pixels extracted as dark portions due to illumination unevenness such as image margins other than the lumen region. However, a minute extraction pixel region may be removed.

  Labeling is also disclosed in, for example, “Introduction to Computer Image Processing P.75-P83, Supervised by Hideyuki Tamura, edited by Japan Industrial Technology Center, published by Soken Publishing Co., Ltd., and released by Nebula.”

Fifth embodiment:
FIGS. 34 to 36 relate to the fifth embodiment of the present invention, FIG. 34 is a flowchart for explaining the insertion direction detection process, and FIG. 35 shows the lumen direction detection based on the movement vector in the process of FIG. FIG. 36 is an explanatory diagram for explaining a case where it is determined that the insertion direction is inappropriate, and FIG. 36 is an explanatory diagram for explaining a result display when the insertion direction detection in the processing of FIG. 34 is inappropriate or impossible.

  Since the fifth embodiment is almost the same as the first embodiment, only different points will be described, and the same components are denoted by the same reference numerals and description thereof will be omitted.

  In the fifth embodiment of the present invention, an insertion direction detection method and an insertion that can improve accuracy by selectively applying an optimal insertion direction detection method according to an endoscopic image to be processed. The present invention relates to a direction detection device.

  In the first to fourth embodiments of the present invention, various lumen direction detection methods based on the light / dark gradient in the image field, the shape feature of the halation, and the field movement tracking have been described. On the other hand, there are a wide variety of observation scenes appearing in an endoscopic image, and in order to effectively use these detection methods, it is important to use them appropriately according to each observation scene.

  Hereinafter, an insertion direction detection method and an insertion direction detection apparatus according to the present embodiment will be described with reference to the drawings.

  FIG. 34 is a flowchart for explaining the operation of the insertion direction detecting device 12 in the present embodiment.

  In step S41, as in step S1 shown in FIG. 8 in the description of the first embodiment, an R image is acquired from the input RGB images of the endoscopic image. Similar processing can be applied using G, B images, luminance images, or the like.

  In step S42, it is determined whether or not the input endoscopic image can detect the lumen or the lumen direction in the visual field. In colonoscopy, a scene in which the focus is lost due to excessive proximity to the mucous membrane and the entire image is reddish occurs (referred to as an “red ball” by endoscopists).

  In such an image, it is inappropriate to display the insertion direction (it can be detected by the method using the movement vector tracking shown in the fourth embodiment, but consideration is given to ensuring a reliable insertion state). It should be prioritized), and it is necessary to retract the endoscope 1 once (Pull Back operation) so that the lumen enters the visual field.

  The state of the red ball is very unique and can be determined from the average value and the standard deviation of the entire R image. Alternatively, the G image may be used in combination, and determination may be made from the average value and standard deviation of the color tone such as R / G of the entire image.

  If it is determined in step S42 that the state is undetectable, the process proceeds to step S53, and if not, the process proceeds to step S43.

  In step S43, a lumen present in the visual field is detected. Here, lumen detection is performed by applying the processing based on the series of dark pixel extraction shown in steps S32 and S33 of FIG. 29 in the fourth embodiment.

    In subsequent step S44, the detection result is determined in the same manner as in step S34 of FIG. 29. If it is determined that a lumen is present in the visual field, the process proceeds to step S45, and steps S35 and 4 in the fourth embodiment are performed. The movement vector is calculated by the same process as S36, and the process returns to step S41.

  On the other hand, if it is determined that there is no lumen in the field of view, the process proceeds to step S46. In step S46, the detection of the luminal direction based on the light / dark gradient by the series of processes described using steps S2, S3, and S4 of FIG. 8 in the first embodiment is applied, and the process proceeds to step S47.

  In step S47, it is determined whether or not the detection of the luminal direction based on the brightness gradient in step S46 is successful. Here, the ratio between the minimum value min (ε (k)) and the next smallest value smin (ε (k)) (second place as a lumen direction candidate) in the error evaluation formula shown in Equation (8). Compare min (ε (k)) / smin (ε (k)) with the threshold THG. min (ε (k))> smin (ε (k)), and the smaller the ratio min (ε (k)) / smin (ε (k)), the higher the reliability of the detection result. Correspond. The threshold value THG is 0 ≦ THG ≦ 1, and THG = 0.5 here. If min (ε (k)) / smin (ε (k)) ≦ THG, it is determined that the detection is successful, and the process proceeds to step S52, and if not, the process proceeds to step S48.

  In step S48, the lumen direction detection processing based on the shape feature of the halation described in the second or third embodiment is applied. Specifically, the series of processes described using steps S12 to S15 in FIG. 18 or steps S22 to S25 in FIG. 26 are applied, and the process proceeds to step S49.

  In step S49, branching is performed based on whether or not the detection based on the shape feature of the halation has succeeded. The determination here is the same as that based on whether or not the halation in step S14 or S25 forms an arc. If it is determined to be successful, the process proceeds to step S52, and if not, the process proceeds to step S50.

  In step S50, the lumen direction detection process based on the movement vector described in the fourth embodiment is applied. Specifically, by applying the same processing as step S37 in FIG. 29, detection is performed based on the moving direction of the lumen existing in the visual field in the previous frame, and the process proceeds to step S51.

  In step S51, it is determined whether or not the detection of the lumen direction in step S50 is successful. For example, changes in the shape of the large intestine due to changes in body position or pulsation, increases in dark areas due to changes in the amount of light due to dimming, or changes in the visual field exceeding the rate of continuous frames (usually 1/30 seconds) due to sudden angle operations, etc. For example, the movement vector may be erroneously detected. In such a case, for example, as shown in FIG. 35, although the movement vector vi is in the center of the visual field, the lumen exists outside the visual field in a direction that cannot be estimated from the movement vector. Therefore, as shown in FIG. 35, a detection exclusion region is set near the center of the image, and it is determined that the detection is not performed when the movement vector vi is within this range.

  In step S51, if it is determined that the lumen direction detection based on the movement vector is successful, the process proceeds to step S52, and if not, the process proceeds to step S53.

  In step S52, based on the detection result of the lumen direction obtained by any of the detection based on the light / dark gradient direction, the shape feature of the halation, or the movement vector, it is obtained in the same manner as in step S5 in the first embodiment. The arrow information indicating the inserted direction is superimposed on the image, displayed on the display device 21, and then returns to step S41 to repeat a series of processing for the next frame.

  In step S53, since it is impossible to detect the lumen direction of the image to be processed, initialization for canceling tracking of the movement vector (discarding of the movement vector information so far) is applied, and step S53 is applied. move on.

  In step S54, a message such as “Pull Back” shown in FIG. 36 is displayed on the display device 21 in order to prompt the doctor to perform a safe and reliable insertion procedure such as retreating the endoscope 1 and securing the lumen in the field of view. Is displayed, the process returns to step S41, and a series of processing is repeated.

  As described above, according to the application of the insertion direction detection method and the insertion direction detection apparatus using the method according to the fifth embodiment of the present invention, an optimum is obtained according to the endoscope image to be processed. A smooth endoscopic examination can be realized by selectively applying the insertion direction detection method.

DESCRIPTION OF SYMBOLS 1 ... Endoscope 2 ... Operation part 3 ... Insertion part 5 ... Connector 6 ... Control apparatus 8 ... Light guide control apparatus 9 ... Monitor 11 for observation ... A / D converter 12 ... Insertion direction detection apparatus 20 ... Computer 21 ... Display Device 22 ... I / O control circuit 23 ... CPU
24 ... Main memory 25 ... Storage device 26 ... Main program

Claims (8)

An operation method of an endoscope insertion direction detection device to which an endoscope image obtained by observing a subject having a lumen structure with an endoscope is input,
Processing unit of the endoscope insertion direction detecting device, a second step of determining whether the endoscope from a first endoscopic image is to observation target in excessively close state,
A movement vector for tracking the movement of the lumen from the first endoscopic image when the processing unit of the endoscope insertion direction detecting device determines that the over-close state is not established in the second step. A third step of detecting
Operation method of the endoscope insertion direction detecting device characterized in that it comprises a. Furthermore, the processing unit of the endoscope insertion direction detection device holds a movement vector detected by the third step, a fourth step,
Further, the processing unit of the endoscope insertion direction detecting device determines whether the endoscope is in an over-proximity state with respect to the observation target from a second endoscopic image different from the first endoscopic image. a sixth step of determining whether,
Further, when the processing unit of the endoscope insertion direction detecting device determines that the over-close state is not present in the sixth step, the movement of the lumen is tracked from the second endoscopic image. A seventh step of detecting a movement vector;
Further, when the processing unit of the endoscope insertion direction detection device determines that the over-close state is present in the sixth step, the movement vector held in the fourth step is initialized. 8 steps,
A method of operating an endoscope insertion direction detecting device, comprising: The said 2nd step determines whether it exists in the said excessive proximity state based on the average value and standard deviation of the predetermined | prescribed color component in a said 1st endoscopic image. Operating method of the endoscope insertion direction detecting device. The 6th step detects whether it is in the over-proximity state based on an average value and standard deviation of a predetermined color component in the 2nd endoscopic image. Operating method of the endoscope insertion direction detecting device.   An input unit for inputting a first endoscopic image obtained by observing a subject having a luminal structure with an endoscope;
  From the first endoscopic image, it is determined whether or not the endoscope is in an over-proximity state with respect to the subject. A processing unit for detecting a movement vector for tracking the movement of the lumen from the mirror image;
  An endoscope insertion direction detecting device comprising:   Furthermore, the input unit receives a second endoscopic image obtained by observing the subject,
  Further, the processing unit holds a movement vector detected from the first endoscopic image, and further, the endoscope is in an in close proximity to the subject from the second endoscopic image. And further, when it is determined from the second endoscopic image that the endoscope is not in the excessively close state, the movement of the lumen is tracked from the second endoscopic image. And a movement detected from the held first endoscope image when it is determined from the second endoscopic image that the endoscope is in the overapproaching state. Initialize vector
  An endoscope insertion direction detecting device characterized by the above.   The internal processing unit according to claim 5, wherein the processing unit determines whether or not the over-close state is based on an average value and a standard deviation of predetermined color components in the first endoscopic image. Endoscope insertion direction detection device.   The processing unit is configured so that, based on an average value and a standard deviation of predetermined color components in the second endoscopic image, the endoscope is in the close proximity state to the subject from the second endoscopic image. The endoscope insertion direction detecting device according to claim 6, wherein the endoscope insertion direction detecting device is detected.