JP5477800B2 - Method of operating rotation state detection device and rotation state detection device - Google Patents

Description

  The present invention relates to a method for detecting the rotation state of an object captured by an image, and in particular, when the rotation state between the operator's hand operation end and the surgical tool is twisted, the rotation state and operation of the surgical tool on the screen. The present invention relates to a rotation state detection method that associates rotation directions of ends.

  As a prior art of a medical manipulator used in a clinical site, as shown in Patent Document 1, for a surgical instrument having a wire drive type joint portion, the distal joint portion is moved by moving the hand operation portion and the handle portion up and down. Operate down and up, move left and right of the hand control and handle to move the tip joint to the right and left, and open and close the tip to open and close the tip. In addition, there is a description of a mechanism that enables a pseudo operation on a multi-degree-of-freedom forceps whose position and posture are operated.

  When a doctor performs an operation while looking at the endoscope screen, when operating the surgical tool displayed on the screen while moving the operating end of the medical instrument at hand, There are times when I don't know how to move. The reason is that when the insertion means of the surgical instrument is soft (flexible), a twist is generated between the operation end and the surgical instrument, and the imaging means and the surgical instrument that are rigidly fixed to the endoscope The relative relationship between the operating end (the doctor's hand) that is the entrance portion and the surgical instrument that is the exit portion are different. For this reason, a doctor moves the surgical tool displayed on the screen to confirm the direction of movement, and performs the work of grasping the relationship between the operation direction and the direction of movement.

  However, if the surgical instrument is moved for confirmation during the operation, the twisting state may change, and it may become impossible to move in the desired direction again. In addition, the operation for confirming the movement of the surgical instrument requires a high skill because there is a risk of contact with the surroundings when the surgical field is narrow, and improvement is also required from the viewpoint of safety.

  Patent Document 2 discloses an endoscopic surgical apparatus that allows a doctor to easily move to a desired position in the visual field direction of an endoscope at hand. Here, a color marker is provided on a treatment tool (gripping forceps) to be inserted into the body, and when the color marker is detected on the screen, an amount of movement for moving the color marker from the current position to the screen center position is obtained, and an actuator circuit To control.

JP 2006-61364 A Japanese Patent Laid-Open No. 10-118015

  However, in the example of Patent Document 2, since the movement amount is obtained from the position of the discontinuous color marker, the detectable angle is also intermittent.

  An object of the present invention is to overcome the above-mentioned problems of the prior art, detect the rotation angle of an object (surgical instrument) simply and continuously, and associate the rotation direction of the object on the screen with the operation direction of the operation end. An object of the present invention is to provide a rotation state detection method and apparatus.

  The present invention that achieves the above object transmits the operation amount of the operation part to the surgical instrument part by the transmission part, and detects the rotation state of the surgical instrument part using an imaging means for photographing the operation of the surgical instrument part in the surgical field. A marker that is provided on the surgical instrument side of the transmission unit and has a pattern that is continuous in the circumferential direction, and performs image processing by capturing marker image information obtained by imaging the marker by the imaging unit. The rotation state of the surgical instrument unit is detected by obtaining the rotation angle of the marker. Here, the marker image information is electronic information of a marker image obtained by imaging the marker, and hereinafter simply referred to as a marker image.

  The marker is a straight line or a curve in which at least one of the boundary lines that divide the region of the continuous pattern continuously changes in the axial direction with respect to the circumferential direction. In addition, the marker has one boundary line that divides the region of the continuous pattern, and the other boundary line with respect to this boundary line has a straight line or a curve in which the axial position continuously changes, Due to these boundary lines, the marker includes a polygon or a curved figure, and the marker has different color feature amounts in the axial direction across the boundary line. The marker may have a plurality of continuous pattern regions in the axial direction of the transmission unit.

  Further, the marker image information is subjected to image processing to obtain a measurement line indicating the center in the depth direction of the marker display area in the marker image information, and the axial position of the area divided in the axial direction along the measurement line is determined. A ratio is obtained, and a rotation angle is obtained from a circumferential position on the marker corresponding to the ratio.

  The depth direction of the marker described here is the direction of the distal end side where the surgical instrument portion is provided. For example, since the medical instrument is extended from the vicinity of the lens of the imaging unit to the surgical field where the affected part is located, the distal end side of the medical instrument extends in the back direction of the screen (the direction away from the lens of the imaging unit). Therefore, in the marker attached to the medical instrument (the transmission unit), the direction closer to the surgical instrument unit is described as the depth direction of the marker image. Further, the measurement line indicating the center in the depth direction of the marker image is a straight line that continuously connects the center points of cross sections obtained by cutting a medical device extending in the back of the screen perpendicularly to the axial direction.

  The marker has a blue and green region across a boundary line that divides the continuous pattern region, and the ratio is a ratio of a blue region and a green region. Here, the expression blue means a color with a strong bluish color, and similarly green means a color with a strong greenness. None of them specify a single color, and any color that can be relatively identified is acceptable.

  When there are two rotation angles corresponding to the ratio, the intersection point between the horizontal axis of the marker display area in the marker image information and the measurement line, and the center of gravity of the surface of the divided one area are connected to the intersection point. An extension line is obtained, and a rotation angle is selected based on an angle formed by the measurement line and the horizontal axis and an angle formed by the extension line and the horizontal axis.

  In the rotational state detection method of the present invention, an instruction image indicating the operation direction of the surgical instrument unit is generated based on the relative angle between the marker and the operation direction and the angle measured by the measurement line, and the technique in the captured image of the imaging unit is generated. Attached to the component display area and displayed. Here, the relative angle between the marker and the movement direction is a relative angle between the movement direction Mx and the position of the pattern where the marker indicates 0 degree. When the marker is pasted on the transmission unit, the pasting is performed in such a relationship that the motion direction is 0 degree of the marker. However, when there is a motion direction My different from Mx and the two motion directions are orthogonal, the motion is performed. The direction My is the 90-degree position of the marker. At this time, the relative angle between the marker and the operation direction My is 90 degrees. For example, if the apparent angle measured on the measurement line is 100 degrees, the movement direction My is deviated from the direction of the measurement line by a difference of 10 degrees, so an arrow rotated by this amount is displayed. . On the other hand, since the movement direction Mx is shifted by 90 degrees, the arrow indicating the movement direction Mx is displayed by being shifted by 90 degrees from the direction of the measurement line.

  A more specific method of the present invention is to transmit the operation direction of the operation unit to the surgical instrument unit by the transmission unit and to rotate the surgical instrument unit using an imaging unit that images the operation of the surgical instrument unit in the surgical field. The transmission unit includes a marker that is provided on the surgical instrument unit side of the transmission unit and has a marker having a continuous pattern in which regions having different feature amounts are continuous in the circumferential direction. When the marker image information obtained by imaging the marker is captured and processed, and the rotation angle of the marker is detected to detect the rotation state of the surgical instrument, the distortion of the marker display area in the captured marker image information is corrected. Then, the corrected image information is generated, and the marker image area extracted by performing the separation of the marker from the background and the pattern of the marker based on the feature amount of the marker image by the filtering means. Boundary information is extracted, an affine transformation matrix is created from the boundary information, the marker image region viewed from the front of the marker is corrected using the affine transformation matrix, and the measurement passes through the center of the corrected marker image region. A ratio of the lengths of the regions having different feature quantities on the line is obtained, and an angle corresponding to the ratio is obtained as a rotation angle of the surgical instrument unit. The regions having different feature amounts are regions arranged in blue and green, and the filtering unit divides the corrected image into R (red image), G (green image), and B (blue image). Then, a combined process of subtracting R from the sum of G and B for each pixel is performed on the divided image to extract a marker area based on a set of a green area and a blue area. Alternatively, the regions having different feature amounts are regions that are separately coated with different fluorescent dyes, and the filtering unit extracts only the excitation wavelength or the fluorescence wavelength of the fluorescent dyes, whereby the fluorescent dye region and the marker region are extracted. Is extracted.

  The rotational state detection device of the present invention observes the operation of the surgical instrument in the surgical field, the surgical instrument that performs work in the surgical field, the operation part thereof, the transmission part that transmits the operation direction of the operation part to the surgical tool part, An imaging means, an apparatus for detecting the rotation state of the surgical instrument portion by the imaging means, a continuous pattern marker provided in a circumferential direction on the surgical instrument side of the transmission portion, and a marker captured by the imaging means An image processing apparatus that performs image processing on an image and obtains the rotation angle of the marker is provided, and the rotation state of the surgical instrument unit is detected by obtaining the rotation angle of the marker.

The image processing apparatus includes: an image capturing unit that captures an image of a surgical instrument part and a marker; an image correcting unit that corrects distortion of the captured image; and a marker and a background that are separated from the corrected image based on a feature amount of the marker. And filtering means for extracting the marker area by separating the marker pattern, extracting the edge of the extracted marker area, creating an affine transformation matrix from the edge, and using the affine transformation matrix to view from the front of the marker And when the marker is a set of a continuous pattern of the first color and the second color, the first color region on the measurement line passing through the center of the enlarged marker image The filtering means comprises the angle detecting means for obtaining a ratio of the lengths of the second color regions and obtaining an angle corresponding to the ratio as a rotation angle of the surgical instrument unit. When the color is green and the second color is blue, the corrected image is divided into R (red image), G (green image), and B (blue image). A combining process of subtracting R from the sum of G and B for each pixel is performed to extract a marker area based on a combination of a green area and a blue area.

  Here, the R, G, and B images are defined in terms of the three color parameters of each pixel of the original color image, red (R), green (G), and blue (B). The R image is obtained by extracting only the R component (red parameter) of this and using this value as the gray value. The same applies to the G image and the B image. The G + B-R combining process is to generate a gray image expressed by the value calculated in the combining process for the gray value parameter of each gray image.

  The rotational state detection device of the present invention generates an image such as an arrow indicating the operation direction of the surgical instrument unit based on a relative angle between the marker and the operation direction and an angle measured by a measurement line. Display information generating means for displaying the image attached thereto is provided.

  The imaging means is an endoscope, and the surgical instrument section is a medical instrument such as a forceps or an electric scalpel that is operated by a doctor and an operation section is operated by the doctor, or an endoscope that is narrower than the endoscope. In the endoscope, the transmission unit is a flexible transmission unit that may cause twisting.

  Instead of displaying the operation direction of the surgical instrument part, the rotation angle of the surgical instrument part detected for the recording of the operation is recorded together with the image of the surgical field at that time, or fed back to the automatic control of the medical instrument. It may be.

  According to the present invention, the rotation state of the transmitting portion (for example, a flexible surgical instrument) having a twist between the operation end and the operated end is indicated as the operation direction of the surgical instrument portion on the screen. The rotation state can be grasped at a glance, and the operation of the surgical instrument part is facilitated, and the safety can be improved.

  Hereinafter, an embodiment of the present invention will be described using a medical endoscope system in which a surgical instrument (surgical instrument) may be twisted inside a parent endoscope as an example. The parent endoscope described here is an endoscope in which a smaller-diameter endoscope can be inserted into the insertion channel of the endoscope. It is a general term for what an endoscope into which a mirror and a child endoscope are inserted is called a parent endoscope. In other words, it is a medical instrument insertion tool that can further insert an endoscope into the insertion channel of the endoscope, is flexible instead of the child endoscope, and has an outer diameter smaller than the inner diameter of the insertion channel. A medical instrument can be inserted. Further, although the positional relationship between the lens surface of the parent endoscope and the outlet of the insertion channel is fixed, it does not necessarily have to be in the same direction. In other words, the medical device may be related so that the medical device comes out obliquely at a predetermined angle from the parent endoscope. By changing the angle between the parent endoscope and the surgical instrument in accordance with the state of the affected part and the treatment content, an effect of improving the ease of treatment work can be expected. Moreover, there is no parent endoscope, and a normal endoscope and a medical instrument may be inserted separately.

  FIG. 1 is an overall configuration diagram of a medical endoscope system according to an embodiment. In the insertion means 3, an insertion portion 2 ′ of an endoscope 2 provided with an endoscope lens (endoscopic tip portion) 6 and a flexible transmission portion 1 ′ provided with a surgical instrument portion 4 are inserted. Yes. The insertion means 3 may be omitted, and the insertion portion 2 ′ of the endoscope 2 and the flexible transmission portion 1 ′ provided with the surgical instrument portion 4 may be separately inserted into the body. By having the insertion means 3, the surgical instrument part 4 can be safely guided to the affected part. When the treatment is desired to be performed over a wide range near the body surface, the configuration in which the insertion means 3 is not used has an advantage that the surgical instrument portion can be freely moved without being limited to the insertion means 3. At the other end of the flexible transmission portion 1 ′, an operation end 1 of a medical instrument that allows an operator to operate the surgical instrument portion 4 is provided. A marker 5 is provided on the surgical instrument portion side of the flexible transmission portion 1 ′.

  6A, 6B, and 6C are examples of development views of markers provided at the distal end portion of the flexible transmission portion. The upper part of the figure is the distal direction of the surgical instrument part. The marker 5 has a continuous pattern in which the square area of the marker sheet is divided by a straight line or a curve represented by a function whose height direction changes continuously with respect to the horizontal width of the marker sheet. Examples of how to divide are indicated by A to C.

  The marker 5 shown in the development view of FIG. 6A has a shape in which a square area (for example, an area formed by combining 53b and 54b) of the marker sheet 50 is divided by the area 53b and the area 54b with a polygonal line such as a vertex of a triangle. is there. Hereinafter, in the description that does not need to be identified by alphabets, those with suffixes 51 to 54 are represented by 51, 52, 53, and 54 without suffixes.

  The marker 5 shown in the development view of FIG. 6B has a shape obtained by dividing the square region of the marker sheet 50 into a region 53 and a region 54 by straight lines.

  As shown in the developed views of FIGS. 6A and 6B, the boundary lines of the markers are triangular, pentagonal, or quadrangular figures. Furthermore, in order to increase the accuracy of a specific angle range, it may be an arbitrary polygonal figure in which the gradient of the corresponding part is increased.

  The marker 5 shown in the developed view of FIG. 6C has a shape obtained by dividing the square region of the marker sheet 50 into a region 53 and a region 54 by a trigonometric function.

  In addition to these, even if the shape is divided by a quadratic curve or the like, the same processing can be performed in the following method.

  Hereinafter, the case of FIG. 6A will be described as a representative example. The region 53 and the region 54 are arranged with colors having feature quantities that can be separated and extracted by the filtering means. For example, if the filtering unit is a unit that extracts a color feature amount, the region 53 and the region 54 are painted with different colors. Alternatively, if the filtering means is means for extracting the wavelength of a specific fluorescent paint, the region 53 and the region 54 are separately coated with fluorescent paints having different wavelengths. As in this example, the region 53 and the region 54 are painted according to the filtering means. Here, as an example, the region 53 is described as green and the region 54 is described as blue.

  The marker-like regions 53 and 54 are shaped like a triangle so that the bottom is 360 ° wide and the marker sheet 50 is in the circumferential direction of the flexible transmission portion 1 ′ (also referred to as the handle of the surgical instrument). It is pasted around once.

  The marker 5 may be printed on a cylindrical elastic member and fitted on the handle of the surgical instrument, or may be printed directly on the handle of the surgical instrument. If it is a means to stick and fix, it becomes possible to detect the angle during treatment only by sticking a marker to a surgical instrument that has been conventionally owned. In this case, it is only necessary to prepare a variation in which the width of the marker is changed and paste a variation that matches the diameter of the surgical instrument. The marker printed on the elastic member can be fixed tightly even if the diameter of the handle of the surgical tool is slightly changed. Less. Furthermore, since direct printing does not cause a gap between the surgical instrument and the marker, stability over time can be obtained.

  In the example shown in FIG. 6A of this embodiment, the marker areas 53 and 54 are formed in a pentagon. That is, a band having a predetermined width (for example, 1 mm) is provided under a pattern having an apex angle such as a triangle. This band is a getter part for avoiding that the length of one of the vertex and the valley becomes zero when calculating the ratio of blue and green, which will be described later, and is a discontinuous point that cannot be calculated. This is to prevent the occurrence of In addition, it is possible to avoid a situation in which it is difficult to distinguish whether it is really 0 or not.

  In the axial direction, one set of the markers 52a to 52b, one set of 52b to 52c, and one set of 52c to 53d, in principle, angle calculation is possible even with one set. Since the marker is reduced and displayed as it goes deeper in the screen, the data error increases. Therefore, even if the depth of the surgical instrument changes, it is better to calculate the angle with the group as close as possible. However, when the marker is contaminated with blood or the like during surgery, there is a merit that the angle can be calculated using the next closest set, and the robustness against the stain is improved.

  Between the markers, bands 51a, 51b, 51c, 51d of predetermined widths and bands 52a, 52b, 52c, 52d are provided, and these bands are landmarks that indicate a start line when detecting an angle. .

  FIG. 7 shows an example in which a state where the marker is attached to the handle of the surgical instrument is observed with an endoscope. This figure is an image after distortion correction by the image correction means 10, and when the handle (flexible transmission part) of the surgical instrument is rotated about its longitudinal axis as a rotation axis, how the attached mark looks This is shown in (a)-(d). That is, when the flexible transmission portion 1 ′ is twisted, the actual image of the marker 4 shows the apparent angle after twisting.

  FIG. 8 is a development view showing a square region of a set of markers. FIG. 7 is a diagram representing one of the three sets of marks in FIG. 6, and a blue pentagonal region 34 and a green region 33 correspond to form a set of continuous patterns. The horizontal width of the blue region 34 is the same as the circumference of the handle of the surgical instrument, and is 360 degrees when expressed in terms of the rotation angle. Assuming that the horizontal axis is θ and the left end of the region 34 is 0 degrees, the oblique line (the length (height) from the base of the pentagon (on the axis of length 0)) at an arbitrary θ can be expressed by an equation. . The upper region 33 corresponding to the region 34 is configured such that the green region 33 forms a pentagon when viewed at a position where the marker 5 wound on the circumference is rotated 180 degrees around the axis.

  The surgical instrument unit 4 of the medical instrument includes a therapeutic instrument having forceps, an electric knife, a scissors and the like for performing a therapeutic action at the tip, and observation instruments such as an endoscope and an imaging means. The endoscope 2 observes the surgical field including the affected part and the surgical instrument part 4 and the marker 5 in the surgical field.

  The image controller 7 is connected to the endoscope 2 to acquire an endoscope image and outputs it to the monitor 40. The image processing device 8 analyzes the input endoscopic image, reads the marker image displayed on the screen, and calculates the rotation angle of the surgical instrument unit 4 (the marker application unit of the flexible transmission unit 1 ′). . Further, the detected rotation angle is output to the monitor 40, and the rotation state of the surgical instrument is displayed as information.

  FIG. 2 is a functional block diagram of the image processing apparatus and also shows a processing flow. The image processing apparatus 8 includes an image capturing unit 9 that captures an original image including a marker from the image controller 7, an image correction unit 10 that corrects luminance and color unevenness of the original image, and distortion of the original image. Image filtering means 11 is provided for extracting and separating the background, the marker area, and the pattern. Based on the separated image, a marker image is extracted, an angle detection means 12 for detecting an apparent angle (rotation angle) at which the center line of the marker image can be read, and an arrow indicating the operation direction of the surgical instrument unit 4 from the apparent angle And the like, a display information output unit 14 for synthesizing and outputting the image of the surgical instrument and the arrow, and a data recording unit 15 for recording the detected rotation angle.

  When the marker 5 is divided into a green region 53 and a blue region 54, the image filtering means divides the image into three images with three feature values of R (red), G (green), and B (blue) of each pixel. To do.

  Filtering methods other than RGB decomposition include a method using a band-pass filter that utilizes the difference in absorption band of the dye, and if the marker contains a fluorescent dye, separate the fluorescent region according to the fluorescent wavelength of the fluorescent dye. An extraction method, a method of physically attaching a filter for selecting a transmission wavelength on the lens surface, and correcting the RGB parameters of the endoscopic image in advance by changing the wavelength of the light to be projected, thereby making it easy to specify the marker region There is a technique, and the feature of the marker can also be extracted and separated from the background by such means. In any case, it is possible to apply a feature amount that does not exist in the body to the marker, and to perform filtering that effectively extracts the feature amount. Therefore, there is an advantage that it is easy to distinguish between the marker and the internal organ and to identify the marker pattern. . In the following, details of the case where filtering by RGB decomposition is performed on the premise that the marker 5 is divided into a green region 53 and a blue region 54 will be described.

  The marker area is identified and extracted from the three images (R, G, B), the marker edge is detected along the detection line set at the axis center of the marker area, and is displayed on the screen from the blue / green edge ratio. Angle detecting means 12 for detecting the angle (θ) indicated by the center line of the marker. Further, based on the angle indicated by the center line of the marker displayed on the screen, an image in which an arrow indicating the operation direction of the surgical instrument unit 4 is superimposed is generated and output to the monitor 40. Is provided. Regarding the angle information, not only the angle information viewed from the screen but also the data recorded by converting the angle into the angle viewed from a specific direction may be used.

  FIG. 3 is a functional block diagram of the image correction means, and also shows the flow of processing. The image correction means 10 is for pre-processing. The brightness correction means 16 corrects the luminance unevenness of the original image including the marker to obtain uniform brightness. If the entire screen is shifted to some color, A color correction unit 17 that corrects the color and a distortion correction unit 18 that corrects the distortion of the image are provided, and the captured original image is corrected by these pre-processing, both of which are known units.

  The color correction unit 17 is a unit that corrects the balance when the entire screen is shifted to a specific color, and checks the color shift state in advance and acquires its parameters. During processing (during program operation), color correction is performed based on this parameter.

  The distortion correction means 18 corrects the curved display of the image depending on the shape of the lens. The distortion correction unit 18 measures the degree of distortion by acquiring a lattice pattern in advance and acquires distortion correction parameters. . Since the image is converted using this parameter during processing, the straight line is displayed as a straight line in the image after distortion correction.

  The preprocessed original image is input to the image filtering means 11. Here, the original image is divided into R, G, and B images using the RGB parameters of each pixel. The G image is an image in which the triangular pattern 33 (region 33) is emphasized, the B image is an image in which the triangular pattern 34 (region 34) is emphasized, and the R image is an image in which the body as a background is emphasized.

  FIG. 4 is a functional block diagram of the angle detection means and also shows the flow of processing. The marker area extraction unit 19 performs a synthesis process of adding the G image and the B image to the RGB image divided by the image filtering unit 11 and subtracting the R image, thereby identifying and extracting the marker area.

  The edge extraction means 20 extracts a straight line component that is an edge of the extracted marker, calculates the inclination of the straight line, and calculates the intersection of the two straight lines of the edge. Since the endoscope shown in the image is in the posture toward the back of the screen, the tip of the surgical instrument (marker) is displayed smaller. The enlargement factor correction unit 21 performs affine transformation so that the intersection portion is in front of the imaging unit, and converts the marker image into an image viewed from the front. The marker reading unit 22 sets an edge detection line at the center of the axis of the marker image, and detects an edge in the marker at a position where brightness changes along this line.

  The angle calculation means 23 reads the lengths of blue and green from the edge in the marker and calculates the ratio. The angle read from this ratio is the apparent angle θ that can be read from the center line of the surgical instrument shown on the screen.

  The marker is attached under a predetermined condition with respect to the operation direction of the surgical instrument unit. In other words, when the swinging direction of the surgical instrument unit 4 is “I” and “B” (not shown), the pasting is performed so that the swinging direction “a” matches 0 degrees of the marker. At this time, the rotation angle is described using the 0-degree position of the marker as a reference.

  The rotation angle (appearance angle) θ detected by the measurement is the amount measured from 0 degrees of the marker, but the relative rotation angle θ ′ over time is calculated by obtaining the difference in the data from the previous detection. You can also

Here, the lengths of blue and green depending on θ are expressed by equations (1) and (2). As for the axial size of the marker used in the angle calculation of the embodiment, the height of the start line is 1 mm, the height of the apex of the triangle is 5 mm, and the height of the band is 1 mm.
When 0 <θ <180 degrees,
Green length = 5-4θ / 180, Blue length = 1 + 4θ / 180 (1)
When 180 degrees <θ <360 degrees,
Green length = −3 + 4θ / 180, Blue length = 9−4θ / 180 (2)
When θ = 0 degrees, the green length = 5, the blue length = 1, and when θ = 180 degrees, the green length = 1 and the blue length = 5.

If Ra = green length ÷ blue length, θ = 0 if Ra = 5, and θ = 180 degrees if Ra = 0.2. In other cases, 0 <θ <180 degrees or 180 <θ <360 degrees. A method for determining which of the current angles is included will be described later. In order to calculate θ from Ra in each case, the relationship between the expressions (3) and (4) is calculated based on the expressions (1) and (2). It becomes.
When 0 <θ <180,
θ = 180 (5-Ra) / (4 (1 + Ra)) (3)
When 180 <θ <360,
θ = 540 (1 + 3Ra) / (4 (1 + Ra)) (4)
By the way, if the measurement point of the marker is a slope, there are two solutions with the same blue-green ratio. FIG. 9 is an explanatory diagram showing a case in which the two blue slopes of the marker have the same blue-green ratio. In the graph of the figure, when the blue height (for example, the length of 0 to 35) at an arbitrary angle θ (for example, θa) is subtracted from the height of the rectangular area on the extension, the green color of the pentagonal area 33 is reduced. I know the height. If the height (length) ratio between green and blue at this time is known, an arbitrary rotation angle θ can be calculated backward.

  In FIG. 9, there are θa and θa ′ in which the ratio of the blue-green length to the axial circumferential rotation angle θ is the same. θa is between 0 and 180 degrees, and θa ′ is between 180 and 360 degrees. Thus, the selection method of the slope with the measurement points 35 and 36 having the same angle is performed as follows.

FIG. 10 is an explanatory diagram when selecting green as a reference, and one of the slopes is selected by the following procedures (1) to (5).
(1) An angle formed by a measurement line (the center line of the surgical instrument 4 or the flexible insertion portion 1 ′) for extracting the boundary between blue and green and the horizontal axis of the screen is φ.
(2) Find the intersection point p between the measurement line and the horizontal axis of the screen.
(3) The center of gravity g of the surface of the green region is calculated.
(4) The angle φ ′ formed by the straight line connecting g and p with the horizontal axis of the screen is compared with the angle φ formed by the measurement line.
(5) When φ ′> φ, blue pentagonal right slope (180 degrees <θ <360 degrees), when φ> φ ′, blue pentagon left slope (0 degrees <θ <180 degrees), φ = φ At ', the measurement line passes through the apex (θ = 0 or 180 degrees), so 0 and 180 degrees are identified from the ratio of green and blue.

  FIG. 5 shows a functional block diagram of the information generating means and a processing flow. Based on the data detected by the angle detection unit 12, the information generation unit 13 performs processing according to the purpose, such as data recording and information display.

  The process distribution means 30 by using information distributes the subsequent processes to the angle correction means 24 and the display image correction means 25 from the reference according to how the detected angle is used. Here, the rotation angle is defined as a rotation angle by determining a reference posture and setting the posture as an initial posture, and how much the surgical instrument is rotated from the initial posture.

  The angle correction unit 24 calculates the difference between the apparent angle and the initial posture, and records the amount of change (rotation angle) from the initial posture in the data recording unit 15. When the angle information of the center line displayed on the screen is stored, it is recorded by the data recording unit 15 as it is without passing through the angle correction unit 24. This rotation angle is fed back to the operation end side and can be used for automatic control of a soft forceps manipulator or the like.

  As an example, the display image correction unit 25 will describe a case where the operation direction is indicated by an arrow. The direction of the arrow to be displayed when the rotation angle is 0 (initial posture) (positive direction) is predetermined, and the arrow is displayed by shifting the direction from the initial arrow direction by the amount of the rotation angle of the surgical instrument. To do.

  In the case of angle information (appearance angle) that can be read from the center line (measurement line) of the surgical instrument part displayed on the screen, the display image correction means 25 rotates the displayed arrow by the angle information with the center line as the axis of rotation. Let The information generating unit 26 generates an arrow after the rotation, and the image synthesizing unit 27 synthesizes the endoscope image with the arrow so that the arrow is in a predetermined position, outputs it to the display information output unit 14, and displays it on the monitor 40. To do. Thus, the direction in which the surgical instrument unit 4 moves can be displayed at a glance by looking at the arrow on the screen. The angle information may be a mark, an icon, etc. in addition to the arrow.

  Next, the procedure of the rotation state detection method according to one embodiment will be described. 11, 12, and 13 are flowcharts illustrating a rotation angle detection procedure by the image processing apparatus 8. 14-17 is explanatory drawing which shows the result of each process.

  Step s101 is an output process from the image correction means 10 to the image filtering means 11. Among the marker images in FIG. 14, the corrected image (a) is output.

  Step s102 is an operation by the image filtering means 11, and the corrected image is decomposed into R, G, and B images. FIGS. 14B, 14C, and 14D are an R image 61, a G image 62, and a B image 63, respectively. The R image, the G image, and the B image are red, This is an image with the green and blue parameters as gray values.

  Steps s103 and after are operations by the angle detection means 12. First, the marker image 64 is extracted from the R image, the G image, and the B image through a synthesis process of subtracting the R image from the sum of the G image and the B image (G + B−R). As a result, as shown in FIG. 15A, only the marker image is lifted and extracted.

  In step s104, a boundary line around the marker region is extracted (FIG. 15B), decomposed into a curve and a straight line, and only a straight line is extracted from the boundary (FIG. 15C). 67 is selected (s105). In step s106, the inclination of the extracted straight line is calculated, and in step s107, the intersection coordinates of the extension lines (dotted lines) 68 and 69 of the two straight lines are calculated (point C in FIG. 15D).

  In step s108, the average value c of the inclinations of the two straight lines (the average value c will be referred to as the inclination c) is calculated. In step s109, the straight line having the inclination c passing through the point C (FIG. 15 (d)). A straight line 70) is generated and this is taken as the center line.

  In step s110, the coordinates (4 points) of both ends of the two straight lines 66 and 67 are acquired, and in step s111, one of the two straight lines is selected, and the front side of the straight line (the front side of the surgical instrument) is selected. The point a1 is selected (FIG. 16A), and the distance L between the point a1 and the center line 70 is calculated in step s112. In step s113, a target point (point A2) sandwiching the center line 70 with respect to the point a1 and a line symmetrical target point (point a4 ′) equidistant from the point a3 with the center line 70 sandwiched therebetween are created. In step s114, a point A3 that passes through the other end point (a3) of the straight line 66 including the point a1 and is orthogonal to the center line 70 is separated from the center line by a distance L. In step s115, a point A3 is created. On the other hand, a target point (A4) sandwiching the center line 70 is created (FIG. 16B).

  In step s116, an affine transformation matrix is created. The affine transformation rotation is such that the points a1, A2, a3, a4 ′ are the four points before conversion, the points a1, A2, A3, A4 are the four points after conversion, and the four points before conversion become the four points after conversion. Create a matrix.

  In step s117, the marker image is transformed using the affine transformation matrix. FIG. 17 shows the image after conversion. In this example, the image of FIG. 14C is converted by an affine transformation matrix. FIG. 14 shows the result of three-dimensional rotational transformation of the image so that the marker pattern reduced as it approaches the distal end side of the surgical instrument portion has the same enlargement ratio from the root side to the distal end side in FIG. Yes.

  In step s118, the sum area of the marker area after the affine transformation and the center line is obtained, and the edge detection line 72 is created. In step s119, the edge of the marker pattern is detected along the edge detection line 72. The detected edge is indicated by a cross on the edge detection line 72 in FIG.

  In step s120, the blue-green ratio is calculated. The distance between the second (73) and the third (74) viewed from the near side of the X mark detected as the edge of the marker pattern is the length of the blue region, and the third (74) The distance of the fourth (75) is the length of the green region. A ratio obtained by dividing the length of the green region by the length of the blue region is Ra (green / blue). Here, the ratio of blue and green used for the marker may be reversed, and may be determined in advance.

  Step s121 is processing related to selection of the two slopes of the marker described above. First, paying attention to the green region 77 whose distance is measured in FIG. 17, the center of gravity (g) 76 of the green region 77 that is visible in the green region is measured. The center of gravity 76 is on the right side of the center line 72, and it can be determined that a portion where the hypotenuse of the triangular portion of the blue region is lowering to the right is visible. Select as described above by comparing the size of the angle φ (FIG. 10) formed by the intersection point p between the center line and the horizontal axis of the screen and the angle φ ′ (FIG. 10) formed by the straight line connecting the center of gravity g and the intersection point p. You can decide the slope to be.

  In step s122, angle information along the center line of the surgical instrument pattern shown on the screen is calculated from the blue-green ratio Ra and the slope of the selected slope.

  FIG. 18 is a flowchart showing the processing of the information generating means, and FIGS. 19 and 20 are explanatory diagrams showing the results of each processing. The information generation unit 13 generates an image indicating the operation direction of the surgical instrument unit 4. Here, an example in which the operation direction is indicated by an arrow will be described.

  In s201, an arc component is extracted from the marker area extracted in s103 (FIG. 11). The circular arc 82 in FIG. 19B is extracted from the circular arc component 64 in FIG. In s202, ellipse fitting is performed on the extracted arc component. That is, an ellipse parameter in which a part of the arc of the ellipse coincides with the arc 82 extracted in s201 is specified as 83 in FIG.

  In s203, as shown in FIG. 19C, intersection points a6, a7, a8, and a9 of the ellipse long axis and short axis passing through the ellipse center point a5 and the arc 82 are extracted. The short axis coincides with the direction of the center line 70.

  In s204, the distance L2 from the point a5 to the point a7 is acquired. In s205, the coordinates of the points A6 and A8 at the distance L2 from the center point (a5) in the short axis direction are acquired. In s206, the rotation matrix is such that the points a6, a7, a8, a9 are the four points before conversion, the points A6, a7, A8, a9 are the four points after conversion, and the four points before the conversion are the four points after conversion. (Affine transformation matrix) is created. After the conversion, the circle shape has the same distance from the center point to each point (circle 84 in FIG. 20).

  In s207, the direction of the arrow indicating the operation direction is specified from the angle indicated by the center line on the converted circle. More specifically, it is assumed that when the marker is provided on the medical instrument, the position of the angle indicated by the marker is determined in advance, and the medical instrument performs the swing motion Mx and the swing motion My. If the direction in which the motion Mx is performed is the position of θx (degrees) indicated on the marker, and the direction in which the motion My is performed is the position of θy (degrees) indicated on the marker, the motion directions of My and Mx are orthogonal. If so, the interval between θx (degrees) and θy (degrees) is 90 degrees. Depending on the medical instrument, if the interval between My and Mx is not orthogonal, the interval between θx (degrees) and θy (degrees) will naturally be a value that matches the direction of movement.

  For example, it is assumed that the angle indicated by the center line is calculated as θa ′ (degrees), and θx (degrees) and θy (degrees) indicating the operation directions Mx and My are also known. Therefore, in order to represent the arrows toward the angles θx (degrees) and θy (degrees), the arrows need only be rotated from the center line by the difference between θx (degrees) and θa ′ (degrees). The same applies to the arrow indicating θy (degrees). Therefore, as shown in FIG. 20, the arrow extending from the center of the circle is rotated around the center point of the circle by the difference between θx (degrees) and θa ′ (degrees) from the direction indicated by the center line, and an arrow 86 is displayed. To do.

  In s208, since the arrows indicating θx (degrees) and θy (degrees) created in s207 are created by overlapping the circle shape 84, a matrix for returning the circle 84 to the original ellipse 83 is used. Affine transform the arrow. The converted arrow is 86 '. The matrix for this conversion is an inverse matrix of the matrix that is converted from the ellipse created in s206 to a circle. The above is the processing operation of the information generating means 26. In order to actually display, image synthesis with the marker image is required.

  In s209, the arrow after the inverse transformation is displayed at a predetermined position on the endoscopic image (arrows 80 and 81 in FIG. 21B).

  FIG. 21 is an explanatory view showing the operation part of the medical instrument and the state of the endoscope screen. As shown in FIG. 21A, the operation unit of the medical instrument can move the surgical instrument unit 4 in directions 78 and 79 of axes orthogonal to each other (see Patent Document 1). Corresponding to this, the surgical instrument part 4 at the tip can swing in two directions, but on the screen, there is a case where it is not known in which direction the head is swung. Therefore, the positive direction (counterclockwise) of the direction 78 is preset as the arrow direction shown in the figure, and the positive direction (counterclockwise) of the direction 79 is preset as the arrow direction shown in the figure. Whether the head is swung in the direction is displayed by arrows 80 and 81 on the screen as shown in FIG.

  The design and display position of the arrow are not limited to those shown in the figure. The posture after movement may be thinly overlapped with CG, and can be changed as needed according to the preference of the operator. The display timing may be always displayed, or may be displayed by a foot switch or the like when the operator needs it.

In the present embodiment, since the information generation unit 13 and the display information output unit 14 display these, even if there is a twist between the operation unit and the surgical instrument, the direction of movement by the operation is actually moved. Even without it, it can be grasped as information on the screen.
When the development view of the marker shown in FIG. 6A shown in the present embodiment is used, there are two rotation angles corresponding to the ratio of the blue area and the green area, and as described above, a step for narrowing down this is necessary. Become. On the other hand, if the development view of the marker in FIG. 6B is used, since the corresponding rotation angle is one based on the ratio of the blue area and the green area, the step for narrowing that is necessary in FIG. Processing speed is improved.

  In addition, in the developed view of the marker in FIG. 6A, the blue region and the green region are separated by a straight line, so the rate at which the ratio between the blue region and the green region changes according to the rotation angle of the surgical instrument is always constant. On the other hand, when the development view of the marker in FIG. 6C is used, since the blue region and the green region are separated by a trigonometric function, there are a range where the ratio changes gradually and a range where the ratio changes rapidly. In a rapidly changing range, the change in the blue region and the green region with respect to the rotation angle becomes large, so that the detection accuracy of the rotation angle becomes high. Therefore, by attaching a marker in accordance with the range in which the detection accuracy of the rotation angle of the surgical instrument is desired to be increased, the rotation angle can be detected with higher accuracy in the range where accuracy is required than when the marker of FIG. 6A is used. There is an effect that can.

  In any of the markers in FIGS. 6A to 6C, the boundary of the region is continuous without interruption, and therefore the rotation angle of the surgical instrument can be continuously detected in the entire range.

  In the present embodiment, the markers are colored in the blue region and the green region, but this has an advantage that when an image is identified by RGB parameters, it can be easily distinguished from red in the body. Apart from this, the method of arranging two regions with fluorescent paints having different wavelengths has an advantage that only the fluorescent region can be extracted by filtering for detecting a specific wavelength regardless of the color in the body.

  In the above embodiment, the control of the soft medical forceps is mainly taken as an example, but the present invention is not limited to this. Needless to say, the present invention can be widely applied to the case where the working end of an industrial robot manipulator, piping, or other inspection / working robot is operated from the operating end via the flexible operation section.

1 is an overall configuration diagram of a medical endoscope system according to an embodiment of the present invention. 1 is a functional block diagram of an image processing apparatus according to an embodiment. The functional block diagram of an image correction means. The functional block diagram of an angle detection means. The functional block diagram of an information generation means. Expanded view of the marker (region segmentation including triangle vertices). Expanded view of the marker (area division by diagonal lines). Expanded view of the marker (region division by trigonometric curve). Explanatory drawing at the time of observing the state which stuck the marker on the handle of the surgical instrument with the endoscope. The development view of a set of markers. Explanatory drawing which shows the case where it becomes the same blue-green ratio on the two slopes of a marker. Explanatory drawing which shows the selection method of a slope. The flowchart which shows the detection procedure of the rotation angle by an image processing means (the 1). 7 is a flowchart (part 2) illustrating a procedure for detecting a rotation angle by the image processing unit. 7 is a flowchart (No. 3) showing the procedure for detecting the rotation angle by the image processing means. Explanatory drawing of a marker image and RGB image. Explanatory drawing of the straight line segment extraction of a marker area | region. Explanatory drawing of affine transformation matrix creation. The enlarged view of the marker after an affine transformation. The flowchart which shows a display image generation process. Explanatory drawing which shows the result of a display image generation process (the 1). Explanatory drawing which shows the result of a display image generation process (the 2). Explanatory drawing which shows the mode of the operation part and endoscope screen of a medical instrument.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Operation part of medical instrument, 1 '... Flexible transmission part, 2 ... Endoscope, 2' ... Endoscope insertion part, 3 ... Insertion means, 4 ... Surgical instrument part, 5 ... Marker, 6 ... Endoscope Mirror lens, 7 ... Image controller, 8 ... Image processing device, 9 ... Image capture means, 10 ... Image correction means, 11 ... Image filtering means, 12 ... Angle detection means, 13 ... Information generation means, 14 ... Display information output means 15 ... Data recording means, 16 ... Luminance correction means, 17 ... Color correction means, 18 ... Distortion correction means, 19 ... Marker area extraction means, 20 ... Edge extraction means, 21 ... Enlargement ratio correction means, 22 ... Marker reading means , 23 ... Angle calculation means, 24 ... Angle correction means from reference, 25 ... Display image correction means, 26 ... Display image creation means, 27 ... Image composition means, 30 ... Processing distribution means, 40 ... Monitor, 50 ... Marker sheet .

Claims (14)

The operation method of the apparatus for detecting the rotation state of the surgical instrument part using an imaging means for imaging the operation of the surgical instrument part in the surgical field while transmitting the operation amount of the operation part to the surgical instrument part by the transmission part,
The transmission unit has a marker provided on the surgical instrument unit side of the transmission unit and provided with a continuous pattern in the circumferential direction, and the marker has at least one boundary line that divides the region of the continuous pattern, It is a straight line or a curve in which the position in the axial direction changes continuously with respect to the circumferential direction, captures the marker image information obtained by imaging the marker by the imaging means, performs image processing, and obtains the rotation angle of the marker An operation method of a rotation state detection device, characterized by detecting a rotation state of a surgical instrument part. The operation method of the apparatus for detecting the rotation state of the surgical instrument part using an imaging means for imaging the operation of the surgical instrument part in the surgical field while transmitting the operation amount of the operation part to the surgical instrument part by the transmission part,
The transmission unit includes a marker provided on the surgical instrument unit side of the transmission unit and provided with a continuous pattern that is continuous in the circumferential direction, and the marker has one boundary line that divides the region of the continuous pattern as a straight line The other boundary line with respect to this boundary line has a straight line or a curve whose axial position continuously changes, and the marker includes a polygon or a curved figure by these boundary lines, and the marker The feature amount of the color is different in the axial direction across the boundary line, the marker image information obtained by imaging the marker by the imaging means is captured, image processing is performed, and the rotation state of the surgical instrument unit is obtained by obtaining the rotation angle of the marker A method for operating a rotational state detection device, characterized in that 3. The measurement line indicating the center in the depth direction of the marker display area in the marker image information is obtained by performing image processing on the marker image information according to claim 1, and divided in the axial direction along the measurement line. An operation method of a rotation state detection device for obtaining a ratio of a region and obtaining a rotation angle from a circumferential position on the marker corresponding to the ratio. 4. The method of operating a rotational state detection device according to claim 3, wherein the marker has a blue and green region across a boundary line that divides the continuous pattern region, and the ratio is a ratio of the blue region and the green region. . The operating method of the rotation state detection device according to claim 1, wherein the marker has a plurality of regions of the continuous pattern in an axial direction of the transmission unit. In Claim 3, when there are two rotation angles corresponding to the ratio, the intersection of the horizontal axis of the marker display area and the measurement line in the marker image information, and the center of gravity of the surface of the divided one area And an extension line connecting the intersections, and an operation method of the rotation state detection device that selects a rotation angle based on an angle formed by the measurement line and the horizontal axis and an angle formed by the extension line and the horizontal axis. 7. The surgical instrument display area in the captured image of the imaging unit according to claim 6, wherein an instruction image indicating the operational direction of the surgical instrument unit is generated based on a relative angle between the marker and the operational direction and an angle measured by a measurement line. The operation method of the rotation state detection apparatus which attaches and displays. The operation method of the apparatus detects the rotation state of the surgical instrument part using an imaging means for imaging the operation direction of the operation part to the surgical instrument part by the transmission part and images the operation of the surgical instrument part in the surgical field. And
An image obtained by capturing marker image information obtained by imaging the marker by the imaging means, having a marker provided on the surgical instrument unit side of the transmission unit and having a continuous pattern in which regions having different feature amounts are continuous in the circumferential direction. When the rotation angle of the marker is detected to detect the rotation state of the surgical instrument unit, the distortion of the marker display area in the captured marker image information is corrected to generate corrected image information, and this corrected image information Filtering means extracts boundary information of the marker image area extracted by separating the marker from the background and separating the marker pattern based on the feature quantity of the marker image, and creates an affine transformation matrix from the boundary information. And using the affine transformation matrix to correct the marker image region viewed from the front of the marker, and the feature on the measurement line passing through the center of the corrected marker image region Method of operating a different seek length ratio of the areas, the rotational state detection device and obtains the angle corresponding to the ratio as the rotation angle of the surgical instrument unit. 9. The region having different feature values is a region colored in blue and green, and the filtering unit converts the correction image into R (red image), G (green image), and B (blue). Rotation state detection characterized by extracting a marker region by a combination of a green region and a blue region by subjecting the divided image to a combination process of subtracting R from the sum of G and B for each pixel. How the device works . 9. The region of the fluorescent dye according to claim 8, wherein the regions having different feature quantities are regions that are separately coated with different fluorescent dyes, and the filtering unit extracts only the excitation wavelength or the fluorescence wavelength of the fluorescent dye. And a method of operating the rotation state detecting device, wherein the marker region is extracted. A surgical tool that performs work in the surgical field, an operation unit thereof, a transmission unit that transmits an operation direction of the operation unit to the surgical tool unit, an imaging unit that observes the operation of the surgical tool unit in the surgical field, and an operation performed by the imaging unit An apparatus for detecting the rotation state of the tool part,
A continuous pattern marker provided in the circumferential direction on the surgical instrument side of the transmission unit, and an image processing device that performs image processing of a marker image captured by the imaging unit and obtains a rotation angle of the marker,
The image processing apparatus includes: an image capturing unit that captures an image of a surgical instrument unit and a marker; an image correcting unit that corrects distortion of the captured image; and a marker and a background that are separated from the corrected image based on a feature amount of the marker. And filtering means for extracting the marker area by separating the marker pattern, extracting the edge of the extracted marker area, creating an affine transformation matrix from the edge, and using the affine transformation matrix to view from the front of the marker And when the marker is a set of a continuous pattern of the first color and the second color, the first color region on the measurement line passing through the center of the enlarged marker image An angle detection means for obtaining a ratio of the length of the second color region and obtaining an angle according to the ratio as a rotation angle of the surgical instrument unit, and determining a rotation angle of the marker to determine a rotation state of the surgical instrument unit. Rotational state detecting apparatus, characterized in that the output.   12. The filtering means according to claim 11, wherein when the first color is green and the second color is blue, the corrected image is R (red image), G (green image), B ( A rotation state characterized by extracting a marker region by a combination of a green region and a blue region by performing a composition process of subtracting R from the sum of G and B for each pixel on the divided image Detection device.   12. The method according to claim 11, wherein an image such as an arrow indicating an operation direction of the surgical instrument unit is generated based on a relative angle between the marker and the operation direction and an angle measured by a measurement line, and the image is attached to the image of the surgical instrument unit. A rotation state detecting device comprising display information generating means for displaying the information.   12. The imaging device according to claim 11, wherein the imaging unit is an endoscope, and the surgical instrument unit is a medical instrument such as a forceps or an electric scalpel in which an operation unit is operated by a doctor and treatment is performed, or the endoscope A rotational state detection apparatus, characterized in that the endoscope is thinner than a mirror, and the transmission section is a flexible transmission section that may cause twisting.

Images