JP2009172003A - Needle point position estimating device and needle point position estimating method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a needle point position estimating device and a needle point position estimating method, capable of accurately estimating the position of the point of a needle of a puncture device pierced into an object for puncture regardless of the structure of the puncture device. <P>SOLUTION: The needle point position estimating device comprises an illumination part, an image acquisition part, and a needle point position estimating part. The illumination part irradiates the needle with light. The image acquisition part acquires the image including the needle, the object for puncture to be punctured with the needle, and the needle projection image projected on the object for puncture illuminated by the illumination part. The needle point position estimating part determines an estimated position of the needle point when the object for puncture is punctured based on the image acquired by the image acquisition part and needle information. The needle information includes the needle image length representing the length of the needle on the image when the position of the needle point in the image acquired by the image acquisition part matches the position of the projected needle point. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an apparatus for supporting a needle puncturing operation, and more particularly to an apparatus and method for estimating the tip position of a needle in a puncture object.

  In order for a doctor or nurse to perform intravenous injection or infusion on the patient's arm, etc., first check the exact position of the patient's vein to puncture and insert the injection needle or catheter, and insert the injection needle. To secure blood vessels. There are the following devices as devices for supporting this work. First, images of blood vessels and injection needles are extracted from image frames of blood vessels and injection needles imaged by a camera using image processing. Next, a blood vessel enhancement image is generated from the extracted blood vessel image. Then, from the extracted information on the injection needle, the tip position (needle tip predicted position) of the needle that is hidden under the puncture and cannot be seen is estimated. Finally, the needle tip mark image schematically representing the estimated needle tip predicted position and the blood vessel emphasized image are combined and displayed on a liquid crystal display or the like (see, for example, Patent Document 1).

  FIG. 13 shows a block diagram of a conventional needle tip position presentation device 1000. The illumination unit 1101 that has received a light emission instruction from the image acquisition unit 1102 alternately emits visible light and near-infrared light. These lights are applied to the arm that is the puncture object. The image acquisition unit 1102 acquires a visible light image by imaging the reflected light from the arm during visible light irradiation. Moreover, the near-infrared image is acquired by imaging the reflected light from the arm during near-infrared light irradiation. Needle tip position estimating means 1103 estimates the tip position of the needle existing under the arm from the visible light image and generates a needle tip mark image. The blood vessel position detecting means 1104 generates a blood vessel emphasized image in which a blood vessel portion is extracted from the near infrared image and emphasized. The display unit 1105 generates and displays a composite image obtained by combining the needle tip mark image with the generated blood vessel emphasized image.

The outline of processing performed by the conventional needle tip position estimating means 1103 will be described below with reference to FIGS. 14, 15 (a), and 15 (b). First, as shown in FIG. 15A, pixel lengths P and Q are extracted as a calibration operation before inserting the syringe into the arm. Next, in FIG. 14, the syringe image extraction means 1201 extracts the outline and scale of the syringe from the visible light image obtained by photographing the syringe. Based on these, the needle tip position is detected by the needle tip position detection means 1202, and the pixel lengths P and Q are extracted and stored. Next, in the state where the nurse has pierced the patient's arm, as shown in FIG. 15 (b), the tip of the needle is subcutaneous, but the pixel length Y that can be detected from the captured visible light image, Then, from the stored pixel lengths P and Q, X is calculated as a pixel length obtained by multiplying P / Q by Y, and predicts a needle tip position that is hidden under the skin and cannot be seen. In FIG. 14, the syringe image extraction means 1201 extracts the outline and scale of the syringe from the visible light image, and the injection needle tip position calculation means 1203 uses these information and the stored information on P and Q. The pixel length X is calculated to calculate the subcutaneous needle tip position. Based on the calculated hypodermic needle tip predicted position, the injection needle tip position image generating unit 1204 generates an image in which this needle tip position predicted position is emphasized and outputs it as a needle tip mark image.
JP 2006-130201 A (published May 25, 2006)

As described above, in the conventional technique, the tip end position of the needle is estimated by analyzing the outline of the syringe provided with the needle and the scale marked on the surface thereof based on the image acquired by the image acquisition means.
However, in the above prior art, when the scale of the syringe is not accurately acquired or when a syringe without a scale is used, the tip position of the needle during puncture cannot be accurately estimated. That is, depending on the configuration of a device (instrument) including a needle, the tip position of the needle present in the puncture object may not be estimated.

  An object of the present invention is to accurately estimate the tip position of a needle punctured inside a puncture object regardless of the configuration of the puncture device.

  The needle tip position estimation device according to the first invention includes an illumination unit, an image acquisition unit, and a needle tip position estimation unit. The illumination unit irradiates the needle with light. The image acquisition unit acquires an image including a needle, a puncture object into which the needle is punctured, and a needle shadow projected onto the puncture object of the needle illuminated by the illumination unit. The needle tip position estimation unit determines a predicted needle tip position when the needle is punctured into the puncture object based on the image acquired by the image acquisition unit and the needle information. The needle information includes a needle image length indicating the length of the needle on the image when the needle tip position and the needle shadow tip position in the image acquired by the image acquisition unit match.

  Here, the needle tip position estimation device that estimates the tip position (needle tip predicted position) of the needle punctured in the puncture target determines the predicted needle tip position regardless of the form of the device (instrument) including the needle. It adopts a configuration that makes it possible to do this.

  Here, the needle is, for example, an injection needle provided in a medical syringe. The puncture object is, for example, a human or animal arm or leg. Therefore, the shadow of the needle is, for example, the shadow of the injection needle reflected on the human skin.

  The illumination unit preferably has a light source, but may have a configuration in which external light is limitedly taken by having a daylighting window without providing a light source. In addition, the position (angle) of the illumination unit may be a position where the image acquisition unit can acquire an image that can identify a needle and a needle shadow.

  The image acquisition unit is preferably a camera that can capture visible light and near infrared light and has high spectral sensitivity, for example. Specifically, it is preferable to use a monochrome CCD camera that represents the luminance of each pixel by 8 bits. In addition, an image may be acquired from data obtained from an external device without including such a camera capable of imaging.

  Then, the needle tip position estimation unit analyzes the image information of the needle and the needle shadow acquired by the image acquisition unit, and determines the tip position (needle tip predicted position) of the needle that is in the puncture object. To do. For example, the needle length (needle image length) on the image when the needle tip position matches the needle shadow tip position is analyzed and stored, and the needle length on the stored image is used for puncturing. Determine the needle tip position (predicted needle tip position). The needle image length is the length of the needle shown in the image acquired by the image acquisition unit, and can be represented by, for example, one pixel as a unit length.

  The needle tip position means the tip position of the needle when the needle is not punctured into the puncture object, and the tip position of the non-puncture portion of the needle when the needle is pierced with the puncture object, That is, it means the intersection of the surface of the puncture object and the needle. The predicted needle tip position means the tip position of the needle in a state where the needle is punctured into the puncture target. Therefore, the predicted needle tip position is the needle tip position within the puncture object.

  With this configuration, when the tip of the needle coincides with the tip of the needle shadow, it is just before the needle is punctured, and thereafter the spatial angle of the needle changes even if the needle is punctured into the puncture object. Therefore, the length of the needle on the image (the sum of the length of the puncture portion and the length of the non-puncture portion) does not change. For this reason, unlike the prior art, it is not necessary to use the non-puncture portion in the puncture device (instrument) as a reference on the image for estimating the needle length and the tip position.

  Therefore, it is not necessary for the puncture device itself to have another reference (for example, the scale) necessary for determining the predicted needle tip position, and any device (instrument) having a needle is used. In addition, it is possible to estimate the predicted needle tip position.

  As a result, for example, when a needle is inserted into a human body like a syringe, the user of this syringe can always grasp the predicted needle tip position. Accordingly, the accuracy and time required for puncturing the needle can be improved, and the burden on the puncture target (puncture target) can be reduced.

  A needle tip position estimation device according to a second invention is the needle tip position estimation device according to the first invention, wherein an image acquired by the image acquisition unit and a predicted needle tip position determined by the needle tip position estimation unit are obtained. A presentation unit for presenting is further provided.

Here, a presentation unit for presenting the predicted needle tip position to the user is further provided.
Here, the presenting unit is preferably a thin display such as a liquid crystal or an organic EL. Alternatively, a printing apparatus that prints on a predetermined sheet may be used.

  Thus, the user can puncture the needle while constantly checking where the tip of the needle is located while puncturing the needle into the puncture object. As a result, it is possible to easily and stably perform puncturing work regardless of the skill level and technical skill of the user.

  A needle tip position estimation device according to a third invention is the needle tip position estimation device according to the first or second invention, and includes a needle information extraction unit and a needle tip predicted position calculation unit. The needle information extraction unit extracts needle information including the needle direction, the needle tip position, and the needle rear end position from the image acquired by the image acquisition unit. The predicted needle tip position calculation unit calculates the predicted needle tip position based on the needle information extracted by the needle information extraction unit.

Here, the predicted needle tip position is calculated by the predicted needle tip position based on the needle information extracted from the image by the needle information extraction unit.
Here, the needle direction is a direction from the rear end to the front end of the needle. The needle tip position is the same as the above-described needle tip position, but may be a pixel in which an end on the puncture side is shown on the image, or a pixel in which the tip of the needle is shown. The surrounding pixels may be included. The needle rear end position is a rear end portion of the needle, that is, a contact portion between the needle and a device (instrument) including the needle, and may be a pixel in which the contact portion is reflected on the image, or the pixel. And surrounding pixels may be included.

Thus, before the needle is punctured, the image acquisition unit acquires an image from a position at which at least the entire needle can be acquired, so that even if the position of the needle changes, the needle tip position and the predicted needle tip position are always automatically obtained. Can be estimated.
As a result, the accuracy when puncturing the needle is further improved, and it becomes possible to puncture even a fine puncture object easily and safely.

  A needle tip position estimation device according to a fourth invention is the needle tip position estimation device according to the third invention, wherein the needle tip position estimation unit includes a needle shadow information extraction unit, a coincidence detection unit, and a flag storage unit. And a needle image length storage unit. The predicted needle tip position calculation unit calculates the predicted needle tip position based on the needle image length stored in the needle image length storage unit when the flag is ON. The needle shadow information extraction unit extracts needle shadow information including the tip position of the needle shadow from the image acquired by the image acquisition unit. The coincidence detection unit outputs a coincidence signal when the needle tip position matches the needle shadow tip position. The flag storage unit stores a flag that is turned on when the coincidence signal is output and that is turned off when the coincidence signal is not output. The needle image length storage unit stores the needle image length when the flag is switched from the OFF state to the ON state.

  Here, the needle shadow tip position is extracted, and when the flag is ON based on the needle image length when the needle shadow tip position and the needle tip position coincide, that is, when the needle is present in the puncture object. The predicted needle tip position is calculated.

Thereby, the needle image length of the image acquired immediately before puncturing the puncture object is stored. Here, the three-dimensional angle of the needle hardly changes between immediately before puncturing and after puncturing. However, while the needle is moved to the point where the tip of the needle is punctured, the angle of the needle changes variously. Since the image acquired by the image acquisition unit is a two-dimensional image, the length of the needle in the image (needle image length) may change depending on the three-dimensional angle of the needle. For this reason, it is difficult to accurately grasp the length of the needle. For these reasons, it is possible to accurately determine the predicted needle tip position after puncturing by storing and using the needle image length when the tip of the needle and the tip of the needle shadow coincide.
Therefore, the predicted needle tip position can be determined even when the needle is punctured into the puncture target and the tip of the needle cannot be seen.

  A needle tip position estimation device according to a fifth invention is the needle tip position estimation device according to any one of the first to fourth inventions, wherein the needle tip position estimation unit displays a predicted needle tip position presentation image. An image forming unit to be formed is further provided.

Here, in order to present on the image the predicted needle tip position determined by the needle tip position estimation unit, an image forming unit that forms a predicted needle tip position presentation image is provided.
Here, the mark indicating the needle tip position in the needle tip predicted position presentation image may be an arrow, for example. In addition, it is preferable that the formed needle tip predicted position presentation image is combined with the image acquired by the image acquisition unit and output. The output destination may be a display or a printing apparatus for printing on a predetermined sheet. And when it is presented on a display or paper, it is preferable to output so that the direction of the needle coincides with the direction indicated by the arrow.

Thereby, since the image forming unit forms an image for easily presenting the predicted needle tip position determined by the needle tip position estimating unit to the user, it is possible to clearly present the predicted needle tip position. Become.
As a result, the user can easily and accurately confirm the location of the needle tip and the direction of the needle.

  A needle tip position estimation device according to a sixth invention is the needle tip position estimation device according to any one of the third to fifth inventions, wherein the predicted needle tip position calculation unit calculates the needle from the needle rear end position. The position separated by the length of the needle image length stored in the needle image length storage unit in the direction is determined as the predicted needle tip position.

Here, the predicted needle tip position calculation unit calculates a position separated from the needle rear end position in the needle direction by the needle image length as the predicted needle tip position.
This makes it possible to determine the predicted needle tip position using the needle image length and needle direction acquired immediately before the needle is punctured.
Accordingly, an accurate predicted needle tip position can be automatically calculated.

  A needle tip position estimation device according to a seventh invention is the needle tip position estimation device according to any one of the first to sixth inventions, wherein the image acquisition unit outputs the acquired image as a monochrome luminance image. To do.

Thereby, if the pixel value of each pixel included in the image acquired by the image acquisition unit is, for example, an 8-bit image, it is graded in 256 steps from 0 to 255. For this reason, for example, the threshold value for performing image processing, the coefficient of the filter, and the like are simplified, and the predicted needle tip position can be determined by simple processing and / or configuration.
As a result, the processing speed of the needle tip position estimation device is improved, and the cost can be reduced.

  A needle tip position estimation device according to an eighth invention is the needle tip position estimation device according to any one of the third to seventh inventions, wherein the needle information extraction unit applies the image acquired by the image acquisition unit. The first filter is scanned to form a needle extraction image, and a first binarized image obtained by binarizing the needle extraction image with a first threshold value is generated. Further, the needle information extraction unit performs line differentiation on the pixel indicating the needle in the first binarized image using the least square method, and from this line segment, the needle image length, the needle direction, the needle tip position, and the needle back Get end position.

  Here, the needle image length, the needle direction, the needle tip position, and the needle back are obtained from the line segment formed by performing the first filter scan, binarization, and least square method on the image acquired by the image acquisition unit. Get end position.

  Here, the first filter is, for example, a sequence of predetermined coefficients, and the coefficients may be −2, −1, 0, 1, and 2. And scanning this 1st filter means filtering by a 1st filter, for example, performing the predetermined calculation with respect to the coefficient contained in a 1st filter with respect to each pixel. Further, the predetermined calculation here may be, for example, a product-sum calculation. The first threshold value is a threshold value for luminance. For example, in an image indicating luminance of 8 bits, that is, 0 to 255, the threshold value may be 150. The binarized image is an image in which each pixel is replaced with “0” or “1” by the threshold value. For example, a color corresponding to “0” is white and a color corresponding to “1” is set. It may be imaged as black.

  Thereby, it is possible to perform image processing by calculating a numerical parameter using each pixel value for a plurality of pixels constituting the acquired image. For this reason, it becomes possible to extract needle information automatically and uniquely from the acquired image. Therefore, it is possible to save the labor of performing calibration every time of use or rewriting each parameter by the user.

  A needle tip position estimation device according to a ninth invention is the needle tip position estimation device according to the eighth invention, wherein the needle information extraction unit includes a needle that does not include a needle based on the line segment in the image. The needle shadow information extraction unit scans a second filter having a predetermined length and having the second threshold value as a threshold value in the first region, and second binarization is determined. An image is generated, and in the pixel matrix of the second binarized image, scanning is performed from both ends of the image along each row direction or each column direction. In each row or each column, a third binarized image is formed by extracting the center pixel of each boundary pixel, and the least square method is applied to the pixel corresponding to the center pixel in the third binarized image. Using the, the line is differentiated, and the needle shadow tip position is obtained from this line segment.

  Here, in the first region determined by the needle information extraction unit, the needle shadow information extraction unit is formed by sequentially performing the second binarization, the boundary pixel specification, the third binarization, and the least squares method. Determine the needle shadow tip position from the line segment.

  Here, the first region is generated based on the line segment generated in the needle tip position estimation device according to the eighth invention, and may be, for example, a rectangular region including the line segment on one side. Moreover, it is preferable that the area | region is an area | region on the opposite side to the illumination part side centering on a needle | hook. The center position of the boundary pixel is, for example, a pixel that is the center of two boundary pixels when the number of pixels between two boundary pixels in the same row is an odd number. Is one of the two pixels adjacent to the pixel. Therefore, the third binarized image is, for example, a binarization process in which a pixel corresponding to the center pixel of two boundary pixels in the same row is set to 1 and other pixels in the same row are set to 0 in each row. It may be a performed image.

Thereby, it is possible to perform image processing by calculating a numerical parameter using each pixel value for a plurality of pixels constituting the acquired image. For this reason, it becomes possible to extract needle shadow information automatically and uniquely from the acquired image.
Therefore, it is possible to save the labor of performing calibration every time of use or rewriting each parameter by the user.

  A needle tip position estimation device according to a tenth invention is the needle tip position estimation device according to any one of the third to ninth inventions, wherein the coincidence detection unit is a second region centered on the needle tip position. When the needle shadow tip position is present in the second region, it is determined that the needle tip position matches the needle shadow tip position.

Here, the coincidence detection unit determines the second region, and when the needle shadow tip position exists in the second region, it is determined that the needle tip position matches the needle shadow tip position.
Here, the second area is an area on the image having a predetermined vertical and horizontal width, and may be, for example, an area of 10 pixels square.

  Thereby, even if it is difficult to recognize the shadow of the puncture part by the image because the puncture part that is the tip part of the needle is extremely thin, before the needle is punctured, the coincidence detection unit is It can be determined that the needle tip position matches the needle shadow tip position.

  Therefore, it is possible to prevent the timing of the coincidence signal output from the coincidence detection unit from being delayed. That is, it is possible to prevent the needle image length stored in the needle image length storage unit from being stored as a needle image length shorter than the actual needle image length. This is because when the needle tip predicted position is presented closer to the needle rear end side than the actual position when the needle is punctured, there is a risk that the puncture will erroneously occur to the area (depth) that should not be punctured. is there. However, according to the configuration of the needle tip position estimating device, there is no such a possibility, and thus it becomes possible to prevent erroneous needle puncture.

  A needle tip position estimating device according to an eleventh invention is the needle tip position estimating device according to any one of the fourth to tenth inventions, and when the flag is OFF, based on the needle tip position. Determine the needle tip position.

Here, the needle tip position is determined based on the needle tip position extracted by the needle information extraction unit from the acquired image in a state before the needle is punctured into the puncture target.
Thereby, the process for presenting the tip of the needle is clearly switched between before and during the puncture of the needle. Therefore, the tip position of the needle can be presented based on the position of the tip of the needle before puncturing the needle and based on the predicted position of the tip of the needle during puncturing of the needle.

  Therefore, it becomes easy for the user to check the position of the tip of the needle before puncturing, and for example, it is possible to prevent a problem that it is difficult to check the tip of the needle depending on the angle of illumination light.

  A needle tip position estimating method according to a twelfth invention includes first to third steps. In the first step, the needle is irradiated with light. In the second step, an image including a needle, a puncture object to be punctured by the needle, and a needle shadow projected on the puncture object of the needle irradiated with light in the first step is acquired. The third step includes needle information including the image acquired in the second step, and a needle image length indicating the length of the needle on the image when the needle tip position matches the needle shadow tip position in the image. Based on the above, the predicted needle tip position when the needle is punctured into the puncture object is determined.

  Here, the needle tip position estimation method for estimating the tip position (needle tip predicted position) of the needle punctured in the puncture object determines the predicted needle tip position regardless of the form of the device (instrument) provided with the needle. Adopting a method that makes it possible to do.

  Here, the needle is, for example, an injection needle provided in a medical syringe. The puncture object is, for example, a human or animal arm or leg. Therefore, the shadow of the needle is, for example, the shadow of the injection needle reflected on the human skin.

  In the third step, information on the needle and the needle shadow is extracted and analyzed from the needle and needle shadow image acquired in the second step, and the tip position (needle tip) of the needle that is in the puncture object is extracted. (Predicted position) is determined. For example, the needle length (needle image length) on the image when the needle tip position matches the needle shadow tip position is analyzed and stored, and the needle length on the stored image is used for puncturing. Determine the needle tip position (predicted needle tip position). The needle image length is the length of the needle shown in the image acquired in the second step, and can be expressed, for example, as a unit length of one pixel.

  The needle tip position means the tip position of the needle when the needle is not punctured into the puncture object, and the tip position of the non-puncture portion of the needle when the needle is pierced with the puncture object, That is, it means the intersection of the surface of the puncture object and the needle. The predicted needle tip position means the tip position of the needle in a state where the needle is punctured into the puncture target. Therefore, the predicted needle tip position is the needle tip position within the puncture object.

  By such a method, when the tip of the needle coincides with the tip of the needle shadow, it is immediately before puncturing the needle, and the spatial angle of the needle changes even after the needle is punctured into the puncture object. Therefore, the length of the needle on the image (the sum of the length of the puncture portion and the length of the non-puncture portion) does not change. Therefore, unlike the prior art, it is not necessary to use a non-punctured portion of the puncture device (instrument) as a reference on the image for estimating the needle length and the tip position.

  Therefore, it is not necessary for the puncture device itself to have another reference (for example, the scale) necessary for determining the predicted needle tip position, and any device (instrument) having a needle is used. In addition, it is possible to estimate the predicted needle tip position.

  As a result, for example, when a needle is inserted into a human body like a syringe, the user of this syringe can always grasp the predicted needle tip position. Accordingly, the accuracy and time required for puncturing the needle can be improved, and the burden on the puncture target (puncture target) can be reduced.

  A needle tip position estimating method according to a thirteenth invention is the needle tip position estimating method according to the twelfth invention, wherein the third step further includes fifth to ninth steps. In the fifth step, needle shadow information including the needle shadow tip position is extracted from the image acquired in the first step. In the sixth step, it is determined whether or not the needle tip position matches the needle shadow tip position. In the seventh step, when it is determined in the sixth step that the needle tip position and the needle shadow tip position match, the flag stored in the OFF state in advance is updated to the ON state. The eighth step stores the needle image length when the flag is switched from the OFF state to the ON state. In the ninth step, the predicted needle tip position is calculated when the flag is ON based on the needle image length stored in the eighth step.

  Here, the needle shadow tip position is extracted, and when the flag is ON based on the needle image length when the needle shadow tip position and the needle tip position coincide, that is, when the needle is present in the puncture object. The predicted needle tip position is calculated.

Thereby, the needle image length of the image acquired immediately before puncturing the puncture object is stored. Here, the three-dimensional angle of the needle hardly changes between immediately before puncturing and after puncturing. However, while the needle is moved to the point where the tip of the needle is punctured, the angle of the needle changes variously. Since the image acquired by the image acquisition unit is a two-dimensional image, the length of the needle in the image (needle image length) may change depending on the three-dimensional angle of the needle. For this reason, it is difficult to accurately grasp the length of the needle. For these reasons, it is possible to accurately determine the predicted needle tip position after puncturing by storing and using the needle image length when the tip of the needle and the tip of the needle shadow coincide.
Therefore, the predicted needle tip position can be determined even when the needle is punctured into the puncture target and the tip of the needle cannot be seen.

  A needle tip position estimation method according to a fourteenth invention is the needle tip position estimation method according to the twelfth or thirteenth invention, further comprising a fourth step of presenting the predicted needle tip position determined in the third step. .

Here, information including the predicted needle tip position is presented.
Here, the presentation in the fourth step may be presented by being displayed on a display, or may be presented by printing on paper. Further, separately acquired images (for example, near-infrared light images) and the like may be combined with the predicted needle tip position and presented simultaneously.

  Accordingly, the user can puncture the needle while confirming where the tip of the needle is located based on the presented information while the needle is being punctured into the puncture object. . As a result, it is possible to easily and stably perform puncturing work regardless of the skill level and technical skill of the user.

A needle tip position estimation method according to a fifteenth aspect of the invention is the needle tip position estimation method according to the fourteenth aspect of the invention, and displays the predicted needle tip position determined in the third step on the display device.
Here, the predicted needle tip position is presented using a display device.

  Here, the display device may be a display as described above, and is preferably a lightweight and thin display device such as a liquid crystal display or an organic EL display. Further, as described above, it is desirable to synthesize and display a separately acquired image (for example, a near-infrared light image) and the predicted needle tip position.

As a result, by observing the display device, it is possible to puncture the needle while constantly checking the predicted needle tip position.
In addition, as described above, a lightweight and thin display can be easily carried and installed (attached), and the implementation place and applications can be expanded.

  According to the needle tip position estimation device and the needle tip position estimation method of the present invention, the position of the needle tip that is invisible and visible in any living body is predicted and displayed together with biological information such as blood vessels. It becomes possible to do.

  Hereinafter, embodiments of a needle tip position estimating apparatus and a needle tip position estimating method of the present invention will be described in detail with reference to the drawings. In the present embodiment, the puncture object is a human arm (skin), the needle is a commonly used injection needle, and the user of the needle tip position estimation device places the needle in the arm. It is assumed that the purpose is to puncture a certain vein.

(Embodiment)
The needle tip position estimation device 1 according to an embodiment of the present invention will be described below with reference to FIGS.
[Configuration of Needle Tip Position Estimation Device 1]
As shown in FIG. 1, the needle tip position estimation device 1 according to the present embodiment mainly includes a device fixing unit 2, an arm unit 3, a head unit 4, a visible light LED (illumination unit) 6, and a liquid crystal display (display). Part) 9.

  The device fixing unit 2 is provided to fix the needle tip position estimating device 1 to the patient's arm 11 which is a puncture object, and is a belt (illustrated) that can be wound around the patient's arm 11. Not).

The arm part 3 is provided to fix the apparatus fixing part 2 and the head part 4 to each other and to keep the distance between the head part 4 and the arm 11 at a predetermined distance.
The visible light LED 6 is provided on one side surface 3 a of the arm unit 3, and the illumination direction thereof is a direction that illuminates an area captured by a CCD camera included in the image acquisition unit 5 described later. That is, the visible light LED 6 irradiates light to a region where the needle 12a is punctured, and projects the shadow (needle shadow) 21 of the needle 11 on the puncture target.

As shown in FIG. 2, the head unit 4 includes an image acquisition unit 5, a blood vessel position detection unit 7, and a needle tip position estimation unit 101.
The image acquisition unit 5 includes a black and white luminance image CCD camera (hereinafter, not shown) that can capture visible light and near infrared light and represents the luminance of each pixel in 8 bits. Images taken by this CCD camera are acquired as a near-infrared image 701 and a visible light image 702 as shown in FIG. The near-infrared image 701 is transmitted to the blood vessel position detection unit 7, and the visible light image 702 is transmitted to the needle tip position estimation unit 101. In addition, this CCD camera captures an area including the skin 11a, the needle 12a of the syringe 12, and the needle shadow 21. Note that each image including the needle 12a and the needle shadow 21 is shown in the visible light image 702, and one side surface irradiated with visible light appears relatively white and the other side surface is relatively black. Reflect. Furthermore, the needle shadow 21 appears black in a region on the skin 11a opposite to the side irradiated with the visible light of the needle 12a. In the near-infrared light image 701, the vein 11b portion appears black compared to other portions, and other regions such as the needle 12a appear relatively white.

  The blood vessel position detection unit 7 receives the near infrared image 701 acquired by the image acquisition unit 5, and generates a blood vessel emphasized image 703 from the near infrared image 701 (see FIG. 3). This blood vessel enhanced image 703 is obtained by detecting a luminance region lower than a predetermined luminance as a blood vessel region from the near-infrared image 701 and imaging the detected region with contrast enhancement. This utilizes the characteristics of the near-infrared image 701 generated by the image acquisition unit 5 as described above.

  As shown in FIG. 2, the needle tip position estimation unit 101 estimates the tip position (needle tip predicted position 15) of the needle 12a located subcutaneously from the visible light image 702, and only the generated arrow image 602 exists at this position. A predicted needle tip predicted position presentation image 704 (see FIG. 3) is formed.

  The liquid crystal display 9 is installed on the upper side of the head unit 4 and is installed so as to be rotatable to a predetermined angle with one side as a rotation axis. As shown in FIG. 3, the liquid crystal display 9 displays a composite image 705 obtained by combining the images generated by the blood vessel position detection unit 7 and the needle tip position estimation unit 101.

[Configuration of Needle Tip Position Estimation Unit 101]
Next, the configuration of the needle tip position estimation unit 101 will be described with reference to FIGS. 4, 5, and 6.

  The needle tip position estimation unit 101 includes a needle information extraction unit 102, a needle shadow information extraction unit 103, a needle tip predicted position calculation unit 107, a needle image length storage unit 105, a coincidence detection unit 104, and a flag storage unit 106. And an image forming unit 108 that forms an image presenting the tip position of the needle.

  The needle information extraction unit 102 detects the shape of the needle in the visible light image 702, and the needle tip position 13, which is the tip of the visible portion of the needle 12a, the root of the needle 12a, that is, the contact point between the needle 12a and the syringe 12. The needle rear end position 14, which is the position, the needle image length Z indicating the length of the needle 12a, and the traveling direction (needle direction) of the needle 12a are extracted. The extracted information is on the image. For example, the needle image length Z is different from the actual length of the needle 12a. In addition, the needle information extraction unit 102 determines a first region 31 that indicates the vicinity of the needle 12 a for searching for the needle shadow 21.

The needle shadow information extraction unit 103 detects the needle shadow 21 from the first region 31 that indicates the vicinity of the needle 12a in the visible light image 702a, and extracts the needle shadow tip position 22 that is the tip position of the needle shadow 21.
The coincidence detection unit 104 always determines whether or not the needle tip position 13 and the needle shadow tip position 22 match. If they match, it is determined that the needle 12a is in contact with the skin 11a of the arm 11. Send a match signal.

  The flag storage unit 106 stores a flag for switching ON / OFF depending on whether or not a coincidence signal is transmitted. This flag is turned on when the coincidence signal is transmitted, and is turned off when the coincidence signal is not transmitted. Therefore, in the initial state, this flag is OFF.

The needle image length storage unit 105 stores the needle image length Z when the flag is switched from the OFF state to the ON state.
The predicted needle tip position calculation unit 107 has the needle image length Z stored in the needle image length storage unit 105 when the needle 12a is stuck under the arm 11, that is, when the coincidence signal is transmitted and the flag is ON. Based on the needle rear end position 14 and the needle direction extracted by the needle information extraction unit 102, the predicted needle tip position 15 that is hidden under the skin and is not visible is calculated.

  When the flag is OFF, that is, when the needle 12a is not yet in contact with the skin 11a of the arm 11, the image forming unit 108 is based on the needle tip position 13 and the needle direction extracted by the needle information extracting unit 102. A needle tip predicted position presentation image 704 is generated. Further, when the flag is ON, the image forming unit 108 generates a predicted needle tip position presentation image 704 based on the predicted needle tip position 15 and the needle direction calculated by the needle tip position calculator 107.

  Here, the outline of the processing performed by the needle tip position estimation unit 101 having the above-described configuration will be described with reference to FIGS. 5A to 5D. First, as shown in FIGS. 5A and 5C, at the moment when the needle 12a contacts the skin 11a, the tip position of the needle 12a and the tip position of the needle shadow 21 coincide. At this time, the flag changes from the OFF state to the ON state, and the needle image length Z at this time is stored in the needle image length storage unit 105. Thereafter, as shown in FIGS. 5B and 5D, the needle 12a advances toward the subcutaneous vein 11b. At this time, the angle formed by the needle 12a and the skin 11a does not substantially change. At this time, the sum of the image length X of the subcutaneous needle 12a portion and the image length Y of the visible needle 12a portion is the needle image length Z. Almost equal. For this reason, by calculating the position ahead by the needle image length Z from the needle rear end position 14 which is the root position of the needle 12a toward the traveling direction of the needle 12a, the needle tip predicted position 15 which is hidden under the skin and is not visible. Can be estimated.

  That is, since the needle tip position estimation unit 101 is in a state where the needle 12a is not stabbed subcutaneously when the flag stored in the flag storage unit 106 is OFF, the needle tip which is the tip of the needle 12a reflected in the visible light image 702a. Based on the position 13, a needle tip predicted position presentation image 704 is output. On the other hand, when the flag is ON, the needle 12a is stuck under the skin, so the needle 12a stored in the needle image length storage unit 105 from the needle rear end position 14 toward the direction indicated by the needle direction. A predicted needle tip position 15 is calculated using the presence of the tip of the needle 12a at a position ahead by the needle image length indicating the entire length, and the predicted needle tip is calculated based on the predicted needle tip position 15. A position presentation image 704 is output. Accordingly, the needle tip position estimation apparatus 1 can always display the tip position of the needle 12a on the blood vessel emphasized image 703 by combining the needle tip predicted position presentation image 704 and the blood vessel emphasized image 703.

[Specific Processing of Each Part in Needle Tip Position Estimation Unit 101]
Hereinafter, specific processing of each part in the needle tip position estimation unit 101 will be described with reference to FIGS. 1 to 11. Here, the processing of each unit will be specifically described on the assumption that the positional relationship among the syringe 12, the arm 11, and the needle tip position estimation device 1 is in the state shown in FIG.

(Needle information extraction unit 102)
Specific processing contents performed by the needle information extraction unit 102 will be described with reference to FIGS. 6A, 6B, 7A, and 7B. Note that the pixel coordinate system in the visible light image 702a shown in FIGS. 6A and 6B has an origin at the upper left in the drawing, the X direction in the horizontal direction, and the Y direction in the vertical direction. Axis. Further, the right direction on the X-axis and the downward direction on the Y-axis are the positive directions. As shown in FIG. 1, due to the positional relationship between the arm 11 and the needle tip position estimation device 1, the needle 12a appears to be approximately parallel to the Y-axis, and the syringe needle 12a is visible imaged during blood collection or infusion. It is inserted from the lower side to the upper side of 702a.

  6A is a visible light image 702a in which the needle 12a and the needle shadow 21 of the syringe 12 are reflected, and FIG. 6B shows the luminance value on a straight line parallel to the X axis including the line segment E1-E2. It is shown. This visible light image is 640 pixels wide and 480 pixels high, the pixel value of each pixel is 8-bit representation, the pixel value 0 represents black, and 255 represents white. As shown in FIG. 9, the visible light LED 6 that illuminates the needle 12a irradiates the needle 12a with light from the side surface of the needle 12a. Therefore, in the visible light image 702a, the side surface of the needle 12a on the visible light LED 6 side is brighter white than the skin 11a, and the side surface of the needle 12a opposite to the visible light LED 6 is darker than the skin 11a. Yes. At the same time, the needle shadow 21 shown in the visible light image 702a is shown as a black line segment on the skin 11a. As shown in FIG. 3, the visible light image is an image captured by irradiating the arm 11 with visible light, and thus does not show a blood vessel image.

  The needle information extraction unit 102 sequentially scans the needle extraction filter 32 for extracting the degree of shading (brightness) of the image of the needle 12a from the upper left pixel position in the visible light image 702 of FIG. That is, pixel by pixel is filtered from left to right in the X-axis direction, and when the pixel reaches the right end, the next pixel line in the positive Y-axis direction is filtered pixel by pixel from the left end to the right. Finally, filtering is performed up to the pixel at the lower right corner of the image. Here, the scanning of the needle extraction filter 32, that is, filtering, performs a product-sum operation on a plurality of coefficients of the needle extraction filter 32 and pixel values corresponding to these coefficients, and the product-sum value is converted into a new pixel value. Image processing. Thus, the image filtered by the needle extraction filter 32 becomes a needle information extraction image. However, when a value of 255 or more is calculated by filtering, the value is forcibly changed to 255, and when a value of 0 or less is calculated, the value is forcibly changed to 0.

  Here, as shown in FIG. 7A, when the luminance value of the surface of the arm 11 is about 30 to 40 by the light irradiated from the right side of the needle 12a, the luminance value of the right side surface of the needle 12a is about 80. It is bright and the left side surface of the needle 12a is dark when the luminance value is near zero. Here, the image of the needle 12a is narrower than the width of the image of the needle shadow 21, which is about 5 pixels in this embodiment. For this reason, the needle extraction filter 32 suitable for extracting needle information in the present embodiment is configured with coefficients of “−1, −2, 0, 2, 1” as shown in FIG. ing.

  In the needle information extraction image generated by the filter, the background is dark around the brightness value of 0, the needle 12a is bright white with a brightness value of about 200, and the boundary of the needle shadow 21 and a part of the side surface of the syringe 12 are brightness values. It becomes as pale as 80-90. A first binarized image is generated by binarizing the needle information extracted image with a first threshold (here, luminance value 150). The first binarized image is an image that has been subjected to image processing in which a pixel value less than the first threshold is set to 0 and a pixel value greater than or equal to the first threshold is set to 1 in the needle information extraction image.

  In the present embodiment, the first threshold value is a luminance value of pixels counted by 10% of the total number of pixels from the pixel with the highest luminance value toward the pixel with the lower luminance value in the needle information extraction image. To do. Here, 10% of the total number of pixels corresponds to the area ratio of the needle 12a in the visible light image 702, and is determined in advance depending on the magnification at the time of photographing by the image acquisition unit 5.

  Then, the needle information extraction unit 102 linearizes the pixel having the pixel value of 1 in the first binarized image using the least square method, and the pixel included in the generated line segment indicates the needle 12a. And Then, a binarized image is generated as a needle thin line image with the pixel value of this pixel being 1 and the pixel values of other pixels being 0.

  Then, the needle thin line image is scanned from the upper left pixel to the lower right pixel, and a pixel having a value of “1” is first determined to be a pixel indicating the tip of the needle 12a, and this pixel position is set as the needle tip position 13. . Further, this needle thin line image is scanned from the lower right pixel to the upper left pixel (that is, in the direction opposite to the above scanning), and the pixel having the value of “1” first indicates the root of the needle 12a (the needle rear end). The pixel position is determined, and this pixel position is set as the needle rear end position 14.

  Subsequently, as illustrated in FIG. 9, the needle information extraction unit 102 sets the extracted needle tip position 13 as pixel coordinates (HSX, HSY) and the needle rear end position as pixel coordinates (HKX, HKY). The needle image length Z is the square root of the sum of the square of (HKX−HSX) and the square of (HKY−HSY). Further, as the needle direction, an X component VX = HSX−HKX and a Y component VY = HSY−HKY in the needle traveling direction are obtained. Further, as shown in FIG. 6 (A1) as a first area 31 that indicates the periphery of the needle 12a, the needle 12a includes the pixels on the left side of the needle tip position 13 and the needle rear end position 14, respectively. A rectangular region (first region 31) of a predetermined size is prepared in a region opposite to the visible light source.

  As described above, if the first region 31 does not include the image of the needle 12a, it is possible to prevent erroneous detection of the image of the needle 12a as the needle shadow 21 in the processing of the needle shadow information extraction unit 103 described later. It becomes possible. For example, when it is known that the pixel width of the image of the needle 12a is 5 pixels in advance, the right side of the first region 31 in FIG. 6A is 3 pixels left from the pixel having the pixel value 1 in the needle thin line image. It consists of the pixel of.

(Needle shadow information extraction unit 103)
Specific processing contents performed by the needle shadow information extraction unit 103 will be described below with reference to FIGS. 6C and 6D. FIG. 6D shows luminance values on a straight line including the line segment E3-E1.

  The needle shadow information extraction unit 103 uses the second threshold value (TH1) binarized with respect to the pixels in the line segment E3-E1 area in the first area 31 indicating the periphery of the needle 12a of the visible light image. A converted image is generated (see FIG. 6C). Specifically, the needle shadow information extraction unit 103 first identifies pixels (pixel groups) that overlap the line segment E3-E1 region. Then, an average luminance value of this pixel group is obtained, and this average luminance value is set as the second threshold value. Therefore, since the second threshold value is based on the luminance value of the pixel overlapping the line segment E3-E1 region, the second threshold value is a value that changes by scanning the line segment E3-E1. Next, the needle shadow information extraction unit 103 determines whether or not there is a pixel having a pixel value lower than the second threshold among the pixel values of these pixels. Then, when there is a pixel having a pixel value lower than the second threshold, the needle shadow information extraction unit 103 determines that the needle shadow 21 exists in the line segment E3-E1 region, and the pixel overlaps the line segment E3-E1. Let the pixel value of the pixel located at the center of 1 be 1. On the other hand, the needle shadow information extraction unit 103 sets the pixel value to 0 when there is no pixel having a pixel value lower than the second threshold among the pixels in the line segment E3-E1 region. This process is performed in order from the upper left pixel (shift one pixel at a time horizontally from left to right, and once it reaches the right end, it is similarly shifted one pixel from the left end of the next line to the right, and finally to the lower right end of the image. The second binarized image is generated. Note that the length of the line segment E3-E1 is about twice the number of pixels as the width of the needle shadow 21 in a known visible light image obtained by imaging in advance.

  Subsequently, the needle shadow information extraction unit 103 generates a third binarized image. Specifically, the needle shadow information extraction unit 103 detects a pixel whose pixel value is first detected as a leftmost pixel when scanning from the left in each pixel line along the X-axis direction of the second binarized image. At the same time, a pixel having a pixel value of 1 detected first by scanning from the right is set as a right end pixel. Then, in the pixel line in which both the left end pixel and the right end pixel are detected, the pixel located at the center of the left end pixel and the right end pixel is determined as the center pixel of the needle shadow width, and the pixel value of this pixel is set to 1. The pixel value of this pixel is set to 0. In this manner, the needle shadow information extraction unit 103 generates a third binarized image having pixel values of 1 and 0.

  Subsequently, the needle shadow information extraction unit 103 line-divides the pixel having the pixel value 1 in the third binarized image using the least square method. The needle shadow information extraction unit 103 generates a binarized needle shadow thin line image by setting this line segment as a line segment indicating the needle shadow 21 and setting the pixels constituting this line segment to 1 and the other pixels to 0. To do. Then, the needle shadow information extraction unit 103 scans along the X-axis direction from the upper left pixel to the lower right pixel in the first region 31 that indicates the periphery of the needle 12a in the needle shadow thin line image, and first has a value of “1”. Is determined as the tip of the needle shadow 21 and is defined as the needle shadow tip position 22.

(Coincidence detection unit 104)
Next, specific processing contents of the coincidence detection unit 104 will be described below with reference to FIG.
The coincidence detection unit 104 generates a second region 33 centered on the needle tip position 13 as a coincidence range between the needle tip position 13 and the needle shadow tip position 22, and the needle shadow tip position 22 exists in the second region 33. When it does, it judges that needle tip position 13 and needle shadow tip position 22 matched, and transmits a coincidence signal. Specifically, as shown in FIG. 5, a 5 × 5 pixel region centered on the needle tip position 13 (black circle in FIG. 8) is set as the second region 33 as the matching range. Here, the coincidence detection unit 104 determines that the needle tip position 13 and the needle shadow tip position 22 match because the tip pixel (white circle in FIG. 8) of the needle shadow 21 is included in the 5 × 5 pixel region. Determination is made, that is, it is determined that the tip of the needle 12a has contacted the skin 11a, and a coincidence signal is transmitted. Here, the reason why the second region 33 that is the coincidence range is made as large as 5 × 5 pixels is that an error of about several pixels is allowed with respect to the needle tip position 13 and the needle shadow tip position 22. .

(Needle tip predicted position calculation unit 107)
Next, specific processing contents of the predicted needle tip position calculation unit 107 will be described below with reference to FIG. In the coordinate system here, the upward direction toward the paper surface is the positive direction of the Y axis, and the right direction is the positive direction of the X axis. In addition, an angle in the counterclockwise direction from the positive X-axis direction is θ.

  Point E is the needle rear end position 14 (HKX, HKY) extracted from the visible light image. Point F is the needle tip position 13 (HSX, HSY) extracted from the visible light image. Point P is the predicted needle tip position 15 (LPX, LPY) calculated by the predicted needle tip position calculation unit 107. The predicted needle tip position calculation unit 107 calculates sin θ = VY / FE and cos θ = VX / FE from the X component VX and the Y component VY in the needle direction. Then, using the image length Z of the line segment EP, the predicted needle tip position 15P is calculated as LPX = HKX + Z · cos θ and LPY = HKY + Z · sin θ. However, in the coordinate systems in FIGS. 6A and 6C, since the positive and negative of the Y axis are opposite, if the angle θ is counterclockwise from the X axis positive direction as in FIG. = HKY-Z · sin θ.

(Image forming unit 108)
Next, specific processing of the image forming unit 108 will be described below with reference to FIGS. 10 (a) and 10 (b).

  As shown in FIG. 10A, the image forming unit 108 stores and holds in advance a cursor composed of 7 pixels horizontally and 9 pixels vertically as an initial arrow image 601. This cursor has an arrow pattern formed by a binary bitmap in which black pixels are “1” and white pixels are “0”.

  When the flag is OFF, the image forming unit 108 matches the pixel position of the arrow tip of the initial arrow image 601 with the needle tip position 13 on the visible light image. Then, the generated arrow image 602 in which the needle direction and the direction indicated by the arrow coincide is generated by rotating the Y axis by an angle φ. Further, when the flag is ON, the image forming unit 108 matches the pixel position of the arrow tip of the initial arrow image 601 with the predicted needle tip position 15 on the visible light image. Then, the generated arrow image 602 in which the needle direction and the direction indicated by the arrow coincide is generated by rotating the Y axis by an angle φ. The angle φ is assumed to be zero in the positive direction of the Y axis and increase in the counterclockwise direction. Therefore, the relationship with the angle θ shown in FIG. 9 is φ = θ−90 °.

  Here, the rotation process of the initial arrow image 601 is performed using a generally used two-dimensional affine transformation. The center of rotation is the position of the arrow tip in the cursor. In the affine transformation in the present embodiment, x ′ = x · cos φ + y, where (x, y) is a coordinate before transformation, (x ′, y ′) is a coordinate after transformation, and φ is a counterclockwise rotation angle. Sinφ, y ′ = − x · sinφ + y · cosφ. That is, if the arrow tip position in the initial arrow image 601 is (LPX, LPY), the Y axis of the coordinate system is reversed, so the final pixel coordinates (x ″, y ″) are x ″ = LPX + x ′, y ″. = LPY−y ′.

[Processing in Needle Tip Position Estimation Device 1]
Next, the processing performed by the needle tip position estimation apparatus 1 will be described below using the flowchart shown in FIG.

In step S1 (first step), the visible light LED 6 irradiates the needle 12a with visible light. In addition, visible light is irradiated until the image acquisition part 5 finishes acquiring a visible light image.
In step S2 (second step), the image acquisition unit 5 acquires a visible light image including the needle 12a, the needle shadow 21, and the skin 11a.

In step S3, the needle information extraction unit 102 extracts needle information from the image acquired in step S2.
In step S4 (fifth step), the needle shadow information extraction unit 103 extracts the needle shadow information from the image acquired in step S2.

In step S5, the needle tip position 13 is displayed on the liquid crystal display 9.
In step S6 (sixth step), the coincidence detection unit 104 determines whether or not the needle tip position 13 and the needle shadow tip position 22 match. If it is determined that they do not match, the process returns to step S1. That is, until the tip of the needle 12a touches the skin 11a, the processes from step S1 to step S6 are repeated, and a visible light image is always displayed on the liquid crystal display 9 during that time. On the other hand, if it is determined that they match, the process proceeds to step S7.

In step S7, the coincidence detection unit 104 determines that the needle tip position 13 and the needle shadow tip position 22 coincide with each other, and transmits a coincidence signal.
In step S8 (seventh step), the transmitted coincidence signal is received, and the flag storage unit 106 updates the flag from the OFF state to the ON state.

In step S9 (eighth step), the needle image length storage unit 105 stores the needle image length when the flag is updated from the OFF state to the ON state in step S8.
In step S10 (9th step), the predicted needle tip position calculation unit 107 uses the needle image length Z stored in step S9 and the needle information (needle rear end position 14 and needle direction) extracted in step S3. Thus, the predicted needle tip position 15 is calculated.

In step S <b> 11, the image forming unit 108 forms a predicted needle tip position presentation image 704 in which an arrow is presented at the predicted needle tip position 15.
That is, in the processing from step S4 to step S10, the needle tip position estimation device 1 displays the image acquired in step S2 on the above image when the needle tip position 13 and the needle shadow tip position 22 in this image match. Based on the needle information including the needle image length Z indicating the length of the needle, the predicted needle tip position 15 when the needle 12a is punctured into the skin 11a is calculated (third step).

In step S12 (fourth step), the blood vessel position detection unit 7 synthesizes and displays the blood vessel emphasized image 703 acquired at this time and the needle tip predicted position presentation image 704.
In step S <b> 17, the coincidence detection unit 104 determines whether or not the needle tip position 13 and the needle shadow tip position 22 match. Here, if it is determined that they match, the process returns to step S10. That is, in the state where the needle 12a is punctured into the skin 11a, the processing from step S10 to step S17 is repeated, and during that time, the predicted needle tip position 15 is displayed on the liquid crystal display 9. On the other hand, if it is determined that they do not match, the process proceeds to step S18.

In step S18, the coincidence detection unit 104 stops sending the coincidence signal. That is, in this state, the needle 12a is in a state of being separated (removed) from the skin 11a.
In step S19, the flag storage unit 106 updates the flag from the ON state to the OFF state.
Note that steps S1 to S19 in the needle tip position estimation apparatus 1 of the present embodiment are repeated until the power is turned off, and can be repeated approximately 20 times per second.

[Features of Needle Tip Position Estimation Device 1]
(1)
As shown in FIGS. 1 and 4, the needle tip position estimation device 1 according to the present embodiment includes a visible light LED 6, an image acquisition unit 5, and a needle tip position estimation unit 101. The visible light LED 6 irradiates light to the needle 12a. The image acquisition unit 5 includes a needle 12a, a skin 11a (an object to be punctured) of an arm 11 into which the needle 12a is punctured, a needle shadow 21 projected onto the skin 11a of the needle 12a illuminated by the visible light LED 6, Take and acquire images containing The needle tip position estimation unit 101 calculates the predicted needle tip position 15 when the arm 11 is punctured based on the image acquired by the image acquisition unit 5 and the needle information. The needle information includes a needle image length Z indicating the length of the needle 12a on the image when the needle tip position 13 and the needle shadow tip position 22 in the image acquired by the image acquisition unit 5 coincide.

  As a result, when the distal end portion of the needle 12a and the distal end portion of the needle shadow 21 coincide with each other, immediately before the needle 12a is punctured, the spatial angle of the needle 12a changes even if the needle 12a is punctured into the arm 11 thereafter. Therefore, the length of the needle 12a on the image (the sum of the length X of the puncture portion and the length Y of the non-puncture portion) does not change. For this reason, unlike the prior art, it is not necessary to use the non-puncture portion in the puncture device (instrument) as an image reference for estimating the length and tip position of the needle 12a.

  Therefore, the puncture device itself does not need to be provided with another reference material (for example, a scale marked on the surface of the syringe) necessary for determining the predicted needle tip position 15, and what is provided with the needle 12 a Even when a device (instrument) is used, the predicted needle tip position 15 can be estimated.

  As a result, for example, when the needle 12a is inserted into the human body such as the syringe 12, the user of the syringe 12 can always grasp the predicted needle tip position 15. Therefore, the accuracy and time required for puncturing the needle 12a can be improved, and the burden on the puncture target (puncture target) can be reduced.

(2)
As shown in FIG. 1, the needle tip position estimation device 1 in the present embodiment further includes a liquid crystal display 9 for presenting a predicted needle tip position 15 to the user.

  Thus, the user can puncture the needle 12a while constantly checking where the tip of the needle 12a is located while puncturing the needle 12a in the skin 11a. As a result, it is possible to easily and stably perform puncturing work regardless of the skill level and technical skill of the user.

(3)
As shown in FIG. 4, in the needle tip position estimation apparatus 1 according to the present embodiment, the needle tip predicted position calculation unit 107 has a needle tip prediction based on the needle information extracted from the visible light image 702a by the needle information extraction unit 102. The position 15 is calculated.

Thus, before the needle 12a is punctured, the image acquisition unit 5 acquires an image from a position where at least the entire needle 12a can be acquired, so that even if the position of the needle 12a changes, the needle tip always automatically. The position 13 and the predicted needle tip position 15 can be estimated.
As a result, since the accuracy when puncturing the needle 12a is further improved, it becomes possible to puncture even a fine puncture object easily and safely.

(4)
As shown in FIGS. 4 and 11, the needle tip position estimation apparatus 1 in the present embodiment extracts a needle shadow tip position 22, and a needle image when the needle shadow tip position 22 and the needle tip position 13 coincide with each other. Based on the length Z, when the flag is ON, that is, when the needle 12a is present in the skin 11a of the arm 11, the predicted needle tip position 15 is calculated.

Thereby, the needle image length Z of the image acquired immediately before puncturing the arm 12 with the needle 12a is stored. Here, the three-dimensional angle of the needle 12a hardly changes between immediately before puncturing and after puncturing. However, while the needle 12a is moved to the point where the tip of the needle 12a is punctured, the angle of the needle 12a changes variously. Since the image acquired by the image acquisition unit 5 is a two-dimensional image, the length of the needle 12a (needle image length) in the image may change depending on the three-dimensional angle of the needle 12a. For this reason, it is difficult to accurately grasp the length of the needle 12a. From these facts, it becomes possible to accurately determine the predicted needle tip position 15 after puncturing by storing and using the needle image length when the tip of the needle 12a and the tip of the needle shadow 21 coincide. .
Therefore, even if the needle 12a is punctured into the arm 11 and the tip of the needle 12a cannot be seen, the predicted needle tip position 15 can be determined.

(5)
As shown in FIGS. 4 and 10, the needle tip position estimation device 1 according to the present embodiment displays the tip of the arrow mark in order to present the predicted needle tip position 15 determined by the needle tip position estimation unit 101 on the image. An image forming unit 108 is provided that forms a needle tip predicted position presentation image 704 arranged at the needle tip position 13 and in which the direction indicated by the arrow mark coincides with the needle direction.

This makes it possible to clearly present the predicted needle tip position 15.
As a result, the user can easily and accurately confirm the location of the tip of the needle 12a and the direction of the needle 12a.

(6)
As shown in FIG. 9, the needle tip position estimation apparatus 1 according to the present embodiment has a position where the needle tip predicted position calculation unit 107 separates the needle image length Z from the needle rear end position 14 toward the needle direction. The predicted position 15 is calculated.

This makes it possible to determine the predicted needle tip position 15 using the image length (needle image length Z) and needle direction of the needle 12a acquired immediately before puncturing the needle 12a.
Therefore, the accurate predicted needle tip position 15 is automatically calculated.

(7)
In the needle tip position estimation apparatus 1 according to the present embodiment, as illustrated in FIG. 2, the image acquisition unit 5 outputs the acquired image as an 8-bit monochrome luminance image.

As a result, the pixel value of each pixel included in the image acquired by the image acquisition unit 5 is stepped in 256 levels from 0 to 255. For this reason, the threshold value for performing image processing, the coefficient of the filter, and the like are simplified, and the predicted needle tip position 15 can be determined by simple processing and / or configuration.
As a result, the processing speed of the needle tip position estimation apparatus 1 is improved, and the cost can be reduced.

(8)
As shown in FIGS. 2, 4, 7, and 9, the needle tip position estimation apparatus 1 according to the present embodiment scans and binarizes the needle extraction filter 32 on the image acquired by the image acquisition unit 5. The needle image length, the needle direction, the needle tip position 13 and the needle rear end position 14 are acquired from the line segment formed by performing the least square method.

Here, the needle extraction filter 32 has a coefficient of “−2, −1, 0, 1, 2”. The first threshold value is a luminance value of 150 for pixels showing luminance from 0 to 255.
Thereby, it becomes possible to extract needle information automatically and uniquely from the acquired image. Therefore, it is possible to save the labor of performing calibration every time of use or rewriting each parameter by the user.

(9)
As shown in FIG. 6, the needle tip position estimation apparatus 1 according to the present embodiment includes a second binarization and boundary pixel in the first region 31 determined by the needle information extraction unit 102. The needle shadow tip position 22 is determined from a line segment formed by sequentially performing identification, third binarization, and least square method.

  Thereby, it becomes possible to extract needle shadow information automatically and uniquely from the acquired image. Therefore, it is possible to save the labor of performing calibration every time of use or rewriting each parameter by the user.

(10)
In the needle tip position estimation device 1 according to the present embodiment, as shown in FIG. 8, when the coincidence detection unit 104 determines the second region 33 and the needle shadow tip position 22 exists in the second region 33, It is determined that the needle tip position 13 and the needle shadow tip position 22 match. The second area 33 is a 5 × 5 pixel area.

  Accordingly, even if it is difficult to recognize the shadow of the puncture portion by an image because the puncture portion that is the tip portion of the needle 12a is extremely thin, the coincidence detection unit is inserted before the needle 12a is punctured. 104, it can be determined that the needle tip position 13 and the needle shadow tip position 22 coincide.

  Therefore, it is possible to prevent the timing of the coincidence signal output from the coincidence detection unit 104 from being delayed. That is, it is possible to prevent the needle image length Z stored in the needle image length storage unit 105 from being stored as a needle image length shorter than the actual needle image length. This is because, when the needle 12a is punctured, if the predicted needle tip position 15 is presented on the needle rear end side from the actual position, the needle is erroneously punctured to a region (depth) that should not be punctured. There is a fear. However, according to the configuration of the needle tip position estimating apparatus 1, since there is no such fear, it is possible to prevent erroneous puncture of the needle 12a.

(11)
As shown in FIG. 4, the needle tip position estimation device 1 according to the present embodiment has a needle information extraction unit 102 from a visible light image 702 a acquired by the image acquisition unit 5 in a state before the needle 12 a is punctured into the arm 11. Based on the extracted needle tip position 13, the tip position of the needle 12a is determined.

  As a result, the tip position of the needle 12a can be determined before the needle 12a is punctured and presented to the user. Therefore, it becomes easy for the user to confirm the tip position of the needle 12a before puncturing. For example, it becomes difficult to confirm the tip of the needle 12a depending on the angle at which the light of the visible light LED 6 or the illumination light of the room hits. Such a problem can be prevented.

[Other Embodiments]
As mentioned above, although one Embodiment of this invention was described, this invention is not limited to the said embodiment, A various change is possible in the range which does not deviate from the main point of invention.

(A)
In the said embodiment, as shown in FIG. 1, the example which used single visible light LED6 as an illumination part was given and demonstrated. However, the present invention is not limited to this.

For example, as shown in FIG. 12, the structure provided with two visible light LED6a, 6b may be sufficient.
FIG. 12 shows a second structure example of the needle tip position estimating apparatus 1a. This is an embodiment in which the arrangement and number of visible light sources are different from those of the first structural example shown in FIG. 1, and in particular, a case is assumed in which the needle 12a and the needle shadow 21 cannot be correctly extracted due to the influence of external light.

  In FIG. 12, the visible light source 6 is a visible light LED 6a and a visible light LED 6b are arranged so as to irradiate light on each side surface or each upper surface of the needle 12a, and the visible light LED 6a and the visible light LED 6b are near red. A visible light image in which the needle 12a and the needle shadow 21b are emitted by the light emission of the visible light LED 6b and a visible light image in which the needle 12a and the needle shadow 21a are emitted by the light emission of the visible light LED 6a are alternately emitted except during the period in which the external light is emitted. An optical image is generated by the image acquisition unit 5 and input to the needle tip position estimation unit 101.

  Unlike the above-described embodiment (see FIG. 1), the first region 31 indicating the periphery of the needle 12a generated by the needle information extraction unit 102 of the needle tip position estimation unit 101 is a needle in the visible light image when the visible light LED 6b emits light. It is generated on the right side of 12a, and is generated on the left side of the needle 12a in the visible light image when the visible light LED 6a emits light.

In this case, the two types of visible light image sequences generated by the visible light LEDs 1 and 2 are processed for each sequence by the needle tip position estimation unit 101, and either one of the visible light image sequences is caused by external light. Even if noise is mixed into the image of the needle 12a or the needle shadow due to the influence and the coincidence detection unit 104 cannot detect coincidence, there is a possibility that the coincidence detection unit 104 can detect coincidence in the other visible light image series. It is. In general, external light is often irradiated from a biased direction with respect to the arm 11 on which the apparatus is mounted. Therefore, the two types of visible light image sequences can be generated as described above with respect to the needle 12a. By arranging two light sources for irradiating light, the possibility of avoiding the influence of external light increases.
Here, the light sources for irradiating the needle 12a with two light sources are arranged, but a configuration in which the influence of external light can be avoided by arranging three or more light sources may be adopted.

(B)
In the said embodiment, the example which used the liquid crystal display 9 as the presentation part was given and demonstrated. However, the present invention is not limited to this.

  For example, a printing apparatus that prints data output from the needle tip position estimation unit 101 may be used. Moreover, the structure which is not provided with these presentation parts may be sufficient. That is, the configuration may be such that data including the predicted needle tip position 15 estimated by the needle tip position estimating device is notified to the user by outputting the data using another output device.

(C)
In the above-described embodiment, the needle extraction filter 32 has been described using an example in which the coefficient is “−1, −2, 0, 2, 1”. However, the present invention is not limited to this.

For example, any filter that can distinguish the image of the needle 12a from the image of the needle shadow 21 and the syringe 12 may be used.
(D)
In the above embodiment, the generation arrow image 602 has been described as an example when the flag is in the OFF state, that is, before the needle 12a is stuck in the arm. However, the present invention is not limited to this.

For example, when the needle 12a is not punctured into the puncture target, only a visible light image may be displayed.
(E)
In the said embodiment, the example using visible light LED6 was given and demonstrated as an illumination part. However, the present invention is not limited to this.

For example, a configuration using a light source with high brightness and long life such as an electrodeless discharge lamp may be used.
As a result, the needle shadow 21 can be reliably and clearly projected onto the puncture target.

(F)
In the above embodiment, the coincidence detection unit 104 and the flag storage unit 106 automatically determine whether or not the needle 12a has been punctured, and automatically perform processing from before the needle 12a is punctured to during the puncture of the needle 12a. The configuration described in (1) has been described as an example. However, the present invention is not limited to this.

For example, the needle image length storage unit 105 acquires the needle image length when the user performs a predetermined operation immediately before the needle 12a is punctured into the puncture object, that is, when the needle tip position 13 is in contact with the puncture object. To do. The needle tip predicted position 15 during puncture may be estimated using the needle image length acquired here.
This makes it possible to prevent misrecognition of information in image processing, and therefore this configuration is suitable from the viewpoint of certainty.

(G)
In the said embodiment, the example using visible light LED6 was given and demonstrated as an illumination part. However, the present invention is not limited to this.

  For example, the light source may be illumination of a room using the needle tip position estimation device, or may be an illumination unit having a daylighting window provided so as to take in the light limitedly.

  The needle tip position estimation device and the needle tip position estimation method according to the present invention are used to determine where in the puncture object the needle tip portion that cannot be seen by piercing the puncture object is located in the puncture object. There is an effect that can be known. For this reason, it can be widely applied as a device and method for supporting a medical practice, particularly for a medical practice that has so far required a high level of skill when using a puncture device.

The perspective view which shows the structure of the needle tip position estimation apparatus which concerns on one Embodiment of this invention. The block diagram which shows the functional structure of the needle | hook tip position estimation apparatus shown in FIG. The figure which shows the outline | summary of the process which the needle | hook tip position estimation apparatus shown in FIG. 1 performs. The block diagram which shows the functional structure in the needle tip position estimation part of the needle tip position estimation apparatus shown in FIG. (A) is a top view transmission diagram of the syringe, skin, and vein immediately before needle puncture. (B) is a top view transmission diagram of the syringe, skin, and vein after needle puncture. (C) is a transparent side view of the syringe and the skin immediately before needle puncture. (D) is a side view transmission diagram of the syringe and the skin after needle puncture. (A) is an image figure explaining the process which a needle information extraction part and a needle shadow information extraction part perform. (B) is a figure which shows the luminance value on the straight line containing the line segment E1-E2 shown to Fig.6 (a). (C) is an image figure explaining the process which a needle information extraction part and a needle shadow information extraction part perform. (D) is a figure which shows the luminance value on the straight line containing line segment E3-E1 shown in FIG.6 (c). (A) is a schematic diagram which shows the pixel which comprises the needle | hook on an image. (B) is a figure which shows the coefficient of the filter which filters with respect to the pixel shown to Fig.7 (a). The image figure which shows the positional relationship of a 2nd area | region, a needle tip position, and a needle shadow tip position in the process which the coincidence detection part in the needle tip position estimation part shown in FIG. 4 performs. The image figure explaining the process which the needle tip estimated position calculation part in the needle tip position estimation part shown in FIG. 4 performs. The image figure explaining the process which an image formation part performs. The flowchart figure of the process which the needle | hook tip position estimation apparatus shown in FIG. 1 performs. The perspective view which shows the structure of the needle tip position estimation apparatus which concerns on other embodiment of this invention. The block diagram which shows the functional structure of the conventional needle tip position estimation apparatus. The block diagram which shows the functional structure of the needle tip position estimation means in the conventional needle tip position presentation apparatus. The image figure for demonstrating the process which the conventional needle | hook tip position estimation means performs.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Needle tip position estimation apparatus 2 Apparatus fixing | fixed part 3 Arm part 3a Side surface 4 Head part 5 Image acquisition part 6 Visible light LED (illumination part)
6a Visible LED (lighting unit)
6b Visible LED (lighting unit)
7 Blood vessel position detection unit 9 Liquid crystal display (display unit, presentation unit)
11 Arm (puncture object)
11a skin 11b vein 12 syringe (puncture device (instrument))
12a Needle 13 Needle tip position 14 Needle rear end position 15 Needle tip predicted position 21 Needle shadow 22 Needle shadow tip position 31 First region 32 Needle extraction filter 33 Second region 101 Needle tip position estimation unit 102 Needle information extraction unit 103 Needle shadow Information extraction unit 104 Match detection unit 105 Needle image length storage unit 106 Flag storage unit 107 Needle tip predicted position calculation unit 108 Image formation unit 601 Initial arrow image 602 Generation arrow image 701 Near-infrared image 702 Visible light image 703 Blood vessel enhancement image 704 Needle tip predicted position presentation image 705 Composite image 1000 Conventional needle tip position presentation device 1101 Illumination means 1102 Image acquisition means 1103 Needle tip position estimation means 1104 Blood vessel position detection means 1105 Display means 1201 Syringe image extraction means 1202 Injection needle tip position detection means 1203 Injection needle tip position calculation means 1204 Injection needle tip position Image generation means

Claims (15)

An illumination unit that emits light to the needle;
An image acquisition unit for acquiring an image including the needle, a puncture object into which the needle is punctured, and a needle shadow projected onto the puncture object of the needle illuminated by the illumination unit;
Based on the image acquired by the image acquisition unit, and needle information including a needle image length indicating the length of the needle on the image when the needle tip position and the needle shadow tip position in the image coincide with each other. A needle tip position estimation unit for determining a needle tip predicted position when the needle is punctured into the puncture object;
A needle tip position estimation device comprising: A presentation unit for presenting the image acquired by the image acquisition unit and the predicted needle tip position determined by the needle tip position estimation unit;
The needle tip position estimating apparatus according to claim 1. The needle tip position estimation unit is
A needle information extraction unit for extracting the needle information;
The needle information further includes a needle direction, the needle tip position, and a needle rear end position in the image acquired by the image acquisition unit,
Based on the needle information extracted by the needle information extraction unit, a predicted needle tip position calculating unit that calculates the predicted needle tip position;
Having
The needle tip position estimation device according to claim 1 or 2. The needle tip position estimation unit is
A needle shadow information extraction unit for extracting needle shadow information including the needle shadow tip position from the image acquired by the image acquisition unit;
A coincidence detection unit that outputs a coincidence signal when the needle tip position is coincident with the needle shadow tip position;
A flag storage unit that stores a flag that is ON when the coincidence signal is output and that is OFF when the coincidence signal is not output;
A needle image length storage unit for storing the needle image length when the flag is switched from the OFF state to the ON state;
Have
The predicted needle tip position calculation unit calculates the predicted needle tip position when the flag is in an ON state based on the needle image length stored in the needle image length storage unit.
The needle tip position estimation device according to claim 3. The needle tip position estimation unit further includes an image forming unit that forms a needle tip predicted position presentation image.
The needle tip position estimation device according to any one of claims 1 to 4. The predicted needle tip position calculating unit calculates a position separated by the length of the needle image length stored in the needle image length storage unit from the needle rear end position toward the needle direction as the needle tip predicted position. decide,
The needle tip position estimation device according to any one of claims 3 to 5. The image acquisition unit outputs the acquired image as a monochrome luminance image.
The needle tip position estimation device according to any one of claims 1 to 6. The needle information extraction unit
Scanning the first filter for the image acquired by the image acquisition unit to form a needle extraction image;
Generating a first binarized image obtained by binarizing the needle extraction image with a first threshold;
The pixel indicating the needle in the first binarized image is subjected to line differentiation using a least square method,
From this line segment, the needle image length, the needle direction, the needle tip position and the needle rear end position are obtained.
The needle tip position estimation device according to any one of claims 3 to 7. The needle information extraction unit determines a first region indicating the periphery of the needle not including the needle based on the line segment in the image,
The needle shadow information extraction unit generates a second binarized image by scanning a second filter having a predetermined length and having a second threshold as a threshold in the first region, and generating the second second image. In the pixel matrix of the binarized image, scanning is performed from both ends of the image along each row direction or each column direction, and the first pixel whose pixel value has changed is set as a boundary pixel, and in each row or each column Forming a third binarized image obtained by extracting the center position of each of the boundary pixels, and performing line segmentation using a least-squares method on pixels corresponding to the center position in the third binarized image. And determine the needle shadow tip position from this line segment,
The needle tip position estimation device according to claim 8. The coincidence detection unit
Determining a second region centered on the needle tip position;
When the needle shadow tip position is present in the second region, it is determined that the needle tip position matches the needle shadow tip position;
The needle tip position estimation device according to any one of claims 4 to 9. When the flag is in an OFF state, the tip position of the needle is determined based on the needle tip position.
The needle tip position estimation device according to any one of claims 4 to 10. A first step of irradiating the needle with light;
A second step of obtaining an image including the needle, a puncture object to which the needle is punctured, and a needle shadow projected onto the puncture object of the needle irradiated with light in the first step;
Based on the image acquired in the second step, and needle information including a needle image length indicating a length of the needle on the image when a needle tip position and a needle shadow tip position in the image coincide with each other. A third step of determining a predicted needle tip position when the needle is punctured into the puncture object;
A needle tip position estimation method comprising: The third step includes
A fifth step of extracting needle shadow information including the needle shadow tip position from the image acquired in the second step;
A sixth step of determining whether or not the needle tip position matches the needle shadow tip position;
A seventh step of updating the flag stored in the OFF state in advance to the ON state when it is determined in the sixth step that the needle tip position is coincident with the needle shadow tip position;
An eighth step of storing the needle image length when the flag is switched from the OFF state to the ON state;
A ninth step of calculating the predicted needle tip position when the flag is in an ON state based on the needle image length stored in the eighth step;
Further having
The needle tip position estimation method according to claim 12. A fourth step of presenting the predicted needle tip position determined in the third step;
The needle tip position estimation method according to claim 12 or 13. In the fourth step, the predicted needle tip position determined in the third step is displayed by a display device.
The needle tip position estimation method according to claim 14.

Images