JP2021165639A - Posture estimation device - Google Patents

Abstract

To estimate the posture of an instrument without providing a marker.SOLUTION: A posture estimation device 50 estimates the posture of an instrument that performs a prescribed operation in a prescribed direction. The posture estimation device 50 includes: a memory 52 that stores a three-dimensional shape of a pre-registered instrument and shape information in an operation direction corresponding to the three-dimensional shape; an image acquisition unit 53 that acquires image information obtained by photographing an instrument and an object of the operation by the instrument; and a posture estimation unit 54 that estimates the operation direction of the instrument with respect to the object based on the shape information stored in the memory 52 and the image information acquired by the image acquisition unit 53.SELECTED DRAWING: Figure 3

Description

The present invention relates to a posture estimation device that estimates the posture of an instrument based on an image.

As a device of this type, a device that estimates the posture of the ultrasonic probe with respect to the measurement target is conventionally known (see, for example, Patent Document 1). The apparatus described in Patent Document 1 recognizes the top and bottom, left and right, and front and back of the ultrasonic probe by recognizing protrusions having a predetermined shape provided on a plurality of side surfaces of the ultrasonic probe along the central axis of the ultrasonic probe. , Estimate the posture of the ultrasonic probe.

Japanese Patent No. 6510570

However, in the device described in Patent Document 1, it is necessary to provide a marker such as a protrusion on the ultrasonic probe, which may hinder the measurement work of the user. In addition, accurate posture estimation may not be possible due to marker dropout, misalignment, shape change due to damage or deterioration, and the like.

One aspect of the present invention is a posture estimation device that estimates the posture of an instrument that performs a predetermined motion in a predetermined direction, and obtains a three-dimensional shape of a pre-registered instrument and shape information of an operation direction corresponding to the three-dimensional shape. The shape acquisition unit to be acquired, the image acquisition unit that acquires image information obtained by photographing the instrument and the object of operation by the instrument, and the shape information acquired by the shape acquisition unit and the image information acquired by the image acquisition unit. Based on this, it is provided with a posture estimation unit that estimates the movement direction of the device with respect to the target.

According to the present invention, the posture of the instrument can be estimated without providing a marker.

The figure which shows typically an example of the apparatus to which the posture estimation apparatus which concerns on embodiment of this invention is applied. FIG. 6 is a perspective view schematically showing a film used for dental radiography of FIG. 1 and a holder for holding the film. The block diagram which shows the main part structure of the posture estimation apparatus which concerns on embodiment of this invention. The figure which shows an example of the 3D shape data of the instrument of FIG. The figure which shows another example of the 3D shape data of the instrument of FIG. The figure which shows an example of the image data which was taken from the front of the patient and acquired by the image acquisition part of FIG. The figure which shows an example of the image data which was taken from the side of a patient and acquired by the image acquisition part of FIG. The figure which shows an example of the display screen of the input / output part of FIG. 3 about the patient photographed from the front. The figure which shows an example of the display screen of the input / output part of FIG. 3 about the patient photographed from the side. The flowchart which shows an example of the process executed by the posture estimation apparatus which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to FIGS. 1 to 7. The posture estimation device according to the embodiment of the present invention performs a predetermined operation in a predetermined direction, such as an irradiation device of an X-ray device that irradiates X-rays in a predetermined direction, a probe of an ultrasonic measuring device that transmits and receives ultrasonic waves in a predetermined direction, and the like. It is applied to various medical instruments and estimates the direction of movement by the device (the posture of the device) with respect to the object of movement such as measurement and treatment. In the following, an example of estimating the posture of the irradiation instrument with respect to the imaging target of the dental X-ray apparatus will be described.

FIG. 1 is a diagram schematically showing an example of an instrument 1 to which the posture estimation device according to the embodiment of the present invention is applied, and shows an X-ray irradiation instrument (hereinafter, instrument) 1 of the dental X-ray apparatus 10. Further, FIG. 2 is a perspective view schematically showing a film 3 used for dental X-ray photography and a holder 40 for holding the film 3.

As shown in FIGS. 1 and 2, in dental X-ray photography, the film 3 is fixed to the oral side of the tooth of the patient 2 (for example, the left molar), and X is taken in a predetermined irradiation direction A from the instrument 1 to the film 3. Irradiate the line.

The film 3 is held by a holder 40 as shown in FIG. As shown in FIG. 2, the holder 40 includes a clip portion 41 for holding the film 3, an occlusal portion 42 adjacent to the clip portion 41, a ring-shaped indicator portion 43, a clip portion 41, an occlusal portion 42, and an indicator portion. It has a stay portion 44 that connects to the 43.

When the patient 2 occludes the occlusal portion 42 of the holder 40 holding the film 3 with the teeth, the film 3 is fixed to the oral side of the teeth of the patient 2, the indicator portion 43 is exposed to the outside of the oral cavity, and the X-ray is performed by the instrument 1. The irradiation direction A to which the line should be irradiated is shown.

By the way, when a follow-up observation is performed on a target tissue such as a tooth of patient 2 or a lesion around it using an X-ray image or the like, the images of the target tissue taken from the same irradiation direction A are compared with each other on a regular basis. There is a need. However, even if the holder 40 of FIG. 2 is used, the position and angle at which the patient 2 occludes the holder 40 changes. Therefore, it is not possible to completely match the irradiation direction A for each imaging, that is, the posture of the instrument 1 with respect to the target tissue. difficult.

Therefore, in the present embodiment, the posture of the medical device with respect to the object of the movement such as measurement or treatment is estimated, and the appropriate posture of the medical device can be guided to the user who performs the movement such as measurement or treatment as follows. A posture estimation device is configured in.

FIG. 3 is a block diagram showing a main configuration of a posture estimation device (hereinafter, device) 50 according to an embodiment of the present invention. The device 50 includes a computer having a CPU 51, a memory 52 such as a ROM and a RAM, and other peripheral circuits such as an I / O interface. The CPU 51 estimates the shape acquisition unit 53 that acquires the three-dimensional shape data of the instrument 1, the image acquisition unit 54 that acquires the image data of the instrument 1 and the patient 2, and the posture estimation that estimates the irradiation direction A of the instrument 1 with respect to the patient 2. It functions as a unit 55 and an information output unit 56 that outputs information on the irradiation direction A. The external memory 57, the input / output unit 58, and the camera 59 can be connected to the device 50 by wire or wirelessly.

4A and 4B are diagrams showing an example of three-dimensional shape data of the instrument 1. The memory 52 stores three-dimensional shape data of the dental X-ray apparatus 10 including the instrument 1. As shown in FIG. 4A, the three-dimensional shape data includes information on the outer shape 1M of the instrument 1, information on the X-ray irradiation direction A (irradiation axis) by the instrument 1, and the intersection P1 of the outer shape of the instrument 1 and the irradiation axis. Information on P2 is included.

Although not shown, the three-dimensional shape data of the dental roentgen device 10 includes three-dimensional data 10M showing the outer shape of the dental roentgen device 10 (whole or main body 10a), when it is fixedly installed on the floor or the like. The three-dimensional shape data CM indicating the installation axis C and the three-dimensional shape data OM indicating the reference point O of the headrest, the switch, etc. fixed to the main body 10a are also included.

Information on the outer shape 1M of the instrument 1 includes, for example, three-dimensional CAD (Computer-Aided Design) data provided by the manufacturer of the dental roentgen device 10 or an actual dental roentgen device 10 (instrument 1) using a 3D scanner or the like. 3D shape data or the like obtained by measuring the above can be used.

The three-dimensional shape data includes data of a plurality of two-dimensional images (first image data) when the instrument 1 is photographed from a plurality of different directions instead of the information of the outer shape 1M of the instrument 1, and features in each image. It is also possible to use the second image data in which the virtual marker Pn is set at the point. FIG. 4B is an example of the second image data. By setting virtual markers Pn (13 virtual markers P1 to P13 in FIG. 4B) at feature points (13 feature points in FIG. 4B) in a plurality of images taken from a plurality of different directions, the entire instrument 1 can be set. It is possible to obtain three-dimensional shape data covering the feature points.

For example, the actual measurement (pre-measurement) is performed with a marker attached to the characteristic points (centers of surfaces and edges, equal division points, corners, etc.) of the outer shape of the actual instrument 1 with a pen or a sticker, and the measurement is in progress. The second image data (FIG. 4B) can be obtained by taking a moving image of the instrument 1 and dividing it into a plurality of still images. Further, by deleting the marker Pn from the second image data by image processing, the first image data (FIG. 4A) is obtained, whereby a data set of the first image data and the second image data is obtained. By performing machine learning using such a data set, it becomes possible to estimate the posture of the instrument 1 based on the learning result thereafter.

As a preliminary measurement for such machine learning, by performing a measurement assuming that the instrument 1 is actually used, the shooting angle of the instrument 1 by the camera 59 (FIG. 3) is reproduced, which is practical. Machine learning based on datasets can be performed. As a result, the accuracy of posture estimation of the instrument 1 based on the learning result can be improved. Further, a data set may be prepared for each measurement item assumed for the instrument 1 (measurement contents such as a measurement target location and a probe scanning direction). In this case, the accuracy of posture estimation of the instrument 1 can be further improved by performing machine learning for each data set annotated for each measurement item and switching the learning result for each measurement item.

The memory 52 stores, for example, three-dimensional shape data of a plurality of dental X-ray devices 10 (instruments 1) having different outer shapes and feature points for each manufacturer and model.

The external memory 57 in FIG. 3 is, for example, a memory of an external computer that can be connected to the device 50 via a network, or a flash memory that can be connected to the device 50 via an input / output port. The input / output unit 58 includes a keyboard, a mouse, a display, a speaker, a touch panel integrated with the display, and the like. The user operates the input / output unit 58 to input the three-dimensional shape data stored in the external memory 57 into the device 50, thereby additionally registering the three-dimensional shape data of the new dental roentgen device 10 (instrument 1). can do.

The shape acquisition unit 53 of the device 50 acquires the three-dimensional shape data input via the input / output unit 58. The three-dimensional shape data acquired by the shape acquisition unit 53 is stored in the memory 52 of the device 50.

The camera 59 is composed of two cameras capable of photographing a moving image, and is mounted on a wall surface facing, for example, a dental X-ray apparatus 10 so that the faces of the instrument 1 and the patient 2 are included in the imaging range. By configuring the camera 59 with two cameras, that is, as a stereo camera, it is possible to secure a sufficient imaging range even when the instrument 1 and the face of the patient 2 overlap each other. The two cameras 59 are arranged so that the shooting directions in the horizontal direction differ from each other by about 90 degrees. The image data captured by the camera 59 is transmitted to the device 50 and the display (input / output unit) 58.

The image acquisition unit 54 of the device 50 acquires the image data of the instrument 1 and the patient 2 transmitted from the camera 59. The image acquisition unit 54 can also acquire the image data of the instrument 1 and the patient 2 input from the external memory 57 via the input / output unit 58. For example, the user can operate the input / output unit 58 to input the image data of the same patient 2 stored in the external memory 57 at the time of the previous photographing to the device 50.

The posture estimation unit 55 is based on the three-dimensional shape data of the dental X-ray apparatus 10 (instrument 1) acquired by the shape acquisition unit 53 and stored in the memory 52 and the image data acquired by the image acquisition unit 54. , The irradiation direction A of the device 1 with respect to the patient 2 is estimated. Specifically, first, the installation axis C and the reference point O of the main body 10a of the dental roentgen device 10 on the image are recognized, the gravity direction parallel to the recognized installation axis C is the Z axis, and the reference point O is the origin. Set the three-dimensional coordinate space to be.

5A and 5B are diagrams showing an example of image data acquired by the image acquisition unit 54, FIG. 5A shows image data taken from the front of the patient 2, and FIG. 5B is the side of the patient 2. Shows the image data taken from. As shown in FIGS. 5A and 5B, the posture estimation unit 55 detects feature points including facial feature points such as body joints, eyes, ears, and nose of patient 2 on each image, and corresponds to each feature point. Calculate the three-dimensional coordinates of the nodes N1 to N18. Further, the outer shape and the feature point of the instrument 1 are detected, and the three-dimensional coordinates of the intersection (feature point) P1 and P2 between the outer shape of the instrument 1 and the irradiation axis are calculated.

For the detection of feature points and the calculation of two-dimensional coordinates in each image, a feature point detection algorithm by deep learning such as the OpenPose library (International Society for Computer Vision CVPR2017, Carnegie Mellon University, Zhe Cao et al.) Stored in the memory 52 in advance is used. It is done using. In the OpenPose library, each feature point is detected based on the relationship of the feature points such as the distance between the feature points and the relative movement.

The detection of the outer shape and feature points of the instrument 1 in each image and the calculation of the two-dimensional coordinates of the intersections (feature points) P1 and P2 between the outer shape of the instrument 1 and the irradiation axis are stored in the three-dimensional shape data stored in the memory 52 in advance. It is done based on. When a plurality of 3D shape data are registered, the manufacturer or the virtual marker Pn can be compared by comparing the outer shape or feature point of the instrument 1 on the image with the outer shape 1M or the virtual marker Pn of the instrument 1 included in the 3D shape data. The model is specified, and the two-dimensional coordinates of the intersections P1 and P2 are calculated based on the corresponding three-dimensional shape data. The user may specify the manufacturer and model used for shooting.

When the two-dimensional coordinates of the nodes N1 to N18 and the intersections P1 and P2 in the pair of images (for example, FIGS. 5A and 5B) taken by the pair of left and right cameras 59 constituting the stereo camera are calculated, the principle of triangulation. The corresponding three-dimensional coordinates are calculated by. These three-dimensional coordinates may be detected by an infrared depth camera such as Kinect (registered trademark).

The calculated three-dimensional coordinates of the nodes N1 to N18 correspond to the posture of the patient 2 in the direction of gravity or the posture of the patient 2 with respect to the main body 10a of the dental roentgen device 10. Further, the calculated three-dimensional coordinates of the intersections P1 and P2 correspond to the posture of the instrument 1 in the direction of gravity or the attitude of the instrument 1 with respect to the main body 10a of the dental roentgen device 10. Further, for example, the relative coordinates of the intersections P1 and P2 of the instrument 1 with respect to the nodes N1 to N3 of the facial feature points of the patient 2 correspond to the posture (irradiation direction A) of the instrument 1 with respect to the patient 2.

Information on the posture of the patient 2 and the posture of the instrument 1 (irradiation direction A) is output to the display (input / output unit) 58 by the information output unit 56. Further, the nodes N1 to N18 and the intersection P1 corresponding to the reference posture of the patient 2 and the reference posture (reference direction Az) of the patient 2 and the reference posture (reference direction Az) of the patient 2 in the image at the time of radiography obtained by irradiating X-rays with the dental X-ray apparatus 10 , P2 three-dimensional coordinates are stored in the memory 52. The information of the reference posture of the patient 2 and the reference posture (reference direction Az) of the instrument 1 stored in the memory 52 is read out at the time of the next shooting and output to the display 58 by the information output unit 56.

6A and 6B are views showing an example of a display screen of the display (input / output unit) 58, FIG. 6A shows a display screen of an image taken from the front of the patient 2, and FIG. 6B is a view of the patient 2. The display screen of the image taken from the side is shown. As shown in FIGS. 6A and 6B, the display 58 superimposes information on the current posture and the previous posture of the patient 2 and the instrument 1 on the image from the camera 59.

Through the display screen of the display 58, the user can see the patient 2's current posture (N1, N2, N3) and the reference posture (N1z, N2z) at the time of the previous radiography in the direction of gravity or with respect to the main body 10a of the dental X-ray apparatus 10. , N3z) can be confirmed at the same time. Thereby, the posture of the patient 2 can be adjusted so as to match the reference posture. Further, since the current irradiation direction A of the instrument 1 and the reference direction A0 at the time of the previous photographing can be confirmed at the same time through the display screen of the display 58, the posture of the instrument 1 with respect to the patient 2 is adjusted so as to match the reference direction A0. can do.

FIG. 7 is a flowchart showing an example of processing executed by the device 50 according to a program stored in advance. The process shown in this flowchart is started when, for example, image data is acquired, and is repeated at a predetermined cycle. First, the image data is read in step S1, then the instrument 1 currently in use is specified in step S2, and then the postures of the patient 2 and the instrument 1 are estimated in step S3. Next, in step S4, it is determined whether or not the information of the reference posture at the time of the previous shooting is stored in the memory. If affirmed in step S4, the process proceeds to step S5, and the current posture information and the reference posture information at the time of the previous shooting are output to the display 58. If denied in step S4, the process proceeds to step S6, and only the current posture information is output to the display 58. Next, in step S7, it is determined whether or not shooting has been performed, and if affirmed, the process proceeds to step S8, and if denied, the process ends. In step S8, the current (that is, shooting time) posture information is stored in the memory.

According to the embodiment of the present invention, the following effects can be obtained.
(1) The device 50 estimates the posture of the instrument 1 that irradiates X-rays in a predetermined irradiation direction A. The device 50 is a shape acquisition unit 53 that acquires the three-dimensional shape of the device 1 registered in advance and the three-dimensional shape data of the irradiation direction A corresponding to the three-dimensional shape, and the device 1 and the object to be photographed by the device 1. Instrument 1 for patient 2 based on the image acquisition unit 54 that acquires the image data of the patient 2 and the three-dimensional shape data acquired by the shape acquisition unit 53 and the image data acquired by the image acquisition unit 54. It is provided with an attitude estimation unit 55 that estimates the irradiation direction A of the above (FIG. 3). Thereby, the posture of the instrument 1 can be estimated based on the three-dimensional shape data of the instrument 1 registered in advance without providing a marker.

(2) The instrument 1 includes a first instrument and a second instrument having different three-dimensional shapes from each other. The posture estimation unit 55 identifies the instrument 1 based on the three-dimensional shape data acquired by the shape acquisition unit 53 and the image data acquired by the image acquisition unit 54, and irradiates the patient 2 with the specified instrument 1. Estimate direction A. As a result, a plurality of instruments 1 having different three-dimensional shapes registered in advance are automatically recognized and the posture is estimated, so that it is not necessary to change the settings or the like each time the instrument 1 to be used is changed.

(3) The device 50 further includes an information output unit 56 that outputs information on the irradiation direction A of the instrument 1 for the patient 2 estimated by the posture estimation unit 55 (FIG. 3). For example, by displaying the irradiation direction A superimposed on the images of the instrument 1 and the patient 2, it becomes easy for the user to grasp and adjust the posture of the instrument 1.

(4) The device 50 further includes a memory 52 that stores the irradiation direction A of the instrument 1 with respect to the patient 2 estimated by the posture estimation unit 55 when the image is taken as the reference direction Az (FIG. 3). The information output unit 56 outputs the reference direction Az stored in the memory 52 together with the information on the irradiation direction A of the instrument 1 for the patient 2 estimated by the posture estimation unit 55.

For example, by displaying the current irradiation direction A and the reference direction AZ, which is the irradiation direction at the time of the previous shooting, on the images of the device 1 and the patient 2, the user can change the posture of the device 1 to that at the time of the previous shooting. It will be easy to match. As a result, it becomes easy to photograph the patient 2 from the same irradiation direction A on a regular basis, and it is possible to preferably support the user who performs the follow-up observation of the specific patient 2.

(5) The image data includes information on the direction of gravity. The posture estimation unit 55 further estimates the posture of the patient 2 in the direction of gravity. For example, by displaying the current posture of the patient 2 and the posture of the patient 2 at the time of the previous radiography on an image including the main body 10a of the dental X-ray apparatus 10 fixed to the floor or the like, the user. However, it becomes easy to match the posture of the patient 2 in the direction of gravity with that at the time of the previous imaging. As a result, it becomes easy to photograph the patient 2 in the same posture on a regular basis, and it is possible to preferably support the user who observes the course of the target tissue such as a lesion whose shape changes according to the direction of gravity.

The above embodiment can be transformed into various forms. Hereinafter, a modified example will be described. In the above embodiment, the information output unit 56 outputs the information of the current posture and the information of the reference posture at the time of the previous shooting to the display 58, but the information output that outputs the information of the operation direction and the reference direction of the instrument. The department is not limited to this. For example, information may be output to a speaker to guide the user by voice, or information may be output to a movable laser light source or the like to guide the user by laser light.

In the above embodiment, an example of applying the device 50 to the X-ray irradiation device 1 of the dental X-ray device 10 has been described, but the posture estimation device that estimates the posture of the device that performs a predetermined operation in a predetermined direction is such a device. Not limited to. For example, it may be applied to forceps for laparoscopic surgery that performs operations such as incision and suturing at a predetermined position in a predetermined direction.

In the above embodiment, an example is shown in which the three-dimensional coordinates of the nodes N1 to N3 corresponding to the facial feature points of the patient 2 calculated by the posture estimation unit 55 are the postures of the patient 2 (FIGS. 6A and 6B). The posture of the object in the direction of gravity is not limited to this. The posture of the target can be the three-dimensional coordinates of a plurality of nodes corresponding to feature points such as body joints according to the target of movement.

The above description is merely an example, and the present invention is not limited to the above-described embodiments and modifications as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or a plurality of the above-described embodiments and the modified examples, and it is also possible to combine the modified examples.

1 instrument (X-ray irradiation instrument), 2 patients, 50 devices (attitude estimation device), 51 CPU, 52 memory, 53 shape acquisition unit, 54 image acquisition unit, 55 attitude estimation unit, 56 information output unit, 57 external memory, 58 I / O, 59 Camera

Claims (5)

A posture estimation device that estimates the posture of an instrument that performs a predetermined movement in a predetermined direction.
A shape acquisition unit that acquires the three-dimensional shape of the instrument registered in advance and the shape information of the operation direction corresponding to the three-dimensional shape, and
An image acquisition unit that acquires image information obtained by photographing the device and an object of operation by the device, and
A posture characterized by including a posture estimation unit that estimates the movement direction of the device with respect to the target based on the shape information acquired by the shape acquisition unit and the image information acquired by the image acquisition unit. Estimator. The posture estimation device according to claim 1.
The instrument includes a first instrument and a second instrument having different three-dimensional shapes from each other.
The posture estimation unit identifies the device based on the shape information acquired by the shape acquisition unit and the image information acquired by the image acquisition unit, and estimates the operating direction of the specified device with respect to the target. A posture estimation device characterized by The posture estimation device according to claim 1 or 2.
A posture estimation device further comprising an information output unit that outputs information on the operating direction of the device with respect to the target estimated by the posture estimation unit. The posture estimation device according to claim 3.
Further provided is a storage unit that stores the operation direction of the device with respect to the target estimated by the posture estimation unit as a reference direction when the predetermined operation is performed.
The information output unit is a posture estimation device that outputs information on a reference direction stored in the storage unit together with information on the operation direction of the device with respect to the target estimated by the posture estimation unit. The posture estimation device according to any one of claims 1 to 4.
The image information includes information on the direction of gravity.
The posture estimation unit is a posture estimation device that further estimates the posture of the target in the direction of gravity.

Images