JP7219243B2 - Posture estimation device - Google Patents

Description

The present invention relates to a posture estimation device that estimates the posture of a tool based on images.

As a device of this type, there is conventionally known a device configured to estimate the posture of an ultrasonic probe with respect to a measurement object (see, for example, Japanese Unexamined Patent Application Publication No. 2002-100003). The apparatus described in Patent Document 1 recognizes the top and bottom, left and right, and front and back of the ultrasonic probe by recognizing projections of a predetermined shape provided on a plurality of side surfaces of the ultrasonic probe along the central axis of the ultrasonic probe. , to estimate the pose of the ultrasound probe.

Japanese Patent No. 6510570

However, in the apparatus described in Patent Document 1, it is necessary to provide a marker such as a projection on the ultrasonic probe, which may hinder the user's measurement work. In addition, there is a possibility that accurate posture estimation may not be possible due to marker omission, positional displacement, shape change due to damage or deterioration, and the like.

One aspect of the present invention is a posture estimation device that includes a first tool and a second tool that are three-dimensionally different from each other and that estimates the posture of the tool that performs a predetermined action in a predetermined direction, the posture estimating apparatus comprising: Acquired by a shape acquisition unit that acquires a dimensional shape and shape information in an operation direction corresponding to the three-dimensional shape, an image acquisition unit that acquires image information obtained by photographing an instrument and an object to be operated by the instrument, and a shape acquisition unit a posture estimating unit that identifies the appliance based on the obtained shape information and the image information acquired by the image acquiring unit, and estimates the movement direction of the specified appliance with respect to the target.

According to the present invention, the posture of the instrument can be estimated without providing markers.

1 is a diagram schematically showing an example of a device to which a posture estimation device according to an embodiment of the invention is applied; FIG. FIG. 2 is a perspective view schematically showing a film and a holder holding the film used for the dental radiography of FIG. 1; 1 is a block diagram showing the configuration of main parts of a posture estimation device according to an embodiment of the present invention; FIG. The figure which shows an example of the three-dimensional shape data of the instrument of FIG. FIG. 2 is a diagram showing another example of three-dimensional shape data of the instrument in FIG. 1; FIG. 4 is a diagram showing an example of image data captured from the front of a patient and acquired by the image acquisition unit in FIG. 3 ; FIG. 4 is a view showing an example of image data captured from the side of a patient and acquired by the image acquisition unit in FIG. 3; FIG. 4 is a diagram showing an example of a display screen of the input/output unit in FIG. 3 for a patient photographed from the front; FIG. 4 is a diagram showing an example of a display screen of the input/output unit in FIG. 3 for a patient imaged from the side; 4 is a flowchart showing an example of processing executed by the posture estimation device according to the embodiment of the present invention;

An embodiment of the present invention will be described below with reference to FIGS. 1 to 7. FIG. A posture estimation apparatus according to an embodiment of the present invention performs a predetermined operation in a predetermined direction, such as an irradiation device of an X-ray apparatus that emits X-rays in a predetermined direction, or a probe of an ultrasonic measurement apparatus that transmits and receives ultrasonic waves in a predetermined direction. Applied to various medical instruments, it estimates the direction of motion (orientation of the instrument) with respect to the target of motion such as measurement or treatment. An example of estimating the orientation of the irradiation device with respect to the imaging target of the dental X-ray apparatus will be described below.

FIG. 1 is a diagram schematically showing an example of a device 1 to which a posture estimation device according to an embodiment of the present invention is applied, and shows an X-ray irradiation device (hereinafter referred to as device) 1 of a dental X-ray apparatus 10. As shown in FIG. FIG. 2 is a perspective view schematically showing a film 3 used for dental radiography and a holder 40 holding the film 3. As shown in FIG.

As shown in FIGS. 1 and 2, in dental radiography, a film 3 is fixed to the oral cavity side of a patient's 2 teeth (for example, left molars), and X is projected in a predetermined irradiation direction A from the instrument 1 toward the film 3. irradiate the line.

The film 3 is held by a holder 40 as shown in FIG. As shown in FIG. 2, the holder 40 includes a clip portion 41 for holding the film 3, an engaging portion 42 adjacent to the clip portion 41, a ring-shaped indicator portion 43, the clip portion 41, the engaging portion 42, and the indicator portion. 43, and a stay portion 44 that connects with 43.

When the patient 2 bites the bite portion 42 of the holder 40 holding the film 3 with the teeth, the film 3 is fixed to the teeth of the patient 2 on the oral cavity side, and the indicator portion 43 is exposed outside the oral cavity. An irradiation direction A is indicated in which the line is to be irradiated.

By the way, when performing follow-up observation of a target tissue such as a tooth of the patient 2 or a lesion around it using an X-ray image or the like, images of the target tissue taken from the same irradiation direction A are periodically compared with each other. There is a need. However, even if the holder 40 of FIG. 2 is used, the position and angle at which the patient 2 bites the holder 40 will change. difficult.

Therefore, in this embodiment, the posture of the medical device with respect to the target of the operation such as measurement or treatment is estimated, and the appropriate posture of the medical device can be guided to the user performing the operation such as measurement or treatment as follows. to construct a pose estimation device.

FIG. 3 is a block diagram showing the main configuration of a posture estimation device (hereinafter referred to as device) 50 according to an embodiment of the present invention. The device 50 includes a computer having a CPU 51, a memory 52 such as ROM and RAM, and other peripheral circuits such as an I/O interface. The CPU 51 includes a shape acquisition unit 53 that acquires three-dimensional shape data of the device 1, an image acquisition unit 54 that acquires image data of the device 1 and the patient 2, and a posture estimation unit that estimates the irradiation direction A of the device 1 with respect to the patient 2. 55 and an information output unit 56 that outputs information on the irradiation direction A. FIG. An external memory 57, an input/output unit 58, and a camera 59 can be connected to the device 50 by wire or wirelessly.

4A and 4B are diagrams showing an example of the three-dimensional shape data of the instrument 1. FIG. The memory 52 stores three-dimensional shape data of the dental X-ray apparatus 10 including the instrument 1 . As shown in FIG. 4A, the three-dimensional shape data includes information on the outline 1M of the instrument 1, information on the X-ray irradiation direction A (irradiation axis) of the instrument 1, an intersection point P1 between the outline of the instrument 1 and the irradiation axis, Information of P2 is included.

Although illustration is omitted, the three-dimensional shape data of the dental X-ray apparatus 10 includes three-dimensional data 10M representing the outer shape of the dental X-ray apparatus 10 (whole or main body 10a), and the shape of the dental X-ray apparatus 10 when fixed to the floor or the like. The three-dimensional shape data CM indicating the installation axis C and the three-dimensional shape data OM indicating the reference points O of the headrest, switches, etc. fixed to the main body 10a are also included.

The information on the external shape 1M of the instrument 1 includes, for example, three-dimensional CAD (Computer-Aided Design) data provided by the manufacturer of the dental X-ray apparatus 10, and the actual dental X-ray apparatus 10 (instrument 1) obtained by a 3D scanner or the like. Three-dimensional shape data or the like obtained by measuring can be used.

As the three-dimensional shape data, instead of the information on the outer shape 1M of the instrument 1, data of a plurality of two-dimensional images (first image data) when the instrument 1 is photographed from a plurality of different directions, and the features in each image. Second image data in which virtual markers Pn are set at points can also be used. FIG. 4B is an example of second image data. By setting virtual markers Pn (13 virtual markers P1 to P13 in FIG. 4B) at feature points (13 feature points in FIG. 4B) in a plurality of images captured from a plurality of different directions, the entire instrument 1 can be visualized. Three-dimensional shape data covering feature points can be obtained.

For example, actual measurement (preliminary measurement) is performed with a marker being applied to a feature point (the center of a surface or an end, an equal division point, a corner, etc.) of the outer shape of the actual instrument 1 with a pen, a sticker, or the like, and the measurement is performed. 2nd image data (FIG. 4B) is obtained by taking a moving image of the instrument 1 and dividing it into a plurality of still images. Furthermore, the first image data (FIG. 4A) is obtained by deleting the marker Pn from the second image data by image processing, thereby obtaining a data set of the first image data and the second image data. By performing machine learning using such a data set, it becomes possible to estimate the orientation of the appliance 1 based on the learning results.

As a preliminary measurement for such machine learning, by performing measurements assuming that the instrument 1 is actually used, the imaging angle of the instrument 1 by the camera 59 (FIG. 3) can be reproduced and practical Can perform machine learning based on datasets. As a result, the accuracy of posture estimation of the appliance 1 based on the learning result can be improved. Furthermore, a data set may be prepared for each measurement item assumed for the instrument 1 (measurement content such as measurement target location and probe scanning direction). In this case, machine learning is performed in units of data sets annotated for each measurement item, and by switching the learning result for each measurement item, the accuracy of posture estimation of the appliance 1 can be further improved.

The memory 52 stores three-dimensional shape data of a plurality of dental X-ray apparatuses 10 (tools 1) having different external shapes and feature points, for example, for each manufacturer and model.

The external memory 57 in FIG. 3 is, for example, an external computer memory connectable to the device 50 via a network, or a flash memory connectable to the device 50 via an input/output port. The input/output unit 58 includes a keyboard, a mouse, a display, a speaker, a touch panel integrated with the display, and the like. By operating the input/output unit 58 and inputting the three-dimensional shape data stored in the external memory 57 to the device 50, the user additionally registers the three-dimensional shape data of the new dental X-ray device 10 (instrument 1). can do.

The shape acquisition unit 53 of the device 50 acquires three-dimensional shape data input via the input/output unit 58 . The three-dimensional shape data acquired by the shape acquisition section 53 is stored in the memory 52 of the device 50 .

The camera 59 is composed of two cameras capable of capturing moving images, and is attached to, for example, a wall surface facing the dental X-ray apparatus 10 so that the instrument 1 and the face of the patient 2 are included in the capturing range. By constructing the camera 59 with two cameras, that is, constructing it as a stereo camera, it is possible to ensure a sufficient photographing range even when the instrument 1 and the face of the patient 2 overlap each other. The two cameras 59 are arranged such that their horizontal shooting directions are different from each other by about 90 degrees. Image data captured by camera 59 is transmitted to device 50 and display (input/output unit) 58 .

The image acquisition unit 54 of the device 50 acquires image data of the instrument 1 and the patient 2 transmitted from the camera 59 . The image acquisition unit 54 can also acquire image data of the instrument 1 and the patient 2 input from the external memory 57 via the input/output unit 58 . For example, the user can operate the input/output unit 58 to input to the device 50 the image data of the same patient 2 stored in the external memory 57 at the time of the previous imaging.

The posture estimation unit 55 is based on the three-dimensional shape data of the dental X-ray apparatus 10 (instrument 1) acquired by the shape acquisition unit 53 and stored in the memory 52 and the image data acquired by the image acquisition unit 54. , estimate the irradiation direction A of the instrument 1 with respect to the patient 2 . Specifically, first, the installation axis C and the reference point O of the main body 10a of the dental X-ray apparatus 10 on the image are recognized, and the direction of gravity parallel to the recognized installation axis C is the Z axis, and the reference point O is the origin. A three-dimensional coordinate space is set.

5A and 5B are diagrams showing an example of image data acquired by the image acquisition unit 54. FIG. 5A shows image data taken from the front of the patient 2, and FIG. shows image data taken from As shown in FIGS. 5A and 5B, the posture estimation unit 55 detects feature points including facial feature points such as body joints, eyes, ears, and nose of the patient 2 on each image, and corresponds to each feature point. Three-dimensional coordinates of nodes N1 to N18 are calculated. Further, the outline and feature points of the instrument 1 are detected, and the three-dimensional coordinates of intersection points (feature points) P1 and P2 between the outline of the instrument 1 and the irradiation axis are calculated.

Detection of feature points in each image and calculation of two-dimensional coordinates are performed using feature point detection algorithms based on deep learning such as the OpenPose library (CVPR2017, Carnegie Mellon University, Zhe Cao et al., International Conference on Computer Vision) stored in memory 52 in advance. performed using In the OpenPose library, each feature point is detected based on the relationship between the feature points, such as the distance between feature points and relative movement.

The detection of the outline and feature points of the instrument 1 in each image and the calculation of the two-dimensional coordinates of the intersection points (feature points) P1 and P2 between the outline of the instrument 1 and the irradiation axis are based on the three-dimensional shape data stored in the memory 52 in advance. done based on When a plurality of three-dimensional shape data are registered, the outline or feature points of the appliance 1 on the image can be compared with the outline 1M or the virtual marker Pn of the appliance 1 included in the three-dimensional shape data. The model is specified, and the two-dimensional coordinates of the intersection points P1 and P2 are calculated based on the corresponding three-dimensional shape data. It should be noted that the user himself/herself may specify the manufacturer and model used for photographing.

When the two-dimensional coordinates of nodes N1 to N18 and intersections P1 and P2 in a pair of images (for example, FIGS. 5A and 5B) captured by a pair of left and right cameras 59 constituting a stereo camera are calculated, the principle of triangulation is calculated. , the corresponding three-dimensional coordinates are calculated. These three-dimensional coordinates may be detected by, for example, an infrared depth camera such as Kinect (registered trademark).

The calculated three-dimensional coordinates of the nodes N1 to N18 correspond to the posture of the patient 2 in the direction of gravity or the posture of the patient 2 with respect to the main body 10a of the dental X-ray apparatus 10. FIG. Further, the calculated three-dimensional coordinates of the points of intersection P1 and P2 correspond to the orientation of the instrument 1 in the direction of gravity or the orientation of the instrument 1 with respect to the main body 10a of the dental X-ray apparatus 10 . Further, for example, the relative coordinates of the intersection points P1 and P2 of the instrument 1 with respect to the nodes N1 to N3 of the facial feature points of the patient 2 correspond to the posture of the instrument 1 with respect to the patient 2 (irradiation direction A).

Information on the posture of the patient 2 and the posture of the appliance 1 (irradiation direction A) is output to the display (input/output unit) 58 by the information output unit 56 . Further, nodes N1 to N18 corresponding to the reference posture of the patient 2 and the reference posture (reference direction Az) of the instrument 1 and the intersection point P1 in the image at the time when X-rays were emitted by the dental X-ray apparatus 10 and X-rays were taken. , P2 are stored in the memory 52 . Information on the reference posture of the patient 2 and the reference posture (reference direction Az) of the instrument 1 stored in the memory 52 is read out at the next imaging and output to the display 58 by the information output unit 56 .

6A and 6B are diagrams showing an example of the display screen of the display (input/output unit) 58. FIG. 6A shows the display screen of the image photographed from the front of the patient 2, and FIG. Fig. 3 shows a display screen of an image taken from the side; As shown in FIGS. 6A and 6B, the image from the camera 59 is superimposed on the display 58 with information on the current and previous postures of the patient 2 and the instrument 1 .

Via the display screen of the display 58, the user can view the current posture (N1, N2, N3) of the patient 2 in the direction of gravity or relative to the main body 10a of the dental X-ray apparatus 10 and the reference posture (N1z, N2z) at the time of the previous imaging. , N3z) can be verified at the same time. Thereby, the posture of the patient 2 can be adjusted so as to match the reference posture. In addition, since the current irradiation direction A of the instrument 1 and the reference direction A0 at the time of the previous imaging can be confirmed simultaneously via the display screen of the display 58, the posture of the instrument 1 with respect to the patient 2 is adjusted so as to match the reference direction A0. can do.

FIG. 7 is a flow chart showing an example of processing executed by device 50 according to a pre-stored program. The processing shown in this flowchart is started, for example, when image data is acquired, and is repeated at a predetermined cycle. First, image data is read in step S1, then the instrument 1 currently in use is specified in step S2, and then the postures of the patient 2 and the instrument 1 are estimated in step S3. Next, in step S4, it is determined whether or not the information of the reference posture at the time of the previous photographing is stored in the memory. If the result in step S4 is affirmative, the process advances to step S5 to output to the display 58 information on the current posture and information on the reference posture at the time of the previous photographing. If the result in step S4 is NO, the process advances to step S6 to output only the current posture information to the display 58. FIG. Next, in step S7, it is determined whether or not photography has been performed. If the result is affirmative, the process proceeds to step S8. In step S8, information about the current posture (that is, at the shooting time) is stored in the memory.

According to the embodiment of the present invention, the following effects can be obtained.
(1) The device 50 estimates the posture of the instrument 1 that irradiates X-rays in a predetermined irradiation direction A. The apparatus 50 includes a shape acquisition unit 53 that acquires a pre-registered three-dimensional shape of the instrument 1 and three-dimensional shape data in the irradiation direction A corresponding to the three-dimensional shape, the instrument 1, and an object to be imaged by the instrument 1. An image acquisition unit 54 that acquires image data obtained by photographing the patient 2, and the instrument 1 for the patient 2 based on the three-dimensional shape data acquired by the shape acquisition unit 53 and the image data acquired by the image acquisition unit 54. and a posture estimation unit 55 for estimating the irradiation direction A (FIG. 3). Thereby, based on the three-dimensional shape data of the tool 1 registered in advance, the posture of the tool 1 can be estimated without providing a marker.

(2) The instrument 1 includes a first instrument and a second instrument having three-dimensional shapes different from each other. The posture estimation unit 55 identifies the device 1 based on the three-dimensional shape data acquired by the shape acquisition unit 53 and the image data acquired by the image acquisition unit 54, and irradiates the patient 2 with the specified device 1. Estimate direction A. As a result, a plurality of pre-registered instruments 1 with different three-dimensional shapes are automatically recognized and the posture is estimated, so there is no need to change the settings each time the instrument 1 to be used is changed.

(3) The apparatus 50 further includes an information output unit 56 that outputs information on the irradiation direction A of the instrument 1 to the patient 2 estimated by the posture estimation unit 55 (FIG. 3). For example, by superimposing the irradiation direction A on the captured image of the instrument 1 and the patient 2, the user can easily grasp the posture of the instrument 1 and adjust it.

(4) The device 50 further includes a memory 52 that stores the irradiation direction A of the appliance 1 toward the patient 2 estimated by the posture estimation unit 55 at the time of imaging as a reference direction Az (FIG. 3). The information output unit 56 outputs the reference direction Az stored in the memory 52 together with the information on the irradiation direction A of the appliance 1 with respect to the patient 2 estimated by the posture estimation unit 55 .

For example, by superimposing the current irradiation direction A and the reference direction Az, which is the irradiation direction at the time of the previous imaging, on an image obtained by photographing the device 1 and the patient 2, the user can change the posture of the device 1 from that at the time of the previous photography. Easy to match. As a result, it becomes easy to photograph the patient 2 periodically from the same irradiation direction A, and a user who performs follow-up observation of a specific patient 2 can be favorably supported.

(5) The image data includes information on the direction of gravity. The posture estimation unit 55 further estimates the posture of the patient 2 in the direction of gravity. For example, by superimposing the current posture of the patient 2 and the posture of the patient 2 at the time of the previous imaging on an image including the main body 10a of the dental X-ray apparatus 10 fixedly installed on the floor or the like, the user can However, it becomes easy to match the posture of the patient 2 in the direction of gravity with that at the time of the previous imaging. As a result, it becomes easy to regularly photograph the patient 2 in the same posture, and it is possible to suitably support the user observing the progress of the target tissue such as a lesion whose shape changes according to the direction of gravity.

The above embodiment can be modified in various forms. Modifications will be described below. In the above-described embodiment, the information output unit 56 outputs to the display 58 information about the current posture and information about the reference posture at the time of the previous imaging. The department is not limited to these. For example, the information may be output to a speaker to guide the user by voice, or the information may be output to a movable laser light source or the like to guide the user by laser light.

In the above embodiment, an example in which the device 50 is applied to the X-ray irradiation device 1 of the dental X-ray device 10 has been described. is not limited to For example, the present invention may be applied to forceps for laparoscopic surgery that performs operations such as incision and suturing at a predetermined position in a predetermined direction.

In the above embodiment, an example is shown in which the three-dimensional coordinates of the nodes N1 to N3 corresponding to the facial feature points of the patient 2 calculated by the posture estimation unit 55 are used as the posture of the patient 2 (FIGS. 6A and 6B). The orientation of the target in the direction of gravity is not limited to this. Three-dimensional coordinates of a plurality of nodes corresponding to feature points such as body joints can be used as the pose of the target.

The above description is merely an example, and the present invention is not limited by the above-described embodiments and modifications as long as the features of the present invention are not impaired. It is also possible to arbitrarily combine one or more of the above embodiments and modifications, and it is also possible to combine modifications with each other.

1 instrument (X-ray irradiation instrument), 2 patient, 50 device (posture estimation device), 51 CPU, 52 memory, 53 shape acquisition unit, 54 image acquisition unit, 55 posture estimation unit, 56 information output unit, 57 external memory, 58 input/output unit, 59 camera

Claims (4)

A posture estimating device including a first tool and a second tool having three-dimensional shapes different from each other and estimating the posture of the tool performing a predetermined action in a predetermined direction,
a shape acquisition unit that acquires a pre-registered three-dimensional shape of the instrument and shape information of an operating direction corresponding to the three-dimensional shape;
an image acquisition unit that acquires image information obtained by photographing the tool and an object to be operated by the tool;
a posture estimation unit that identifies the appliance based on the shape information acquired by the shape acquisition unit and the image information acquired by the image acquisition unit, and estimates the movement direction of the specified appliance with respect to the target; A posture estimation device comprising: The posture estimation device according to claim 1,
A posture estimation apparatus, further comprising: an information output unit that outputs information on the movement direction of the appliance with respect to the object estimated by the posture estimation unit. The posture estimation device according to claim 2,
further comprising a storage unit that stores, as a reference direction, the direction of movement of the appliance with respect to the object estimated by the posture estimation unit when the predetermined action is performed,
The posture estimation device, wherein the information output unit outputs information on the reference direction stored in the storage unit together with information on the movement direction of the appliance with respect to the object estimated by the posture estimation unit. The posture estimation device according to any one of claims 1 to 3,
The image information includes information on the direction of gravity,
The posture estimation device, wherein the posture estimation unit further estimates the posture of the object in a direction of gravity.

Images