JP7465469B2 - Learning device, estimation device, learning program, and estimation program - Google Patents

Description

The present invention relates to a learning device, an estimation device, a learning program, and an estimation program for estimating the position of a joint in an image of bone tissue inside a living body.

In recent years, the number of rheumatoid arthritis patients has been increasing. Rheumatoid arthritis is an inflammatory autoimmune disease in which the body's own immune system attacks the joints, mainly of the hands and feet, causing joint pain and deformation.

Rheumatoid arthritis is a disease in which early treatment can improve the prognosis. Therefore, in order to provide appropriate treatment to rheumatoid arthritis patients, it is necessary to accurately evaluate the degree of disease progression.

The progression of rheumatoid arthritis is evaluated, for example, by the mTS (modified Total Sharp) score.

JP2017-534397A (paragraphs 0009 and 0010)

Conventionally, to calculate the mTS score, doctors had to manually identify the joint positions in X-ray images of the fingers or toes of a rheumatoid arthritis patient. This task of manually identifying joint positions was time-consuming for doctors. There was also the problem that the accuracy of identifying joint positions depended on the doctor's skill.

Therefore, there is a demand for the development of image processing that can automatically identify joint positions. However, if the computational load is large in image processing that can automatically identify joint positions, new problems such as increased processing time will arise.

Patent Document 1 discloses a technology for automatically identifying the positions of finger joints, but this technology requires a hand fixation device, sensors, etc., and is therefore difficult to use.

In view of the above, the present invention aims to provide a learning device and a learning program that can obtain the learning results necessary to estimate the position of joints while reducing the computational load in image processing.

In view of the above, the present invention also aims to provide an estimation device and an estimation program that can estimate the position of a joint while reducing the computational load in image processing.

To achieve the above object, a learning device according to a first aspect of the present invention includes a first identification unit that identifies the position of each of the multiple joints in an image including the area of each of the multiple joints, and a second identification unit that identifies the relative positions between the multiple joints as model position information, the model position information having a predetermined number of parameters, the predetermined number being smaller than the product of the number of the multiple joints and the number of dimensions of each coordinate indicating the position of each of the multiple joints (first configuration).

To achieve the above object, the learning device according to the second aspect of the present invention includes an acquisition unit that acquires first and second captured images of bone tissue in a living body, a calculation unit that divides the first captured image into a plurality of regions and determines image features of the divided regions, a generation unit that generates a classifier for identifying which of at least one class including a first class that is a joint region an arbitrary region belongs to by machine learning the image features, and an extraction unit that extracts the relative positions between a plurality of joints from the second captured image as model position information, the model position information having a predetermined number of parameters, and the predetermined number is configured to be smaller than the product of the number of the plurality of joints and the number of dimensions of each coordinate indicating the position of each of the plurality of joints (second configuration).

In the learning device of the second configuration, the second captured images may be multiple, each of the multiple second captured images being captured with a different living body as a subject, and the extraction unit may be configured to extract the model position information by performing principal component analysis on the joint positions of the multiple second captured images (third configuration).

In the learning device of the second or third configuration, the at least one class may be configured to include a second class that is a fingertip region (fourth configuration).

The learning device of any one of the second to fourth configurations may further include a pre-processing unit provided between the acquisition unit and the calculation unit and the extraction unit, and the pre-processing unit may divide the first captured image and the second captured image into a plurality of blocks by horizontal lines, obtain pixel values that are the boundaries between the subject area in which the living body is captured and the background area in which the living body is not captured in each of the plurality of blocks based on a pixel value histogram, and correct the pixel values on a block-by-block basis so that the shift in pixel values at the boundaries between the plurality of blocks is small (fifth configuration).

To achieve the above object, the learning program according to the third aspect of the present invention is a learning program that causes a computer to function as a first identification unit that identifies the position of each of the multiple joints in an image including the area of each of the multiple joints, and a second identification unit that identifies the relative positions between the multiple joints as model position information, the model position information having a predetermined number of parameters, the predetermined number being smaller than the product of the number of the multiple joints and the number of dimensions of each coordinate indicating the position of each of the multiple joints (sixth configuration).

In order to achieve the above object, the learning program according to the fourth aspect of the present invention is a learning program that causes a computer to function as an acquisition unit that acquires first and second captured images of bone tissue in a living body, a calculation unit that divides the first captured image into multiple regions and determines image features of the divided regions, a generation unit that generates a classifier for identifying which of at least one class including a first class that is a joint region an arbitrary region belongs to by machine learning the image features, and an extraction unit that extracts the relative positions between multiple joints as model position information from the second captured image, where the model position information has a predetermined number of parameters, and the predetermined number is configured to be smaller than the product of the number of the multiple joints and the number of dimensions of each coordinate indicating the position of each of the multiple joints (seventh configuration).

In order to achieve the above object, the estimation device according to the fifth aspect of the present invention is an estimation device having a classifier generated by a learning device and model position information extracted by the learning device, the learning device being a learning device having any of the second to fifth configurations described above, and the estimation device is configured to include a first processing unit that divides an estimation target image in which bone tissue in a living body is photographed into a plurality of regions and obtains image features for each of the plurality of regions, a second processing unit that uses the classifier to obtain a degree of joint-likeness for each of the plurality of regions from the image features obtained by the first processing unit, a third processing unit that obtains joint candidates in the estimation target image based on the degree of joint-likeness, and a fourth processing unit that changes the parameters of the model position information based on the joint candidates and converges the update of the parameters based on maximizing the sum of the degrees of joint-likeness obtained by the classifier corresponding to each joint in the model position information (eighth configuration).

In order to achieve the above object, the estimation program according to the sixth aspect of the present invention is an estimation program that causes a computer to function as an estimation device having a classifier generated by a learning device and model position information extracted by the learning device, the learning device being a learning device having any of the first to fourth configurations described above, and the estimation device is configured to include a first processing unit that divides an estimation target image in which bone tissue in a living body is photographed into a plurality of regions and obtains image features for each of the plurality of regions, a second processing unit that uses the classifier to obtain a degree of joint-likeness for each of the plurality of regions from the image features obtained by the first processing unit, a third processing unit that obtains joint candidates in the estimation target image based on the degree of joint-likeness, and a fourth processing unit that changes the parameters of the model position information based on the joint candidates and converges the update of the parameters so as to maximize the sum of the degrees of joint-likeness obtained by the classifier corresponding to each joint in the model position information (ninth configuration).

The learning device and learning program of the present invention can obtain the learning results necessary to estimate the position of joints while reducing the computational load in image processing.

The estimation device and estimation program of the present invention can estimate the position of joints while reducing the computational load in image processing.

FIG. 1 is a diagram showing a configuration of an information processing device according to an embodiment of the present invention. FIG. 2 is a functional block diagram showing an example of a function of the information processing device shown in FIG. 1 ; FIG. 1 is a diagram showing an example of an X-ray image acquired by an acquisition unit; FIG. 13 is a diagram showing another example of an X-ray image acquired by the acquisition unit; FIG. 13 is a schematic diagram showing how a first captured image is divided by a horizontal line; A diagram showing the relationship between block numbers and boundary pixel values. 1 is a flowchart showing an example of a general operation of a learning device; 1 is a flowchart showing an example of a schematic operation of an estimation device;

An embodiment of the present invention will be described below with reference to the drawings.

1. Configuration of information processing device
1 is a diagram showing the configuration of an information processing device according to an embodiment of the present invention. The information processing device 1 according to an embodiment of the present invention (hereinafter referred to as information processing device 1) includes a control unit 2, a storage unit 3, a communication unit 4, a display unit 5, and an operation unit 6.

The control unit 2 is, for example, a microcomputer. The control unit 2 performs overall control of the information processing device 1. The control unit 2 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory), all of which are not shown.

The storage unit 3 is, for example, a flash memory, a hard disk drive, etc. It stores various data, learning programs and estimation programs executed by the information processing device 1, etc.

The communication unit 4 is a communication interface for communicating with an external device. The communication method between the communication unit 4 and the external device may be wired communication, wireless communication, or a combination of wired and wireless communication. Examples of the external device include an X-ray imaging device that captures X-ray images, a storage device that stores X-ray images, etc.

The display unit 5 is, for example, a liquid crystal display device, an organic EL (Electro Luminescence) display device, etc. The display unit 5 displays various images based on the control of the control unit 2.

The operation unit 6 is, for example, a keyboard, a pointing device, etc. The operation unit 6 outputs a signal according to the user's operation to the control unit 2.

2. Functions of the information processing device
2 is a functional block diagram showing an example of the functions of the information processing device 1. The information processing device 1 functions as a learning device 10 by executing a learning program. The information processing device 1 functions as an estimation device 20 by executing an estimation program. In this embodiment, one information processing device 1 includes the learning device 10 and the estimation device 20 as functional units, but unlike this embodiment, the information processing device including the learning device 10 as a functional unit and the information processing device including the estimation device 20 as a functional unit may be separate information processing devices.

<2-1. Learning device>
The learning device 10 includes an acquisition unit 11, a preprocessing unit 12, a calculation unit 13, a generation unit 14, and an extraction unit 15.

The acquisition unit 11 acquires the first and second captured images of bone tissue in a living body. Specifically, the acquisition unit 11 acquires the first and second captured images from an external device via the communication unit 4 (see FIG. 1).

In this embodiment, the first captured image and the second captured image are the same image. However, unlike this embodiment, the first captured image and the second captured image may be different captured images.

In addition, in this embodiment, the acquisition unit 11 acquires a plurality of first captured images (= second captured images). More specifically, the acquisition unit 11 acquires a plurality of X-ray captured images as the plurality of first captured images (= second captured images). In this embodiment, the plurality of X-ray captured images acquired by the acquisition unit 11 are images of both hands of a plurality of different people captured by an X-ray imaging device.

Figure 3 is an example of an X-ray image acquired by the acquisition unit 11, which is an image of both hands of a person with non-advanced rheumatoid arthritis. Figure 4 is another example of an X-ray image acquired by the acquisition unit 11, which is an image of both hands of a person with advanced rheumatoid arthritis.

As can be seen from Figures 3 and 4, the upper part of the X-ray image is dark, and the lower part of the X-ray image is bright. This occurrence of light and dark in the vertical direction is due to the characteristics of the X-ray imaging device.

Therefore, the preprocessing unit 12 divides the first captured image (= second captured image) into blocks B1 to B10 by horizontal lines H1 to H9 arranged at equal intervals, as shown in the schematic diagram of FIG. 5. Here, the number of divisions is set to 10 for simplicity of illustration, but the actual number of divisions is preferably about 100 to 400, for example. If the number of divisions is too small, smooth correction cannot be performed, and conversely, if the number of divisions is too large, the area of each block becomes narrow and noise resistance deteriorates. In this embodiment, the actual number of divisions is set to 200. Then, the preprocessing unit 12 obtains pixel values (threshold values) that are the boundaries between the hand area (foreground) and the non-hand area (background) in each of the blocks B1 to B10. There is no particular limitation on the method of obtaining pixel values (threshold values) that are the boundaries between the hand area (foreground) and the non-hand area (background), but in this embodiment, Otsu's threshold method is used. Then, the preprocessing unit 12 corrects the pixel values in units of blocks B1 to B10 so that the shift in pixel values that are the boundaries between the blocks B1 to B10 is small. For example, if the relationship between the block number and the pixel value at the boundary is as shown in Figure 6 and can be approximated by the exponential function E1, different parameters are set for each block B1 to B10 so that the exponential function E1 is converted to a straight line L1, and the pixel values are corrected by calculating the parameters to the pixel values. In other words, after correcting the pixel values, if the pixel value (threshold value) at the boundary between the hand area (foreground) and the non-hand area (background) in each block B1 to B10 is found using Otsu's threshold method based on the pixel value histogram, the relationship between the block number and the pixel value (threshold value) at the boundary can be approximated by the straight line L1.

The correction process for X-ray images described above is not a process that can only be performed by the learning device 10, but can be applied to all image processing devices that perform image processing on X-ray images.

Next, the pre-processing unit 12 automatically extracts a hand region from the first captured image (= second captured image). Specifically, the pre-processing unit 12 determines that a pixel value higher than the boundary pixel value (threshold value) TH1, which is the vertical axis coordinate value of the straight line L1 shown in FIG. 6, is the hand region. Furthermore, the pre-processing unit 12 determines that a pixel value equal to or lower than the boundary pixel value (threshold value) TH1 is the non-hand region, and determines the hand region and non-hand region by removing holes and isolated points, smoothing shapes, and removing small area regions in each of the hand region and non-hand region.

Next, the preprocessing unit 12 divides the left hand into the right hand. The method for dividing the right hand into the left hand is not particularly limited, but in this embodiment, Otsu's threshold method is used. Specifically, in this embodiment, the preprocessing unit 12 creates a pixel value histogram of the hand region in the horizontal direction from the first captured image (= second captured image), and determines a dividing line between the left hand and the right hand in the first captured image (= second captured image) using Otsu's threshold method based on the pixel value histogram of the hand region. The right hand region is left-right inverted by the preprocessing unit 12, and is treated as the left hand thereafter. In other words, after the right hand region is left-right inverted, two left hands exist in one first captured image (= second captured image). As a result, two pieces of data related to the left hand are obtained from one first captured image (= second captured image). By doubling the amount of data in this way, the learning accuracy in the learning device 10 and the estimation accuracy in the estimation device 20 can be improved.

In this embodiment, the calculation unit 13 sets an area of 100 pixels vertically by 100 pixels horizontally, and divides each of the first captured images into a plurality of areas. The number of vertical pixels of the area set by the calculation unit 13 is not limited to 100. Similarly, the number of horizontal pixels of the area set by the calculation unit 13 is not limited to 100. The number of vertical pixels of the area set by the calculation unit 13 and the number of horizontal pixels of the area set by the calculation unit 13 may or may not be the same. Also, in this embodiment, each of the first captured images is divided into a plurality of areas such that there is no overlap between adjacent areas, but each of the first captured images may be divided into a plurality of areas such that there is overlap between adjacent areas.

Next, the calculation unit 13 calculates image features for each region. In this embodiment, HOG (Histograms of Oriented Gradients) features are used as the image features. The reason for using HOG features is that they are resistant to geometric transformations and robust to variations in brightness of the first captured image. Note that, unlike this embodiment, the calculation unit 13 may calculate image features other than HOG features.

The generation unit 14 sets multiple classes including a first class which is a joint region.

In this embodiment, the generation unit 14 sets a first class which is a joint region, a second class which is a fingertip region, and a third class which is a region which is neither a joint region nor a fingertip region. By the generation unit 14 setting the second class which is a fingertip region, it is possible to prevent the learning device 10 from erroneously learning that the fingertip region is a joint region. Note that, unlike this embodiment, the generation unit 14 may set only two classes, the first class which is a joint region and a class which is a non-joint region. Also, unlike this embodiment, the generation unit 14 may set only the first class which is a joint region.

The generating unit 14 selects 14 regions in the first class from each left hand region in each first captured image based on the output signal of the operating unit 6 (see FIG. 1). That is, the region selection in the first class is manual. The 14 regions are a region corresponding to the first joint of the thumb, a region corresponding to the second joint of the thumb, a region corresponding to the first joint of the index finger, a region corresponding to the second joint of the index finger, a region corresponding to the third joint of the index finger, a region corresponding to the first joint of the middle finger, a region corresponding to the second joint of the middle finger, a region corresponding to the third joint of the middle finger, a region corresponding to the first joint of the ring finger, a region corresponding to the second joint of the ring finger, a region corresponding to the third joint of the ring finger, a region corresponding to the first joint of the little finger, a region corresponding to the second joint of the little finger, and a region corresponding to the third joint of the little finger. As described above, the calculating unit 13 selects 14 regions in the first class from each left hand region in each first captured image, so it can also be said to be a first identifying unit that identifies the position of each of the multiple joints in an image including the regions of each of the multiple joints.

The generation unit 14 selects five regions in the second class from each left hand region in each first captured image based on the output signal of the operation unit 6 (see FIG. 1). In other words, the region selection in the second class is manual. The five regions are a region corresponding to the tip of the thumb, a region corresponding to the tip of the index finger, a region corresponding to the tip of the middle finger, a region corresponding to the tip of the ring finger, and a region corresponding to the tip of the little finger.

The generation unit 14 removes the first class area and the second class area from each left hand area in each first captured image, and then randomly selects, for example, 25 areas in the third class. In other words, area selection in the third class is automatic. In this embodiment, the calculation unit 13 randomly selects multiple areas in the third class that satisfy a predetermined condition. The predetermined condition is that each area selected in the third class does not overlap with other areas selected in the third class, and each area selected in the third class is evenly arranged so as not to be unevenly distributed.

Next, the generation unit 14 changes the contrast for each of the multiple first captured images. The contrast may be changed in one way or in multiple ways. The original first captured image and the first captured image with the contrast changed are used as learning data.

The generation unit 14 receives the above-mentioned learning data and image features of each region from the calculation unit 13. Using the above-mentioned learning data, the generation unit 14 performs machine learning on the image features of each region of the first class, the image features of each region of the second class, and the image features of each region of the third class, thereby generating a classifier for identifying which of the first to third classes an arbitrary region belongs to. There are no particular limitations on the method for generating the classifier, but in this embodiment, the classifier is generated using a support vector machine.

The extraction unit 15 receives the center position of each finger joint in the second captured image (= first captured image) from the pre-processing unit 12. The pre-processing unit 12 selects the center position of each finger joint in the second captured image (= first captured image) based on the output signal of the operation unit 6 (see FIG. 1). In other words, the selection of the center position of each finger joint in the second captured image (= first captured image) is manual. The extraction unit 15 calculates the average value of the coordinates for each joint for the multiple second captured images, and performs rigid body registration for each of the multiple second captured images based on the average value of the coordinates for each joint. The extraction unit 15 calculates the average value of the coordinates for each joint for the multiple second captured images after the rigid body registration, and performs rigid body registration for each of the multiple second captured images based on the average value of the coordinates for each joint. The extraction unit 15 repeats the calculation of the average value of the coordinates for each joint and the rigid body registration until the rigid body registration converges. The extraction unit 15 extracts the relative positions between the multiple joints as model position information from the second captured image (= first captured image) in which the rigid body registration has converged. The model position information has a predetermined number of parameters. The predetermined number is smaller than the product of the number of joints and the number of dimensions of each coordinate indicating the position of each of the joints. This makes it possible to estimate the positions of the joints while reducing the computational load in image processing, compared to when model position information is constructed using each coordinate indicating the position of each of the joints. As described above, the extraction unit 15 extracts the relative positions between the multiple joints from the second captured image (= first captured image) as model position information, and therefore can also be said to be a second identification unit that identifies the relative positions between the multiple joints as model position information.

In this embodiment, the extraction unit 15 extracts the relative positions between multiple finger joints as model position information from images of both hands of multiple different people captured by an X-ray imaging device. In this embodiment, the model position information has three parameters, the number of multiple finger joints in the model position information is 14, and each coordinate indicating the position of each of the multiple finger joints is two-dimensional. Note that the predetermined number of parameters in the model position information is not limited to three parameters, and as described above, it is sufficient that the predetermined number is smaller than the product of the number of multiple joints and the number of dimensions of each coordinate indicating the position of each of the multiple joints.

In this embodiment, the extraction unit 15 extracts model position information by subjecting the joint positions of the M second captured images to principal component analysis. Since there are 14 joint points in one hand and each joint point has a two-dimensional coordinate in the second captured images, the coordinates of each joint point of one hand (e.g., the right hand) are converted into a one-dimensional vector, resulting in vector a as follows:
a = [ _x1R1 , _y1R1 , _x1R2 , _y1R2 , ..., _x1R14 , _y1R14 ]

Therefore, the information on the joint positions of the M second captured images is expressed as a matrix A with M rows and 28 columns as shown below.

The extraction unit 15 extracts model position information by performing principal component analysis on matrix A. In this embodiment, the model position information is expressed as a mean vector + principal component score × principal component vector. The mean vector and the principal component vector are the same. The principal component vector is an n-dimensional vector composed of the direction with the largest variance (first principal component vector), the direction with the second largest variance (second principal component vector), ..., and the direction with the nth largest variance (nth principal component vector). In this embodiment, by setting n = 3, the model position information has three parameters as described above.

The principal component score is composed of a first parameter that is the coefficient of the first principal component vector, a second parameter that is the coefficient of the second principal component vector, and a third parameter that is the coefficient of the third principal component vector. The principal component score is a parameter that indicates the variation from the average relative position between the finger joints of each individual.

The learning device 10 described above operates, for example, as shown in the flowchart in FIG. 7. The following describes an example of the operation shown in FIG. 7.

First, the learning device 10 acquires the first and second captured images (step S1).

Next, the learning device 10 performs a correction process on the first and second captured images to reduce the brightness and darkness in the vertical direction of the first and second captured images (step S2).

Next, the learning device 10 determines the hand and non-hand regions of each of the first and second captured images (step S3).

Next, the learning device 10 divides the left hand and the right hand and performs left-right inversion processing on the right hand region (step S4).

Next, the learning device 10 divides the first captured image into a plurality of regions and obtains image features for each of the divided regions (step S5).

Next, the learning device 10 sets the first to third classes and selects areas for each of the first to third classes (step S6).

Next, the learning device 10 changes the contrast of the first captured image to increase the number of pieces of learning data (step S7).

Next, the learning device 10 uses the original first captured image and the first captured image with the contrast changed as learning data to generate a classifier for identifying which of the first to third classes an arbitrary region belongs to (step S8).

Next, the learning device 10 performs rigid body registration on the second captured image (step S9).

Finally, the learning device 10 extracts the relative positions between the multiple joints as model position information from the second captured image where the rigid body registration has converged (step S10).

<2-2. Estimation device>
The estimation device 20 includes a first processing unit 21, a second processing unit 22, a third processing unit 23, and a fourth processing unit 24.

The first processing unit 21 acquires the estimation target image. Specifically, the first processing unit 21 acquires the estimation target image from an external device via the communication unit 4 (see FIG. 1). In this embodiment, the estimation target image acquired by the first processing unit 21 is an image of both hands of a person captured by an X-ray imaging device.

Next, the first processing unit 21 performs the same processing as the pre-processing unit 12 on the estimation target image.

Next, the first processing unit 21 divides the estimation target image after the same processing as the pre-processing unit 12 has been performed into a plurality of regions, and obtains image features for each of the plurality of regions. In this embodiment, the first processing unit 21 sets a region of 100 pixels vertically by 100 pixels horizontally, and divides the estimation target image into a plurality of regions. Note that in this embodiment, the estimation target image is divided into a plurality of regions such that there is no overlap between adjacent regions, but the estimation target image may also be divided into a plurality of regions such that there is overlap between adjacent regions. Also, in this embodiment, the first processing unit 21 uses HOG features as image features, just like the calculation unit 13.

The second processing unit 22 uses the classifier generated by the generation unit 14 to determine the degree of joint-likeness for each of the multiple regions from the image features of each of the multiple regions determined by the first processing unit 21. Note that the above-mentioned first class means joint-likeness, the above-mentioned second class means fingertip-likeness, and the above-mentioned third class means background-likeness. In this embodiment, the degree of joint-likeness takes any value between 0 and 1, and the more joint-like an area is, the greater the value of the degree of joint-likeness.

The third processing unit 23 finds candidates for joints in the estimation target image based on the degree of joint-likeness.

Specifically, the third processing unit 23 uses the degree of joint-likeness to extract the area that is most like a joint as a joint candidate, and excludes the surrounding areas of the area extracted as a joint candidate from the joint candidates. The third processing unit 23 then uses the degree of joint-likeness to extract the area that is most like a joint from the remaining areas (areas other than the areas extracted as joint candidates and the areas excluded from the joint candidates) as a joint candidate, and excludes the surrounding areas of the area extracted as a joint candidate from the joint candidates. By repeating the above process, the third processing unit 23 extracts 20 joint candidates for one hand area.

The third processing unit 23 selects 14 joint positions from 20 joint candidates for one hand region. There is no particular limitation on the method of selecting 14 joint positions from 20 joint candidates for one hand region, but in this embodiment, the ICP (Iterative Closest Point) algorithm is used. Specifically, in this embodiment, the third processing unit 23 randomly selects 10 from the 20 joint candidates for one hand region, compares the average position of the randomly selected 10 with the position represented by the average vector of the model position information extracted by the extraction unit 15 to check the degree of match, and selects 14 joint positions using the ICP (Iterative Closest Point) algorithm.

The fourth processing unit 24 changes the parameters of the model position information based on the joint candidates, and converges the update of the parameters based on the sum of the degrees of joint-likeness determined by the classifiers corresponding to each joint in the model position information. Specifically, the fourth processing unit 24 changes the first to third parameters of the model position information extracted by the extraction unit 15 based on 14 joint positions selected from the 20 joint candidates. Then, the fourth classifier 24 converges the update of the first to third parameters so as to maximize the sum of the degrees of joint-likeness determined by the classifiers corresponding to each joint in the model position information. The model position information having the first to third parameters for which the updates have converged becomes the estimation result of the joint positions of the fingers in the estimation target image.

The estimation device 20 described above operates, for example, as shown in the flowchart in FIG. 8. The following describes an example of the operation shown in FIG. 8.

First, the estimation device 20 acquires an image to be estimated (step S11).

Next, the estimation device 20 performs a correction process on the estimation target image to reduce brightness in the vertical direction of the estimation target image (step S12).

Next, the estimation device 20 determines the hand region and non-hand region of the estimation target image (step S13).

Next, the estimation device 20 divides the left hand and the right hand, and performs left-right inversion processing on the right hand region (step S14).

Next, the estimation device 20 divides the target image into multiple regions and obtains image features for each of the multiple regions (step S15).

Next, the estimation device 20 uses the classifier generated by the learning device 10 to determine the degree of joint-likeness for each of the multiple regions from the image features of each of the multiple regions (step S16).

Next, the estimation device 20 finds candidates for joints in the estimation target image based on the degree of joint-likeness (step S17).

Finally, the estimation device 20 changes the parameters of the model position information based on the joint candidates, and converges the parameter update based on the sum of the degrees of joint-likeness determined by the classifiers corresponding to each joint in the model position information (step S18).

<3. Other>
In addition to the above-described embodiment, the configuration of the present invention can be modified in various ways without departing from the spirit of the invention. The above-described embodiment is illustrative in all respects and should be considered as not limiting, and the technical scope of the present invention is indicated by the claims, not the description of the above-described embodiment, and should be understood to include all modifications that fall within the meaning and scope of the claims.

For example, in the above-described embodiment, the first captured image, the second captured image, and the estimation target image are X-ray images, but they may be ultrasound images, etc. Also, they may be 3D images such as MRI (Magnetic Resonance Imaging) images and 3D-CT (Computed Tomography) images. When the first captured image, the second captured image, and the estimation target image are 3D images, the coordinates of the joint are three-dimensional.

The subject to be imaged is not limited to humans, but can be any living organism that has bone tissue.

REFERENCE SIGNS LIST 10 Learning device 11 Acquisition unit 12 Preprocessing unit 13 Calculation unit 14 Generation unit 15 Extraction unit 20 Estimation device 21 First processing unit 22 Second processing unit 23 Third processing unit 24 Fourth processing unit

Claims (7)

an acquisition unit that acquires first and second captured images of bone tissue in a living body;
A calculation unit that divides the first captured image into a plurality of regions and obtains image feature amounts of the divided regions;
a generation unit that generates a classifier for identifying to which class an arbitrary region belongs among at least one class including a first class that is a joint region, by machine learning the image feature amount;
an extraction unit that extracts relative positions between a plurality of joints from the second captured image as model position information;
Equipped with
The model position information has a predetermined number of parameters;
A learning device, characterized in that the predetermined number is smaller than a product of the number of the plurality of joints and the number of dimensions of each coordinate indicating the position of each of the plurality of joints. The second captured images include a plurality of images, each of which is an image captured using a different living body as a subject;
The learning device according to claim 1 , wherein the extraction unit extracts the model position information by performing principal component analysis on joint positions in the plurality of second captured images. The learning device according to claim 1 or 2 , wherein the at least one class includes a second class that is a fingertip region. a pre-processing unit provided between the acquisition unit and the calculation unit and the extraction unit;
The pre-treatment unit includes:
Dividing the first captured image and the second captured image into a plurality of blocks by horizontal lines;
determining a pixel value that is a boundary between a subject region in which the living body is captured and a background region in which the living body is not captured in each of the plurality of blocks based on a pixel value histogram;
4. The learning device according to claim 1, wherein pixel values are corrected on a block-by-block basis so that a shift in pixel values at the boundaries between the plurality of blocks is reduced. Computer,
an acquisition unit that acquires first and second captured images of bone tissue in a living body;
A calculation unit that divides the first captured image into a plurality of regions and obtains image feature amounts of the divided regions;
a generation unit that generates a classifier for identifying to which class an arbitrary region belongs among at least one class including a first class that is a joint region, by machine learning the image feature amount; and an extraction unit that extracts relative positions between a plurality of joints from the second captured image as model position information.
A learning program that functions as
The model position information has a predetermined number of parameters;
The learning program, wherein the predetermined number is smaller than a product of the number of the plurality of joints and the number of dimensions of each coordinate indicating the position of each of the plurality of joints. An estimation device having a classifier generated by a learning device and model position information extracted by the learning device,
The learning device is a learning device according to any one of claims 1 to 4 ,
The estimation device comprises:
a first processing unit that divides an estimation target image obtained by capturing an image of bone tissue in a living body into a plurality of regions and obtains an image feature amount for each of the plurality of regions;
a second processing unit that uses the classifier to determine a degree of likelihood of each of the plurality of regions being a joint from the image feature amount determined by the first processing unit;
a third processing unit that obtains candidates for the joint in the estimation target image based on the degree of joint resemblance;
a fourth processing unit that changes the parameters of the model position information based on the joint candidates and converges the update of the parameters so as to maximize a sum of the degrees of joint-likeness determined by the classifier corresponding to each joint in the model position information;
An estimation device comprising: Computer,
An estimation program for causing an estimation device to function as an estimation device having a classifier generated by a learning device and model position information extracted by the learning device,
The learning device is a learning device according to any one of claims 1 to 4 ,
The estimation device comprises:
a first processing unit that divides an estimation target image obtained by capturing an image of bone tissue in a living body into a plurality of regions and obtains an image feature amount for each of the plurality of regions;
a second processing unit that uses the classifier to determine a degree of likelihood of each of the plurality of regions being a joint from the image feature amount determined by the first processing unit;
a third processing unit that obtains candidates for the joint in the estimation target image based on the degree of joint resemblance;
a fourth processing unit that changes the parameters of the model position information based on the joint candidates and converges the update of the parameters so as to maximize a sum of the degrees of joint-likeness determined by the classifier corresponding to each joint in the model position information;
An estimation program comprising:

Images