JP2022175787A - Image processing device, image processing method and program - Google Patents

Abstract

To provide an image processing device which more quickly and highly accurately acquires a reference cross section group used for observation and image processing of a three-dimensional image.SOLUTION: An image processing device comprises: an image acquisition unit which acquires an input image being a three-dimensional image of an object; and a cross section acquisition unit which acquires a second cross section group formed of a plurality of second cross sections on the basis of a first cross section group formed from a plurality of first cross sections that are set for observing the object in the input image and are in a prescribed angle relation with each other, and a constraint condition based on the prescribed angle relation.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device, an image processing method, and a program.

In the medical field, three-dimensional images acquired by various image diagnostic apparatuses (modalities) such as ultrasonic diagnostic imaging apparatuses are used for diagnosis. In diagnosis using a three-dimensional image, a doctor sets a plurality of cross-sections (hereinafter also referred to as reference cross-sections) to be observed in the three-dimensional image, and diagnoses, measures, or performs functions on the reference cross-sections. Various image processing such as evaluation is applied. A plurality of reference cross-sections (hereinafter also referred to as a group of reference cross-sections) have predetermined positional and angular relationships with each other. For example, three-dimensional transesophageal echocardiographic observation uses three reference planes that are approximately orthogonal to each other.

As a technique for acquiring a cross section of a three-dimensional image, for example, Patent Document 1 discloses a technique for detecting feature points of a search site from candidate cross sections and acquiring observation cross sections using information on a plurality of detected feature points. do. Further, Patent Literature 2 discloses a technique for adjusting the neck surface of an aneurysm by superimposing and displaying mutually different cross-sectional images on two non-orthogonal cross-sectional images.

WO2016/195110 JP 2017-35379 A

A reference slice group for observing and measuring a three-dimensional image can be set manually, for example. That is, an operator such as a doctor specifies which cross section is the reference cross section on the three-dimensional image. Manually setting the reference cross section increases the workload of the operator. Also, manually set fiducial cross-sections may be inconsistent and reproducible. Furthermore, even if observation cross-sections are obtained using information on feature points of search regions, it may not be possible to efficiently search for reference cross-sections.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus that acquires a group of reference cross-sections used for observation and image processing of a three-dimensional image at a higher speed and with a higher degree of accuracy.

In order to solve the above problems, an image processing apparatus according to the present invention includes:
an image acquisition unit that acquires an input image that is a three-dimensional image of an object;
Based on a first cross-section group consisting of a plurality of first cross-sections set to observe the object in the input image and having a predetermined angular relationship with each other, and a constraint condition based on the predetermined angular relationship , a cross-section acquisition unit that acquires a second cross-section group consisting of a plurality of second cross-sections;
characterized by comprising

According to the present invention, it is possible to obtain a group of reference cross-sections used for observation and image processing of a three-dimensional image at a higher speed and with a higher degree of accuracy.

It is a figure explaining a reference cross section. 1 is a diagram illustrating the configuration of an image processing system according to a first embodiment; FIG. 9 is a flowchart illustrating reference cross-section acquisition processing according to the first embodiment; 9 is a flowchart illustrating learning model generation processing; 9 is a flowchart illustrating first cross-section acquisition processing; 9 is a flowchart illustrating second cross-section acquisition processing; FIG. 10 is a diagram illustrating the configuration of an image processing system according to a second embodiment; FIG. 10 is a flowchart illustrating reference cross-section acquisition processing according to the second embodiment;

Embodiments of the present invention will be exemplarily described in detail below with reference to the drawings. However, the components described in the following embodiments are merely examples, and the technical scope of the present invention is not limited by individual embodiments.

The image processing apparatus according to the embodiment provides a function of inputting a three-dimensional image (three-dimensional volume) of a subject (object) and acquiring (estimating) two or more types of reference cross sections. The input image to be processed is a medical image, that is, an image of a subject (such as a human body) captured or generated for purposes such as medical diagnosis, examination, and research. 1 is an image acquired by the system; For example, input images that can be processed include ultrasonic images obtained by an ultrasonic diagnostic apparatus, X-ray CT images obtained by an X-ray CT apparatus, MRI images obtained by an MRI apparatus, and the like.

In the following embodiments, an image processing apparatus acquires a group of reference cross sections using an example of a three-dimensional image of a heart region (three-dimensional transesophageal echocardiography) captured by a three-dimensional transesophageal ultrasound probe. A specific example of processing will be described in detail.

<First Embodiment>
The image processing apparatus according to the first embodiment acquires three reference cross-sections suitable for observing the mitral valve captured in the three-dimensional image from the three-dimensional image captured by the three-dimensional transesophageal probe. It is assumed that the three reference cross sections are approximately orthogonal to each other.

This embodiment takes advantage of the property that the reference cross-sections are approximately orthogonal to each other. That is, the image processing apparatus first acquires the first cross section (temporary reference cross section) under the constraint that the three cross sections are orthogonal to each other. Next, the image processing apparatus changes the first cross-sections of the first cross-section group (provisional reference cross-section group) so as not to deviate too much from the state in which the first cross-sections are orthogonal to each other. , obtain a second cross-section (reference cross-section) corresponding to each first cross-section.

An example of three second cross sections will be described with reference to FIG. FIG. 1 schematically shows a cross-sectional image obtained by slicing a three-dimensional image of the mitral valve using a transesophageal probe at the junction of the left ventricle and the left atrium. FIG. 1 shows an aortic valve 101 and a mitral valve 102 . The mitral valve 102 consists of two membranes, an anterior leaflet 103 and a posterior leaflet 104 .

A cross section 111 represented by a straight line in FIG. 1 is a second cross section passing through the center of the aortic valve 101 and the center of the mitral valve 102, and is called the A plane (YZ plane). A cross-section 112 represented by a straight line perpendicular to the straight line representing the cross-section 111 is a second cross-section passing through two connecting positions between the anterior leaflet and the posterior leaflet, and is called the B plane (XZ plane). A cross-section 113 is a second cross-section obtained by cutting the junction of the left ventricle and the left atrium, and is called the C-plane (XY plane). The A-plane, B-plane, and C-plane are a second group of cross sections that are substantially perpendicular to each other.

When the doctor manually sets the second cross-sections, each second cross-section may be set to a cross-section that deviates from the orthogonal relationship to some extent (the angle is shifted) in consideration of the appearance. In the present embodiment, the image processing apparatus obtains a plurality of first cross sections that are orthogonal to each other, corrects the angles of the obtained first cross sections, and obtains corresponding second cross sections. .

[Device configuration]
The configuration and processing of the image processing apparatus according to this embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating the configuration of an image processing system (also referred to as a medical image processing system) according to the first embodiment. The image processing system includes an image processing device 10 and a database 22 .

The image processing apparatus 10 is communicably connected to a database 22 via a network 21 . The network 21 includes, for example, a LAN (Local Area Network) and a WAN (Wide Area Network). Note that the image processing apparatus 10 may be configured integrally with the database 22 .

The database 22 holds and manages multiple images and information. Information managed by the database 22 includes three-dimensional images to be processed by the image processing apparatus 10 and learning data (also referred to as teacher data) for generating learning models (learned models). The learning data is composed of data of a plurality of cases prepared for learning (learning cases). Below, the data of individual cases included in the learning data will be referred to as learning case data. Each piece of learning case data includes a three-dimensional image of each case and correct second cross-sectional information in each image. The information managed by the database 22 may include information on learning models generated from learning data instead of learning data.

The image processing apparatus 10 can acquire data held in the database 22 via the network 21 . Information stored in the database 22 or part of the information stored in the database 22 may be stored in the internal storage (ROM 32 or storage unit 34) of the image processing apparatus 10. FIG.

The image processing apparatus 10 includes a communication IF (Interface) 31 , a ROM (Read Only Memory) 32 , a RAM (Random Access Memory) 33 , a storage section 34 , an operation section 35 , a display section 36 and a control section 37 .

The communication IF 31 (communication unit) is, for example, a LAN card. The communication IF 31 realizes communication between an external device such as the database 22 and the image processing apparatus 10, for example. The ROM 32 is a non-volatile memory and stores various programs and various data. The RAM 33 is a volatile memory and used as a work memory for temporarily storing programs and data being executed.

The storage unit 34 is, for example, an HDD (Hard Disk Drive), and stores various programs and various data. The operation unit 35 is a keyboard, a mouse, a touch panel, or the like, and receives input of instructions for each block of the image processing apparatus 10 from a user such as a doctor or a laboratory technician.

The display unit 36 is a display or the like, and displays various information to the user. The control unit 37 includes one or more processors such as a CPU (Central Processing Unit), and controls the processing of the image processing apparatus 10 in an integrated manner. The control unit 37 includes a GPU (Graphics Processing Unit), a DSP (Digital
Signal Processor), FPGA (Field-Programmable Gate Array), and the like. The control unit 37 includes an image acquisition unit 41, a learning model generation unit 42, a first cross-section acquisition unit 43, a second cross-section acquisition unit 44, and a display control unit 45 as functional configurations.

The image acquisition unit 41 acquires from the database 22 an input image to be processed (a three-dimensional image in which the second cross section is not set). Input images to be processed are images of a subject acquired by various modalities (imaging systems). Assume that the input image in this embodiment is a three-dimensional ultrasound image of the heart. The image acquisition unit 41 may directly acquire the input image from the modality. In this case, the image processing apparatus 10 may be implemented as a modality console. In this embodiment, an example in which the input image is a three-dimensional ultrasound image will be described, but the input image may be another type of image.

The learning model generation unit 42 (first generation unit, second generation unit) acquires learning data from the database 22 and builds a learning model. The learning case data constituting the learning data includes pixel value information of the three-dimensional image of each case and correct second cross-section information (hereinafter referred to as correct second cross-section information). The learning model generation unit 42 uses the learning data acquired from the database 22 to generate a learning model for acquiring a first cross section (first learning model) and a learning model for acquiring a second cross section (second learning model). We build two types of learning models.

The first cross-section acquiring unit 43 acquires the first cross-section group using the learning model for acquiring the first cross-section. The first group of cross-sections includes a plurality of first cross-sections orthogonal to each other. The first learning model for cross-section acquisition is a learning model for acquiring cross-sections from a three-dimensional image under the constraint that the cross-sections are orthogonal to each other. Details will be described later in the description of step S303.

The second cross-section acquisition unit 44 acquires a second cross-section group including second cross-sections corresponding to the first cross-sections of the first cross-section group using the second cross-section acquisition learning model. . The second cross-section acquisition unit 44 acquires a second cross-section using the first cross-section group acquired by the first cross-section acquisition unit 43 as an initial value. The learning model for acquiring the second cross-section corresponds to each first cross-section under the constraint that the positional relationship between the second cross-sections does not deviate too much from the orthogonal relationship, with each first cross-section as an initial value. It is a learning model that acquires the second cross section that Details will be described later in the description of step S304.

The display control unit 45 displays the result obtained by the second cross-section obtaining unit 44 within the image display area of the display unit 36 . The display control unit 45 displays the input image and the acquired (estimated) information of the second cross section in the image display area of the display unit 36 in a display form that allows the user to easily visually recognize them.

Each component of the image processing apparatus 10 functions according to a computer program. For example, the control unit 37 (CPU) reads and executes a computer program stored in the ROM 32 or the storage unit 34 using the RAM 33 as a work area, thereby implementing the functions of each component.

Note that some or all of the functions of the components of the image processing apparatus 10 may be realized by using a dedicated circuit. Moreover, some functions of the components of the control unit 37 may be realized by using a cloud computer. For example, the image processing apparatus 10 is communicably connected to a computing device located at a different location via the network 21, and data is transmitted and received between the image processing device 10 and the computing device. Element functionality may be implemented.

[Second Cross-Section Acquisition Processing]
The second cross-section acquisition process will be described with reference to FIG. FIG. 3 is a flowchart illustrating second cross-section acquisition processing according to the first embodiment.

(Step S301: Image Acquisition/Display)
In step S<b>301 , when the user instructs image acquisition via the operation unit 35 , the image acquisition unit 41 acquires the input image specified by the user from the database 22 and stores it in the RAM 33 . The display control unit 45 also displays the input image acquired from the database 22 within the image display area of the display unit 36 .

Any known method may be used to specify the input image. For example, the input image may be specified directly by the user. Also, the input image may be automatically selected based on a predetermined criterion such as the shooting date and time. For example, the image acquisition unit 41 can select, as an input image, an image captured on the most recent date and time among unobserved images.

(Step S302: Generate Learning Model)
In step S302, the learning model generation unit 42 acquires learning data from the database 22 and executes learning processing using the acquired learning data to construct a learning model as a learning device. The learning data desirably includes case data obtained by photographing a plurality of different patients, as cases for learning, in order to increase the robustness of the learning model. The learning data may also include each frame image in a series of moving image data obtained by imaging the same patient, and images obtained by imaging the same patient at different times.

The processing of the learning model generation unit 42 in this step will be described in detail with reference to FIG. FIG. 4 is a flowchart illustrating learning model acquisition processing.

(Step S3021: Acquisition of learning data image and correct second cross-sectional information)
In step S<b>3021 , the learning model generation unit 42 acquires learning data from the database 22 . The learning data includes a three-dimensional image of each learning case and correct second cross-sectional information.

There are three types of 3D images of each learning case included in the learning data: the number of voxels in each axis direction (width, height, depth), the physical size per voxel, and the array that stores the pixel value of each voxel. information.

The correct second slice information of each learning case included in the learning data is information defining each second slice of the learning case. As described with reference to FIG. 1, in each learning case of learning data, three types of cross-sections of A plane, B plane, and C plane are defined as second cross-sections. The three types of second cross sections are defined by four vectors having three parameters, for example, as shown in Equation (1) below. That is, the three types of second cross sections are defined by a total of 12 parameters.

In equation (1), C is a position vector representing the coordinates of the intersection point of the three second cross-sections, that is, the coordinates of a point in space where the three second cross-sections intersect each other. Also, N ^A to N ^C represent the normal vectors of the A plane, B plane, and C plane, respectively.

(Step S3022: Orthogonalization of correct second cross-sectional information)
In step S3022, the learning model generation unit 42 orthogonalizes (corrects) the correct second cross-sectional information for each learning case of the learning data acquired in step S3021. Specifically, the learning model generation unit 42 approximates and converts information defining each second cross section of each learning case (correct second cross section information) into parameters representing mutually orthogonal cross sections. If the cross-sections are orthogonal to each other, the information defining each second cross-section can be represented by a total of six parameters shown in equation (2) below.

In equation (2), C is a position vector representing the coordinates of the intersection of the three second cross sections, like C in equation (1). Also, R represents the rotation angle in each axial direction from the X-axis, Y-axis, and Z-axis to the normal vector of each corresponding second cross section. The correct second cross section acquired in S3021 is obtained by approximating the second cross sections (normal vectors thereof) so that they are orthogonal to each other. It can be expressed as a formula (2) as two cross sections.

The conversion processing for orthogonalizing the correct second cross-sectional information in this step will be specifically described. The learning model generation unit 42 can orthogonalize the correct second cross-sectional information by obtaining the parameter R in Equation (2).

Here, an example in which priority is given to the C-plane and the orientation of the C-plane does not change before and after conversion will be described. First, the learning model generation unit 42 calculates the outer product of the normal vector N ^C of the C plane and the normal vector N ^B of the B plane, and acquires the vector N ^A ′. Vector N ^A ' is orthogonal to both vector N ^C and vector N ^B by the well-known theorem orthogonality of outer products.

Next, the learning model generation unit 42 calculates the outer product of the vector N ^C and the vector N ^A ', and obtains the vector N ^B '. Vector N ^B ' is orthogonal to both vector N ^A ' and vector N ^C due to the orthogonality of the cross product. In this way, the learning model generation unit 42 acquires three mutually orthogonal vectors, vector N ^A ', vector N ^B ', and vector N ^C .

Note that the learning model generation unit 42 is not limited to giving priority to the C plane, and may give priority to the A plane or the B plane to obtain three mutually orthogonal vectors. When the ^A plane is prioritized, the learning model generation unit 42 first acquires ^NB ' that is orthogonal to ^NA and NC, and then acquires ^NC ' that is orthogonal to ^NA and ^NB '. Similarly, when priority is given to surface B, the learning model generator 42 first acquires N ^C ' orthogonal to N ^B and N ^A , and then acquires N ^A ' orthogonal to N ^B and N ^C '. .

The learning model generator 42 acquires the angles that the vector N ^C forms with the Z axis on the XZ plane and the YZ plane as r _x and r _y , respectively. The learning model generation unit 42 also acquires, as r _z , the angle formed by the vector N ^B ′ and the X axis on the plane defined by the vector N ^C . The learning model generation unit 42 can acquire six parameters representing three cross sections orthogonal to each other by using the coordinates of the intersection of the three second cross sections before conversion as the position vector C.

It should be noted that the parameters for orthogonalizing the correct second cross-sectional information should be able to express the postures formed in space by three mutually orthogonal vectors, vector N ^A ', vector N ^B ', and vector N ^C . Not limited to R. The orientation (orientation) of the second group of cross sections orthogonal to each other may be represented by a parameter other than R. For example, the orientation of the second group of cross sections that are orthogonal to each other can be expressed using a general method such as expression using a rotation axis and rotation angle, expression using a quaternion, or the like.

Further, the learning model generation unit 42 is not limited to orthogonalizing the correct second cross-section information with priority on the C plane, and may also orthogonalize other cross-sections with priority. Furthermore, instead of prioritizing a specific cross section, the learning model generation unit 42 may search for parameters that minimize the overall change in the direction of the axis before and after conversion processing. For example, the learning model generation unit 42 uses the appropriate six parameters in Equation (2) as initial values, and obtains the error of the angle formed with each axis vector of the second cross-section group. Then, the learning model generation unit 42 searches for a set of parameters that minimizes the total value of each axis angle error using a known parameter search method such as the gradient method.

(Step S3023: Construction of learning model for acquiring first cross section)
In step S3023, the learning model generation unit 42 (first generation unit) constructs a learning model using the learning data obtained in step S3022 in which the correct second cross-sectional information is orthogonalized. The learning model for acquiring the first cross section is input with a partial three-dimensional image (sample) obtained by extracting a part from the three-dimensional image to be processed, and the input sample is a sample that specifies the first cross section group. It is a learning device that outputs whether or not it is appropriate as

The learning model generation unit 42 builds a learning model using, for example, Extremely Randomized Tree (ERT), which is a known technique derived from the Random Forest algorithm. ERT is described, for example, in Geurts et al. , "Extremely Randomized Trees", Machine Learning, vol. 36, number 1, pp. 3-42, 2006. It is a method disclosed in.

A specific procedure for constructing a learning model is shown below. First, the learning model generation unit 42 uses the three-dimensional image of each learning case data included in the learning data and the information of the orthogonalized second cross-sectional parameter (hereinafter referred to as the correct position) to generate the learning A sample is generated by cutting out a part from the three-dimensional image of the case data. Specifically, the learning model generation unit 42 cuts out a region (partial three-dimensional image) of a predetermined size from the three-dimensional image according to the orientation indicated by the parameter R with the position of the position vector C as the center. A region cut out from the three-dimensional image is, for example, 50×50×50 voxels.

The size of the sample (partial three-dimensional image) to be cut out should be large enough to include the subject (in the example of FIG. 1, the mitral valve region) to be observed when acquiring the second cross section. desirable. The extracted sample is resized to, for example, 10×10×10. The learning model generation unit 42 obtains a vector in which pixel values of 10×10×10=1000 voxels are arranged in a line, and uses the obtained vector as a correct sample. On the other hand, a non-correct sample is a sample cut out at a position or orientation greatly deviated from the correct position.

The learning model generation unit 42 cuts out a plurality of (for example, 500 each) correct samples and non-correct samples from one piece of learning case data. While non-correct samples can be generated almost infinitely, only one true correct sample can be obtained from one set of learning case data.

In order to obtain a plurality of correct samples, the learning model generation unit 42 also generates samples obtained by adding some positional deviation, posture deviation, scale change, etc. to the correct position (second cross-sectional parameter). and For example, the allowable range of correct samples is ±5 pixels for position, ±5 degrees for angle, and 0.95 times to 1.05 times for scale, based on the correct position. That is, a correct sample is a sample extracted by randomly changing the position, posture, and scale of the correct position within the allowable range, and an incorrect sample is a sample extracted outside the allowable range.

The learning model generation unit 42 cuts out correct samples and non-correct samples from each of the learning data orthogonalized in S3022. The learning model generation unit 42 generates, for example, 500 correct samples and 500 non-correct samples for each case of learning data.

The learning model generation unit 42 constructs a discriminator (learning model) using the generated correct and non-correct samples. The learning model generation unit 42 can use ERT, for example, to construct a discriminator that discriminates whether an input unknown sample is a correct sample or a non-correct sample.

Note that construction of the classifier is not limited to the case of using ERT, and any method can be applied as long as it is a method of estimating parameters based on learning data. For example, the discriminator may be constructed by machine learning using a decision tree, neural network, deep learning, or SVM (Support Vector Machine).

In addition, although the learning model is constructed using the pixel values of the three-dimensional image of the case data for learning, it is also possible to construct the learning model using data obtained by applying predetermined image processing to the three-dimensional image. . For example, the learning model generation unit 42 may generate a differential image by differentiating pixel values in a predetermined direction, and cut out samples from the differential image. The learning model generation unit 42 may also cut out samples from an image to which image quality improvement processing such as noise reduction or sharpening has been applied.

(Step S3024: Construction of second cross-section acquisition learning model)
In step 2024, the learning model generation unit 42 (second generation unit) constructs a learning model using the learning data (before being orthogonalized) obtained in step S3021. As in step S3023, the learning model generation unit 42 can construct a learning model by ERT. A learning model for acquiring a reference cross section is a learning device that receives a cross-sectional image as an input and outputs whether or not the input cross-sectional image is appropriate as a reference cross-section, and is generated for each cross-section.

The difference between the first learning model for cross-section acquisition built in step S3023 and the second learning model for cross-section acquisition built in step S2024 will be described. The first cross-section acquisition learning model is a learning model for estimating a total of six parameters included in the position vector C and the parameter R in Equation (2).

On the other hand, in the second cross-section acquisition learning model, it is assumed that the position vector C and the parameter R in equation (2) are known from acquisition of the first cross-section. The second cross-section acquisition learning model is a learning model for obtaining a total of 12 parameters of the position vector ^C and the normal vectors N ^A to N C in Equation (1).

In constructing the learning model for acquiring the second cross section, the learning model generation unit 42 first adopts the value obtained by acquiring the first cross section as it is as the position vector C of the intersection coordinates. Next, the learning model generating unit 42, in order to estimate (obtain) three normal vectors of vector N ^A to vector N ^C , obtain second planes A, B, and C, respectively. We use three learning models for each cross-section of . The model for acquiring the first cross section is constructed using a three-dimensional image (partial three-dimensional image) as a sample, while the three learning models for acquiring the second cross section of planes A, B, and C are uses a two-dimensional image (cross-sectional image) as a sample.

The procedure for constructing the learning model for the three second cross sections will be described below. First, the learning model generation unit 42 uses the 3D image and the second cross-sectional parameter (correct position) of each piece of learning case data included in the learning data to convert the 3D image of the learning case data into plane A, Samples are generated by cutting out cross-sectional images of the B plane and the C plane.

Specifically, when generating a cross-sectional image sample of the plane ^A , the learning model generating unit 42 converts the cross-sectional image to a predetermined size (for example, 50 x 50 pixels). The learning model generation unit 42 resizes the extracted sample to a size of 10×10 pixels, for example. The learning model generating unit 42 acquires a vector in which pixel values of 10×10=100 pixels are arranged in a line as a correct sample.

The learning model generation unit 42 similarly acquires vectors of correct samples for the B plane and the C plane. In addition, the learning model generation unit 42 generates a predetermined number (for example, 500 each) of correct samples and non-correct samples from one piece of learning case data using the same procedure as in step S3023.

The classifier construction method is the same as in step S3023. However, while the learning model generating unit 42 generates one learning model in step S3023, in step S3024 it generates three learning models for obtaining the second cross-sections of planes A, B, and C. .

As in the case of step S3023, generation of the learning model is not limited to the case of using ERT, and any method can be applied as long as it is a method of estimating parameters based on learning data.

Also, the learning model generating unit 42 may construct the learning model using different methods in steps S3023 and S3024. For example, the learning model generation unit 42 may build a learning model based on ERT in step S3023, and build a learning model based on deep learning in step S3024.

In addition, as in step S3023, the learning model generation unit 42 applies arbitrary image processing such as differentiation processing for differentiating pixel values, noise reduction, and sharpening instead of pixel values of the three-dimensional image of the case data for learning. You may cut out a sample from the image. The learning model generation unit 42 may use images to which different image processing is applied in steps S3023 and S3024. For example, the learning model generation unit 42 may build the learning model using the differential image in step S3023, and build the learning model using the sharpened image in step S3024.

The learning model generation unit 42 executes the learning model generation processing in step S302 as described above. Note that the learning model generation processing in step S302 is processing independent of the second cross-section acquisition processing that the image processing apparatus 10 executes on the input image. Therefore, the process of step S302 can be performed in advance. The generated learning model is stored in a storage device such as the database 22 or the storage unit 34, for example.

When the learning model is generated in advance, the learning model generation unit 42 reads the learning model from the storage device and stores it in the RAM 33 in step S302. By generating a learning model in advance, the processing time of the second cross-section acquisition process for the input image can be shortened. Note that the learning model generation process may be executed by a device other than the image processing device 10 according to the procedure shown in step S302.

(Step S303: Acquisition of first group of cross sections)
In step S303, the first cross-section acquisition unit 43 acquires a first cross-section group using the input image acquired in step S301 and the first learning model for cross-section acquisition generated in step S3023. The first cross-section obtaining unit 43 calculates the positions and orientations of three first cross-sections that are orthogonal to each other. That is, the first cross-section acquiring unit 43 calculates a total of six parameters represented by Equation (2) for the input image.

The first cross-section acquiring unit 43 acquires a first group of cross-sections using an ERT, which is a known discriminator. When an unknown sample (partial three-dimensional image) cut out from the input image is input, the classifier determines whether the input sample is an appropriate sample (correct sample) for the first cross section group, or Inappropriate samples (non-correct samples) are identified. The first cross-section acquiring unit 43 acquires a first group of cross-sections based on samples determined to be correct by the discriminator.

The processing of the first cross-section acquisition unit 43 in this step will be described in detail with reference to FIG. FIG. 5 is a flowchart illustrating the first cross-section acquisition process.

In step S3031, the first cross-section acquisition unit 43 determines whether or not image processing has been applied when constructing the learning model. When image processing is applied during learning model construction, the process proceeds to S3032. If image processing was not applied when constructing the learning model, the process proceeds to S3033.

In step S3032, the first cross-section acquisition unit 43 applies the same image processing as the image processing applied when constructing the learning model to the input image. For example, when the learning model is constructed using a differential image, the first cross-section acquiring unit 43 also performs differential processing on the input image. An image that has been subjected to the same image processing as the image processing applied when building the learning model is referred to as a preprocessed image.

In step S3033, the first cross-section acquisition unit 43 cuts out a sample (partial three-dimensional image) by randomly varying parameters within a predetermined range from the input image or the preprocessed image. The predetermined range can be, for example, a range of ±100 voxels in each axis direction and ±45 degrees in each axis direction with respect to the center of the input image (preprocessed image). Specifically, the sample is cut out in the same manner as in the construction of the first learning model for cross-section acquisition (step S3023). Each sample extracted here represents a candidate (hypothesis) for the first cross-section group defined by the parameters (position and orientation) used for extraction. The first cross-section acquisition unit 43 inputs the extracted sample to the first discriminator for cross-section acquisition, and determines whether the input sample is a correct sample or an incorrect sample. A series of processes of sample extraction and identification are performed a predetermined number of times (eg, 10,000 times).

In step S3034, the first cross-section acquiring unit 43 acquires a first group of cross-sections from samples determined to be correct (candidates for the first group of cross-sections). If only one sample is determined to be correct, the first cross-section acquisition unit 43 outputs the position and orientation of the sample as parameters of the first cross-section group (hereinafter also referred to as first cross-section parameters). On the other hand, if there are a plurality of samples determined to be correct, the first cross-section acquiring unit 43 outputs the average position and average angle of the samples determined to be correct as parameters of the first cross-section group. The acquired first cross-sectional parameters are the position vector Cc and the parameter Rc.

It should be noted that if no sample determined to be correct is found, the first cross-section acquisition unit 43 may display a dialog box on the display unit 36 to notify the user that the first group of cross-sections has not been found. good.

In addition, when there are a plurality of samples determined to be correct, the first cross-section acquisition unit 43 can output the average position and the average angle in this step, but the present invention is not limited to this. For example, the first cross-section acquiring unit 43 applies known clustering such as the K nearest neighbor method to the sample group determined to be correct, and calculates the average position and average angle using the samples included in the cluster with a large number of samples. may

Further, the first cross-section acquiring unit 43 may select samples whose pixel value contrast is higher than the threshold value among the samples determined to be correct, and calculate the average position and average angle. In this way, the first cross-section acquisition unit 43 can select samples that satisfy a predetermined criterion among the samples determined to be correct, and calculate the average position and average angle.

Further, when no sample determined to be correct is found, the first cross-section acquiring unit 43 does not notify the user that the first cross-section was not found, but outputs an appropriate parameter in this step ( The process may be continued as the first section). Appropriate parameters can be, for example, the average position and average angle in learning data of correct samples used in building the first learning model for cross-section acquisition.

Alternatively, the first cross-section acquiring unit 43 may calculate, for example, the three parameters of the position, fix the parameters of the position that have already been calculated, and calculate the three parameters of the angle. Conversely, the first cross-section acquisition unit 43 may calculate the three parameters of the angle, fix the calculated parameters of the angle, and calculate the three parameters of the position.

Furthermore, the acquisition processing of the first slice group in this step is regarded as a problem of obtaining an orthogonal coordinate system that defines the position and orientation of the mitral valve to be observed. may be performed by a known method of estimating For example, a three-dimensional image template of the mitral valve may be created, and the position and orientation of the mitral valve may be estimated by template matching techniques.

(Step S304: Acquisition of second group of cross sections)
In step S304, the second cross-section obtaining unit 44 obtains each second cross-section. The second cross-section acquisition unit 44 acquires the input image acquired in step S301, the first cross-section parameters Cc and Rc calculated in step S303, and the second cross-section acquisition learning model generated in step S3024. , to obtain each second cross-section.

For example, the second cross-section acquiring unit 44 calculates (corrects) the orientation of each second cross-section represented by a total of 12 parameters shown in Equation (1) for the input image, thereby obtaining the second can be obtained. The second cross-section acquisition unit 44 acquires a second cross-section group using the ERT discriminator, as in step S303.

The processing of the second cross-section acquisition unit 44 in this step will be described in detail with reference to FIG. FIG. 6 is a flowchart illustrating the second cross-section acquisition process.

In S3041, the second cross-section acquisition unit 44 converts the angle parameter Rc among the first cross-section parameters into normal vectors ^NAc , ^NBc , and ^NCc of the respective first cross-sections. The second cross-section acquiring unit 44 acquires the normal vector of each first cross-section by rotating the X-axis, Y-axis, and Z-axis vectors in the input image based on each parameter of the angle parameter Rc. can be done.

In step S3042, the second cross-section acquiring unit 44 determines whether or not image processing has been applied when constructing the learning model. When image processing is applied during learning model construction, the process proceeds to S3043. If image processing was not applied when constructing the learning model, the process proceeds to S3044.

In step S3043, the second cross-section acquisition unit 44 applies the same image processing as the image processing applied when constructing the learning model to the input image. For example, if the learning model is constructed using a differential image, the second cross-section acquiring unit 44 also performs differential processing on the input image. An image that has been subjected to the same image processing as the image processing applied when building the learning model is referred to as a preprocessed image.

In step S3044, the second cross-section acquiring unit 44 cuts out samples of each second cross-section from the input image or the preprocessed image based on each first cross-section. Unlike step S3033, the sample cut out in this step is a two-dimensional cross-sectional image. The second cross-section acquiring unit 44 cuts out samples of each of the A plane, B plane, and C plane while the position vector Cc is fixed.

The second cross-section acquiring unit 44 cuts out samples randomly within a predetermined angle range centered on the normal vector N ^A c (that is, with the normal vector N ^A c of the first cross section as an initial value). , A-side samples. Similarly, the second cross-section acquiring unit 44 cuts out B-plane and C-plane samples. Each sample extracted here represents a candidate (hypothesis) for each second cross section defined by the parameter (orientation) used for extraction. Note that the sample of the A plane does not necessarily have to be cut out centered on N ^A c (that is, as an initial value) as long as it is cut out based on the position C and the normal vector N ^A c. good. For example, a cost function is defined that is calculated by a weighted sum of differences for position C and normal vector N ^A c and an image quality evaluation index (eg, luminance contrast) given a sample of a certain A plane. Then, samples of the A plane can be randomly cut out from the input image, and samples of the A plane whose cost function value is equal to or less than a certain value can be adopted.

The second cross-section acquiring unit 44 cuts out the samples of the A plane, the B plane, and the C plane so that the respective two-dimensional cross-section samples do not deviate too much from the mutually orthogonal relationship. For example, the second cross-section acquisition unit 44 sets the range of the predetermined angle for cutting out the sample to ±5 degrees around each normal vector, thereby cutting out the sample so as not to deviate too much from the orthogonal relationship. can be done.

Another specific example of a method of cutting out two-dimensional cross-sectional samples of each surface so as not to deviate too much from the relationship of being perpendicular to each other will be described. First, the second cross-section acquisition unit 44 cuts out samples of the A plane, and then rejects samples that deviate from the average value of the normal vector of each sample of the A plane by a predetermined angle (for example, ±10 degrees) or more. do. The second cross-section acquiring unit 44 may reject samples that deviate from the normal vector N ^A by a predetermined angle or more, instead of the average value of the normal vector of each sample of the A plane.

Next, the second cross-section acquiring unit 44 cuts out samples of the B plane, and calculates the angle between the average value of the normal vectors of the A plane and the normal vectors of the cut out samples of the B plane. The second cross-section acquiring unit 44 deviates by a predetermined angle (eg ±10 degrees) from the orthogonality (90 degrees) relationship from the average value of the normal vectors of the cut-out B plane and the A plane. If so, reject the B-side sample.

Similarly, the second cross-section acquisition unit 44 cuts out a sample of the C plane, the average value of the normal vectors of the A plane and the average value of the normal vectors of the B plane, and the normal vector of the cut out sample of the C plane Calculate the angle between The second cross-section acquisition unit 44 determines that the normal vector of the cut-out C plane is orthogonal (90 degrees) to the average value of the normal vectors of the A plane or the average value of the normal vectors of the B plane, and is at a predetermined angle ( For example, ±10 degrees) or more, the C-plane sample is rejected.

In this way, the second cross-section acquisition unit 44 selects a set of samples in which the A plane, the B plane, and the C plane deviate from the orthogonal relationship with each other within a predetermined angle range as candidates for the second cross-section group. get. In the above example, the second cross-section acquisition unit 44 determines whether or not the B plane and the C plane deviate from the orthogonal relationship with the A plane as the reference, but other planes are determined with the B plane or the C plane as the reference. It may be determined whether or not the two planes of are deviated from the orthogonal relationship.

The second cross-section acquisition unit 44 inputs the set of samples acquired as candidates for the second cross-section group to the second cross-section acquisition discriminator. The input set of samples is determined as to whether it is a set of samples suitable for the second cross-section group (correct sample set) or a sample set unsuitable for the second cross-section group (incorrect sample set). For this purpose, the second cross-section acquisition unit 44 first inputs the samples of the A-plane, B-plane, and C-plane into the second cross-section identifiers for the A-plane, B-plane, and C-plane, respectively, Determine whether the sample is correct or non-correct. Then, the second cross-section acquisition unit 44 selects a set of samples (candidates for the second cross-section group) in which each sample of the A plane, B plane, and C plane is determined to be a correct sample by each classifier, Obtained as a correct sample set. As in step S3033, the process of S3044 (the process of determining whether a correct sample set or an incorrect sample set) is performed a predetermined number of times (for example, 10,000 times).

In step S3045, the second cross-section acquisition unit 44 acquires the second cross-section group to be output from the set of samples determined to be correct (candidates for the second cross-section group). If there is only one set of samples determined to be correct, the second cross-section acquiring unit 44 outputs parameters of the second cross-section represented by the set of samples (hereinafter also referred to as second cross-section parameters). On the other hand, if there are a plurality of sets of samples determined to be correct, the second cross-section acquiring unit 44 outputs the average normal vector of each cross-section sample in the plurality of correct sample sets as the respective second cross-section parameters. do.

Note that if no sample determined to be correct is found, the second cross-section acquisition unit 44 may display a dialog box on the display unit 36 to notify the user that the second cross-section was not found. .

Also, as in step S303, the second cross-section acquisition unit 44 may use the average normal vector of the samples included in the cluster with the large number of samples as a result of clustering. Alternatively, the second cross-section acquiring unit 44 may use a sample selected based on a criterion such as brightness contrast of the image. Furthermore, if no sample determined to be correct is found, the second cross-section acquiring unit 44 acquires the first cross-section (that is, the position vector Cc and the normal vectors ^NA c, ^{NB c} , and NC c). can be output as is.

(Step S305: Display of Second Cross-Section Acquisition Result)
In step S<b>305 , the display control unit 45 displays in the image display area of the display unit 36 cross-sectional images obtained by cutting out the input image in step S<b>304 at the respective second cross-sections. In addition to the cross-sectional image, the display control unit 45 may display an image obtained by converting a three-dimensional image, which is an input image, into a two-dimensional image using a known display method such as surface rendering or volume rendering.

As illustrated in FIG. 1, when displaying a cross-sectional image of a certain second cross-section, the display control unit 45 displays a straight line representing another second cross-section that intersects the second cross-section on the cross-sectional image. may be drawn superimposed on the . FIG. 1 shows an example in which straight lines representing plane A (cross section 111) and plane B (cross section 112) are drawn on an image of plane C (cross section 113).

In addition, the display control unit 45 may further display information on the degree of divergence, which indicates how much the second cross section represented by the straight line deviates from the angle orthogonal to the displayed cross section. good. For example, the display control unit 45 may display text information such as "10 degrees deviation from orthogonal" near the straight line representing the second cross section, and the degree of deviation is indicated by a display mode such as the color or thickness of the straight line. can be expressed. By displaying the degree of divergence of the second cross-sections, the user can easily observe how the second cross-sections maintain their orthogonality with each other in the second cross-section acquisition results. is possible.

Note that if the purpose is to analyze or measure a three-dimensional image, the process of step S305 can be omitted. In this case, the image processing apparatus 10 saves the acquired second cross-sectional information in the storage device, and ends the processing shown in FIG.

Note that, when a plurality of correct samples are obtained in step S303, the first cross-section acquisition unit 43 limits the parameters to one set of first cross-section group by averaging, but is not limited to this. . The first cross-section acquisition unit 43 may pass the parameters of the multiple sets of correct samples (first cross-section group) to the second cross-section acquisition unit 44 as they are. The second cross-section acquiring unit 44 executes the process of step S304 for the number of transferred first cross-section groups. The second cross-section acquiring unit 44 can acquire a set of second cross-sections by averaging a plurality of second cross-sections acquired based on each first cross-section group. By obtaining the second set of cross-sections based on multiple sets of the first set of cross-sections, the risk of falling into inappropriate local minima is reduced.

In the first embodiment, the second cross-section group is acquired so as not to deviate too much from the angular relationship that the second cross-sections are orthogonal to each other. That is, the image processing apparatus 10 restricts the search range of the solution so that the angular relationship between the second cross sections deviates from the orthogonal relationship within a predetermined angular range. As a result, the image processing apparatus 10 can acquire the second cross-section group used for observation or image processing of the three-dimensional image at high speed and with higher accuracy.

(Modification 1)
The first embodiment shows an example of acquiring a second group of cross-sections in which the second cross-sections are in an approximately orthogonal angular relationship. On the other hand, in Modification 1, the angular relationship between the acquired second cross sections does not have to be an orthogonal relationship. In Modification 1, the angular relationship between the second cross sections can be a predetermined angular relationship set in advance. In the following, among the processing of each step of the second cross-section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3, processing that differs from the first embodiment will be described.

As an example in which the second slices have an angular relationship other than orthogonal, three-dimensional images of three-dimensional transsternal echocardiography, four-chamber image (A4C), two-chamber image (A2C), and three-chamber image (A3C), are obtained. An example is taking a second cross-section of a species. The four-chamber view (A4C) and the two-chamber view (A2C) are approximately orthogonal. However, the three-chamber view (A3C) is a cross-section rotated approximately 50 degrees with respect to the four-chamber view (A4C), and the angular relationship with the four-chamber view (A4C) is not orthogonal.

In Modified Example 1, in the orthogonalization of the correct second cross-section information in step S3022, the cross-section defining the three-chamber view (A3C) is not orthogonal to the other two cross-sections, but to the cross-section of the four-chamber view (A4C). are aligned so as to maintain an angle of 50 degrees. The section of the three-chamber image (A3C) can be uniquely represented by the six parameters shown in Equation (2) even if the angular relationship with the section of the four-chamber image (A4C) is not orthogonal.

In steps S3023 and S303, when samples are generated by randomly varying the parameters, the section of the three-chamber image (A3C) deviates from the section of the four-chamber image (A4C) by a predetermined angle or more from 50 degrees. Samples are rejected from both correct and non-correct samples.

In step S304, when cutting out a sample of each cross section, the sample of the two-chamber view (A2C) that deviates from the cross section of the four-chamber view (A4C) by a predetermined angle or more from the cross section of the four-chamber view (A4C) is selected from the candidates for the second cross section. reject. In addition, the sample of the three-chamber view (A3C), which deviates from the cross-section of the four-chamber view (A4C) by a predetermined angle or more from 50 degrees, is rejected from the candidates for the second cross-section.

In Modified Example 1, even if the second cross sections are not in a substantially orthogonal angular relationship, the image processing device 10 acquires the second cross section group used for observation or image processing of the three-dimensional image at high speed and with higher accuracy. can do.

(Modification 2)
The first embodiment shows an example in which the image processing apparatus 10 automatically acquires the first group of cross-sections and the second group of cross-sections. On the other hand, in Modification 2, the image processing apparatus 10 receives an input of a first group of cross-sections from a user such as a doctor, and acquires a second group of cross-sections based on the input first group of cross-sections. In the following, among the processing of each step of the second cross-section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3, processing that differs from the first embodiment will be described.

In Modified Example 2, the first cross-section group is manually input by the user. In the acquisition process of the first cross-section group in step S303, the display control unit 45 displays a user interface (UI) for manually inputting the first cross-section on the display unit . The user inputs the first cross section via the operation unit 35 to the UI for inputting the first cross section.

Any UI for inputting the first cross-section should be able to specify three mutually orthogonal first cross-sections. For example, the display control unit 45 arranges and displays cross-sectional images obtained by cutting a three-dimensional image along three planes, the YZ plane, the XZ plane, and the XY plane, in tiles. The display control unit 45 displays a cross that can be moved or rotated by mouse operation on each cross-sectional image.

For example, in FIG. 1, the cross displayed on the YZ plane represents the line of intersection where the B plane and C plane cross the YZ plane. Similarly, the cross displayed on the XZ plane represents the line of intersection of the A plane and the C plane with the XZ plane, and the cross displayed on the XY plane represents the intersection of the A plane and the B plane with the XY plane. Represents intersecting intersection lines.

The user can input the first cross section by moving or rotating the cross with the mouse in each cross section image. When the position and angle of the cross in one of the cross-sectional images are changed, the display control unit 45 reflects the change results in the other two cross-sectional images. The display control unit 45 accepts the input of the first cross section, so that the user can intuitively and manually input the first cross section group while maintaining the orthogonality constraint.

(Modification 3)
In the first embodiment, the learning model for acquiring the second cross section is constructed in step S3024, and when acquiring the second group of cross sections in step S304, the second cross section acquiring unit 44 acquires the two-dimensional cross section from the input image. Cut out the image and use it. On the other hand, in Modification 3, the second cross-section acquiring unit 44 may cut out and use a three-dimensional image (partial three-dimensional image). In the following, among the processing of each step of the second cross-section acquisition processing in FIG. 3 and the learning model generation processing in FIG. 3, processing that differs from the first embodiment will be described.

In steps S3024 and S304, the sample cut out by the second cross-section acquisition unit 44 is not a two-dimensional image (eg, 10×10 pixels) but a three-dimensional image (eg, 10×10×10 pixels). Further, in step S304, the process of determining that the planes A, B, and C are approximately orthogonal to each other is a process of determining by calculating the angle between the normal vectors used when extracting each three-dimensional image sample. becomes. That is, the second cross-section acquiring unit 44 rejects samples in which the angles of the normal vectors deviate from each other by a predetermined angle or more from the relationship of being orthogonal to each other.

In Modification 3, even when the second group of cross-sections is acquired using samples of partial three-dimensional images, the image processing apparatus 10 acquires the second group of cross-sections in the same manner as when samples of two-dimensional images are used. It can be acquired at high speed and with higher accuracy.

<Second embodiment>
As in the first embodiment, the image processing apparatus 10 according to the second embodiment acquires (estimates) a predetermined second cross section from a three-dimensional image (input image) to be processed. The first embodiment is an embodiment in which after acquiring a first group of cross-sections, a second cross-section is acquired with the acquired first group of cross-sections as the initial position. In contrast, the second embodiment evaluates the quality of the first cross-section group, and if a predetermined condition is satisfied, the acquired first cross-section is used as is without executing the second cross-section acquisition process. This is an embodiment in which a cross section is output as a second cross section.

[Device configuration]
The configuration and processing of the image processing apparatus of this embodiment will be described with reference to FIG. FIG. 7 is a block diagram illustrating the configuration of an image processing system according to the second embodiment. The same reference numerals as in FIG. 2 are assigned to the same configuration as in the first embodiment, and the description thereof is omitted. Differences from the first embodiment will be described below.

The image processing apparatus 10 includes a determination unit 51 in addition to the configuration shown in FIG. The determination unit 51 determines whether or not the first cross-section group acquired by the first cross-section acquisition unit 43 is appropriate as the second cross-section group. Details of the determination method will be described later in the description of step S804 in FIG.

If the acquired first cross-section group is determined to be appropriate as the second cross-section group, the second cross-section acquisition process is not executed. The display control unit 45 regards the first group of cross-sections as the second group of cross-sections, and displays them in the image display area of the display unit 36 as in the first embodiment.

[Second Cross-Section Acquisition Processing]
The second cross-section acquisition process will be described with reference to FIG. FIG. 8 is a flowchart illustrating second cross-section acquisition processing according to the second embodiment. Since each process of steps S801 to S803 and S805 in FIG. 8 is the same as the process of S301 to S303 and S304 in FIG. 3, description thereof is omitted. Differences from the first embodiment will be described below.

(Step S804: Determine whether or not to execute second cross-section acquisition)
In step S804, the determination unit 51 evaluates the first cross-section group calculated in step S803, and determines whether or not to execute the second cross-section acquisition. The determination unit 51 determines to execute the acquisition of the second cross section when the first group of cross sections acquired in step S803 satisfies the predetermined condition, and when the first group of cross sections does not satisfy the predetermined condition, , it can be determined not to perform the second cross-section acquisition. The first group of cross-sections satisfies the predetermined condition when the first group of cross-sections is appropriate as the second group of cross-sections. The image processing apparatus 10 advances the process to step S805 when executing the second cross-section acquisition, and advances the process to step S806 when not executing the second cross-section acquisition.

A method of determining whether or not to acquire the reference cross section, that is, whether or not the first group of cross sections satisfies a predetermined condition will be described. The determination unit 51 uses a known statistical process, Principal Component Analysis (PCA), to obtain a model ( subspace). The model is constructed by generating cross-sectional images obtained by cutting a three-dimensional image with correct second cross-sectional information for all learning case data, and performing principal component analysis of the generated cross-sectional images.

The determination unit 51 re-expresses the first cross section obtained in step S803 using the corresponding model. For example, the first cross-section of the A plane is reproduced using the model of the A plane. The determination unit 51 calculates an error (reconstruction error) between the first cross-sectional image and the reconstructed image. If the first cross-section is close to the average trend of the corresponding cross-section (eg, plane A), the model will have a small reconstruction error. Conversely, if the first cross-section deviates far from the average trend of the corresponding cross-sections, the reconstruction error will be large. By calculating the reconstruction error, the determination unit 51 can quantify "how likely the first cross section is to be the corresponding cross section".

If the reconstruction error is smaller than a predetermined threshold for each of the three slices, it is determined that "the acquisition of the second slice is not performed". Conversely, if the reconstruction error is equal to or greater than the predetermined value, it is determined that 'second cross-section acquisition is to be performed'. In this case, the first slice group satisfies a predetermined condition when the reconstruction error is smaller than a predetermined threshold. The predetermined threshold value for evaluating the reconstruction error can be, for example, the average value+standard deviation value when similar reconstruction errors are calculated for the learning data.

Note that the method of constructing a model that acquires the statistical tendency of pixel values of cross-sectional images is not limited to the method using PCA. The determination unit 51 can use any model as long as it is a model that quantifies the “probability that the input image is an image of a predetermined type”. For example, the determination unit 51 may construct and use a model for determining the probability that an input image belongs to a predetermined class by deep learning. In this case, the first cross-section group satisfies the predetermined condition when the probability that the first cross-section group belongs to the predetermined class is greater than the threshold.

In addition, the determination unit 51 may use a simple method that is not based on a model, such as determining that “the second cross-section acquisition is not performed” when the brightness contrast of the cross-section image is equal to or greater than a predetermined threshold. The predetermined threshold can be, for example, the average value of luminance contrasts in cross-sectional images of learning data. In this case, the first cross-section group satisfies the predetermined condition when the brightness contrast of the first cross-section group is equal to or greater than a predetermined threshold.

Also, a model for determining whether or not to execute the second cross-section acquisition can be constructed in advance. The model may be constructed by a device other than the image processing device 10 . The constructed model is stored in the database 22 or the storage unit 34. FIG. In step S804, the determination unit 51 reads the constructed model into the RAM 33 and determines whether or not to execute the second cross-section acquisition.

Also, in the determination in step S804, an example using the pixel values of the cross-sectional image has been described, but the determination unit 51 may use data obtained by applying predetermined image processing to the cross-sectional image for determination. For example, the determination unit 51 may extract a sample from a differential image obtained by differentiating pixel values in a predetermined direction, or from an image subjected to image quality improvement processing such as noise reduction and sharpening.

By applying predetermined image processing to the cross-sectional image, the determination unit 51 can perform determination with higher accuracy than when determination is performed using pixel values. Further, the determination unit 51 is not limited to the example using only the cross-sectional image, and may perform similar determination processing, that is, determination processing based on probability quantification, on a three-dimensional image including regions before and after the cross section.

(Step S806: Display of Second Cross-Section Acquisition Result)
In step S806, the display control unit 45 displays cross-sectional images obtained by cutting out the input image in the second cross-sectional group obtained in step S805 in the image display area. When acquisition of the second cross section is performed (when step S805 is performed), the display control unit 45 performs display using the acquisition result of the second cross section, as in the first embodiment.

If it is determined in step S804 that "the acquisition of the second cross section is not executed", the display control unit 45 regards the first cross section, which is a cross section perpendicular to each other, as the second cross section and displays it. The display processing is the same as step S305 of the first embodiment.

When displaying the first cross-section group acquired in step S803 as the second cross-section, the display control unit 45 may notify the user that the second cross-section acquisition process has not been executed. For example, the display control unit 45 may display text information to the effect that the second cross-section acquisition process is not being executed on the display unit. In addition, the display control unit 45 displays a mathematical symbol representing a right angle near the intersection of the straight lines representing cross sections on the display area, thereby displaying the first cross sections orthogonal to each other as the second cross section. You may indicate that

According to the above-described second embodiment, when it is determined that the first group of cross-sections acquired in step S803 can be used as the second cross-section, execution of acquisition of the second cross-section is omitted. , the image processing apparatus 10 can shorten the processing time.

<Third Embodiment>
The image processing apparatus 10 according to the third embodiment acquires (estimates) two or more predetermined second cross-sections from the three-dimensional image, as in the first and second embodiments. The second embodiment is an embodiment in which it is determined whether or not to execute the second cross-section calculation for the acquired first cross-section group. In contrast, the third embodiment is an embodiment in which it is determined whether or not to execute the second cross-section acquisition for each cross-section. For example, the image processing apparatus 10 according to the third embodiment uses the first cross section as the second cross section without executing the second cross section acquisition for the A plane, but uses the second cross section for the B plane and the C plane. It is possible to determine whether to acquire the cross section of

The configuration of the image processing apparatus 10 according to the third embodiment is the same as the configuration of the second embodiment shown in FIG. However, the processing by the determination unit 51 is different from that of the second embodiment in that it is determined whether or not to execute the second cross-section acquisition for each cross-section.

The second cross-section acquisition process according to the third embodiment is the same as the flowchart of the second embodiment shown in FIG. However, the processing of steps S804 and S805 is different from the processing in the second embodiment. Differences will be described below.

(Step S804: Determine whether or not to execute second cross-section acquisition)
In step S804, the determination unit 51 evaluates the first group of cross-sections calculated in step S803. The determination unit 51 determines whether or not to execute the second cross-section acquisition for each cross-section of the A-plane, B-plane, and C-plane.

If it is determined that “the acquisition of the second cross section is not performed” for any cross section, the image processing apparatus 10 advances the process to step S806. If it is determined to "perform acquisition of the second cross section" for any cross section, the image processing apparatus 10 advances the process to step S805.

A determination method for each cross section will be described. The determination unit 51 calculates the reconstruction error for each cross section in the same procedure as in step S804. The determination unit 51 determines that “the second cross-section acquisition is not performed” for cross-sections whose reconstruction error is less than a predetermined threshold. Conversely, the determination unit 51 determines to “perform second cross-section acquisition” for cross-sections whose reconstruction error is equal to or greater than the predetermined threshold value, as in the first embodiment.

The predetermined threshold value for evaluating the reconstruction error can be, for example, the average value+standard deviation value when similar reconstruction errors are calculated for the learning data. The information of the determination result in step S804 is used for the processing in step S805.

(Step S805: Acquisition of second cross-section group)
In step S805, the second cross-section acquisition unit 44 "performs acquisition of the second cross-section".
A second cross section is acquired for the cross section determined as The second cross-section acquisition unit 44 uses the input image, the first cross-section parameters Cc and Rc, the second learning model for cross-section acquisition, and the determination information calculated in step S804 to obtain each second Get a cross section of

The processing of step S804 is partially common to step S304 of the first embodiment. Hereinafter, the processing of step S804 will be described while clarifying common points and differences from step S304.

First, the second cross-section acquisition unit 44 converts the angle parameter Rc among the first cross-section parameters into normal vectors ^NAc , ^NBc , and ^NCc of the respective first cross-sections. . The first process in step S804 is the same as the first process in step S304.

Second, the second cross-section acquisition unit 44 cuts out samples of each second cross-section from the input image. If there is a section among the three planes A, B, and C for which it is determined that "the second section acquisition is not to be performed" in step S804, the second section acquisition unit 44 performs the second section acquisition. A normal vector corresponding to a cross-section for which cross-section acquisition is not performed is not changed. Normal vectors corresponding to other cross sections are calculated so that the normal vectors of each cross section maintain an angular relationship within a predetermined angle range from a mutually orthogonal relationship, as in the second process in step S304. . The second cross-section acquiring unit 44 acquires a sample extracted based on the normal vector of each cross-section as a second cross-section candidate.

Third, the second cross-section acquiring unit 44 inputs the sample acquired as the candidate for the second cross-section in the preceding process into the second classifier for acquiring cross-section, and determines whether the input sample is a correct sample or an incorrect sample. Determine if it is a sample. The third process in step S804 is the same as the third process in step S304.

Similar to step S304, the second process and the third process are executed a predetermined number of times (eg, 10,000 times). If there are a plurality of samples determined to be correct, the second cross-section acquiring unit 44 outputs the average position and the average normal vector of each cross-section as the output of this step. It should be noted that if no sample determined to be correct is found, the second cross-section acquisition unit 44 displays a dialog box on the display unit 36 to notify the user that the second cross-section was not found.

It should be noted that the second cross section is fixed to the first cross section acquired in step S803 for the cross section for which the reconstruction error is smaller than the predetermined threshold and it is determined not to perform the acquisition of the second cross section. , but not limited to. Instead of using the first cross section as the second cross section as it is, the second cross section acquiring unit 44 can also give minute variations when cutting out the sample in step S805. That is, when extracting the sample in step S805, the second cross-section acquiring unit 44 selects a range (eg, ±3 degrees) smaller than the range (eg, ±10 degrees) that a cross section having a reconstruction error equal to or greater than a predetermined threshold can take. , the first cross-section may vary. The second cross-section acquiring unit 44 can more flexibly estimate the second cross-section by setting a different variation range for each cross-section.

Also, in steps S3024 and S805, the second cross-section acquisition unit 44 acquires a second cross-section using ERT, that is, by randomly cutting out a sample and determining correct or incorrect answers. is not limited to The second cross-section acquiring unit 44 may acquire the second cross-section by optimizing a predetermined evaluation function.

In the method of optimizing the predetermined evaluation function, in step S3024, the second cross-section acquisition unit 44 applies principal component analysis (PCA) to the correct sample to obtain cross-sections of the A plane, the B plane, and the C plane. build a model that captures the statistical trends of the image features of

In step S804, the second cross-section acquisition unit 44 uses two terms as costs: an error when the input cross-section image is re-expressed by each model, and a regularization term (angular difference from the first cross-section). Use an evaluation function. The second cross-section acquisition unit 44 increases the weight of the regularization term for the cross-section of the case for which it is determined that “the second cross-section acquisition is not performed” in step S804, so as not to deviate from the first cross-section as much as possible. to The second cross-section acquiring unit 44 searches for the parameter that minimizes the cost value by a known optimization method such as the gradient method or the Levenburg-Marquardt method.

According to the above-described third embodiment, it is possible to acquire the second group of cross-sections with higher accuracy because it is determined whether or not to acquire the second cross-section for each cross-section.

<Fourth Embodiment>
The image processing apparatus 10 according to the fourth embodiment acquires (estimates) two or more predetermined second cross-sections from the three-dimensional image, as in the first to third embodiments. In the first embodiment, when acquiring the second cross-sections, the second cross-section acquiring unit 44 does not deviate too much from the angular relationship in which the second cross-sections are substantially perpendicular to each other. The search range of the solution is restricted so that the deviation from the angular relationship is within a predetermined range.

On the other hand, in the fourth embodiment, the image processing apparatus 10 uses an evaluation function for acquiring the second group of cross-sections in order to prevent the second cross-sections from deviating too much from the angular relationship of being orthogonal to each other. to evaluate at least the degree of divergence from the orthogonal relationship between cross sections. That is, the second cross-section acquiring unit 44 uses an evaluation function that includes a term that quantifies the degree of deviation from the orthogonal relationship, and optimizes (minimizes or maximizes) the value of the evaluation function to obtain the first Get a first cross-section and a second cross-section.

The configuration of the image processing apparatus 10 according to the fourth embodiment is the same as the configuration of the first embodiment shown in FIG. Also, the second cross-section acquisition process according to the fourth embodiment is the same as the flow charts of the first embodiment shown in FIGS. 3 and 3 . However, part of the learning model construction processing shown in steps S3023 and S3024 and the cross-section acquisition processing shown in steps S303 and S304 are different from those of the first embodiment. Hereinafter, each process will be described while clarifying the differences from the first embodiment.

(Step S3023: Construction of learning model for acquiring first cross section)
In step S3023, the learning model generation unit 42 constructs a learning model using the learning data in which the second cross-sectional information is orthogonalized, which is obtained in step S3022. The learning model generation unit 42 uses principal component analysis (PCA), which is a known statistical process, to build a learning model that acquires the statistical tendency of the first cross-sectional image.

A specific procedure for constructing a learning model is shown below. First, as in step S3023 of the first embodiment, the learning model generation unit 42 uses the 3D image of the learning case data and the second cross-sectional parameter (correct position) orthogonalized from the 3D image to 1 Generate a sample that cuts out the part. As in the first embodiment, the learning model generation unit 42 regards samples with slight positional deviation, posture deviation, scale change, etc. from the correct position as correct samples, and padded the learning data.

However, unlike step S3023 of the first embodiment, the learning model generation unit 42 does not generate non-correct samples. The learning model generation unit 42 generates learning samples by extracting correct samples from all learning case data. The learning model generation unit 42 constructs a learning model that expresses the statistical tendency of the first cross-sectional image by performing principal component analysis on the generated learning sample.

(Step S3024: Construction of second cross-section acquisition learning model)
In step S3024, the learning model generation unit 42 constructs a learning model using the learning data (before being orthogonalized) obtained in step S3021. The learning model generation unit 42 constructs a learning model by principal component analysis (PCA) as in step S3023 of the fourth embodiment.

The process of cutting out the sample image from the input image is the same as step S3024 in the first embodiment. However, the learning model generation unit 42 does not generate non-correct samples, and builds a learning model using only correct samples, as in step S3023 of the fourth embodiment. The learning model generation unit 42 constructs a learning model that expresses the statistical tendency of the second cross-sections of the A plane, B plane, and C plane by performing principal component analysis on the generated learning samples.

(Step S303: Calculation of first group of cross sections)
In step S303, the first cross-section acquisition unit 43 acquires a first cross-section group using the input image acquired in step S301 and the first learning model for cross-section acquisition generated in step S3023. As in the first embodiment, the first cross-section acquiring unit 43 calculates a total of six parameters represented by Equation (2).

The first cross-section acquisition unit 43 acquires the first cross-section by searching for parameters (six parameters represented by Equation (2)) that minimize the cost function using the generated learning model. The cost function is the difference (reconstruction error) when samples extracted from the input image are re-expressed using the model. In other words, the cost function is an index that quantifies "how likely the sample extracted from the input image is the first section based on the statistical tendency of the learning model". The processing procedure for acquiring the first cross section is shown below.

First, the first cross-section acquisition unit 43 applies the same image processing to the input image when image processing was applied when constructing the learning model. For example, when the learning model is constructed using a differential image, the first cross-section acquiring unit 43 also performs differential processing on the input image.

Next, the first cross-section acquisition unit 43 determines initial values (hereinafter referred to as initial parameters) of the six parameters shown in Equation (2). For example, the initial value of the position vector C is assumed to be the center of the image, and the initial value of the parameter R indicating the orientation is assumed to be parallel to each axis of the input image. The first cross-section acquisition unit 43 searches for the parameter that minimizes the cost value using a known optimization method such as the gradient method or the Levenburg-Marquardt method. The parameters of the searched first cross section are the position vector Cc and the parameter Rc.

Note that the first cross-section acquiring unit 43 may, for example, calculate the three parameters of the position, fix the parameters of the calculated position, and calculate the three parameters of the angle (orientation). Conversely, the first cross-section acquisition unit 43 may calculate the three parameters of the angle, fix the calculated parameters of the angle, and calculate the three parameters of the position.

(Step S304: Acquisition of second group of cross sections)
In step S304, the second cross-section obtaining unit 44 obtains each second cross-section. The second cross-section acquisition unit 44 acquires the input image acquired in step S301, the first cross-section parameters Cc and Rc calculated in step S303, and the second cross-section acquisition learning model generated in step S3024. , to obtain each second cross-section.

For example, the second cross-section acquisition unit 44 minimizes the cost function and calculates (corrects) the orientation of each second cross-section represented by a total of 12 parameters shown in Equation (1) in the first embodiment. Thus, a second cross section can be obtained. The cost function includes two types of terms: the reconstruction error described in step S303 of the fourth embodiment, and a constraint term that gives a penalty when the positional relationship between cross sections deviates from the orthogonal relationship.

The reconstruction error is the same as described in step S303 of the fourth embodiment. The constraint term can be the absolute value of the inner product of the normal vectors representing the cross section. When the cross sections are orthogonal to each other, the inner product of the normal vectors is 0, so the degree of deviation from the orthogonal relationship can be quantified by taking the absolute value of the inner product. If there are three cross-sections A, B, and C, the constraint terms are the A-plane normal vector and B-plane normal vector, the B-plane normal vector and C-plane normal vector, and the C-plane normal vector and It is represented by the sum of the inner products of three combinations of normal vectors of the A plane. The processing procedure is shown below.

First, the second cross-section obtaining unit 44 converts the angle parameter Rc among the first cross-section parameters into normal vectors ^NAc , ^NBc , and ^NCc of the respective cross-sections. This process is the same as the process of step S3041 of the first embodiment.

Second, the second cross-section acquisition unit 44 applies the same image processing to the input image when image processing was applied when the learning model was constructed. For example, if the learning model is constructed using a differential image, the second cross-section acquiring unit 44 also performs differential processing on the input image.

Next, the second cross-section acquiring unit 44 determines initial parameters that serve as starting points for acquiring the A plane, B plane, and C plane. The second cross-section acquiring unit 44 fixes the position vector Cc, sets the normal vector N ^A c to the initial parameter of the A surface, sets the normal vector N ^B c to the initial parameter of the B surface, sets the normal vector N ^C c to C be the initial parameters of the surface.

Third, the second cross-section acquiring unit 44 searches for the parameters (normal vectors) of the A plane, B plane, and C plane by minimizing the cost function. The cost function is defined as (A-plane reconstruction error+B-plane reconstruction error+C-plane reconstruction error)+constraint term. In the cost function, the reconstruction error term and the constraint term are adjusted by preset weighting parameters. The second cross-section acquisition unit 44 searches for parameters of each of the A plane, B plane, and C plane by known optimization techniques such as the gradient method and the Levenburg-Marquardt method.

In step S304, a constraint term is introduced in which the cost increases as the cross sections deviate from the orthogonal relationship. Defined according to relationship. For example, if the A and B planes should satisfy an 80 degree angular relationship, the value of the constraint term can be the absolute value of the angular difference between the A and B planes minus 80 degrees. .

As described above, the image processing apparatus 10 can calculate the parameters of the A plane, the B plane, and the C plane, and acquire the second group of cross sections.

Both the first cross-section group and the second cross-section group were acquired using a learning model based on principal component analysis (PCA), but either one can be obtained using a learning model generated by a method based on ERT. good. For example, a second group of cross-sections may be acquired using a PCA-based learning model and a first group of cross-sections may be acquired using an ERT-based learning model. When the existence range of the first cross section is narrowed down to some extent, it is possible to acquire the first cross section at a higher speed by using ERT, which can build a learning model with a smaller number of cases than PCA. .

According to the above-described fourth embodiment, in order to prevent the second cross-sections from deviating too much from the orthogonal relationship, the evaluation function for calculating the second cross-section group expresses the degree of deviation from the orthogonal relationship. Contains constraint terms to evaluate. In acquiring the second cross-sections, the cost (constraint term) increases as the cross-sections deviate from the orthogonality, so the image processing apparatus 10 balances the statistical probability obtained from the learning model and the orthogonality between the cross-sections. A second cross-section can be obtained.

Each of the above-described embodiments merely exemplifies the configuration of the present invention. The present invention is not limited to the specific forms described above, and various combinations or modifications are possible within the scope of its technical ideas.

<Other embodiments>
In addition, the technology disclosed can be embodied as, for example, a system, device, method, program, recording medium (storage medium), or the like. Specifically, the technology disclosed herein may be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to a device composed of a single device. may be applied to

Moreover, the object of the present invention is achieved as follows. That is, a recording medium recording software program code (computer program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus. Such a recording medium is computer readable. A computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium implements the functions of the above embodiments. A recording medium recording the program code constitutes the present invention.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

10 image processing device 32 ROM
33 RAM
37 control unit 41 image acquisition unit 43 first cross-section acquisition unit 44 second cross-section acquisition unit

Claims (27)

an image acquisition unit that acquires an input image that is a three-dimensional image of an object;
Based on a first cross-section group consisting of a plurality of first cross-sections set to observe the object in the input image and having a predetermined angular relationship with each other, and a constraint condition based on the predetermined angular relationship , a cross-section acquisition unit that acquires a second cross-section group consisting of a plurality of second cross-sections;
An image processing device comprising: 2. The image processing apparatus according to claim 1, wherein said second cross section is a cross section obtained by varying said first cross section. The cross-section acquiring unit
a first learning model that receives as input a partial three-dimensional image obtained by cutting out a portion of the input image, and outputs whether or not the partial three-dimensional image is appropriate as a three-dimensional image for specifying the first cross-section group; and acquiring the first cross-section group by inputting the partial three-dimensional image cut out from the input image into the first learning model,
A second learning model for each cross section that receives a cross-sectional image as an input and outputs whether or not the cross-sectional image is appropriate as the second cross-section, and is extracted from the input image based on the first cross-sectional group. Obtaining the second group of cross-sections by inputting the cross-sectional images obtained into the second learning model,
3. The image processing apparatus according to claim 1, wherein: When a plurality of first cross-section group candidates determined to be appropriate by the first learning model are obtained, the cross-section acquiring unit averages the plurality of first cross-section group candidates. 4. The image processing apparatus according to claim 3, wherein the first cross-section group is acquired. When the second learning model for each cross section obtains a plurality of candidates for the second cross section determined to be appropriate, the cross section acquiring unit averages the plurality of candidates for the second cross section. 5. The image processing apparatus according to claim 3, wherein the second cross section corresponding to each cross section is acquired by: The cross-section acquiring unit inputs cross-sections cut out within a range of a predetermined angle around a normal vector of each first cross-section of the first cross-section group to the second learning model. The image processing apparatus according to any one of claims 3 to 5. The cross-section acquiring unit
A first learning model that receives as an input a partial three-dimensional image obtained by cutting out a part of the input image and outputs whether or not the partial three-dimensional image is appropriate as a three-dimensional image for specifying the plurality of first cross sections. and obtaining the first cross-section group by inputting a partial three-dimensional image cut out from the input image into the first learning model,
a second learning model that receives as an input a partial three-dimensional image obtained by cutting out a part of the input image and outputs whether or not the partial three-dimensional image is appropriate as a three-dimensional image for specifying the second cross section; and obtaining the second cross-section group by inputting a partial three-dimensional image obtained by cutting out a part of the input image based on the first cross-section group into the second learning model.
2. The image processing apparatus according to claim 1, wherein: 8. The method according to any one of claims 3 to 7, wherein the first learning model and the second learning model are generated using an image to which at least one of image processing of differential processing, noise reduction, and sharpening is applied. The image processing device according to any one of items 1 and 2. The cross-section acquiring unit
obtaining the first cross-section group by applying the image processing applied to the image used to generate the first learning model to the input image;
9. The image processing according to claim 8, wherein the image processing applied to the image used to generate the second learning model is applied to the input image to acquire the second cross-section group. Device. The cross-section acquiring unit uses the plurality of first cross-sections as initial values of the respective second cross-sections, and obtains a plurality of 10. The image processing apparatus according to any one of claims 1 to 9, wherein a second cross-section group consisting of the second cross-sections of . The cross-section acquisition unit acquires the second cross-sections so that the angular relationship of the second cross-sections included in the second cross-section group deviates from the predetermined angular relationship within a range of a predetermined angle. 11. The image processing apparatus according to any one of claims 1 to 10, wherein a group is acquired. The evaluation function used by the cross-section acquisition unit to acquire the second cross-section group is the degree of divergence between the angular relationship of the second cross-sections included in the second cross-section group and the predetermined angle relationship. 12. The image processing apparatus according to any one of claims 1 to 11, wherein the image processing apparatus evaluates A cross-sectional image obtained by cutting the input image with the second cross section included in the second cross section group, and an angular relationship between the second cross section included in the second cross section group and the predetermined angular relationship 13. The image processing apparatus according to any one of claims 1 to 12, further comprising a display control unit that displays information indicating the degree of divergence on a display unit. A determination unit that determines whether the first group of cross-sections satisfies a predetermined condition for being used as the second group of cross-sections,
14. The cross-section acquiring unit, if the first group of cross-sections satisfies the predetermined condition, sets the first group of cross-sections as the second group of cross-sections. 2. The image processing device according to item 1. A determination unit that determines whether each of the first cross-sections included in the first cross-section group satisfies a predetermined condition for being used as the second cross-section,
15. The image according to any one of claims 1 to 14, wherein the cross-section acquisition unit acquires the second cross-section group without changing the first cross-section that satisfies the predetermined condition. processing equipment. The cross-section acquiring unit, in the evaluation function used to acquire the second cross-section group, assigns the weight of the regularization term to the first cross-section that satisfies the predetermined condition as the weight of the regularization term that does not satisfy the predetermined condition. By setting a weight greater than the regularization term for the first cross section, the second cross section does not deviate from the first cross section that satisfies the predetermined condition.
16. The image processing apparatus according to claim 15, characterized by: 17. The image processing apparatus according to any one of claims 1 to 16, further comprising a display control unit that displays information on the second slice group acquired by the slice acquisition unit on a display unit. The display control unit displays one second cross section included in the second cross section group on the display unit, and displays a straight line representing another second cross section that intersects with the one second cross section. 18. The image processing apparatus according to claim 17, wherein the second cross section is superimposed on the first cross section. The display control unit displays one second cross section included in the second cross section group on the display unit, and another second cross section that intersects with the one second cross section is displayed on the first cross section group. 19. The method according to claim 17 or 18, wherein information on the degree of divergence, which indicates the degree of divergence from an angle perpendicular to the two cross sections, is displayed superimposed on the one second cross section. Image processing device. 20. The display control unit according to claim 19, wherein the display control unit displays the information on the degree of divergence in the form of character information or a display mode of a color or thickness of a straight line representing the other second cross section. Image processing device. 21. The image processing apparatus according to any one of claims 1 to 20, wherein the cross-section acquiring unit acquires images input by a user's operation as the first group of cross-sections. 22. The image acquisition unit according to any one of claims 1 to 21, wherein the image acquisition unit acquires, as the input image, an image designated by a user's operation or an image automatically selected based on a predetermined criterion. The image processing device according to . A three-dimensional image is used as an input, and a partial three-dimensional image obtained by cutting out a part of the three-dimensional image is a three-dimensional image specifying a first cross-sectional group including a plurality of first cross-sections having a predetermined angular relationship with each other. a first generation unit that generates a first learning model for acquiring a first cross section that outputs whether it is appropriate;
Generating a second learning model for acquiring a second cross section for each cross section, which outputs whether or not the cross section image is appropriate as a second cross section for observing the target object, using the cross section image as an input. a second generator;
An image processing device comprising: The first generating unit inputs the information of the three-dimensional image and the second cross section of the correct answer, and based on the information of the correct position corrected so that the second cross section of the correct answer has the predetermined angular relationship, 24. The image processing apparatus according to claim 23, wherein the first learning model is generated by generating correct samples and incorrect samples, and learning the correct samples and the incorrect samples. The second generating unit receives the three-dimensional image and second correct cross-sectional information as input, generates a correct sample and an incorrect sample of the two-dimensional image for each cross-section, and makes the correct sample and the incorrect sample learn. 25. The image processing apparatus according to claim 23 or 24, wherein the second learning model is generated for each cross section. the computer
Acquiring an input image that is a three-dimensional image of an object,
Based on a first cross-section group consisting of a plurality of first cross-sections set to observe the object in the input image and having a predetermined angular relationship with each other, and a constraint condition based on the predetermined angular relationship and obtaining a second cross-section group consisting of a plurality of second cross-sections. A program for causing a computer to execute the image processing method according to claim 26.

Images