KR102453028B1 - Method for providing three-dimensional animated video using respiratory motion tracking-based lung CT images and recording medium thereof - Google Patents

Abstract

The present invention relates to a method for providing a three-dimensional animation video using a lung CT image based on respiratory movement tracking, comprising the steps of: receiving spirometry curve data according to spirometry performed on a patient; Tomography) receiving maximal inspiration CT image data as images and maximal expiratory CT image data as CT images taken during peak expiration, CT using the spirometry curve data, the maximal inspiratory CT image data, and the maximal expiratory CT image data Reconstructing the image into a three-dimensional image and providing a three-dimensional animation video in a manner that synchronizes the three-dimensional image according to the patient's respiratory movement data detected and input in real time.
According to the present invention, it is possible to describe and predict the movement of internal organs during the actual patient's breathing exercise as a 3D CT video by using the change in the height of the patient's forearm skin according to the respiratory movement as a reference index of the respiratory movement.

Description

{Method for providing three-dimensional animated video using respiratory motion tracking-based lung CT images and recording medium thereof}

The present invention relates to a technology for providing a 3D animation video using a lung CT image.

Lung cancer is the most common cancer in the world and a fatal disease that kills more than 1.3 million people annually. In Korea, the death toll from lung cancer exceeded 12,000 in 2002, and the death rate from lung cancer was also the highest. Pulmonary nodules are commonly found on chest radiographs and have clinical forms such as inflammatory granulomas, benign tumors, and malignant tumors (lung cancer). Therefore, not only the simple detection of lung nodules, but also the determination of benign/malignant pulmonary nodules is very important for early diagnosis and rapid treatment of lung cancer.

Distinguishing benign/malignant lung nodules is difficult even for experienced experts, and even when using computed tomography (CT) images, it is known that the identification accuracy is about 2/3. In the actual medical field, for this reason, for the final determination of benign/malignant pulmonary nodules, it is highly dependent on direct verification of the living body.

On the other hand, since the patient continues to breathe as long as he is alive, the volume and shape of the lungs change according to the breathing state of the patient. Lung registration between images of the area around the lung tissue obtained from exhalation or inspiration, i.e., in different respiration states, is essential for clinical applications such as treatment planning, quantitative evaluation of obstructive pulmonary disease, and analysis of lung movement.

In particular, it is difficult to correct the deformation of the lungs due to respiration by conventional rigid body registration or affine registration, and for correction, a non-rigid body registration process must be performed. Rigid registration or affine registration is used to correct for translation mismatch or global expansion of lung tissue, but with sufficient degrees of freedom to compensate for local deformation of the lungs due to respiration. ) because it does not have

As an example of non-rigid registration, a method of registering each pixel by calculating a dense displacement vector from all pixels of a CT image has been proposed. However, this method has a problem in that, when the resolution of a CT image is high, the amount of data to be processed is too large, so the amount of calculation required for the registration process is too large and the execution time is long.

1 is a graph showing a lung capacity curve of a patient measured through a lung function test.

In the spirometry curve graph of FIG. 1, IC (Inspiratory capacity), FRC (Functional Residual Capacity), TLC (Total Lung Capacity), RV (Residual Volume), ERV (Expitory Reserve Volume), VC (Vital Capacity), etc. are displayed . Here, capacity means the sum of lung volumes.

TLC is the lung volume at full inhalation (RFC + IC = TLC).

FRC is the lung volume after the end of exhalation, and serves as a standard for measuring lung volume.

RV is the lung volume at maximum exhalation, IC is the amount of air inhaled from FRC to TLC, and ERV is the amount of air exhaled from FRC to RV.

VC is the maximum amount of air that can be inhaled in an RV, and is the maximum amount of air that can be exhaled in a TLC.

Spirometry measures the amount of airflow obstruction present and is usually performed after administration of a bronchodilator, a drug that opens the airways. At this time, two main indicators are diagnosed: forced expiratory flow per second (FEV1), which is the maximum volume of air that can be exhaled in the first second of one breath, and forced vital capacity (FVC), which is the maximum volume of air that can be exhaled with one big breath. is measured for Usually 75-80% of FVC is released in the first second, and when an FEV1/FVC ratio of 70% or less is measured in patients with symptoms of obstructive pulmonary disease (COPD), it is defined as a chronic obstructive pulmonary disease patient. Meanwhile, studies comparing expiratory computed tomography (expiratory CT) with lung function test results are being actively conducted for the diagnosis and severity of chronic obstructive pulmonary disease. In this study, CT at maximal inspiration and CT at maximal expiration were analyzed and lung volume changes were compared to estimate the severity of COPD lesions by lung area.

Republic of Korea Patent Registration 10-1697430

The present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a method for providing a three-dimensionally reconstructed CT image as an animation video based on the results of an expiratory CT and lung function test. have.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

The present invention for achieving the above object relates to a method for providing a 3D animation video using a lung CT image based on respiratory movement tracking, the step of receiving spirometry curve data according to spirometry performed on a patient, the maximum for the patient receiving CT image data of maximum inspiration, which is a computed tomography (CT) image taken during inspiration, and CT image data of maximum expiration, which is a CT image taken during maximum expiration, the spirometry curve data, the maximum inspiration CT image data, and the maximum expiration Reconstructing a CT image into a 3D image using CT image data and providing a 3D animation video in a manner that synchronizes the 3D image according to the patient's respiratory movement data detected and input in real time .

Segmentation for each major organ region from the maximum inspiratory CT image data and the maximum expiratory CT image data, reconstructing the segmented images into a three-dimensional mesh, meshes at maximal inspiration and maximal expiration for each major organ site Classifying the meshes of the city to generate a corresponding pair, mapping a vertex correlation between two meshes for each corresponding pair, and interpolating all vertices based on the mapped vertex correlation data to generate animation data may be further included.

The step of providing the three-dimensional animation video includes: confirming the position data at the time of maximum inspiration and the position data at the time of maximum exhalation in the respiratory movement data of the patient; Normalizing to, normalizing the patient's respiratory movement data sensed and input in real time, and selecting and outputting a 3D animation frame at a position corresponding to the normalized value.

According to the present invention, it is possible to describe and predict the movement of internal organs during the actual patient's breathing exercise as a 3D CT video by using the change in the height of the patient's forearm skin according to the respiratory movement as a reference index of the respiratory movement.

1 is a graph showing a lung capacity curve of a patient measured through a lung function test.
2 is a flowchart illustrating a method for providing a 3D animation video using a lung CT image based on respiratory movement tracking according to an embodiment of the present invention.
3 is a flowchart illustrating an animation data generation process according to an embodiment of the present invention.
4 is a flowchart illustrating an animation synchronization process based on respiratory movement tracking according to an embodiment of the present invention.
5 is a diagram for explaining a mapping process according to an embodiment of the present invention.
6 is a diagram for explaining an interpolation process according to an embodiment of the present invention.
7 is a view showing a state in which a patient with a marker attached according to an embodiment of the present invention is lying on the CT imaging equipment.
8 is a view for explaining a process of matching with a marker according to an embodiment of the present invention.
9 is a 3D animation image of a lung according to an embodiment of the present invention.
10 is an exemplary view for explaining a displacement measurement process using a marker during respiration according to an embodiment of the present invention.
11 illustrates a marker according to an embodiment of the present invention.
12 illustrates a side view of a marker according to an embodiment of the present invention.

Advantages and features of the embodiments disclosed herein, and methods for achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the embodiments to be proposed in the present disclosure are not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments are provided to those of ordinary skill in the art. It is only provided to be a complete indication of the category.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in this specification have been selected as currently widely used general terms as possible while considering the functions of the disclosed embodiments, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the detailed description of the corresponding specification. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the content throughout the present specification, rather than the name of a simple term.

Expressions in the singular herein include plural expressions unless the context clearly dictates the singular.

In the entire specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

In the present invention, the subject performing the method for providing a 3D animation video using a lung CT image based on respiratory movement tracking may be a general computer device. That is, in the present invention, a computer performing the method for providing a 3D animation video using a lung CT image based on respiratory movement tracking may be a computer, a controller of the computer, or a processor.

2 is a flowchart illustrating a method for providing a 3D animation video using a lung CT image based on respiratory movement tracking according to an embodiment of the present invention.

Referring to FIG. 2 , in the method for providing a 3D animation video of the present invention, spirometry curve data according to spirometry performed on a patient is input (S201), and a computed tomography (CT) image taken during maximum inspiration of the patient is obtained. The maximum inspiration CT image data and the maximum expiration CT image data, which is a CT image taken during maximum expiration, are received ( S203 ).

Then, the CT image is reconstructed into a three-dimensional image by using the spirometry curve data, the maximum inspiratory CT image data, and the maximum expiratory CT image data (S205).

Then, a 3D animation video is provided in a manner that synchronizes the 3D image according to the patient's respiratory movement data detected and input in real time (S207, S209).

3 is a flowchart illustrating an animation data generation process according to an embodiment of the present invention.

Referring to FIG. 3 , the maximum inspiration CT image data and the maximum expiration CT image data are segmented for each major organ region ( S301 ).

Then, the segmented images are reconstructed into a three-dimensional mesh (S303).

Then, the mesh at the time of maximal inspiration and the mesh at the time of maximal exhalation are classified for each major organ region to generate a corresponding pair (S305).

Then, for each corresponding pair, a vertex correlation between the two meshes is mapped ( S307 ).

5 is a diagram for explaining a mapping process according to an embodiment of the present invention.

Referring to FIG. 5 , in the present invention, vertex correlations between two meshes are mapped for each pair. In this case, an existing vertex to vertex correlation method may be used, and for more accurate correlation, a relationship between some vertices may be manually specified by directly taking a picture.

Next, animation data is generated by interpolating all vertices based on the mapped vertex correlation data (S309, S311).

6 is a diagram for explaining an interpolation process according to an embodiment of the present invention.

Referring to FIG. 6 , animation data is generated by interpolating all vertices based on mapped vertex correlation data. In this case, the number of interpolation may be determined according to the target volume change amount output precision. For example, to express the amount of volume change in units of 0.01%, interpolation is repeated until the number of interpolated data becomes 10000. In an embodiment of the present invention, the interpolation method uses linear interpolation when no information is given, and nonlinear interpolation using the rate of change as a parameter can be performed if information on rate of change of specific vertices is provided.

4 is a flowchart illustrating an animation synchronization process based on respiratory movement tracking according to an embodiment of the present invention.

Referring to FIG. 4 , the step of providing a 3D animation video ( S209 ) includes the following process.

First, the position data at the time of maximum inspiration and the position data at the time of maximum exhalation are checked in the patient's respiratory movement data (S401).

Then, the position data at the time of maximum inspiration and the position data at the time of the maximum exhalation are normalized as a percentage based on the position data (S403).

Then, the patient's respiratory movement data detected and inputted in real time is normalized, and a 3D animation frame of a position corresponding to the normalized value is selected and output (S405).

In the present invention, the 3D model file generally includes information such as points, faces, and textures. Among them, some model file formats such as fbx and c4d also include 3D animation information, and the animation video file in the present invention refers to such a 3D animation model file. 3D Animated Model can be produced in various 3D modeling programs such as blender, 3ds max, cinema4d, etc.

In the present invention, the respiratory movement of the patient is measured and synchronized with the animation data. In this case, an Inertial Measurement Unit (IMU) sensor or an image marker may be used to measure the respiratory movement.

In the present invention, the displacement during respiration of the patient may be measured using an IMU sensor or an image marker, but the present invention is not limited thereto, and various methods for measuring the displacement of an object may be used.

For example, when measuring the degree of respiration using a marker, after photographing a fiducial marker with a patterned pattern with a camera, detecting 4 or more points from the marker, and obtaining a PnP solution Absolute displacement can be obtained through

7 is a view showing a state in which a patient with a marker attached according to an embodiment of the present invention is lying on the CT imaging equipment.

In FIG. 7 , a state in which the patient 10 to which the marker 710 is attached is lying on the CT imaging equipment is shown.

7 illustrates a method of measuring the respiratory movement of the patient 10 using the marker 710 .

If the patient's maximum inspiration is Z _M , the patient's maximum expiration is Z _m , and the current respiratory displacement is z, the current respiration state can be expressed as the following percentage.

{(zZ _M )/(Z _M -Z _m )}*100 %

8 is a view for explaining a process of matching with a marker according to an embodiment of the present invention.

Referring to FIG. 8 , in the present invention, a CT is taken by attaching a marker 710 to the patient 10 , and the relative positions between the lung 810 and the marker 710 in the CT image are matched.

9 is a 3D animation image of a lung according to an embodiment of the present invention.

Referring to FIG. 9 , it can be confirmed that a 3D animation image synchronized according to the patient's breathing movement is displayed.

10 is an exemplary view for explaining a displacement measurement process using a marker during respiration according to an embodiment of the present invention.

In FIG. 10, (a) is a side view showing the displacement of the marker 710 during maximum inspiration of the patient, and (b) is a side view showing the displacement of the marker 710 during maximum expiration of the patient.

As such, in the present invention, the displacement during respiration of the patient may be measured using the marker 710 .

In the present invention, in order to register the animation video at the same position as the actual lung, the reference marker for registration must be clearly displayed on the CT image.

In order to ensure clarity in such a CT image, in the present invention, a material such as acrylic or Teflon having a thickness greater than or equal to a certain thickness is used, and a pattern is drawn image in which a total of three three-dimensional landmarks for axis alignment are formed. markers are used.

Here, the landmark means a shape that can be identified in the 3D CT reconstructed image. For example, the landmark should be perforated, dented, or protruding, and if the relative position between the landmark and the origin coordinates of the marker pattern is determined, it can be matched in reality.

That is, in the present invention, in order to match the body image and the lung animation image, the same reference point and axis must be shared with each other. To create these common fiducial points and axes, fiducial markers are used.

Usually, a rectangular pattern image is used as a reference marker. Since most of the reference markers have an asymmetric pattern, the direction of each vertex is always set to one regardless of rotation or movement. If one of the vertices is taken as a reference point, and the straight lines extending from the vertex in the vertical direction are called the x-axis and y-axis, respectively, the coordinates of both sides can be precisely synchronized only with the coordinates of the vertices of the markers for each body image and lung animation image. can

However, in the body image, since the external camera directly captures the marker, it is easy to find the vertex position by detecting the pattern of the rectangular reference marker. is difficult

In order to detect a square marker in a CT image, it is necessary to use a method of drawing a marker pattern on a square plate of a certain thickness or more made of acrylic or Teflon. Since it cannot be seen, in order to accurately know the position of the vertex, it is necessary to place a sculpture of an identifiable form nearby. These sculptures are called landmarks, and landmarks can be formed using various materials and shapes such as holes, columns, and iron cores.

11 illustrates a marker according to an embodiment of the present invention.

In FIG. 11, (a) shows a top view and a side view of the marker 710 in a CT image, and (b) shows an actual marker.

In FIG. 11 , a landmark 712 and an origin 714 can be identified from the marker 710 .

12 illustrates a side view of a marker according to an embodiment of the present invention.

12 shows a side view of the marker 710 in the CT image, the shape of the marker 710 including a landmark 712 in which a hole is formed and a square plate 716 formed with a predetermined thickness or more. can be checked In this case, the square plate 716 may be made of acrylic or Teflon.

Meanwhile, the method for providing a 3D animation video using a lung CT image based on respiratory movement tracking according to an embodiment of the present invention may be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

For example, computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, hard disk, floppy disk, removable storage device, and non-volatile memory (Flash Memory). , optical data storage devices, and the like.

In addition, the computer-readable recording medium may be distributed in computer systems connected through a computer communication network, and stored and executed as readable codes in a distributed manner.

The present invention has been described above using several preferred embodiments, but these embodiments are illustrative and not restrictive. Those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made without departing from the spirit of the present invention and the scope of the appended claims.

10 patient 710 markers
810 lung

Claims (4)

receiving spirometry curve data according to spirometry performed on the patient;
receiving maximum inspiration CT image data, which is a computed tomography (CT) image taken during maximum inspiration, and maximum expiration CT image data, which is a CT image taken during maximum inspiration, of the patient;
reconstructing a CT image into a three-dimensional image using the spirometry curve data, the maximum inspiratory CT image data, and the maximum expiratory CT image data;
providing a 3D animation video in a manner that synchronizes the 3D image according to the patient's respiratory movement data detected and input in real time;
segmenting the maximal inspiratory CT image data and the maximal expiratory CT image data for each major organ region;
reconstructing the segmented images into a 3D mesh;
classifying the mesh at the time of maximum inspiration and the mesh at the time of maximum expiration for each major organ region to generate a corresponding pair;
mapping a vertex correlation between the two meshes for each corresponding pair; and
Interpolating all vertices based on the mapped vertex correlation data to generate animation data
3D animation video providing method comprising a.
delete The method according to claim 1,
The step of providing the 3D animation video includes:
confirming position data during maximum inspiration and position data during maximum expiration in the patient's respiratory movement data;
Normalizing the position data during maximum inspiration and position data during maximum expiration as a percentage based on the position data;
Normalizing the patient's respiratory movement data detected and input in real time, selecting and outputting a 3D animation frame at a position corresponding to the normalized value
3D animation video providing method, characterized in that it comprises a.
A computer-readable recording medium in which a program capable of executing the method of any one of claims 1 and 3 to a computer is recorded.

Images