KR102007656B1 - Method for measuring length using motion sensor and apparatus for the same - Google Patents

Abstract

An actual length measuring method using a motion sensor performed by an endoscope device is provided. The actual length measuring method may include obtaining first motion sensor data, a virtual lesion image, and training data indicating a virtual lesion size based on a movement of a first user's hand instructing an endoscopy trainer; Generating a lesion size calculation model based on the training data; Acquiring second motion sensor data based on a movement of a second user's hand instructing the actual endoscope operator through the motion sensor of the endoscope device; Acquiring a lesion image through a camera of the endoscope device; And calculating the size of the lesion by inputting the lesion image according to the second motion sensor data to the lesion size calculation model.

Description

METHOD FOR MEASURING LENGTH USING MOTION SENSOR AND APPARATUS FOR THE SAME}

The present invention relates to an actual length measuring method and apparatus using a motion sensor.

In general, endoscopy is performed by inserting an endoscope and various surgical tools installed with a camera through a small hole without largely dissecting the human body, and performing the procedure while looking at the affected part through the image seen through the endoscope. Since the endoscopic procedure mostly uses the oral cavity, anus, etc., unlike the open surgery, there is no incision site and no scar, and the recovery time is fast. As the development of endoscopy and tools has progressed, endoscopy has been developed in many of the diseases requiring open surgery, and endoscopy has been increasingly applied in other medical fields.

For example, the endoscope is inserted into the oral cavity, the esophagus and duodenum are observed, and if necessary, the small intestine can be inserted into the small intestine. Internal organs can be observed, diagnosed, or treated. In many cases, endoscopy has been used to diagnose various diseases occurring inside the organs, and furthermore, treatments such as hemostasis at bleeding, ablation of early cancer or polyps, anastomosis such as fistulas, etc. are possible.

When viewing lesions through endoscopy, measuring the actual size of the lesions is a very important item in diagnosis, treatment, and patient prognosis. Currently, most endoscopy equipment does not provide a way to measure the size of the lesion. Specifically, most endoscopy equipment uses the length of the operator's biopsy forceps fully open to measure the actual size of the lesion. Some endoscopy instruments use a method of attaching an endoscope to the side of the lens, but they are expensive and are not commonly used. If the lesion is large and difficult to measure at once, or if you need to measure different parts of the image, It is not easy to change it, so it is not used well. In summary, conventionally, when viewing a lesion through an endoscope, there was a problem of estimating size depending on experience.

Publication No. 10-2017-0078489, 2017.07.07

The problem to be solved by the present invention is to provide a method and apparatus for measuring the actual length of the lesion through a combination of motion sensor and image change.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The length measuring method using a motion sensor performed by the endoscope device according to an embodiment of the present invention for solving the above problems, the first stored based on the movement of the hand of the first user instructing the endoscope training trainer Obtaining training data indicating motion sensor data, virtual lesion image, and virtual lesion size; Generating a lesion size calculation model based on the training data; Acquiring second motion sensor data based on a movement of a hand of a second user instructing an endoscope operator through a motion sensor of the endoscope device; Acquiring a lesion image through a camera of the endoscope device; And calculating the size of the lesion by inputting the lesion image according to the second motion sensor data to the lesion size calculation model.

Here, the learning data, the size of the lesion image in the endoscope surgical simulator according to the first motion sensor data when the endoscope device to explore the inside of the endoscope surgical simulator indicating a virtual internal model of the lesion It may include information about the correlation with the size.

Here, the learning data may include information about a correlation between the size change of the lesion image and the size of the lesion according to the first motion sensor data when the endoscope device detects the inside of the body. have.

Here, the lesion size calculation model may be constructed through a deep learning algorithm using a deep neural network (DNN).

Here, when obtaining information on the type of the lesion from the second user,

The method may further include outputting a treatment method according to the type of the lesion and the size of the lesion.

The method may further include outputting information about the size of the lesion.

Here, the 3D image of the lesion may be extracted and output based on the lesion image according to the second motion sensor data.

An endoscope apparatus equipped with a motion sensor for measuring a length according to another embodiment of the present invention for solving the above problems is a processor; And a memory in which at least one instruction executed by the processor is stored, wherein the at least one instruction is stored in the first motion sensor data based on a movement of a hand of a first user instructing an endoscopy trainer. Acquiring training data indicating a virtual lesion image, virtual lesion size, generating a lesion size calculation model based on the training data, and instructing an endoscope operator through a motion sensor of the endoscope device. Obtaining second motion sensor data based on movement, acquiring a lesion image through a camera of the endoscope device, and calculating a size of the lesion based on the lesion image according to the second motion sensor data. .

Here, the learning data, the size of the lesion image in the endoscope surgical simulator according to the first motion sensor data when the endoscope device to explore the inside of the endoscope surgical simulator indicating a virtual internal model of the lesion It may be feasible to include information regarding correlation with size.

Here, when the endoscope device detects the inside of the body, the learning data is executed to include information on a correlation between a change in the size of the lesion image and the size of the lesion in the body according to the first motion sensor data. It may be possible.

Length measurement method using a motion sensor performed by the endoscope device according to another embodiment of the present invention for solving the above problems, the hand of the second user instructing the endoscope operator through the motion sensor of the endoscope device Acquiring second motion sensor data based on a motion of the second motion sensor data; Acquiring a lesion image through a camera of the endoscope device; And calculating the size of the lesion by inputting the lesion image according to the second motion sensor data to a previously obtained lesion size calculation model, wherein the lesion size calculation model includes a first user's instruction indicating an endoscopy training trainer. It is a calculation model that models the correlation between the stored first motion sensor data, the virtual lesion image, and the virtual lesion size based on the movement of the hand.

According to another aspect of the present invention, there is provided a method of measuring length using a motion sensor performed by an endoscope apparatus, wherein the length measurement method is performed based on a movement of a first user's hand instructing an endoscopy trainee. Acquiring one motion sensor data; Obtaining a virtual lesion image corresponding to the first motion sensor data; Obtaining a preset virtual lesion size; Calculating lesion size indicating correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size based on the first motion sensor data, the virtual lesion image, and training data indicating the virtual lesion size And generating a model, wherein the lesion size calculation model is input with second motion sensor data and a lesion image based on a movement of a second user's hand instructing an endoscope operator and used to calculate a disease size.

An endoscope apparatus equipped with a motion sensor for measuring a length according to another embodiment of the present invention for solving the above problems is a processor; And a memory storing at least one instruction executed by the processor, wherein the at least one instruction is based on a movement of a second user's hand instructing the endoscope operator through a motion sensor of the endoscope device. Acquire a second motion sensor data, obtain a lesion image through the camera of the endoscope device, and input the lesion image according to the second motion sensor data into a previously obtained lesion size calculation model to determine the size of the lesion. The lesion size calculation model is a calculation model modeling a correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size stored based on the movement of the first user's hand instructing the endoscopy trainer. to be.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, accurate and objective lesion size can be measured in endoscopy and findings, thereby increasing diagnostic accuracy.

In addition, since the direction of the future treatment plan can be easily established based on the size of the lesion, there is an effect helpful for treatment.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram showing the configuration of an endoscope apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of operating an endoscope device according to an embodiment of the present invention.
3 is a conceptual diagram illustrating the structure of an endoscopic procedure simulator.
4 is a conceptual diagram illustrating a structure in which the lesion models are male and female coupled to the coupling holes of the endoscopic procedure simulator.
5 is a front view showing an embodiment of the lesion model.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

The spatially relative terms " below ", " beneath ", " lower ", " above ", " upper " It can be used to easily describe a component's correlation with other components. Spatially relative terms are to be understood as including terms in different directions of components in use or operation in addition to the directions shown in the figures. For example, when flipping a component shown in the drawing, a component described as "below" or "beneath" of another component may be placed "above" the other component. Can be. Thus, the exemplary term "below" can encompass both an orientation of above and below. Components may be oriented in other directions as well, so spatially relative terms may be interpreted according to orientation.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The invention can be carried out on a computer. In the present specification, a computer includes all of various devices capable of performing arithmetic processing to present a result visually to a user. For example, a computer may also be a medical device for acquiring or observing a medical image (eg, an endoscope device). The method of the present invention may be performed on the endoscope device 10 as well as a computer. In the following, the process of measuring the actual length through the endoscope device 10 as an embodiment of the present invention is described in detail.

Referring to FIG. 1, the endoscope device 10 may include at least one processor (or control device) 100, a storage device 110, a motion sensor 120, a camera 130, a display 140, and a transceiver ( 150) and the like.

The processor 100 may execute a program command stored in the storage device 110. The processor 100 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each storage device 110 may be configured as at least one of a volatile storage medium and a nonvolatile storage medium. For example, the storage device 110 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

The endoscope device 10 may include a motion sensor 120 in the handle portion. The movement of the input portion of the endoscope device 10 may be determined based on a hand movement of a user who uses the endoscope device 10. Here, the input unit may instruct the configuration that is input into the body of the person of the configuration of the endoscope device 10. The endoscope device 10 may recognize a movement of a user's hand that manipulates the endoscope device 10 through the handle of the endoscope device 10 through the motion sensor 120. The endoscope device 10 may acquire first motion sensor data or second motion sensor data indicating a movement of a user's hand.

The motion sensor data may include first motion sensor data obtained from a first user and second motion sensor data obtained from a second user. Here, the first user may be a person who performs an endoscope based on the endoscope operation simulator 30 or a person who performs an endoscope on an actual human body. The first user may simply be an accumulator of learning data. Here, the second user may indicate a person who performs an endoscope on a body of an actual person.

Here, the training data may indicate a data set (or data set) that may be used to extract the size of the virtual lesion based on the lesion image according to the first motion sensor data. That is, the training data may be data indicating the first motion sensor data, the lesion image, and the lesion size at the same time. In addition, the training data may be data stored by the first user based on the endoscopic surgery simulator 30 indicating the virtual body model.

The training data may include obtaining first motion sensor data stored on the basis of a movement of a first user's hand instructing the endoscope surgical trainer, acquiring a virtual lesion image corresponding to the first motion sensor data, and preset virtual It may be obtained through the step of obtaining the lesion size.

Here, the first motion sensor data refers to data obtained from the motion sensor 120 attached to the handle portion of the endoscope device 10. The motion sensor 120 may recognize the movement of the handle of the endoscope device 10 by the first user, and may data it.

The endoscope device 10 may include a camera 130 capable of capturing an endoscope object. The endoscope device 10 may acquire a lesion image through the camera 130. In order for the camera 130 to distinguish the lesion from other components inside the body, the following processing may be necessary.

The camera 130 of the endoscope apparatus 10 may acquire a lesion image in image data through image processing. Image processing may refer to a series of processes for creating or modifying a new image using an endoscope device for an existing image. The image processing method receives the image from the camera 130 and is performed simultaneously with subsequent processing consisting of grabbing, filtering, thresholding, thinning, and profiling. Can be.

The endoscope apparatus 10 may calculate the size of the lesion based on the lesion image according to the second motion sensor data. An image captured by the camera 130 included in the endoscope device 10 may be output through the display 140 included in the endoscope device 10. In addition, the endoscope apparatus 10 may extract a 3D image of the lesion based on the lesion image according to the second motion sensor data.

When the endoscope device 10 is a device that can be independently operated, the endoscope device 10 may itself acquire first motion sensor data, a lesion image, and learning data indicating a lesion size stored by the first user.

When the endoscope device 10 is a device independent of the endoscope device, the endoscope device 10 indicates the first motion sensor data, the lesion image, the lesion size stored by the first user from the endoscope device through the transceiver 150. Learning data can be obtained.

Hereinafter, an actual length measuring method using a motion sensor according to an embodiment of the present invention will be described with reference to the drawings.

Referring to FIG. 2, the endoscope apparatus 10 acquires stored first motion sensor data, a lesion image, and learning data indicating a lesion size based on an endoscopic surgery simulator 30 that indicates a virtual internal model of the body ( S200: learning data acquisition step). Here, the lesion image may be a real lesion image that may be found in a real person's body or a virtual lesion image that may be found in an endoscopy simulator 30, and the lesion size may be found in a real person's body. It may be the actual lesion size or the virtual lesion size that can be found within the endoscopic procedure simulator 30.

In an embodiment, when the endoscope device 10 is a device that can be independently operated, the endoscope device 10 may itself include first motion sensor data, a lesion image, and learning data indicating a lesion size stored by a first user. Can be obtained by.

In another embodiment, when the endoscope device 10 is a separate computing device that is wired or wirelessly connected to the computer, the endoscope device 10 may have a first motion stored by the first user from the computer through the transceiver 150. Sensor data, a lesion image, and learning data indicating a lesion size may be acquired.

The training data may be a data set including first motion sensor data, a lesion image, and data indicating a lesion size. The first motion sensor data may be measured by the motion sensor 120 mounted on the handle portion of the endoscope device 10. The first motion sensor data may be data obtained by measuring the movement of the endoscope device 10 by the operation of the first user with the motion sensor 120 over time.

The lesion image may be measured by the camera 130 mounted on the input unit of the endoscope device 10. The lesion image may be an image obtained by measuring an image of the inside of the human body (or the endoscope operation simulator 30 to be described later) photographed according to the movement of the camera 130 by a first user's operation over time. The training data and the lesion image may be acquired at the same time point, and the training data and the lesion image acquired at the same time point may be used to calculate the lesion size in a lesion size calculation model to be described later. That is, the lesion size may be calculated based on the lesion image according to the motion sensor data at a specific time point. The lesion size may be the size of the lesion inside the human body (or endoscopy simulator 30).

That is, the lesion size calculation model may indicate a correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size. The lesion size calculation model may be generated based on the first motion sensor data, the virtual lesion image, and the training data indicating the virtual lesion size.

As a method of acquiring the learning data by the endoscope device 10, various methods may be applied. In one embodiment, the training data may be acquired by inserting the endoscope into the endoscope surgical simulator 30 which may apply lesions of various sizes to various locations.

The endoscope surgical simulator 30 is a device that is generated similar to an actual body organ to perform endoscope training, and may be a device that can detach a virtual lesion module. The endoscope surgical simulator 30 may perform a process of combining virtual lesion modules of various sizes and acquiring learning data to the plurality of coupling holes 320 to which the virtual lesion module may be attached.

3 to 5, a specific embodiment of the endoscope procedure simulator 30 may include an organ model unit 300 and a virtual lesion module 321.

As shown in FIG. 3, the organ model unit 300 has a simulated organ shape. In the present embodiment, the organ model unit 300 of the stomach shape is connected to the esophagus connected to the oral cavity of the human body, but is not limited thereto, and the organ model unit 300 may have an organ shape such as a small intestine, a large intestine, and an anus. have. For example, in particular, the organ model according to the present invention is formed in the same manner as the actual body structure, it is possible to learn realistic endoscopic procedures.

Inside the organ model unit 300 is formed an insertion space for inserting and moving the endoscope. In addition, a plurality of coupling holes 320 may be formed through the surface of the organ model 300. The plurality of coupling holes 320 may indicate a hole capable of detachable virtual lesion.

In the present embodiment, although the plurality of coupling holes 320 are illustrated as being formed in the organ model unit 300, only one coupling hole 320 may be formed in the organ model unit 300. In addition, the organ model 300 has a plurality of coupling holes 320 through which the virtual lesion module 321 is selectively detachably coupled. Each coupling hole 320 is formed through the surface of the organ model unit 300 to communicate with the insertion space. Coupling hole 320 has a size of the diameter that each coupling portion 320 of the virtual lesion module 321 is fitted to fit and fit.

By varying and attaching the lesion size of the virtual lesion module 321 to the plurality of coupling holes 320, learning data about the size of various lesions at various positions in the body may be accumulated.

The virtual lesion module 321 is fitted into the coupling hole 320 of the organ model 300 to expose the lesion display unit 321-1 of FIG. 5 to the insertion space. In addition, the virtual lesion module 321 is selectively coupled while changing the position of the plurality of coupling holes 320 of the organ model unit 300, it is possible to produce a variety of lesion phenomenon at various locations.

In addition, in another embodiment, the endoscope surgery simulator 30 may further include a fixing frame 310 surrounding and fixing the organ model unit 300.

In addition, the fixing frame 310 is formed with a through hole communicating with the coupling hole 320 at a position corresponding to each coupling hole 320 of the organ model 300. The through-hole virtual lesion module 321 is fitted into the coupling hole 320 of the organ model unit 300 is fitted.

By such a configuration, the learning data acquisition step (S200) using the endoscope surgery simulator in a state in which the virtual lesion module 321 is mounted in each coupling hole 320 of the organ model unit 300 is described as follows. .

The first user may determine in advance the size of the virtual lesion module 321 to be installed in the endoscopic procedure simulator 30. For example, the first user may install the virtual lesion module 321 having the size of the protrusion 321-2 to the endoscope surgery simulator 30. The endoscope device 10 may receive an input from the first user that the size of the protrusion 321-2 of the virtual lesion module 321 installed by the first user is 5 mm. The first user may move along the insertion space while observing the inside of the organ model unit 300 through the camera 130 of the endoscope device 10. When the camera 130 of the endoscope device 10 reaches the virtual lesion module 321 to produce a lesion state during the movement process, the first user may operate the endoscope device 10. Through this process, the endoscope device 10 may acquire learning data.

In this way, by producing a variety of lesions through the module detachably coupled to the organ model unit 300, by inserting and moving the endoscope device 10 along the insertion space of the organ model unit 300, to acquire the learning data can do.

The camera 130 of the endoscope device 10 may be inserted into the organ model unit 300. When the first user grabs the handle of the endoscope device 10 and moves back and forth, the first user may acquire an image captured by the camera 130. The endoscope device 10 may store information of an image that may occur when photographing lesions of various sizes (eg, lesion sizes of 5 mm, 6 mm, 7 mm, ..., 10 mm). When this learning is repeated, the endoscope apparatus 10 may include i) first motion sensor data by the motion sensor 120, ii) lesion image data by the endoscope camera 130, and iii) within the endoscope procedure simulator 30. Learning data indicating lesion size may be obtained.

The endoscope apparatus 10 constructs a lesion size calculation model by applying the learning data obtained from the endoscope surgery simulator 30 indicating a virtual internal model to the machine learning algorithm (S210). The endoscope device 10 may apply the learning data to various machine learning algorithms. In one embodiment, the endoscope device 10 may utilize a deep learning algorithm using a deep neural network.

For example, the endoscope apparatus 10 may acquire a plurality of motion sensor data and data regarding a plurality of lesion images while photographing lesions having sizes of 5 mm, 6 mm, 7 mm,..., 10 mm. The endoscope apparatus 10 applies a plurality of motion sensor data and a plurality of image information obtained when photographing lesions having sizes of 5 mm, 6 mm, 7 mm, ... 10 mm to a deep learning algorithm using a deep neural network. Build output models. In another embodiment, the lesion size calculation model may be obtained in advance from the outside.

In embodiments of the present invention, a deep neural network (DNN) refers to a system or a network that performs a determination based on a plurality of data by constructing one or more layers in one or more endoscope devices. For example, the deep neural network may be implemented as a set of layers including a convolutional pooling layer, a locally-connected layer, and a fully-connected layer. The convolution pooling layer or the local access layer can be configured to extract features in the image. The fully connected layer can determine the correlation between the features of the image. In some embodiments, the overall structure of the deep neural network may be in the form of a local connection layer followed by a convolution pooling layer and a complete connection layer in the local connection layer. The deep neural network may include various criteria (ie, parameters), and may add new criteria (ie, parameters) through an input image analysis.

In the embodiments of the present invention, the deep neural network is a structure called a convolutional neural network suitable for acquiring training data, and is a feature extraction layer that self-learns a feature having the highest discriminative power from given image data. ) And a prediction layer that learns a prediction model to have the highest prediction performance based on the extracted feature.

The feature extraction layer spatially integrates a convolution layer and a feature map to create a feature map by applying a plurality of filters to each region of an image, thereby allowing a feature to be changed to change in position or rotation. It can be formed in a structure that repeats the alternating layer (Pooling Layer) to be extracted several times. Through this, it is possible to extract various levels of features, from low level features such as points, lines and planes to complex high level features.

The convolutional layer obtains a feature map by taking a nonlinear activation function on the inner product of a filter and a local receptive field for each patch of the input image, compared to other network structures. Thus, CNN is characterized by using filters with sparse connectivity and shared weights. This connection structure reduces the number of parameters to be learned and makes the learning through the backpropagation algorithm more efficient, resulting in better prediction performance.

The pooling layer or sub-sampling layer creates a new feature map using the local information of the feature map obtained from the previous convolutional layer. In general, the feature map newly created by the integration layer is reduced to a smaller size than the original feature map. Typical integration methods include the maximum pooling to select the maximum value of the corresponding area in the feature map and the corresponding feature map in the feature map. Average pooling, which finds the mean of a domain. The feature map of the unified layer is generally less affected by the location of any structure or pattern present in the input image than the feature map of the previous layer. That is, the integrated layer can extract features that are more robust to regional changes such as noise or distortion in the input image or previous feature maps, and these features can play an important role in classification performance. Another role of the integration layer is to allow for the reflection of a wider range of features as one goes up to the upper learning layer in the deeper structure, and as feature extraction layers accumulate, the lower layer reflects local features and rises to the upper layer. Increasingly, it is possible to generate a feature that reflects a feature of the entire abstract image.

As such, the final feature extracted through the iteration of the convolutional layer and the integrated layer is that the classification model such as Multi-layer Perception (MLP) or Support Vector Machine (SVM) is fully connected. Combined in the form of -connected layer, it can be used for classification model training and prediction.

However, the structure of the deep neural network described in the embodiments of the present invention is not limited thereto, and may be formed of a neural network having various structures.

For example, a plurality of training data finally extracted through the repetition of the convolutional layer and the integrated layer may be classified by a classification model such as Multi-layer Perception (MLP) or Support Vector Machine (SVM). It may be combined in the form of a fully-connected layer and used to measure the size of a lesion through the lesion image according to the first motion sensor data.

Referring back to FIG. 2, the endoscope device 10 obtains second motion sensor data by the motion sensor 120 (S220). A motion sensor 120 may be attached to a handle portion of the endoscope device 10. The endoscope device 10 may acquire second motion sensor data storing, as a vector value, the motion of a second user's hand using the endoscope device 10 through the motion sensor 120 of the endoscope device 10. .

The endoscope device 10 obtains the lesion image by the camera 130 (S230). The camera 130 may be attached to a portion of the endoscope device 10 inserted into the human body. The endoscope device 10 may capture an image of an inside of an organ of an actual person or an image of a lesion indicating a shape of an inside of an organ through a camera 130 of the endoscope device 10. The endoscope device 10 may acquire an organ internal image or a lesion image corresponding to the second motion sensor data. Here, the correspondence may mean a value measured at the same time. For example, the lesion image at a specific time point may include a motion sensor value at a specific time point.

The endoscope apparatus 10 calculates the size of the lesion by inputting the motion sensor data and the lesion image value to the lesion size calculation model (S240). For example, after the lesion size calculation model is constructed, the endoscope apparatus 10 may acquire specific motion sensor data and data regarding a specific lesion image while photographing a lesion of unknown size. The endoscope device 10 may input the acquired specific motion sensor data and data regarding the specific lesion image into the lesion size calculation model. In this case, the lesion size calculation model may determine that the specific motion sensor data and the specific lesion image acquired through the lesion of unknown size correspond to the motion sensor data and the lesion image acquired while photographing the lesion having a size of 6 mm. . In this case, the endoscope apparatus 10 may calculate that the size of the lesion whose size is unknown is 6 mm through the lesion size calculation model.

In addition, in another embodiment of the present invention, the endoscope apparatus 10 may output the acquired size information of the lesion through the display 140. In addition, the endoscope apparatus 10 may extract the 3D image of the lesion based on the lesion image according to the second motion sensor data. On the other hand, the endoscope device 10 may output a three-dimensional image of the lesion using a 3D printer. For example, the endoscope device 10 may output the size of the calculated lesion so that the second user or the like may visually check the display 140. Also, in another embodiment, the endoscope device 10 may obtain information about the type of lesion from the second user. For example, when the second user recognizes which disease is identified through the display 140 while performing an endoscope operation, the second user may input information regarding the type of the lesion to the endoscope device 10.

In this case, the endoscope apparatus 10 may recognize the size of the lesion, and may output which treatment is appropriate based on the type of the lesion through the display 140. Depending on the size of the lesion, the treatment may be applied differently. Below may be a specific explanation that the treatment may be applied differently depending on the size of the lesion.

1) Colorectal adenoma larger than 1 cm may be classified as advanced adenoma with a high risk of developing colorectal cancer. In this case, it may be advisable to advance the follow-up period after ablation with an endoscope. In other words, when 1-2 tubular adenomas less than 1 cm in size are removed, colonoscopy should be performed after 5 years. However, colonoscopy should be performed after 3 years if colonic adenoma greater than 1 cm is removed. However, at present, the timing of follow-up is largely determined by the endoscopic doctor's estimate, not by direct measurement.

2) For lateral developmental tumors of the large intestine, effective treatment can be expected with submucosal dissection, a special form of colon polypectomy. However, the treatment policy is also decided based on the size estimated by the endoscope doctor.

Therefore, the endoscope device 10 recognizes the size of the lesion, and outputting which treatment is appropriate based on the type of the lesion through the display 140 can be very helpful for the treatment.

The actual lesion length measuring method using the deep neural network according to an embodiment of the present invention described above may be implemented as a program (or an application) to be executed in combination with a computer which is hardware and stored in a medium.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. Software modules may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims (13)

Length measurement method using a motion sensor performed by an endoscope device,
Acquiring the stored first motion sensor data, the virtual lesion image, and the training data indicating the virtual lesion size based on the movement of the hand of the first user instructing the endoscope treatment trainer;
Generating a lesion size calculation model based on the training data;
Acquiring second motion sensor data based on a movement of a hand of a second user instructing an endoscope operator through a motion sensor of the endoscope device;
Acquiring a lesion image through a camera of the endoscope device; And
And calculating a size of a lesion by inputting the lesion image according to the second motion sensor data to the lesion size calculation model. The method according to claim 1,
The training data may include the size of the lesion image and the size of the lesion in the endoscope surgery simulator based on the first motion sensor data when the endoscope device explores the inside of the endoscope surgery simulator indicating a virtual internal model. Length measurement method using a motion sensor, including the information on the correlation of. The method according to claim 1,
The learning data includes information on a correlation between a change in size of the lesion image and a size of the lesion according to the first motion sensor data when the endoscope device detects the inside of the body. Method of measuring length using The method according to claim 1,
The lesion size calculation model is constructed through a deep learning algorithm using a deep neural network (DNN), the length measurement method using a motion sensor. The method according to claim 1,
When obtaining information on the type of the lesion from a second user,
And outputting a treatment method according to the type of the lesion and the size of the lesion. The method according to claim 1,
And outputting information about the size of the lesion. The method according to claim 6,
And extracting and outputting a 3D image of the lesion based on the lesion image according to the second motion sensor data. An endoscope device equipped with a motion sensor for measuring the length,
A processor; And
At least one instruction executed by the processor includes a memory (memory),
The at least one command is
Obtaining first motion sensor data, virtual lesion image, and training data indicating virtual lesion size stored based on the movement of the first user's hand instructing the endoscope treatment trainer;
Generate a lesion size calculation model based on the training data,
Acquiring second motion sensor data based on the movement of the hand of the second user who instructs the endoscope operator through the motion sensor of the endoscope device,
Acquire lesion image through the camera of the endoscope device, and
And inputting the lesion image according to the second motion sensor data into the lesion size calculation model to calculate the size of the lesion. The method according to claim 8,
The training data may include the size change of the lesion image and the size of the lesion in the endoscope surgery simulator according to the first motion sensor data when the endoscope device explores the inside of the endoscope surgery simulator indicating a virtual internal model. Endoscopic devices, including information about the correlation of The method according to claim 8,
The learning data may include information regarding a correlation between a change in size of the lesion image and a size of the lesion according to the first motion sensor data when the endoscope device detects the inside of the body. Length measurement method using a motion sensor performed by an endoscope device,
Acquiring second motion sensor data based on a movement of a hand of a second user instructing an endoscope operator through a motion sensor of the endoscope device;
Acquiring a lesion image through a camera of the endoscope device; And
Calculating the size of the lesion by inputting the lesion image according to the second motion sensor data into a previously obtained lesion size calculation model;
The lesion size calculation model is a motion sensor, which is a calculation model modeling a correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size stored based on the movement of the first user's hand instructing the endoscope. Length measurement method utilized. Obtaining stored first motion sensor data based on a movement of a hand of a first user instructing the endoscope surgical trainer;
Obtaining a virtual lesion image corresponding to the first motion sensor data;
Obtaining a preset virtual lesion size;
Calculating lesion size indicating correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size based on the first motion sensor data, the virtual lesion image, and training data indicating the virtual lesion size Creating a model,
The lesion size calculation model is a length measurement method using a motion sensor is input to the second motion sensor data and the lesion image based on the movement of the hand of the second user instructing the endoscope to calculate the size of the disease. An endoscope device equipped with a motion sensor for measuring the length,
A processor; And
At least one instruction executed by the processor includes a memory (memory),
The at least one command is
Acquiring second motion sensor data based on the movement of the hand of the second user who instructs the endoscope operator through the motion sensor of the endoscope device,
Acquire lesion image through the camera of the endoscope device, and
The lesion image according to the second motion sensor data is input to a previously obtained lesion size calculation model to calculate the size of the lesion.
The lesion size calculation model is an endoscope apparatus, which is a calculation model modeling a correlation between the first motion sensor data, the virtual lesion image, and the virtual lesion size stored on the basis of the movement of the first user's hand instructing the endoscope.

Images