KR102495838B1 - Method, apparatus and computer program for controlling endoscope based on medical image - Google Patents

Abstract

According to an embodiment of the present disclosure, provided is a method of controlling an endoscope performed by a computing device including at least one processor, which includes the steps of: calculating a distance from an end of a scope to a first reference position based on medical images captured by the endoscope and positional information of the end of the scope of the endoscope; and controlling the end of the scope so that the end of the scope reaches the first reference position based on the distance. Accordingly, medical staff can control the location and degree of bending of an endoscopic scope according to a region to be photographed while performing an endoscopic procedure.

Description

METHOD, APPARATUS AND COMPUTER PROGRAM FOR CONTROLLING ENDOSCOPE BASED ON MEDICAL IMAGE}

The present disclosure relates to a method, apparatus, and computer program for controlling an endoscopic apparatus, and more specifically, a method for controlling an endoscopic apparatus based on a medical image for controlling the position and bending of an endoscopic scope based on a medical image when performing endoscopic imaging; It relates to devices and computer programs.

Endoscopy is a collective term for medical devices that observe organs by inserting a scope into the body without performing surgery or autopsy. In the endoscope, a scope is inserted into the human body, and light is irradiated to visualize the light reflected from the surface of the inner wall. Types of endoscopes are classified according to the purpose and body part, and can be largely divided into a rigid endoscope in which an endoscope tube is formed of metal and a flexible endoscope represented by a digestive endoscope.

Since the flexible endoscope device includes various devices inside, it is vulnerable to impact, and the inside of the digestive tract into which the flexible endoscope is inserted is also a very soft tissue and has an atypical shape. In addition, since the shape of the inside of the digestive tract differs depending on the patient, it may not be easy to insert the endoscope even for experienced medical staff. In this situation, a high level of concentration is required because the medical staff must safely insert the endoscope and explore the lesion.

The direction and degree of bending of the endoscopic scope are controlled by the manipulation of the endoscopic operator. In this case, when the endoscopic procedure is repeatedly performed due to the tension of the wire in the scope, the degree of fatigue of the endoscopic operator inevitably increases.

Republic of Korea Patent Registration No. 10-2185886 (2020.11.26)

The present disclosure is directed to solving the above-described problems of the prior art, and relates to a method, apparatus, and computer program for controlling an endoscope device based on medical images to control the position and bending of an endoscope scope based on medical images during endoscopic imaging. will be.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

A control method of an endoscope apparatus performed by a computing device according to an embodiment of the present disclosure for realizing the above object is disclosed. The method includes calculating a distance from the end of the scope to a first reference position based on a medical image taken from the endoscope device and positional information of the end of the scope of the endoscope device, and based on the distance, the end of the scope is and controlling the end of the scope to reach the first reference position.

Alternatively, the calculating may include determining the first reference position in the medical image using a first artificial intelligence model learned based on the medical image and the location information. .

Alternatively, the first artificial intelligence model controls the medical image, an encoder value of a first motor for controlling an x-axis motion of the scope with respect to the medical image, and a y-axis motion of the scope with respect to the medical image. The body part is inferred from the medical image by using learning data including the encoder value of the second motor, 6-dimensional posture information obtained from a posture sensor installed at the end of the scope, and matching information of the body part with respect to the medical image. It is characterized in that it is a model learned to do.

Alternatively, the controlling may include converting a distance between a central point of the medical image and the first reference position in the medical image into real coordinates, and moving each axis of the scope based on the real coordinates. It is characterized in that it comprises the step of generating a control signal for.

Alternatively, the control signal comprises a first control signal for a first motor for controlling the x-axis motion of the scope and a second control signal for a second motor for controlling the y-axis motion for the scope. to be characterized

Alternatively, the controlling may include controlling a degree of bending of the scope so that an end of the scope is separated from an inner wall of the body by a predetermined distance.

Alternatively, the controlling of the degree of bending may include starting an operation of controlling the degree of bending of the scope when an end of the scope reaches a second reference position.

Alternatively, the second reference location may include the esophagus.

Alternatively, controlling the degree of bending of the scope may include determining a bending direction of the scope so that the scope faces a dark region in the medical image.

Alternatively, controlling the degree of bending of the scope may include determining a bending direction of the scope using a second artificial intelligence model learned to determine the inner wall of the body based on the medical image. to be characterized

Alternatively, the first reference location may include the larynx.

Alternatively, the method may further include switching to an automatic control mode when it is determined that the end of the scope is inserted into the oral cavity of the patient based on the medical image.

Alternatively, the method may further include switching to a manual control mode when the end of the scope reaches the third reference position.

Alternatively, the third reference position may include false statements.

A computing device for controlling an endoscope device is disclosed according to an embodiment of the present disclosure for realizing the above object. The device may include a memory for storing a medical image captured by the endoscope device and location information of the end of the scope of the endoscope device, and a distance from the end of the scope to a first reference position based on the medical image and the location information. and a processor for calculating and controlling the end of the scope so that the end of the scope reaches the first reference position based on the distance.

A computer program stored in a computer readable storage medium is disclosed according to an embodiment of the present disclosure for realizing the above object. When the computer program is executed on one or more processors, operations for controlling the endoscope apparatus are performed. In this case, the operations may include an operation of calculating a distance from the end of the scope to a first reference position based on a medical image captured from the endoscope device and location information of the end of the scope of the endoscope device, and based on the distance and controlling an end of the scope so that the end of the scope reaches the first reference position.

According to an embodiment of the present disclosure, the convenience of the medical staff can be improved by allowing the endoscopic scope to reach a desired body part by controlling the location and degree of bending of the endoscopic scope according to the capturing region while the medical staff performs the endoscopic procedure.

1 is a block diagram of an endoscope system according to an embodiment of the present disclosure.
2 is a block diagram of an endoscope device according to an embodiment of the present disclosure.
3 is a flowchart illustrating a method of controlling an endoscope scope according to an embodiment of the present disclosure.
4 is a flowchart illustrating an operation of an automatic control mode of an endoscope scope according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method for controlling bending of an endoscope scope according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail so that those skilled in the art (hereinafter, those skilled in the art) can easily practice with reference to the accompanying drawings. The embodiments presented in this disclosure are provided so that those skilled in the art can use or practice the contents of this disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in many different forms, and is not limited to the following embodiments.

Same or similar reference numerals designate the same or similar elements throughout the specification of this disclosure. In addition, in order to clearly describe the present disclosure, reference numerals of parts not related to the description of the present disclosure may be omitted in the drawings.

The term “or” as used in this disclosure is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified in this disclosure or the meaning is not clear from the context, "X employs A or B" should be understood to mean one of the natural inclusive substitutions. For example, unless otherwise specified in this disclosure or where the meaning is not clear from the context, "X employs A or B" means that X employs A, X employs B, or X employs A and It can be interpreted as any one of the cases of using all of B.

The term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the listed related concepts.

The terms "comprises" and/or "comprising" as used in this disclosure should be understood to mean that certain features and/or elements are present. However, it should be understood that the terms "comprising" and/or "comprising" do not exclude the presence or addition of one or more other features, other elements, and/or combinations thereof.

Unless otherwise specified in this disclosure, or where the context clearly indicates that a singular form is indicated, the singular shall generally be construed as possibly including “one or more.”

The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to distinguish the components of the present disclosure from each other according to a predetermined criterion such as a functional point of view, a structural point of view, or explanatory convenience. there is. For example, components performing different functional roles in the present disclosure may be classified as first components or second components. However, elements that are substantially the same within the technical spirit of the present disclosure but should be distinguished for convenience of description may also be distinguished as a first element or a second element.

Meanwhile, the term "module" or "unit" used in the present disclosure refers to a computer-related entity, firmware, software or part thereof, hardware or part thereof. , It can be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. In this case, a “module” or “unit” may be a unit composed of a single element or a unit expressed as a combination or set of a plurality of elements. For example, as a narrow concept, a "module" or "unit" is a hardware element or set thereof of a computing device, an application program that performs a specific function of software, a process implemented through software execution, or a program. It may refer to a set of instructions for execution. Also, as a concept in a broad sense, a “module” or “unit” may refer to a computing device constituting a system or an application executed in the computing device. However, since the above concept is only an example, the concept of “module” or “unit” may be defined in various ways within a range understandable by those skilled in the art based on the content of the present disclosure.

The term "model" used in this disclosure refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model for a process. For example, a neural network “model” may refer to an overall system implemented as a neural network having problem-solving capabilities through learning. At this time, the neural network may have problem solving ability by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or may include a neural network set in which a plurality of neural networks are combined.

The term "image" used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, "image" can be understood as a term referring to a digital representation of an object that is visible to the human eye. For example, “image” may refer to multidimensional data composed of elements corresponding to pixels in a 2D image.

Explanations of the foregoing terms are intended to facilitate understanding of the present disclosure. Therefore, it should be noted that, when the above terms are not explicitly described as matters limiting the content of the present disclosure, the content of the present disclosure is not used in the sense of limiting the technical idea.

1 is a block diagram of an endoscope system according to an embodiment of the present disclosure.

Referring to FIG. 1 , an endoscope system 10 includes an endoscope apparatus 100 that acquires medical images, receives and analyzes medical images from the endoscope apparatus 100, and controls the endoscope apparatus 100 based on the medical images. Computing device 200 is included. The configuration of the endoscope device 100 will be described later in detail with reference to FIG. 2 .

The computing device 200 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected through a communication network. For example, the computing device 200 may be a server that performs intensive data processing functions and shares resources, or may be a client that shares resources through interaction with the server. In addition, the computing device 200 may be a cloud system in which a plurality of servers and clients interact with each other to comprehensively process data. Since the above description is only one example related to the type of the computing device 200, the type of the computing device 200 may be configured in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure. Since the above description is only one example related to the type of the computing device 200, the type of the computing device 200 may be configured in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1 , a computing device 200 according to an embodiment of the present disclosure may include a processor 210, a memory 220, and a network unit 230. there is. However, since FIG. 1 is only an example, the computing device 200 may include other configurations for implementing a computing environment. Also, only some of the components disclosed above may be included in the computing device 200 .

The processor 210 according to an embodiment of the present disclosure may be understood as a unit including hardware and/or software for performing computing operations. For example, the processor 210 may read a computer program and process data for machine learning. The processor 210 may process input data processing for machine learning, feature extraction for machine learning, calculation of an error based on backpropagation, and the like. The processor 210 for performing such data processing includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), and on-demand It may include a semiconductor (application specific integrated circuit (ASIC)) or a field programmable gate array (FPGA). Since the above-described type of processor 210 is just one example, the type of processor 210 may be variously configured within a range understandable by those skilled in the art based on the contents of the present disclosure.

The processor 210 acquires the medical image captured from the endoscope device 100 and position information of the scope of the endoscope device 100, and based on this, which organ the scope is located in the body or the medical image captured by the scope is obtained. You can figure out what organ this is for. The processor 210 may control the position or bending direction of the scope so that the scope reaches a desired reference position. The processor 210 may determine the degree of bending in consideration of characteristics of organs so that the scope does not damage the inner wall of the body. The processor 210 may determine a reference position to be reached by the scope in the medical image and control an end of the scope to reach the reference position. There may be a plurality of reference positions and may correspond to body parts. The processor 210 may control a driving unit responsible for movement of each axis of the scope in order to control the end of the scope.

The processor 210 may determine the reference position in the medical image using the learned first artificial intelligence model. To this end, the processor 210 may train the first artificial intelligence model using learning data including medical images, posture information of the scope, and body part matching information for the medical images. The processor 210 may determine the inner wall of the body in the medical image by using the learned second artificial intelligence model. To this end, the processor 210 may train the second artificial intelligence model using training data including medical images, posture information of the scope, and matching information on the inner wall of the body with respect to the medical images.

Depending on the body part, the processor 210 may operate the endoscope apparatus 100 in an automatic control mode that determines the position or bending of the scope, or may operate the endoscope apparatus 100 in a manual control mode.

According to an embodiment of the present disclosure, the processor 210 may determine a body part being captured in a medical image, and control the position of an endoscope scope according to characteristics of the body part. That is, when the scope is inserted into the body, the medical staff does not have to consider the direction and bending degree of the scope for endoscopic imaging, and the scope can be moved and bent toward a desired region for imaging. Therefore, the physical force required to manipulate the scope is reduced, and the convenience of operation by medical staff can be greatly improved. In addition, since the scope moves toward the reference position corresponding to the essential imaging area, the possibility of missing the essential imaging area during endoscopic imaging is reduced.

The processor 210 may train the first artificial intelligence model and the second artificial intelligence model through supervised learning using the learning data as an input value. Alternatively, the first artificial intelligence model and the second artificial intelligence model can be trained through unsupervised learning that discovers the criteria for data recognition by self-learning the type of data necessary for data recognition without any guidance. there is. Alternatively, the first artificial intelligence model and the second artificial intelligence model may be trained through reinforcement learning using feedback about whether a result of data recognition according to learning is correct.

The first artificial intelligence model and the second artificial intelligence model may include at least one neural network. Neural networks are network models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), Bidirectional Recurrent Deep Neural Network (BRDNN), Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), and Transformer. It may include, but is not limited to.

The memory 220 according to an embodiment of the present disclosure may be understood as a unit including hardware and/or software for storing and managing data processed by the computing device 200 . That is, the memory 220 may store any type of data generated or determined by the processor 210 and any type of data received by the network unit 230 . For example, the memory 220 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, and random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory , a magnetic disk, or an optical disk may include at least one type of storage medium. Also, the memory 220 may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory 220 is just one example, the type of memory 220 may be configured in various ways within a range understandable by those skilled in the art based on the contents of the present disclosure.

The memory 220 may organize and manage data necessary for the processor 210 to perform calculations, data combinations, program codes executable by the processor 210, and the like. Also, the memory 220 may store program codes that operate the processor 210 to generate learning data.

The memory 220 may store medical images captured by the endoscope device 100 and location information of the end of the scope of the endoscope device 100 .

The network unit 230 according to an embodiment of the present disclosure may be understood as a unit that transmits and receives data through any type of known wired/wireless communication system. For example, the network unit 230 may include a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), and WiBro (wireless). broadband internet), 5th generation mobile communication (5G), ultra wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), near field communication (NFC), or data transmission/reception may be performed using a wired/wireless communication system such as Bluetooth. Since the above-described communication systems are only examples, a wired/wireless communication system for data transmission and reception of the network unit 230 may be applied in various ways other than the above-described examples.

The network unit 230 may receive data necessary for the processor 210 to perform calculations through wired/wireless communication with an arbitrary system or an arbitrary client. In addition, the network unit 230 may transmit data generated through the operation of the processor 210 through wired/wireless communication with an arbitrary system or an arbitrary client. For example, the network unit 230 may receive medical images through communication with a cloud server or the endoscope apparatus 100 that performs tasks such as a medical image storage and transmission system and standardization of medical data. The network unit 230 may transmit various types of data generated through the operation of the processor 210 through communication with the aforementioned system, server, or endoscope apparatus 100 .

Meanwhile, although the endoscope device 100 and the computing device 200 are represented as being distinguished in FIG. 1, this is exemplary and the computing device 200 is provided inside the endoscope device 100 and constitutes a part of the endoscope device 100 You may.

2 is a block diagram of an endoscope device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the endoscope device 100 according to an embodiment of the present disclosure may be a flexible endoscope, specifically, a digestive endoscope. The endoscope device 100 may include a component capable of acquiring a medical image of the inside of a digestive tract and a component capable of performing treatment or treatment while viewing the medical image by inserting a tool, if necessary.

The endoscope device 100 may include an output unit 110 , a control unit 120 , a driving unit 130 , a pump unit 140 and a scope 150 .

The output unit 110 may include a display displaying medical images. The output unit 110 may include a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. A display module capable of outputting visualized information or implementing a touch screen, such as a display, a 3D display, or the like, may be included.

The output unit 110 may include various means for providing information on medical images. For example, a speaker may be included to audibly provide an alarm for a medical image.

The output unit 110 may display a medical image acquired by the scope 150 or a medical image processed by the controller 120 . The output unit 110 may display a moving path of the endoscope scope 150 while endoscopic imaging is being performed. For example, an organ may be displayed in the form of a map, and a position of the scope 150 inside the organ may be displayed. In the long-term map, the current position of the tip of the scope 150 may be displayed, and at least one reference position may be displayed. The reference position may mean a position where an automatic control mode or a manual control mode starts.

The output unit 110 may display the bending degree as well as the current position of the scope 150 . Inside a narrow tube-shaped organ such as the esophagus, the scope 150 needs to be bent so that the scope 150 does not touch the inner wall of the organ. It is difficult to intuitively grasp the degree of bending of the scope 150 only with an image captured at the end of the scope 150 . Accordingly, the controller 120 or the computing device may calculate the degree of bending of the scope 150 in consideration of an entry direction of the scope 150 and a body part where a medical image is captured. The controller 120 or the computing device may calculate the bending degree or bending direction based on the wire tension of the driving unit 130 . The output unit 110 may provide the degree of bending as a visual alarm such as a number or color code or an audible alarm such as a guide sound.

The output unit 110 may display an operation mode in which the endoscope device operates. In the case of the automatic control mode, a message guiding that the operation of the endoscopist is ignored may be displayed. In the case of the manual control mode, an alarm may be displayed to an endoscopist before a predetermined time before the manual control mode starts. As a result, the endoscopist can recognize that the automatic control mode is terminated and prepare to control the manipulation unit 151 .

Meanwhile, although one output unit 110 is shown in FIG. 2 , the number of output units 110 may be plural. In this case, an output unit displaying medical images acquired by the scope 150 and an output unit displaying information processed by the control unit 120 may be distinguished.

The controller 120 may control overall operations of the endoscope apparatus 100 . The controller 120 may include all types of devices capable of processing data. According to an exemplary embodiment, the control unit 120 may be a data processing device embedded in hardware having a physically structured circuit to perform functions expressed by codes or commands included in a program. As an example of a data processing unit built into hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC) , A processing device such as a field programmable gate array (FPGA) may be included, but the technical spirit of the present disclosure is not limited thereto.

The control unit 120 may control the movement of the scope 150 through the driving unit 130 connected to the scope 150 . The controller 120 may perform various control operations for photographing the inside of the fire extinguisher through the scope 150 . The controller 120 may perform various processing of medical images obtained through the scope 150 . The controller 120 may control the endoscope device 100 based on a control signal received from the computing device.

The drive unit 130 may provide power necessary for the scope 150 to be inserted into the body or to move within the body. For example, the driving unit 130 may include a plurality of motors connected to wires inside the scope 150 and a tension adjusting unit that adjusts the tension of the wires. The driving unit 130 may control the scope 150 in various directions by controlling power of each of the plurality of motors.

The driving unit 130 may include a first motor that determines the x-axis motion of the scope 150 and a second motor that determines the y-axis motion of the scope 150 . Under the control of the driving unit 130, the x-axis position, y-axis position, roll, pitch, and yaw values of the end of the scope 150 may be determined.

The driver 130 may control the bending direction or degree of bending of the scope 150 by adjusting the tension of the wire. For example, the driver 130 may increase the degree of bending of the scope 150 by increasing the tension according to a signal of the controller 120, and decrease the degree of bending of the scope 150 by decreasing the tension. The bent portion may be a portion of the insertion portion 152 .

The pump unit 140 includes an air pump that injects air into the body through the scope 150, a suction pump that sucks air from the inside of the body through the scope 150 by providing negative pressure or vacuum, and the scope 150. ) to inject washing water into the body. Each pump may include a valve to control the flow of fluid. The pump unit 140 may be opened and closed by the control unit 120 . At least one of the suction pump, water pump, and air pump may be opened or closed based on a control signal of the computing device or control of the controller 120 .

The scope 150 may include an insertion unit 152 inserted into the fire extinguisher and a manipulation unit 151 that receives input from a user to control the movement of the insertion unit 152 and perform various procedures inside the fire extinguisher. .

The insertion part 152 is configured to be flexible, and since one end is connected to the driving part 130, the degree or direction of bending can be determined by the driving part 130. Since medical imaging and procedures are performed at the end of the insertion part 152 , the scope 150 may include several cables and tubes extending to the end of the insertion part 152 .

A light source 153, a lens 154, a working channel 155, and air and water channels 156 may be provided inside the scope 150. Through the working channel 155, tools for treatment and treatment of lesions may be inserted during an endoscopic procedure. Air may be injected through the air/water channel 156 and washing water may be supplied.

Meanwhile, in FIG. 2 , air and water channels 156 are shown as passages through which washing water is supplied, but the present invention is not limited thereto. Illustratively, a separate water jet channel (not shown) may be provided inside the scope 150, and washing water may also be supplied through the water jet channel.

Meanwhile, the endoscope device according to an embodiment of the present disclosure may further include a posture sensor for obtaining posture information of the tip of the scope 150 in order to train the first artificial intelligence model and the second artificial intelligence model. The posture sensor is provided at the end of the scope 150 to measure coordinate information including the x-axis position, y-axis position, and z-axis position of the end of the scope 150 and rotational amounts including roll, pitch, and yaw. Since the posture information obtained from the posture sensor is used as training data for the first artificial intelligence model and the second artificial intelligence model, the posture sensor may be removed from the end of the scope 150 during an inference process after the learning data is secured.

The manipulation unit 151 may include a plurality of input buttons providing various functions so that an endoscopist controls the steering of the insertion unit 152 and performs a procedure through the working channel 155 and the air and water channels 156. can

In the automatic control mode, when an endoscopist controls the steering of the insertion unit 152 through the manipulation unit 151, the controller 120 may ignore the input of the manipulation unit 151. In the manual control mode, the input of the manipulation unit 151 may be reflected.

Meanwhile, the operation of the control unit 120 described above may be performed by the driving unit 130, and for this purpose, the driving unit 130 may have a necessary processing device such as a microprocessor or CPU. In this case, the controller 120 may process the medical images acquired through the scope 150, and the driving unit 130 may control the endoscope device 100 based on a control signal received from the computing device.

3 is a flowchart illustrating a method of controlling an endoscope scope according to an embodiment of the present disclosure.

Referring to FIG. 3 , the computing device may calculate the distance from the end of the scope to the first reference position based on the medical image and the location information of the end of the scope (S110). For example, the first reference location may include the larynx, but is not necessarily limited thereto. The first reference position may be set by an endoscopist and may be set according to the type of endoscope. The first reference position may be one of essential imaging parts, a clinically characteristic part, or a part having distinctive image characteristics in an image.

The computing device may control the end of the scope so that the end reaches the first reference position (S120). For example, the computing device may regard the position of the tip of the scope as a specific point in the image and control the position of the scope so that the corresponding point reaches the first reference position. The computing device may generate a control signal for controlling a driving unit connected to a wire in the scope so that the end of the scope reaches the first reference position. The computing device may control the first motor for controlling the x-axis motion of the scope and the second motor for controlling the y-axis motion of the scope.

Steps S110 and S120 may be repeatedly performed while the reference position is changed. That is, when the end of the scope reaches the first reference position, the computing device may set a new reference position, for example, a second reference position, and control the scope so that the end of the scope reaches the second reference position.

The repeated execution may end when the end of the scope reaches the third reference position. In this case, the computing device may set the endoscope device to a manual control mode in which the endoscope scope is controlled by an endoscopist.

4 is a flowchart illustrating an operation of an automatic control mode of an endoscope scope according to an embodiment of the present disclosure.

Referring to FIG. 4 , the computing device may determine whether the end of the scope is inserted into the oral cavity of the patient (S210). For example, an endoscopist may directly input whether the endoscope is inserted into the oral cavity through a manipulation unit or a control unit. Alternatively, the computing device may determine whether the scope is inserted into the oral cavity based on location information of the end of the scope and medical images. When the scope is inserted into the oral cavity, the automatic control mode of the endoscope device may be started, but the start point of the automatic control mode may be changed.

The computing device may determine the first reference position in the medical image by using the first artificial intelligence model based on the medical image and the location information of the end of the scope (S220).

For example, the first artificial intelligence model includes a medical image, an encoder value of a first motor that controls the scope's x-axis motion for the medical image, and an encoder value of a second motor that controls the scope's y-axis motion for the medical image. , 6-dimensional posture information acquired from a posture sensor installed at the end of the scope can be learned using learning data including matching information of body parts with respect to the medical image. The first artificial intelligence model may be a model learned to infer a body part appearing in a medical image by using a medical image. Accordingly, the first artificial intelligence model may identify various body parts as well as the first reference position in the medical image. For example, the first artificial intelligence model may have a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), and a matching cost-CNN (MC-CNN).

The computing device may determine which body part is displayed on the medical image by using the first artificial intelligence model. When the displayed body part is at the first reference position, the next step is performed, and when it is not at the first reference position, endoscopic imaging may continue.

The computing device may convert the distance between the central point of the medical image and the first reference position into actual coordinates (S230). The computing device may regard the central point of the medical image as the position of the end of the scope. Accordingly, the distance between the central point on the image and the first reference position in the medical image may correspond to the distance between the end of the scope and the first reference position in the real coordinate system. The computing device may convert a distance in coordinates on an image into a distance in a real coordinate system. In this case, the computing device may additionally use depth information. Depth information may be extracted from a medical image or obtained through a depth sensor provided at the end of an endoscope.

The computing device may generate control signals for movement of each axis of the scope based on the actual coordinates (S240). The computing device may generate control signals for motors that control movement of each axis of the scope. The computing device may also generate control signals for the scope's orientation.

The computing device may determine whether the end of the scope has reached the third reference position (S250). The third reference position may include, for example, a cardia, but is not necessarily limited thereto. The third reference position may be set by an endoscopist and may be set according to the type of endoscope. The third reference position may be one of essential imaging parts, may be a clinically characteristic part, or may be a part with distinctive image characteristics in an image.

The computing device may switch the endoscope device to a manual control mode (S260). That is, the computing device may set the position for starting the manual control mode as the third reference position. The third reference position may correspond to a site where endoscopic imaging ends or to a site where automatic control is not possible.

5 is a flowchart illustrating a method for controlling bending of an endoscope scope according to an embodiment of the present disclosure.

Referring to FIG. 5 , the computing device may control the degree of bending of the scope so that the end of the scope is separated from the inner wall of the body by a predetermined distance.

The computing device may check whether the end of the scope has reached the second reference position (S310). The computing device may determine whether the end of the scope has reached the second reference position by using the first artificial intelligence model. The second reference location may include, for example, the esophagus, but is not necessarily limited thereto. The second reference position may be set by an endoscopist and may be set according to the type of endoscope.

The computing device may determine a bending direction of the scope so that the scope faces a dark area in the medical image (S320). For example, when the scope is positioned in the esophagus, a dark area in the medical image may correspond to a deep portion of the esophagus, and a bright portion may correspond to an inner wall of the esophagus. Therefore, when the scope is bent toward the bright area, the inner wall of the esophagus may be scratched by the scope, resulting in internal injury. Accordingly, the computing device may protect the inner wall of the organ by bending the scope toward the dark side.

In order to accurately identify the depth of the esophagus, shape information along with brightness on the image may be used. For example, the computing device may determine the depth of the esophagus by considering the shape and size of a portion having brightness less than or equal to a reference value in the medical image.

The computing device may determine the bending direction of the scope by using the second artificial intelligence model learned to determine the inner wall of the body based on the medical image (S330). The second artificial intelligence model may be a model learned to recognize the inner wall and tunnel of the body based on medical images. The second artificial intelligence model may have a network structure such as a fully convolutional network (FCN), a conditional adversarial network (CAN), a recurrent neural network (RNN), and a matching cost-CNN (MC-CNN).

The computing device may identify the inner wall of the body through an image processing algorithm or use a second artificial intelligence model in step S320. That is, steps S320 and S330 may be selectively performed.

The computing device may control the degree of bending of the scope so that the end of the scope is separated from the inner wall of the body by a predetermined distance (S340). The predetermined distance may vary depending on the characteristics of the body organ. For example, when the second reference position is the esophagus and the stomach, since the diameters of the esophagus and the stomach are different, the distance at which the scope should be separated may also be different. The computing device may control the degree of bending by adjusting the tension of the wire in the scope. The computing device may additionally control the bending direction.

Steps S310 to S340 may operate in an automatic control mode. That is, the step S310 of determining whether the second reference position has been reached is omitted, and an operation of controlling the degree of bending of the scope may be performed in parallel in all steps of the automatic control mode.

The description of the present disclosure described above is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present disclosure is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

10: endoscope system
100: endoscope device
110: output unit
120: control unit
130: driving unit
140: pump unit
150: scope
151: control panel
152: insertion part
153: light source
154: lens
155: working channel
156: air, water channel
200: computing device
210: processor
220: memory
230: network unit

Claims (16)

A control method of an endoscope apparatus, performed by a computing device including at least one processor, comprising:
calculating a distance from an end of the scope to a first reference position based on a medical image captured by the endoscope and positional information of the end of the scope of the endoscope; and
controlling a tip of the scope to reach the first reference position based on the distance;
A method comprising a. According to claim 1,
The calculation step is
identifying the first reference position in the medical image using a first artificial intelligence model learned based on the medical image and the location information;
A method comprising a. According to claim 2,
The first artificial intelligence model,
The encoder value of the first motor for controlling the x-axis motion of the scope with respect to the medical image and the medical image, the encoder value of the second motor for controlling the y-axis motion of the scope with respect to the medical image, at the end of the scope The method of claim 1 , wherein the model is learned to infer the body part from the medical image using learning data including 6-dimensional posture information acquired from an installed posture sensor and matching information of the body part with respect to the medical image. According to claim 1,
The control step is
converting a distance between a central point of the medical image and the first reference position in the medical image into actual coordinates; and
generating control signals for movement of each axis of the scope based on the actual coordinates;
A method comprising a. According to claim 4,
The control signal is
and a first control signal for a first motor for controlling an x-axis motion of the scope and a second control signal for a second motor for controlling a y-axis motion for the scope. According to claim 1,
The control step is
controlling a degree of bending of the scope so that an end of the scope is separated from an inner wall of the body by a predetermined distance;
A method comprising a. According to claim 6,
In the step of controlling the degree of bending,
starting an operation of controlling a degree of bending of the scope when the end of the scope reaches a second reference position;
A method comprising a. According to claim 7,
The method of claim 1 , wherein the second reference location comprises the esophagus. According to claim 6,
The step of controlling the bending degree of the scope,
determining a bending direction of the scope so that the scope faces a dark area in the medical image;
A method comprising a. According to claim 6,
The step of controlling the bending degree of the scope,
determining a bending direction of the scope using a second artificial intelligence model learned to determine the inner wall of the body based on the medical image;
A method comprising a. According to claim 1,
The method of claim 1 , wherein the first reference location comprises the larynx. According to claim 1,
switching to an automatic control mode when it is determined that the end of the scope is inserted into the oral cavity of the patient based on the medical image;
A method characterized in that it further comprises. According to claim 1,
switching to a manual control mode when the tip of the scope reaches a third reference position;
A method characterized in that it further comprises. According to claim 13,
The method of claim 1, wherein the third reference position comprises a false door. A computing device for controlling an endoscopy device, comprising:
a memory for storing a medical image captured by the endoscope device and positional information of an end of the scope of the endoscope device; and
A processor for calculating a distance from an end of the scope to a first reference position based on the medical image and the location information, and controlling an end of the scope to reach the first reference position based on the distance ;
A device comprising a. A computer program stored on a computer readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for controlling an endoscope device,
These actions are
calculating a distance from an end of the scope to a first reference position based on a medical image captured by the endoscope and positional information of the end of the scope of the endoscope; and
controlling an end of the scope to reach the first reference position based on the distance;
A computer program comprising a.

Images