KR102253398B1 - Method for providing user interface through artificial intelligence-based image processing and user events, and image receiving apparatus using the method - Google Patents

Abstract

According to an embodiment, provided is an image receiving device, which includes: a communication unit for receiving an image frame from a medical imaging device for photographing a digestive organ; a control unit for outputting result values by performing an operation using a neural network model on the image frame; and a display unit for outputting at least one of a first result in which image processing information generated from the result values is overlapped on the image frame and a second result obtained as a user event signal for the image frame.

Description

TECHNICAL FIELD [Method for providing user interface through artificial intelligence-based image processing and user events, and image receiving apparatus using the method]

The present invention relates to a method for providing a user interface through artificial intelligence-based image processing and user events, and an image receiving apparatus using the method. More specifically, an image in which a user event occurs and a processing result through a neural network model can be recognized together. A method for providing a user interface through artificial intelligence-based image processing and user events to provide a UI, and an image receiving apparatus using the method.

There are several types of malignant tumors that occur in the stomach, such as gastric adenocarcinoma, gastric lymphoma, and gastrointestinal stromal tumor. Among these, gastric adenocarcinoma is the most common. In this case, gastric cancer can be classified into early gastric cancer (EGC) and advanced gastric cancer (AGC) according to the degree of tumor progression.

More specifically, early gastric cancer is when tumor cells invade to the upper mucosa or submucosa regardless of metastasis to the lymph nodes, and advanced gastric cancer is when tumor cells pass through the submucosa and penetrate the muscle layer or not. Invasion of the furnace is possible.

As a method of diagnosing gastric cancer, gastroscopy is the most important. More specifically, the medical staff can diagnose gastric cancer by reading a gastroscopy image taken on an individual suspected of gastric cancer using an endoscope scope, and provide treatment according to a lesion based on this. Meanwhile, in the diagnosis system based on an endoscopic image, there may be deviations in diagnosis according to the skill level of a medical staff. Such deviations in diagnosis may also be related to medical accidents such as misdiagnosis, delay in treatment timing, and inappropriate treatment.

In particular, in the case of early gastric cancer, the lesion site may be very small or it may be difficult to distinguish it from a normal tissue, and thus the deviation of diagnosis according to the skill level of the medical staff may be greater.

And, as the accuracy of diagnosis is more demanded for the prevention of medical accidents and improvement of medical services, the development of new diagnostic methods capable of providing not only the onset of disease but also the location of the lesion is continuously required. There is a situation.

In particular, it is also important to prevent medical accidents by improving the precision of diagnosis when the user checks the output image.

The present invention has been derived from the above-described necessity, and an object thereof is to allow the location of the lesion to be identified as a whole and more precisely at the same time.

In addition, it is an object of the present invention to provide a UI that allows a user to more intuitively determine the location of a lesion when checking an output image.

In particular, the purpose of this is to provide a UI that can grasp an image in which a user event has occurred and a processing result through a neural network model together so that a lesion can be identified more accurately and quickly.

An image receiving apparatus according to an embodiment includes a communication unit for receiving an image frame from a medical imaging device for photographing a digestive organ, a control unit for outputting result values by performing an operation using a neural network model on the image frame, and an image frame image. And a display unit that outputs at least one of a first result in which the image processing information generated from the result values overlaps and a second result obtained as a user event signal for the image frame.

The controller may list and output a set including the first result and the second result in a row unit on the screen of the display unit.

The controller may generate a plurality of sets including the first result and the second result, and output a set selected by a user from among the plurality of sets on the screen of the display unit.

The controller may enlarge and output any one selected by the user from among the first result and the second result included in the set selected by the user on the screen of the display unit.

The user event signal may be for capturing the image frame.

The controller may receive a plurality of user event signals and store second results corresponding to each of the plurality of user event signals for each user event signal.

The controller may output time information related to at least one of the first result and the second result in the form of a time-line on the screen of the display unit.

There are a plurality of neural network models, and different types of result values may be output through the operation using each of the plurality of neural network models.

One of the plurality of neural network models is a model for detecting the location of the lesion of the digestive tract, and the controller records the removal state of the lesion in memory when it is determined that the lesion has been removed, and detects the location of the lesion It may not be possible to determine whether a new lesion exists at the location of the digestive tract through the model to be described.

The control unit generates two identical indicators corresponding to the image processing information together with an image of the front wall or the rear wall of the digestive tract and an image of a cutout shape of the digestive tract, and the display unit generates an image of the front wall or the image of the digestive tract. One of the two indicators may be displayed on the image of the rear wall, and the other one of the two indicators may be displayed on the image having the digestive tract cut out.

The controller may output at least one of an index image corresponding to the first result and an index image corresponding to the second result on the screen of the display unit.

According to an embodiment, the first result of the image processing information generated from the result values overlapped on the image frame and the second result obtained as a user event signal for the image frame are displayed together, so that the result of the user's direct work and There is an advantage in that a more accurate analysis can be made by allowing the results of analysis through the neural network model to be compared and judged.

In addition, it is possible to more precisely check the location of the lesion by checking the location of the lesion through various types of UI. In addition, when the user checks the output image, it is possible to more intuitively determine the location of the lesion.

In addition, there is an advantage in that the analyzed result can be displayed in the anatomical structure and provided as a video file and a separate report is provided so that the result management can be carried out in various forms and more securely stored.

In addition to simply indicating the location of the lesion, it is possible to provide convenience for the examination by providing information on a site with a risk factor such as perforation.

1 is a conceptual diagram illustrating an image processing system according to an exemplary embodiment.
2 is a flowchart illustrating a process of outputting an image processing result by an image receiving apparatus according to an exemplary embodiment to a display unit.
3 to 10 are diagrams referenced to describe a UI providing screen of an image in which a user event signal is reflected and a result value processed through a neural network model according to an exemplary embodiment.
11 is a diagram referred to for describing a process of outputting image processing information generated from result values on an image frame using a plurality of neural network models according to an embodiment.
12 is a diagram showing the structure of a neural network model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

1 is a conceptual diagram illustrating an image processing system according to an exemplary embodiment.

1, the medical imaging apparatus 110 may transmit an image frame obtained by photographing an object to the image receiving apparatus 120. The image frame according to the embodiment may be an image obtained from a medical device such as a gastrointestinal endoscopy equipment, but the scope of the present invention is not limited thereto and may be applied to other types of medical devices such as ultrasound or MRI in the same or similar manner. .

The image receiving device 120 may include various electronic devices such as a PC, a laptop computer, and a tablet capable of running software for analyzing an input image frame by performing lesion prediction using the neural network model 121a.

The image receiving device 120 may include a communication unit 122, a control unit 121, a memory 123, and a display unit 124.

The communication unit 122 receives an image frame from the medical imaging apparatus 110 and transmits it to the controller 121.

The controller 121 may perform lesion prediction using the neural network model 121a to obtain a result value (a characteristic value, etc.) of a lesion on an input image frame, and predict a lesion based on this. Also, the controller 121 may output a specific result based on a user event signal for an image frame.

The memory 123 stores various types of data processed by the controller 121 such as result values obtained from the controller 121.

When a result value or image stored in the memory 123 is read under the control of the controller 121, the display unit 124 may output the corresponding data and provide it to the user.

On the other hand, hereinafter, when imaging of the stomach, especially the stomach, of the digestive tract is performed and the result is displayed on the display screen, the lesion is displayed on the front/rear of the stomach or the anatomical structure of the stomach, making it easier to locate the lesion. The user interface to identify it will be described. However, in the present invention, the same/similar application may be applied not only to the stomach, but also to images obtained from various types of other digestive organs, such as the large intestine, duodenum, and the like.

2 is a flowchart illustrating a process of outputting an image processing result of the image receiving apparatus 120 to the display unit 124 according to an exemplary embodiment.

As shown in FIG. 2, the controller 121 may receive an image frame from the medical imaging apparatus 110 through the communication unit 122 (s210). Specifically, when the image receiving device 120 is driven to start analysis of an image, the image analysis start time may be measured and the image frame may be read.

In this case, the controller 121 may monitor the computational speed (frame rate) of the neural network model 121a and read the image frames from the medical imaging apparatus 110 frame by frame. Depending on the embodiment, an operation on a plurality of neural network models 121a for processing one image frame may be processed in parallel or serially so as to enable a plurality of analyzes at a time.

The controller 121 may output result values by performing an operation using a neural network model on an image frame (S220). A process of performing an operation using a neural network model according to an embodiment and output of result values thereof will be described later with reference to FIGS. 11 and 12.

The controller 121 may output at least one of a first result in which image processing information generated from the result values overlapped on an image frame and a second result obtained as a user event signal for the image frame (s230). ).

For example, referring to FIG. 3, (a) a first result of overlapping image processing information generated from result values and (b) a second result obtained as a user event signal for an image frame may be output together. have.

Regarding (a), the controller 121 generates a heat map based on the result values output by performing an operation using the neural network model 121a from the image frame, and applying a threshold value to the heat map from the heat map. By extracting (or separating) the lesion area, it is possible to detect the lesion area from the image frame. In addition, with reference to the detected lesion area, the first result in which image processing information including image information of the boundary information and/or result values of the lesion area is overlapped on the image frame may be output to the display unit 124. . Details of this will be described later in FIG. 11.

With respect to (b), when it is determined that the first user event signal for capturing the image frame is input, the controller 121 performs the capturing of the image frame as shown in FIG. 10A. I can. And, when it is determined that a second user event signal for storing the captured image generated by the capture of the video frame is input, an index image corresponding to the captured image is generated (Fig. )), the captured image is stored as a second result (FIG. 10(c)) and can be output to the display unit 124.

As a result, it is possible to determine whether an event by the user has occurred in the medical imaging device 110 by directly analyzing the change information on the image frame in the image receiving device 120 and using only the corresponding result.

Hereinafter, the output of the first result and the second result according to the embodiment will be described with reference to FIGS. 3 to 10.

According to an embodiment, the control unit 121 includes (a) a first result of overlapping image processing information generated from result values on an image frame, (b) a second result obtained as a user event signal for the image frame, (c) Heat map image (c1,c2), (d) video information of (b), (e) index image and other patient information, environment setting information, connection status with the medical imaging device 110, etc. 124). In addition, each of the pieces of information may be divided and displayed in a separate box area on the entire display screen. And, in particular, in (a) and (b), the borders are highlighted and provided to the user, and the border colors of (a) and (b) are treated with different colors to easily identify them from each other. can do. In addition, it is possible to inform the user that the image corresponds to the recommended minimum inspection image item through the border color of (b). For example, it can be seen that (b) images arranged in the first to third rows correspond to the minimum examination image items, and (b) images arranged in the fourth row do not correspond to the minimum examination image items. Regarding (c), the heat map image may be displayed by overlapping on the image frame (c1), or may be displayed as a separate image separate from the image frame (c2).

In addition, the controller 121 may list and output a set including the first result (a) and the second result (b) on the screen of the display unit 124 in row units.

In the case of FIG. 3, it can be seen that the first set corresponding to the first lesion is arranged in the first row, and the second set corresponding to the second lesion is arranged in the second row.

Meanwhile, with respect to (e), an index image may be generated for each lesion. In addition, each index image generated for each lesion distinguishes borders with different colors so that it is possible to identify whether the corresponding index image is a recommended minimum examination image item and/or whether it corresponds to a lesion area. For example, in the case of (e1) and (e2), the index image represents a plurality of borders in different colors, so that it is possible to identify that the index image is the recommended minimum examination image item and corresponds to the lesion area at the same time. . In addition, in the case of (e3), since the index image represents only one border, it is possible to identify that the index image corresponds to the recommended minimum inspection image item.

According to another embodiment, depending on whether the corresponding index image corresponds to (a) and/or (b), the border of the index image is the same as the border of (a) and/or (b). You can highlight them with color so they can be distinguished from each other. According to an embodiment, when the lesion corresponds to (a) and (b) as well, each border may be displayed together in the index image. For reference, (b) can also be identified as a recommended minimum inspection image item.

On the other hand, the control unit 121 generates two identical indicators corresponding to the image processing information together with an image of the front wall of the digestive tract or an image f1 of the rear wall of the digestive tract and an image f2 in which the digestive tract is cut. , The display unit 124 displays one of the two indicators on the image of the front wall or the image of the rear wall, and displays the other of the two indicators on the image of the digestive tract cut out. can do.

According to FIG. 3, when it is difficult to accurately determine whether the lesion of the indicator occurred in either the anterior wall or the posterior wall only with the image of the incision in the digestive tract, it is possible to refer to the anterior wall image or the posterior wall image together. You will be able to determine the location. In addition, it is possible to implement different shapes, such as color, size, and pattern, of each indicator.

In the case of FIG. 3, by checking the location of each lesion of the digestive tract in the form of two different types of images, not only can it be confirmed in more detail and precisely, but also can be confirmed more intuitively.

According to an embodiment, the control unit 121 generates a plurality of sets including the first result and the second result, and outputs only the set selected by the user among the plurality of sets on the screen of the display unit 124 can do. In addition, the controller 121 may enlarge and output any one selected by the user among the first result and the second result included in the set selected by the user on the screen of the display unit 124.

As shown in FIG. 4A, the control unit 121 may output only a set for the first lesion on the screen of the display unit 124 and enlarge and output the first result selected by the user. According to an embodiment, the first result or the second result selected by the user may be implemented so that the video can be reproduced by the user selection.

According to another embodiment, as shown in FIG. 4B, the first result selected by the user is more clearly displayed by blurring or reducing the image for other items other than the first result selected by the user. Can be provided to.

Meanwhile, in FIGS. 4A and 4B, a separate mark is added to the border area of the index image corresponding to the first lesion selected by the user (eg, a dotted box) to inform the user that the corresponding first lesion is selected. I can tell.

In addition, in FIGS. 4A and 4B, the same indicator as shown in FIG. 3 is displayed on the image of the anterior wall or the posterior wall of the digestive tract and the image of the digestive tract incision, respectively, and the recommended photographing position of the lesion image. The part corresponding to is also displayed as an indicator and can be provided to the user.

Meanwhile, as shown in FIG. 5, at least a part of a set corresponding to a lesion selected by a user among a plurality of lesions may be uploaded to a hospital server (not shown), or unnecessary images that are not selected may be deleted.

In addition, in FIG. 5, indicators corresponding to lesions selected by a user may be displayed on an image of an anterior wall or a posterior wall of the digestive tract, and an image of a shape in which the digestive tract is cut.

As shown in Fig. 6, after selecting the moving picture information related to the second result obtained as the user event signal for the image frame of Fig. 3 (d) and storing it in accordance with the DICOM standard through various storage media (e.g., DVD) So it can be provided to the patient. However, this is only an exemplary embodiment, and at least one of FIGS. 3A to 3C, or a recommended minimum examination image or an entire examination video may be provided to a patient through a storage medium.

(a) is an image of the anterior wall or posterior wall of the digestive tract in the case of the image in which the lesion was found, along with the first and second results of the lesion, and the portion corresponding to the location of the lesion discovery and the recommended minimum imaging location. And it shows that the digestive tract is displayed with indicators on each of the cut-out image.

(b) is an image of the anterior wall or posterior wall of the digestive tract, along with the first and second results for the image, and the portion corresponding to the recommended minimum photographing position in the case of an image in which no lesion is found. It shows that the digestive tract is marked with indicators on the cut-out image.

As shown in FIG. 7, the test result may be stored and managed as an encrypted digital document, and the corresponding information may be stored in the memory 123 in the form of a separate report.

For example, the report may include at least one of patient information, information on the number and location of lesions, information on a recommended minimum examination site, and information on analysis software related to a neural network model.

Particularly, in addition to the first and second results of the lesion described above in FIGS. 3 to 5, an image of the anterior wall of the digestive tract or an image of the posterior wall of the digestive tract and the above A report may also be provided with indicators on each image of the digestive tract incision.

According to the embodiments described above with reference to FIGS. 3 to 5, 1) not only an endoscope image provided in real time, but also 2) a pre-stored endoscope image may be processed and/or managed similarly.

For example, 1) at least one of a first result and a second result may be obtained while providing a real-time endoscope image, and it may be stored together with the user's event information to aid in diagnosis in a later analysis process. And, as the test result, based on the event that the user proceeded, (1) the test information, (2) the captured image, (3) the time taken by the test equipment, and (4) the prediction result through the neural network model 121a (a Predicted probability, b. lesion stroke, c. activation map, d. position in anatomical structure, e. recommended image, etc.) It can be connected and saved as a file that can be managed separately.

According to another embodiment, 2) at least one of the first result and the second result may be obtained by reading a previously stored endoscope image, and even at this time, it is stored together with the user's event information to aid in diagnosis in a later analysis process. You can do this. And, as a result of the inspection, the above-described (1) to (4) may be applied in the same/similar manner.

In the above 1) and 2), analysis can be performed not only in a local PC but also in a cloud environment, and processing can be performed faster than real-time processing (for example, a processing speed of 30 frames or more per second using a GPU). In addition, when a certain amount of analysis is performed, the analysis can be performed simultaneously while checking the results.

According to an embodiment, the controller 121 may receive a plurality of user event signals and store second results corresponding to each of the plurality of user event signals for each user event signal.

According to the embodiments of FIGS. 3 to 7, it is possible to more accurately confirm the location of the lesion by checking the location of the lesion through various types of UI. In addition, when the user checks the output image, it is possible to more intuitively determine the location of the lesion.

In particular, (a) the first result in which the image processing information generated from the result values overlapped on the image frame and (b) the second result obtained as a user event signal for the image frame are displayed together, so that the user works directly. There is an advantage that a more accurate analysis can be made by allowing one result and the result analyzed through the neural network model to be compared and judged.

In addition, (a) (b) results of the analyzed results are displayed in the anatomical structure, and by providing a separate report while providing a video file, the result management can be carried out in various forms and can be stored more safely. There is an advantage.

In addition to simply indicating the location of the lesion, it is possible to provide convenience for the examination by providing information on a site with a risk factor such as perforation.

According to an embodiment, the control unit 121 may output time information related to at least one of the first result and the second result in the form of a time-line on the screen of the display unit 124. .

FIG. 8 is a diagram referenced to illustrate a timeline output on the screen of the display unit 124, and the examples described above in FIGS. 3 to 5 may be applied in the same/similar manner.

(a) shows a first result of overlapping image processing information generated from result values on an image frame, and (b) shows a second result obtained as a user event signal for the image frame.

An image corresponding to the selected indicator (including at least one of (a) and (b)) when each of the indicators displayed on the image of the front wall or the back wall of the digestive tract and the image of the digestive tract is selected Can be printed.

Alternatively, when an index image corresponding to the lesion is selected, an image (including at least one of (a) and (b)) corresponding to the lesion may be output.

(B ₁ ) is a box area containing an image corresponding to the selected indicator and/or index image, and the box area includes buttons (forward-forward, rewind, stop, double-speed playback, etc.) for controlling video playback. Thus, it is possible to control the reproduction of images included in the box area through the user's control of the corresponding button. In addition, it is possible to control the image through direct selection of (a) or (b).

Meanwhile, referring to FIG. 8, the box area B ₁ is time information such as a test start time (s), a current time (t1), and a time elapsed from the test start (s) (t2) that the endoscopy equipment inspects. It may include. Depending on the embodiment, the corresponding time and time information may be confirmed by applying a technique such as optical character recognition (OCR).

(B2) is a separate box area including the timelines (T1, T2), and the output result of (a) and/or (b) is displayed while moving the slide line (S) in the timeline (T1, T2). I can confirm.

On the other hand, referring to FIG. 8, (T1) represents a timeline when an operation using a neural network model and image acquisition due to a user event signal are performed together, and (T ₂ ) is only an operation using a neural network model without a user event signal. Shows the timeline when this is performed. And, the slide line S indicates the position of the current analysis screen.

In (T ₁ ), a bar bar indicating that the first result obtained by the operation using the neural network model for the first lesion is output at the time is displayed at the lower part of the timeline, and the first lesion is displayed at the upper part of the timeline. A bar bar indicating that a second result obtained as a user event signal is output at a corresponding time may be displayed. As described above, the first result can also be identified as a recommended minimum inspection position.

In (T ₂ ), a bar bar indicating that the first result obtained by the operation using the neural network model for the first lesion is output at a corresponding time is displayed at the lower part of the timeline, and the neural network model is displayed at the upper part of the timeline. A recommended minimum inspection position obtained by the used calculation can be displayed.

In (T ₁ ) and (T ₂ ), the bar bars may be displayed in different sizes depending on the elapsed time interval.

Meanwhile, FIG. 8 illustrates that an index image and/or an indicator corresponding to the first lesion is selected and the corresponding image is output to the box area, and the location of the lesion and the examination time information on the timeline are interlocked and displayed together. can do.

According to an embodiment, there are a plurality of neural network models 121a, and different types of result values may be output through an operation using each of the plurality of neural network models. A plurality of neural network models according to an embodiment include (a) a lesion detection model, (b) a lesion type determination model, (c) a location information determination model of the lesion on the image frame, and (d) location information of the lesion. It may be any one of a model for determining whether to proceed with a test related to the lesion, (e) a recommended test time notification model, and (f) a risk notification model of the lesion occurrence. However, this is an example and can be applied in the same/similar manner to the present invention if it is a model implemented in relation to lesion detection.

On the other hand, (c) in the case of a model for determining location information of a lesion on an image frame, if the control unit 121 determines that the lesion has been removed in the process of driving the model by performing an operation to determine the location of the lesion , When the removal state of the lesion is recorded in the memory 123, and when the operation is performed through the model for a new lesion, the lesion removal state information (including location information) recorded in the memory 123 is read and a new It can be controlled not to determine whether the lesion is present at the location of the digestive tract.

For example, when the first lesion located in the stomach and bottom is deleted, the model does not determine whether it is in the stomach and in the stomach when driving the model for the second lesion, but only determines whether it is in the stomach or in the anterior portion of the stomach. Can be implemented.

By doing this, unnecessary calculations are reduced and faster and more efficient calculations are possible.

9 illustrates a screen of the display unit 124 displaying a region of a digestive tract from which a lesion has been removed according to an embodiment.

When the lesion is removed by surgery or the like, the portion of the digestive tract from which the lesion is removed may be marked differently from the rest of the normal portion to be identified from each other. The top portion and the removed portion may be displayed in different shapes such as different colors or patterns. In this case, the user can easily identify the portion of the digestive tract that has been removed.

11 is a diagram referred to for describing a process of outputting image processing information generated from result values on an image frame using a plurality of neural network models according to an embodiment.

Referring to FIG. 11, when an image frame is input to an input node of a neural network model 121a such as a lesion detection model or a lesion determination model, inference is performed in the neural network model 121a and result values (L,N, M, SM, D, UD) can be output. In addition, a model for determining location information of a lesion on an image frame, a model for determining whether to perform an examination related to a lesion corresponding to the location information of the lesion, a recommended examination time notification model, and a risk notification model for the occurrence of the lesion are implemented in relation to lesion detection. If it is a model, the same/similar application can be applied to the present invention. For reference, according to an embodiment, when a plurality of neural network models 121a are operated, each neural network model 121a may be simultaneously processed in parallel.

In addition, Fig. 11 illustrates that the neural network model 121a uses a convolutional neural network (CNN), but the present invention is not limited thereto, and a neural network model generated from various types of neural networks such as FCN, RNN, DNN, GNN, etc. The same/similar application can be applied to.

And, by performing an operation using the neural network model 121a from the image frame, generating a heat map based on the output results, and extracting (or separating) the lesion area from the heat map by applying a threshold value to the heat map, The lesion area can be detected from the image frame. In addition, with reference to the detected lesion area, image processing information including image information of boundary information and/or result values of the lesion area may be overlapped and output on the image frame.

Hereinafter, features of the neural network model of FIG. 11 will be described with reference to FIG. 12.

The (1) lesion detection model of FIG. 11 represents a one-to-one classification model according to an embodiment, and the (2) lesion determination model of FIG. 11 represents a one-to-many classification model according to the embodiment.

In FIG. 11, M (Mucosa) and SM (Submucosa) represent characteristics belonging to class I (depth of lesion), and D (Differentiated) and UD (Undifferentiated) represent characteristics belonging to class II (type of lesion). . The classification criteria for Class I and Class II may be different. In this specification, characteristics belonging to a plurality of classes are referred to as complex characteristics. Inputs with complex characteristics may have a plurality of characteristics, that is, characteristics for each class.

Referring to the lesion discrimination model of FIG. 11 (2), in the one-to-many model, some layers, that is, a convolutional layer and a pooling layer, are shared between classes, and a complete coupling layer is provided for each class. At this time, since the fully coupled layer is provided for each class, the sum of the predicted probabilities of the features M and SM is 100%, and the sum of the predicted probabilities of the features D and UD is 100%. Therefore, since the prediction result in class I and the prediction result in class II are provided respectively, a more accurate prediction result can be obtained. In addition, since the convolution layer is shared between classes, the correlation between classes can be reflected while further reducing the computational amount.

12 is a diagram showing the structure of a neural network model 121a according to an embodiment of the present invention. In FIG. 12, there are NT classes (T ₁ , T ₂ , ... T _t , ... T _NT ), class T ₁ has two characteristics, and class T ₂ has two characteristics. It was assumed that it belongs to, and n characteristics belong to class T _t , and three characteristics belong to class T _NT.

Referring to FIG. 12, a neural network model 121a according to an embodiment of the present invention includes a convolutional layer 10 for performing a convolution operation on an input image xi, and the convolutional layer 10. Each corresponding to a pooling layer 20 for performing pooling on the output (oconv) and a plurality of classes into which complex characteristics are classified, and a weight for each class (w _fc (T _{t)) for the output of the pooling layer 20} It includes a plurality of fully coupled layers 30 for each class outputting a value multiplied by )).

The neural network model 121a corresponds to each of the plurality of fully coupled layers 30 for each class and calculates a characteristic probability for each class according to the output of the fully coupled layer 30 for each class. It may further include.

The convolution layer 10 extracts a feature map by performing a convolution operation on an input image using a plurality of convolution filters. As shown in FIG. 12, the convolution operation may be performed a plurality of times. It is assumed that the output of the convolutional layer 10, that is, the feature map (oconv), has a height (H), a width (W), and the number of channels (C).

The pooling layer 20 is located between the convolutional layer 10 and the fully coupled layer 30, and reduces the size of the feature map (oconv) to reduce the operation required in the fully coupled layer 30 to be described later, It serves to prevent overfitting. The pooling layer 20 may perform global average pooling in which an average value is output for each channel of the feature map oconv.

The class-specific fully coupled layer 30 has weights for each class (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _{fc) on the output of the pooling layer 20} (T _NT )) multiplied by it. At this time, each of the weights for each class (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _fc (T _NT )) is a plurality of Can be a value.

The classifiers 40 for each class correspond to each of the fully coupled layers 30 for each class, and calculate a characteristic probability for each class according to the output of the fully coupled layer 30 for each class. 12, the sorter for the class T ₁ corresponds to the class T probability corresponding to the characteristic of belonging to the _{1 (P 1 (T 1)} , P 2 (T 1)) computing a, and class T ₂ classifier to classifier for the class T ₂ probability _{(P 1 (T 2),} P 2 (T 2)) computing a, and class T _t for the characteristics belonging to the each of the characteristics belongs to the class T _t probability that corresponds to the sorter for calculating the _{(P 1 (T t),} P 2 (T t), ... Pn (T t)) and corresponds to the class T _NT is equivalent to each of the attributes belonging to the class T _NT Calculate the probability (P ₁ (T _NT ), P ₂ (T _NT ), P ₃ (T _NT )). As the class-specific classifier 40, for example, a Softmax function, a Sigmoid function, or the like may be used.

The above-described embodiments may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. optical media), and hardware devices specially configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to execute the processing according to the present invention, and vice versa.

Aspects of the present specification may take the form of hardware as a whole, software as a whole (including firmware, resident software, microcode, etc.), or a computer program product embodied in one or more computer-readable media in which computer-readable program code is implemented. .

Features, structures, effects, and the like described in the above embodiments are included in one embodiment of the present invention, and are not necessarily limited to only one embodiment. Furthermore, the features, structures, effects, and the like illustrated in each embodiment may be combined or modified for other embodiments by a person having ordinary knowledge in the field to which the embodiments belong. Therefore, contents related to such combinations and modifications should be construed as being included in the scope of the present invention.

Claims (11)

A communication unit for receiving an image frame from a medical imaging device for photographing the digestive organs;
A control unit for outputting result values by performing an operation using a neural network model on the image frame; And
A display unit for outputting at least one of a first result in which image processing information generated from the result values overlapped on the image frame and a second result obtained as a user event signal for the image frame; and
The control unit generates two identical indicators corresponding to the image processing information together with an image of the front wall or the rear wall of the digestive tract and an image of a shape in which the digestive tract is cut,
The display unit displays any one of the two indicators on the image of the front wall or the image of the rear wall, and displays the other of the two indicators on the image of the digestive tract cut out,
Video receiving device.
The method of claim 1,
The control unit,
Listing and outputting a set including the first result and the second result in row units on the screen of the display unit,
Video receiving device.
The method of claim 1,
The control unit,
Generating a plurality of sets including the first result and the second result,
Outputting a set selected by a user from among the plurality of sets on the screen of the display unit,
Video receiving device.
The method of claim 3,
The control unit enlarges and outputs any one selected by the user from among the first result and the second result included in the set selected by the user on the screen of the display unit,
Video receiving device.
The method of claim 1,
The user event signal,
To perform the capture of the image frame,
Video receiving device.
The method of claim 1,
The control unit receives a plurality of user event signals,
Storing second results corresponding to each of the plurality of user event signals for each user event signal,
Video receiving device.
The method of claim 1,
The control unit outputs time information related to at least one of the first result and the second result in the form of a time-line on the screen of the display unit,
Video receiving device.
The method of claim 1,
There are a plurality of neural network models,
Outputting different types of result values through the operation using each of the plurality of neural network models,
Video receiving device.
The method of claim 8,
One of the plurality of neural network models is a model for detecting the location of a lesion of the digestive tract,
When it is determined that the lesion has been removed, the control unit records the removal state of the lesion in the memory, and does not determine whether a new lesion exists at the location of the digestive tract through a model for detecting the location of the lesion. ,
Video receiving device.
delete The method of claim 1,
The controller outputs at least one of an index image corresponding to the first result and an index image corresponding to the second result on the screen of the display unit,
Video receiving device.

Images