KR20210089983A - Apparatus and method for comparing similarity between surgical video based on tool recognition - Google Patents

Abstract

Disclosed are a similarity comparing device between surgical videos based on tool recognition and a method thereof. According to an embodiment of the present application, the similarity comparing device between surgery videos based on tool recognition comprises the following steps of: obtaining surgery-related metadata and tool appearance information of each of an inputted reference surgery video and a comparison target surgery video; and calculating similarity between the reference surgery video and the comparison target surgery video based on the metadata and the tool appearance information.

Description

Apparatus and method for comparing similarity between surgical videos based on tool recognition {APPARATUS AND METHOD FOR COMPARING SIMILARITY BETWEEN SURGICAL VIDEO BASED ON TOOL RECOGNITION}

The present application relates to an apparatus and method for comparing similarity between surgical videos based on tool recognition.

Recently, endoscopic equipment with a camera is used in various fields, and as imaging equipment inserted into the body of a subject or photographing a procedure or surgery site has been miniaturized and universalized, there are more cases of acquiring surgical videos in internal and external surgery. Acquired surgical videos are only used in extremely limited circumstances, such as presentations at academic conferences.

In addition, with the recent development of big data and artificial intelligence, numerous medical document records and photos are developing into important medical information. the current situation.

In addition, since surgical videos contain professional medical behavior data, they are highly valuable as medical education and academic research data. , surgical evaluation and validation models, etc. can be established.

In addition, if you want to provide surgery-related knowledge through surgery videos, such as the type of lecture in the medical school curriculum, research areas in hospitals, and review of reference videos for specific surgery prior to surgery, surgery-related knowledge is the most professional knowledge among medical technologies. Therefore, it is required to establish an efficient standard model for surgery through discussion and correction of surgical videos by various medical staff. However, no platform that supports the exchange of opinions on surgery videos has been introduced so far. In order to compare the surgical videos with each other, it is necessary to watch all the video data for a long time or to search the necessary parts one by one, which is very cumbersome and takes a long time.

The technology that is the background of the present application is disclosed in Korean Patent Publication No. 10-1111046.

The present application is to solve the problems of the prior art described above, and it is a tool recognition-based surgery that can recognize and data the tools used in the corresponding surgery from the surgical video, and compare the similarity and correlation for different surgical videos based on this. An object of the present invention is to provide an apparatus and method for comparing similarity between videos.

The present application is to solve the problems of the prior art described above, and by providing a wiki-based web platform that supports discussion and correction of uploaded surgical videos, intelligent knowledge sharing about surgery videos that can be provided by capitalizing surgery-related knowledge The purpose is to provide a system.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the tool recognition-based similarity comparison method between surgery videos according to an embodiment of the present application, each of the input reference surgery video and the comparison target surgery video, surgery-related metadata and tools appear It may include obtaining information and calculating a similarity between the reference surgery video and the comparison target surgery video based on the metadata and the tool appearance information.

In addition, the tool recognition-based similarity comparison method between surgical videos according to an embodiment of the present application includes the steps of determining a correlation of the comparison target surgery video to the reference surgery video based on the similarity, and comparing the determined association relationship It may include the step of uploading the comparison target surgery video to a pre-established wiki (Wiki)-based web platform, including including.

In addition, the surgery-related metadata includes surgery information including operation type information, operation time information, operation location information, and surgical medical staff information associated with the reference operation video or the comparison target operation video and the reference operation video or the comparison It may include patient information including personal information information of the patient and disease information of the patient associated with the target surgery video.

In addition, the tool appearance information may include tool trajectory information associated with a motion of a tool used for a predetermined operation and surgical action information inferred based on the motion of the tool.

In addition, the step of obtaining the tool appearance information may include dividing the reference surgery video or the comparison target surgery video into a plurality of index images, identifying the tool appearing in each of the divided index images, the identified The method may include deriving the tool trajectory information for the tool and deriving the surgical action information corresponding to the tool trajectory information by inputting the tool trajectory information into a pre-generated artificial intelligence algorithm.

In addition, in the determining of the correlation, the correlation may be determined by further considering frame similarity between the representative frames for each of the plurality of index images for each of the reference surgical video and the comparison target surgery video.

In addition, the tool recognition-based similarity comparison method between surgery videos according to an embodiment of the present application may include uploading the reference surgery video to the wiki-based web platform.

In addition, the step of uploading the reference surgery video may be performed through a content creator terminal to which authentication authority is previously granted according to a predetermined condition.

In addition, the tool recognition-based similarity comparison method between surgical videos according to an embodiment of the present application receives feedback data for the comparison target surgery video from at least one or more participant terminals for the wiki-based web platform, and the feedback data It may include correcting at least one of the correlation and the degree of similarity with respect to the comparison target surgery video based on the .

In addition, the tool recognition-based similarity comparison method between surgery videos according to an embodiment of the present application may include providing the comparison target surgery video to a content consumer terminal linked to the wiki-based web platform.

On the other hand, the apparatus for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application includes an image information extracting unit that obtains operation-related metadata and tool appearance information for each of an input reference surgery video and a comparison target surgery video, and the and a similarity calculator configured to calculate a degree of similarity between the reference surgery video and the comparison target surgery video based on metadata and the tool appearance information.

In addition, the apparatus for comparing the degree of similarity between surgical videos based on tool recognition according to an embodiment of the present application determines a relationship between the comparison target surgery video and the reference surgery video based on the similarity, and includes the determined relationship Thus, it may include an uploading unit for uploading the comparison target surgery video to a pre-established wiki (Wiki)-based web platform.

In addition, the image information extraction unit divides the reference surgery video or the comparison target surgery video into a plurality of index images, identifies the tools appearing in each of the divided index images, and the tools for the identified tools It is possible to derive the trajectory information of the tool and input the tool trajectory information into a pre-generated artificial intelligence algorithm to derive the surgical action information corresponding to the tool trajectory information.

In addition, the upload unit may receive the reference surgery video from the content creator terminal to which the authorization is granted in advance according to a predetermined condition, and upload the received reference surgery video to the wiki-based web platform.

In addition, the apparatus for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application receives feedback data for the comparison target surgical video from at least one participant terminal for the wiki-based web platform, and the feedback data It may include a review unit for correcting at least one of the correlation and the degree of similarity with respect to the comparison target surgery video based on the.

In addition, the apparatus for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application may include a content providing unit that provides the comparison target surgical video to a content consumer terminal linked to the wiki-based web platform.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, tool recognition-based similarity comparison between surgical videos that can recognize and data the tools used in the corresponding surgery from the surgical video and compare the similarity and correlation for different surgical videos based on this Apparatus and methods may be provided.

According to the above-described problem solving means of the present application, by providing a wiki-based web platform that supports discussion and correction of uploaded surgical videos, it is possible to provide an intelligent knowledge sharing system for surgical videos that can capitalize and provide surgical-related knowledge. can

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a schematic configuration diagram of an intelligent knowledge sharing system for a wiki-based surgical video according to an embodiment of the present application.
2 is a conceptual diagram for explaining a process of generating and providing surgical video-based content by an intelligent knowledge sharing system for a wiki-based surgical video according to an embodiment of the present application.
3 is a conceptual diagram for explaining a wiki-based web platform according to an embodiment of the present application.
4 is a view for explaining a pre-processing process for an input surgical video according to an embodiment of the present application.
5 is a view showing information about the appearance of a tool in a specific frame in a surgical video by way of example.
6 is a view comprehensively showing information about a plurality of tools appearing in a specific surgical video.
7 is a diagram exemplarily illustrating a user interface for providing a degree of similarity or correlation between a reference surgery video and a comparison target surgery video.
8 is a schematic configuration diagram of an apparatus for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application.
9 is an operation flowchart for a method for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to an apparatus and method for comparing similarity between surgical videos based on tool recognition.

1 is a schematic configuration diagram of an intelligent knowledge sharing system for a wiki-based surgical video according to an embodiment of the present application.

1, an intelligent knowledge sharing system 10 for a wiki-based surgical video according to an embodiment of the present application is a tool recognition-based similarity comparison device 100 between surgical videos (hereinafter, 'similarity comparison device (hereinafter referred to as 'similarity comparison device)') 100)'), the content creator terminal 200 , the comparison target surgical video providing device 300 , the participant terminal 400 , and the content consumer terminal 500 .

The similarity comparison apparatus 100 , the content creator terminal 200 , the comparison target surgical video providing apparatus 300 , the participant terminal 400 , and the content consumer terminal 500 may communicate with each other through the network 20 . The network 20 refers to a connection structure in which information exchange is possible between each node, such as terminals and servers, and an example of such a network 20 includes a 3rd Generation Partnership Project (3GPP) network, a long-term LTE (LTE) network. Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but are not limited thereto.

In addition, the similarity comparison device 100 uploads a reference surgery video, a comparison target surgery video, etc. to be described later on the Wiki-based web platform 30 through the network 20, and various information about each uploaded surgery video ( tool appearance information, metadata, etc.) can be managed in an integrated way.

The content creator terminal 200, the comparison target surgical video providing device 300, the participant terminal 400, and the content consumer terminal 500 are, for example, a smartphone, a smartpad, a tablet PC, and the like and a PCS. (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple) Access)-2000, W-CDMA (W-Code Division Multiple Access), and Wibro (Wireless Broadband Internet) terminal may be any type of wireless communication device.

In addition, according to an embodiment of the present application, the apparatus for providing a comparison target surgery video 300 is a device for providing a comparison target surgery video that is a target of similarity analysis and correlation analysis for a reference surgery video to the similarity comparison device 100 . may mean For example, the comparison target surgical video providing apparatus 300 may be an imaging device, a recording device, etc. for recording a corresponding surgical scene when performing a specific surgery, such as laparoscopic equipment, robotic surgery equipment, and a surgical camera. As another example, the comparison target surgical video providing device 300 may mean a separate storage device (eg, external hard drive, etc.) provided in connection with the above-described imaging equipment, recording device, and the like. As another example, the comparison target surgical video providing apparatus 300 may be any one of the participant terminals 400 participating in the Wiki-based web platform 30 of the present application.

For reference, participating in the wiki-based web platform 30 in the description of the embodiment of the present application means having (registration) an account for the wiki-based web platform 30, or a provider of the wiki-based web platform 30 It means an act of providing a surgical video through the wiki-based web platform 30 or performing corrections, discussions, and opinions on the provided surgical video by being granted permission to upload a surgical video, provide feedback data on the surgical video, etc. can be understood as In this regard, depending on the type of authority granted to the participating subject of the wiki-based web platform 30 (eg, the owner of the participant terminal 400, etc.), the range of actions that can use the wiki-based web platform 30 is can be determined in various ways.

In addition, in the description of the embodiment of the present application, the content creator terminal 200 is pre-authorized according to a predetermined condition from the provider of the wiki-based web platform 30 and serves as a comparison standard for other surgical videos. It may mean a terminal capable of uploading a surgical video. As another example, the content creator terminal 200 refers to a terminal owned by a subject in a position to select a predetermined reference surgical video from among a plurality of surgical videos uploaded to the wiki-based web platform 30 based on the above-described authentication authority. can do. According to an embodiment, the authentication authority granted according to a predetermined condition may be granted to the plurality of content creator terminals 200 . For example, the content creator terminal 200 may be determined differently for each operation type, and a plurality of content creator terminals 200 may be designated for any one specific operation type. According to an embodiment of the present application, when a plurality of content creator terminals 200 are designated for a specific surgery type, the reference video set for the corresponding operation type is uploaded by each of the plurality of content creator terminals 200 individually. It may be selected or uploaded or selected through agreement between a plurality of content creator terminals 200 .

A predetermined condition for granting authentication authority to the content creator terminal 200 may be determined based on, for example, the medical history, status, rank, and number of operations performed for a specific operation of the user of the terminal.

In addition, in the description of the embodiment of the present application, the content consumer terminal 500 watches a surgical video uploaded to the wiki-based web platform 30 for learning purposes, research purposes, academic purposes, and for reference prior to a specific operation. It may be a terminal owned by the subject who wants to do so. According to an embodiment, the content consumer terminal 500 may correspond to a part of the participant terminals 400 . In addition, according to the authority granted by the provider of the wiki-based web platform 30 , the content consumer terminal 500 may be a terminal that does not have the authority to provide feedback data and can only watch the operation video.

2 is a conceptual diagram for explaining a process of generating and providing surgical video-based content by an intelligent knowledge sharing system for a wiki-based surgical video according to an embodiment of the present application.

Referring to FIG. 2 , the user (holder) of the above-described content creator terminal 200 may ^{correspond to the 1st} Creator shown in FIG. 2 , and the user of the participant terminal 400 is a co-worker shown in FIG. 2 . may correspond to, and the user of the content consumer terminal 500 may correspond to the Subscriber and Department, Hospital and University, etc. shown in FIG. 2 .

3 is a conceptual diagram for explaining a wiki-based web platform according to an embodiment of the present application.

3, the wiki-based web platform 30 is a video analysis and conversion system 31, a wiki web server (Wiki Web Server, 32), a video curation web server (Video curation Web server, 33), video storage 34 and a data DB 35 .

Specifically, according to an embodiment of the present application, the video analysis and conversion system 31 may generate indexing information by performing indexing on the uploaded surgical video. In addition, the video analysis and conversion system 31 may generate and store a compressed summary video (cataloging video) of the main scenes for the uploaded surgical video. 3 is to be understood as an exemplary diagram for explaining the function and operation of the wiki-based web platform 30 of the present application, and in this regard, according to an embodiment of the present application, some functions of the video analysis and conversion system 31 are It may be performed by the similarity comparison apparatus 100 of the present application or may be performed by overlapping with the video analysis and conversion system 31 .

In addition, according to an embodiment of the present application, the wiki web server 32 is provided to the above-described content creator terminal 200 , the comparison target surgical video providing device 300 , the participant terminal 400 , and the content consumer terminal 500 , etc. It is possible to manage various privileges and manage accounts for each terminal. In addition, the wiki web server 32 reflects this when the degree of similarity and correlation to a predetermined surgical video based on the feedback data provided by the participant terminal 400 is modified, as described later, to content information and video storage. Stored data can be updated.

In addition, according to an embodiment of the present application, the video curation web server 33 stores similarity, correlation, metadata, etc. with respect to a reference surgical video determined for a plurality of surgical videos uploaded to the video storage 34 as a data DB Obtained from (35), classify and categorize a plurality of surgical videos based on the obtained similarity, correlation, metadata, etc., and provide the surgical video search results to the content consumer terminal 500 based on the classification and categorization results It may mean a server provided to provide and support download of a specific surgical video or to provide a streaming service.

Hereinafter, specific operations and functions of the similarity comparison apparatus 100 will be described in detail with reference to FIGS. 4 to 7 .

First, the similarity comparison apparatus 100 obtains (receives) a reference surgery video from the content creator terminal 200 to which authentication is granted in advance according to a predetermined condition, and uses the received reference surgery video on the wiki-based web platform 30 can be uploaded to

In addition, the similarity comparison device 100 acquires (receives) a comparison target surgery video, which is a target of similarity analysis and correlation between the surgery video, from the comparison target surgery video providing device 300 based on the received reference surgery video. can

The similarity comparison apparatus 100 may acquire operation-related metadata and tool appearance information for each of the input reference surgery video and the comparison target surgery video.

According to an embodiment of the present application, the aforementioned surgery-related metadata may include surgery information and patient information. Here, the operation information includes operation type information, operation time information, operation location information, and surgical staff information associated with a reference operation video or a comparison target operation video, and the patient information is a patient associated with a reference operation video or a comparison target operation video personal information of the patient and the patient's disease information, but is not limited thereto.

According to an embodiment of the present application, the aforementioned tool appearance information may include tool trajectory information associated with the motion of a tool used for a predetermined operation and surgical action information inferred based on the motion of the tool.

In this regard, the similarity comparison apparatus 100 may first divide a reference surgery video or a comparison target surgery video into a plurality of index images. Here, the index image may refer to a plurality of clip images divided based on the set boundary by setting a boundary between successive similar images corresponding to the characteristics of the reference surgical video or the comparison target surgical video. The similarity comparison apparatus 100 does not mutually compare the reference surgery video and the comparison target surgery video for the entire section, but divides each of the reference surgery video and the comparison target surgery video into a plurality of index images, and compares the index images corresponding to each other. By performing the analysis, it is possible to minimize the operation of the CPU for calculating the similarity.

Specifically, the similarity comparison apparatus 100 detects a boundary frame based on the similarity between adjacent frames of each of the reference surgery video or the comparison target surgery video, and based on the detected boundary frame, the reference surgery video or the comparison target surgery video may be divided into a plurality of index images, respectively.

According to an embodiment of the present application, the similarity between adjacent frames may include a structural similarity between adjacent frames and a color histogram similarity between adjacent frames. However, the present invention is not limited thereto, and the similarity between adjacent frames may be calculated in various ways based on color information between adjacent frames and information on appearance objects (eg, surgical tools, etc.).

Also, when the structural similarity between two adjacent frames is less than a preset first threshold or the color histogram similarity is less than a preset second threshold, the similarity comparison apparatus 100 selects any one of the two adjacent frames as the border frame. can operate to detect.

According to an embodiment of the present application, at least one of the first threshold value and the second threshold value is the variance information of the degree of similarity calculated within a window set as a predetermined sequence section of the reference surgical video or the comparison target surgery video. It may be adaptively determined using

For example, a predetermined sequence period for window setting may be 7 frames, but is not limited thereto. As another example, the predetermined sequence section for setting the window may include a different number of frames selectively according to the type of the reference surgery video or the comparison target surgery video.

4 is a view for explaining a pre-processing process for an input surgical video according to an embodiment of the present application.

Referring to FIG. 4 , the similarity comparison apparatus 100 may perform preprocessing for extracting features for each frame in an input surgical video (reference surgery video and comparison target surgery video) prior to index image segmentation process. . Specifically, FIG. 4(a) shows a predetermined original frame of a surgical video input to the similarity comparison device 100, and FIG. 4(b) shows the organ of interest from the original frame inputted by the similarity comparison device 100. The results of segmentation and display are shown, and FIG. 4C shows that the similarity comparison apparatus 100 extracts features such as lesions and lesions based on the segmentation result of the organs of interest and displays them.

That is, the similarity comparison apparatus 100 may determine the characteristic information (position and shape of the organ of interest in the frame, the position and shape of the lesion) for each frame of the surgical video, and furthermore, the similarity comparison apparatus 100 may determine the Based on the feature information, it can be used to select a boundary frame, which is a criterion for indexing image segmentation, or to identify an appearance tool in each frame.

5 is a view showing information about the appearance of a tool in a specific frame in a surgical video by way of example.

Referring to FIG. 5 , the similarity comparison apparatus 100 may identify a tool appearing in each of the divided index images. Specifically, according to an embodiment of the present application, the similarity comparison apparatus 100 may select a tool appearance frame including a predetermined tool from among a plurality of frames in the index image for each of the divided index images. Specifically, according to an embodiment of the present disclosure, the similarity comparison apparatus 100 may acquire an appearance time (appearance frame) and exit time (exit frame) of a predetermined tool for each tool. For reference, various tools that can appear in the surgery video may repeatedly appear and exit within the frame of the surgery video according to the type of surgery, the surgical environment, the proximity of the imaging device, and the like. Accordingly, the similarity comparison apparatus 100 according to an embodiment of the present application may operate to acquire a plurality of entry frames and exit frames for one tool. As an example for better understanding, it is searched based on the start frame (frame 0) of the surgical video, between the first appearance frame where the tool first appears and the first exit frame where the tool exits in the frame closest to the first appearance frame. A section can be identified as a section range in which the tool appears. Also, when the corresponding tool reappears after the first exit frame, similarly, a section between the second appearance frame and the second exit frame may be identified as a section range in which the corresponding tool appears second.

In other words, the appearing frame may mean a frame in which the tool appears in a state in which it does not appear in the operation video, and the exit frame is a frame in which the tool is in a state in which it does not appear in the operation video. can mean

Also, referring to FIG. 5 , when a tool is identified, the similarity comparison apparatus 100 may also calculate the identification accuracy of the corresponding tool. For example, referring to FIG. 5 , the identification accuracy may be calculated in % units.

6 is a view comprehensively showing information about a plurality of tools appearing in a specific surgical video.

Referring to FIG. 6 , the tools identified by the similarity comparison device 100 are Scissors, tweezers, an inhaler, an operator's hand, an endoscope, a catheter, a guide wire, a probe, Grasper, Bipolar, Hook, Clipper, Irrigator, Specimen It can include a wide range of various medical devices, tools, etc. that can be used in a surgical procedure, such as a bag. In addition, according to an embodiment of the present application, the similarity comparison apparatus 100 provides a corresponding surgical video for each predetermined tool based on metadata (eg, surgery type, disease subject, subject's age, etc.) for the corresponding surgical video. It may be to calculate the probability of appearing in , and perform identification of the tool based on the calculated probability. For example, when there is a tool that is not utilized in a specific surgical type, the similarity comparison apparatus 100 may operate such that the tool is not considered when identifying a tool for a surgical video for the corresponding surgical type.

Also, the similarity comparison apparatus 100 may derive tool trajectory information for the identified tool. Specifically, the similarity comparison apparatus 100 may acquire tool trajectory information based on a movement path of the tool in the tool appearance frame adjacent to the tool identified in the surgical video.

According to an embodiment of the present disclosure, the similarity comparison apparatus 100 may input the derived tool trajectory information into a pre-generated artificial intelligence algorithm to derive surgical action information corresponding to the corresponding tool trajectory information.

According to an embodiment of the present application, the similarity comparison apparatus 100 may analyze (derive) a surgical action based on tool trajectory information obtained based on a pre-generated artificial intelligence algorithm. According to an embodiment of the present application, the pre-generated artificial intelligence algorithm specifically performs a surgical action corresponding to the movement of the analyzed tool by inputting the analysis result of the movement within the tool appearance frame of the tool that can appear in the surgical video. It may be an algorithm provided as an output.

According to an embodiment of the present application, a pre-generated artificial intelligence algorithm uses a predetermined surgical video and information on a surgical action displayed in the corresponding surgical video as learning data, and provides an analysis result of the motion of a tool that can appear in the surgery video and the tool. It may be generated by a supervised learning-based machine learning method that learns a correlation (eg, a function form) with a surgical action corresponding to a movement. However, the present invention is not limited thereto, and the previously generated artificial intelligence algorithms in the present application are Logistic Regression, Random Forest, Deep Learning, and Support Vector Machine (SVM). ), artificial neural network (ANN), reinforcement learning, etc., may be generated (learned) through all kinds of machine learning methods that are known or developed in the future.

According to an embodiment of the present application, the aforementioned pre-generated artificial intelligence algorithm guides the annotations written by the operator (doctor, medical staff), the examiner for the operation video, etc. with respect to the operation video corresponding to the learning data. It may be learned as label data for the creation of a learning-based artificial intelligence algorithm.

According to an embodiment of the present application, the similarity comparison apparatus 100 analyzes a predetermined frame adjacent to frames in which a specific tool appears (specifically, frames from an appearance frame to an exit frame for the corresponding tool) and identifies a tool The movement of the tool may be grasped by deriving direction information, distance information, etc. moved within the frame set.

In addition, according to an embodiment of the present application, the similarity comparison apparatus 100 may operate to estimate the movement of the identified tool based on the shape change pattern of the affected part (surgery/operation site) of the subject in the operation video. Here, the shape change pattern of the affected part may be derived by using the characteristic information for each frame described above with reference to FIG. 4 . For example, the similarity comparison device 100 may detect bleeding occurrence or a situation in which an incised affected area is sutured based on the color information of the affected area of the subject as well as the movement information of the tool itself, and the detected By considering the shape change pattern of the affected part together with the movement information of the tool itself, it is possible to more accurately derive the surgical action associated with the tool.

For reference, the surgical action analyzed by the similarity comparison apparatus 100 may include, but is not limited to, ligation, dissection, resection, and suture.

Also, the similarity comparison apparatus 100 may calculate a similarity between the reference surgery video and the comparison target surgery video based on the obtained metadata and tool appearance information.

According to an embodiment of the present application, the similarity comparison apparatus 100 determines the operation type (type), disease type (type), etc. of the comparison target surgery video based on the metadata of the reference surgery video and the comparison target surgery video as the reference surgery video. , and if it is judged that it is inappropriate to compare the similarity between the reference surgery video and the comparison target surgery video (for example, when the type of surgery is slightly different, etc.), the similarity calculation process is not performed. , may operate to re-search the reference surgery video for the corresponding comparison target surgery video. On the other hand, when it is determined that the comparison target surgery video requires a significantly similarity determination with respect to the reference surgery video based on the metadata of the reference surgery video and the comparison target surgery video, the similarity comparison apparatus 100 is used in the two surgery videos as follows. similarity can be calculated.

According to an embodiment of the present application, the similarity comparison apparatus 100 may match the surgical action information identified with respect to each of the segmented index images for the reference surgery video and the comparison target surgery video. For example, when surgical action information is derived by 'ligation' so that a specific index image corresponds to each other between the reference surgery video and the comparison target surgery video, the similarity comparison apparatus 100 performs surgery on the reference surgery video and the comparison target surgery video. The behavior information may operate to compare between index images corresponding to 'ligation'.

Also, the similarity comparison apparatus 100 may calculate the similarity between the two index images based on the tool appearance type and the tool trajectory information in the two index images to which the surgical action information matches. Here, the tool appearance type may mean which tool is used for each corresponding surgical action, and the tool trajectory information may mean how the movement path of the tool and the use (movement) time of the tool are compared, respectively.

According to an embodiment of the present application, the similarity comparison device 100 sums up (comprehensively consider) the similarity between the index images that match each other for the reference surgery video and the comparison target surgery video, and adds it to the entire reference surgery video and the comparison target surgery video. similarity can be derived. In other words, the degree of similarity between the reference surgical video and the comparison target surgery video derived by the similarity comparison device 100 is the partial similarity calculated for each corresponding (matching) index image and the entire reference surgery video and the comparison target surgery video. may include an overall degree of similarity that is comprehensively considered with respect to , and the degree of similarity (partial similarity and total similarity) may be calculated based on a numerical value (eg, probability, etc.) assigned based on a predetermined criterion.

According to an embodiment of the present disclosure, the similarity comparison apparatus 100 may adjust a weight for calculating the partial similarity or the total similarity based on patient information among the metadata for the reference surgery video and the comparison target surgery video. For example, when patient information targeted for each operation is similar to or greater than a predetermined level based on metadata for two surgical videos (in other words, when patient type, patient gender, disease information, patient age, etc. are similar); Weights are adjusted so that partial or total similarity is calculated based on more stringent criteria, and conversely, when patient information targeted for each operation differs by more than a predetermined level (in other words, patient type, patient gender, disease information, patient age, etc.) is different), the weights may be adjusted so that partial or total similarity is calculated on a more relaxed basis. However, the present invention is not limited thereto, and the weight adjustment pattern when calculating the degree of similarity based on patient information may be variously changed based on the specificity of the type of surgery and the purpose of comparing the similarity.

In other words, the similarity comparison apparatus 100 operates to mutually compare tool-based events identified in the index image within the surgery video in consideration of respective metadata in order to calculate the similarity between the comparison target surgery video and the reference surgery video. can

Also, the similarity comparison apparatus 100 may determine a correlation between the comparison target surgery video and the reference surgery video based on the calculated similarity. For example, the correlation is based on the level (level) of similarity calculated for the comparison target surgery video and the reference surgery video, such as "video about the same surgery", "video with high relevance", "reference video", etc. It may mean tag information given to a comparison target surgical video. However, the name of each term (tag) indicating the above-described relationship is not limited thereto.

According to an embodiment of the present disclosure, the similarity comparison apparatus 100 may determine the correlation by considering the frame similarity between the representative frames for each index image for each of the reference surgery video and the comparison target surgery video. Here, the representative frame for each index image is, for example, a frame that has decisively contributed to deriving surgical action information for the index image (eg, a frame in which the movement of the tool is greatest between adjacent frames for the corresponding frame, etc.) can

Also, the similarity comparison apparatus 100 may upload a corresponding comparison target surgery video including the determined correlation relationship to the previously established Wiki-based web platform 30 .

In addition, the similarity comparison device 100 may receive feedback data for a comparison target surgery video uploaded on the wiki-based web platform 30 from at least one participant terminal 400 for the wiki-based web platform 30 . have. For example, the feedback data indicates the degree of similarity (partial similarity) for each index image between the two surgical videos provided by the similarity comparison device 100 of the present application, the overall similarity between the two surgical videos, and the appropriateness of the association determined for the surgical video. It may be data including information that correction is necessary, information on a correction direction, and the like if the partial similarity, the total similarity, or the determined correlation with respect to a specific index image is inappropriate. According to an embodiment of the present application, the feedback data may be received from each of the plurality of participant terminals 400 or may be received in a single form agreed upon through discussion and exchange of opinions among users of the plurality of participant terminals 400 .

Thereafter, the similarity comparison apparatus 100 may correct at least one of the correlation with the reference video and the degree of similarity with the reference video for the uploaded comparison target surgery video based on the received feedback data. According to an embodiment, the similarity comparison apparatus 100 provides feedback data to the content creator terminal 200, and based on an approval request for the feedback data received from the content creator terminal 200, a reference video for a comparison target surgery video The operation may be performed to correct at least one of a relationship with respect to and a degree of similarity to a reference video, but is not limited thereto.

Also, the similarity comparison apparatus 100 may provide a comparison target surgery video to the content consumer terminal 500 linked to the wiki-based web platform 30 .

7 is a diagram exemplarily illustrating a user interface for providing a degree of similarity or correlation between a reference surgery video and a comparison target surgery video.

According to an embodiment of the present application, the user interface for providing the similarity or correlation between the reference surgery video and the comparison target surgery video shown in FIG. 7 includes the participant terminal 400 for providing the aforementioned feedback data, the corresponding reference surgery The video may be initially uploaded or displayed on the display provided in the content creator terminal 200 or the content consumer terminal 500 to view the comparison target surgical video.

Referring to FIG. 7 , the degree of similarity or correlation between the reference surgery video and the comparison target surgery video may be displayed in the form of a graph comparing tool appearance information for each section (in other words, each index image segmentation section), but is limited thereto. it is not In addition, in the user interface for providing the similarity or correlation between the reference surgery video and the comparison target surgery video, information about the appearance of each tool according to the passage of time in the corresponding comparison target surgery video may be displayed in the form of a timeline.

8 is a schematic configuration diagram of an apparatus for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application.

Referring to FIG. 8 , the similarity comparison apparatus 100 may include an image information extracting unit 110 , a similarity calculating unit 120 , an uploading unit 130 , a reviewing unit 140 , and a content providing unit 150 . can

The image information extractor 110 may acquire operation-related metadata and tool appearance information for each of the input reference surgery video and the comparison target surgery video.

According to an embodiment of the present application, the image information extractor 110 may divide a reference surgery video or a comparison target surgery video into a plurality of index images. Also, the image information extractor 110 may identify a tool appearing in each of the divided index images. Also, the image information extractor 110 may derive tool trajectory information for each of the identified tools. In addition, the image information extractor 110 may input the derived tool trajectory information into a pre-generated artificial intelligence algorithm to derive surgical action information corresponding to each tool trajectory information.

The similarity calculator 120 may calculate a similarity between the reference surgery video and the comparison target surgery video based on the acquired metadata and tool appearance information.

The upload unit 130 may determine a correlation between the comparison target surgery video and the reference surgery video based on the calculated similarity. In addition, the upload unit 130 may upload the corresponding comparison target surgery video to the pre-established wiki (Wiki)-based web platform 30 to include the determined correlation.

The review unit 140 may receive feedback data for a comparison target surgery video uploaded from at least one participant terminal 400 for the wiki-based web platform. Also, the review unit 140 may correct at least one of a correlation between the uploaded comparison target surgery video and the reference surgery video and a degree of similarity with the reference surgery video based on the received feedback data.

The content providing unit 150 may provide a comparison target surgery video uploaded to the content consumer terminal 500 linked to the wiki-based web platform 30 .

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

9 is an operation flowchart for a method for comparing similarity between surgical videos based on tool recognition according to an embodiment of the present application.

The tool recognition-based similarity comparison method between surgical videos shown in FIG. 9 may be performed by the similarity comparison apparatus 100 described above. Therefore, even if omitted below, the description of the similarity comparison apparatus 100 may be equally applied to the description of the tool recognition-based similarity comparison method between surgical videos.

Referring to FIG. 9 , in step S910 , the upload unit 130 may obtain (receive) a reference surgical video from the content creator terminal 200 to which authentication authority is previously granted according to a predetermined condition. Also, in step S910 , the upload unit 130 may upload the received reference surgery video to the wiki-based web platform 30 .

Next, in step S920 , the image information extraction unit 110 may acquire surgery-related metadata and tool appearance information for each of the input reference surgery video and the comparison target surgery video.

Next, in step S930, the similarity calculator 120 may calculate a similarity between the reference surgery video and the comparison target surgery video based on the acquired metadata and tool appearance information.

Next, in step S940, the upload unit 130 may determine a correlation between the comparison target surgery video and the reference surgery video based on the calculated similarity.

Next, in step S950 , the upload unit 130 may upload the corresponding comparison target surgery video to the pre-established Wiki-based web platform 30 to include the determined correlation.

Next, in step S960, the review unit 140 may receive the feedback data for the comparison target surgery video uploaded from at least one participant terminal 400 for the wiki-based web platform.

Next, in step S970 , the review unit 140 may correct at least one of a correlation between the uploaded comparison target surgery video and the reference surgery video and the degree of similarity with the reference surgery video based on the received feedback data.

Next, in step S980 , the content providing unit 150 may provide a comparison target surgery video uploaded to the content consumer terminal 500 linked to the wiki-based web platform 30 .

In the above description, steps S910 to S980 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The tool recognition-based similarity comparison method between surgical videos according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the tool recognition-based similarity comparison method between surgical videos may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: Wiki-based intelligent knowledge sharing system for surgical videos
100: Device for comparing similarity between surgical videos based on tool recognition
110: image information extraction unit
120: similarity calculator
130: upload unit
140: review unit
150: content provider
30: Wiki-based web platform
200: content creator terminal
300: Comparison target surgical video providing device
400: participant terminal
500: content consumer terminal
20: network

Claims (15)

In a tool recognition-based similarity comparison method between surgical videos,
acquiring operation-related metadata and tool appearance information for each of the input reference surgery video and the comparison target surgery video; and
calculating a similarity between the reference surgery video and the comparison target surgery video based on the metadata and the tool appearance information;
Including, a similarity comparison method. According to claim 1,
determining a correlation between the comparison target surgery video and the reference surgery video based on the degree of similarity; and
Uploading the comparison target surgery video including the determined correlation relationship to a pre-established Wiki-based web platform;
Further comprising a, similarity comparison method. 3. The method of claim 2,
The surgery-related metadata is
Surgery information including operation type information, operation time information, operation place information, and surgical medical staff information linked to the reference operation video or the comparison target operation video and the patient's personal information linked to the reference operation video or the comparison target operation video The similarity comparison method, which includes patient information including information about the matter and the patient's disease information. 4. The method of claim 3,
The tool appearance information includes tool trajectory information associated with the motion of a tool used for a predetermined operation and surgical action information inferred based on the motion of the tool,
The step of obtaining the tool appearance information includes:
dividing the reference surgery video or the comparison target surgery video into a plurality of index images;
identifying the tool appearing in each of the segmented index images;
deriving the tool trajectory information for the identified tool; and
inputting the tool trajectory information into a pre-generated artificial intelligence algorithm to derive the surgical action information corresponding to the tool trajectory information;
A similarity comparison method comprising a. 5. The method of claim 4,
The step of determining the association relationship is
The similarity comparison method of determining the correlation by further considering frame similarity between the representative frames for each of the plurality of index images for each of the reference surgical video and the comparison target surgical video. 3. The method of claim 2,
Further comprising the step of uploading the reference surgery video to the wiki-based web platform,
The step of uploading the reference surgery video is,
A similarity comparison method, characterized in that it is performed through a content creator terminal to which authentication authority is previously granted according to a predetermined condition. 7. The method of claim 6,
Receive feedback data on the comparison target surgical video from at least one participant terminal for the wiki-based web platform, and modify at least one of the correlation and the similarity for the comparison target surgery video based on the feedback data A similarity comparison method further comprising the step of: 8. The method of claim 7,
The similarity comparison method, further comprising the step of providing the comparison target surgery video to the content consumer terminal linked to the wiki-based web platform. In an apparatus for comparing similarity between surgical videos based on tool recognition,
an image information extractor for acquiring surgery-related metadata and tool appearance information for each of the input reference surgery video and the comparison target surgery video; and
a similarity calculator configured to calculate a degree of similarity between the reference surgery video and the comparison target surgery video based on the metadata and the tool appearance information;
Including, similarity comparison device. 10. The method of claim 9,
Upload to determine a correlation of the comparison target surgery video to the reference surgery video based on the degree of similarity, and upload the comparison target surgery video to a pre-established Wiki-based web platform, including the determined correlation part,
Further comprising a, similarity comparison device. 10. The method of claim 9,
The tool appearance information includes tool trajectory information associated with the motion of a tool used for a predetermined operation and surgical action information inferred based on the motion of the tool,
The image information extraction unit,
Divide the reference surgery video or the comparison target surgery video into a plurality of index images, identify the tool appearing in each of the divided index images, derive the tool trajectory information for the identified tool, and the tool The similarity comparison apparatus, which input the trajectory information into a pre-generated artificial intelligence algorithm to derive the surgical action information corresponding to the tool trajectory information. 11. The method of claim 10,
The upload unit,
A similarity comparison apparatus that receives the reference surgery video from a content creator terminal to which authentication is granted in advance according to a predetermined condition and uploads the received reference surgery video to the wiki-based web platform. 13. The method of claim 12,
Receive feedback data on the comparison target surgical video from at least one participant terminal for the wiki-based web platform, and modify at least one of the correlation and the similarity for the comparison target surgery video based on the feedback data review department,
Further comprising a, similarity comparison device. 14. The method of claim 13,
A content providing unit that provides the comparison target surgery video to a content consumer terminal linked to the wiki-based web platform;
Further comprising a, similarity comparison device. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 8 on a computer.

Images