KR102386497B1 - Apparatus and method for detecting personal identification information for surgical video de-identification - Google Patents

Abstract

Disclosed is an apparatus and method for detecting personal identification information for de-identification of a surgical video, and the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application, for each frame of an input surgery video detecting an object including text and a person corresponding to a personal identification information candidate requiring de-identification, and when the object is detected, an index including a frame number of a frame in which the object is detected and a time code associated with the frame It may include generating information.

Description

Device and method for detecting personally identifiable information for de-identification of surgical video

The present application relates to an apparatus and method for detecting personal identification information for de-identification of a surgical video.

Medical information technology is developing rapidly by combining with big data and artificial intelligence technology in recent years. The production and utilization of unstructured data such as videos shot and stored by

As the number of cases of video recording of such surgical procedures is increasing, recently, due to the abundant clinical information of video recording surgical procedures, unstructured data such as video is important for new service creation and R&D of medical information technology. Its value as data is being evaluated anew.

On the other hand, in the health and medical field, the handling and protection of personal information is very important as in other fields such as administration, safety, and finance. For the standards for personal information and de-identification in medical science research, you can refer to the Personal Information-related Bioethics Act, IRB Operation Guidelines, and Personal Information De-identification Measures Guidelines. However, in the case of the anonymity method suggested in these guidelines, there is a limitation in that it is difficult to apply it collectively because the format is different for each video equipment.

Therefore, in order to use video as data, there is a growing need for a method of detecting collection target information (personal identification information) included in each frame of a video, indexing it, and interworking with an appropriate de-identification processing method.

However, in the past, it was difficult to actively utilize video data containing abundant clinical information produced during surgical procedures, etc. because the series of processes of screening the entire video frame and selecting personally identifiable information depended on visual inspection and manual operation. .

In this regard, in order to build a system for utilizing video data of medical information, a method capable of supporting the processing of personally identifiable information included in unstructured data such as video or decision-making for this is required. For example, if the patient's face or the appearance of the medical staff is unintentionally taken while the camera is staying outside the body for the purpose of replacing surgical tools during laparoscopic surgery, it is necessary to separately record these identification elements and the time of occurrence. However, in general, video management in the medical field is performed at the discretion of the individual doctor, so it is difficult to enforce it, and there is a problem in that each time a medical information video is used for research, there is no choice but to rely on visual inspection.

In addition, unlike still images, large-capacity videos consist of a number of consecutive frames. In the case of a surgical operation, it may take from about 30 minutes to several hours or longer, so the amount of time required for inspection and analysis is proportional to the playback time of the video. Time is bound to increase rapidly. In addition, videos produced in the medical field have diversity in content, such as varying the expression format of medical information according to the variety of shooting environments and methods, characteristics and uses of imaging devices used for examination or surgery. In the end, visual inspection of video is inefficient in terms of cost, and depending on the skill of the inspector, it may be difficult to guarantee the quality of inspection and analysis results.

In addition, the conventional automated personal information de-identification method for medical images mainly targets still images, and is only limited to acquiring structured data such as information included in image headers. There is a problem that is difficult to apply to the video of

The technology that is the background of the present application is disclosed in Korean Patent Publication No. 10-1946333.

The present application is to solve the problems of the prior art described above, by automatically detecting personally identifiable information that can be included in a surgical video and providing indexing for the detected personally identifiable information, so that the fast and effective work required for de-identification of personal information An object of the present invention is to provide a device and method for detecting personally identifiable information for de-identification of a surgical video that can assist in decision-making.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application is personal identification information requiring de-identification for each frame of an inputted surgical video. Detecting an object including text and a person corresponding to a candidate, and when the object is detected, generating index information including a frame number of a frame in which the object is detected and a time code associated with the frame can do.

In addition, in the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application, before the step of detecting the object, the surgical video is divided into a plurality of scenes each including a predetermined frame and classifying the plurality of divided scenes into an in-vitro scene and an in-vitro scene.

In addition, the dividing includes dividing the surgical video based on color information determined based on a preset weight for each RGB channel for each frame of the surgery video and regional direction pattern information determined based on a Kirsch edge detector. can do.

In addition, the dividing may include calculating a distance difference between adjacent frames of the surgical video based on a histogram for each frame corresponding to each of a plurality of preset features and comparing the calculated distance difference with a preset threshold value. It may include determining a scene boundary frame from among the frames included in the operation video.

In addition, the detecting of the object may include detecting a text with respect to a frame included in the in-vitro photographing scene and the in-vitro photographing scene, and detecting a person with respect to a frame included in the in-vitro photographing scene. there is.

Also, in the detecting of the text, when a character string having a predetermined length or longer is detected based on an optical character reader (OCR), it may be determined that the text is detected.

Also, in the detecting of the person, when a body part of a person including a patient or medical staff is detected based on a deep learning-based artificial intelligence model, it may be determined that the person is detected.

In addition, in the step of classifying the scene taken inside the body and the scene taken outside the body, each scene is divided into the scene shot inside the body based on Hue, Saturation, and Intensity of each of the divided scenes. and determining one of the in vitro photographed scenes.

In addition, in the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application, by matching the index information with metadata associated with the surgery video, the surgical video is preset together with the index information It may include the step of storing to storage.

In addition, the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application performs de-identification of at least some of the personal identification information candidates appearing in the operation video based on the index information. may include the step of

On the other hand, the apparatus for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application includes a scene dividing unit for dividing the surgical video into a plurality of scenes each including a predetermined frame, the divided plurality of A scene classification unit for classifying scenes of in-vitro and in-vitro shooting scenes, an object detection unit for detecting objects including text and people corresponding to personal identification information candidates that require de-identification for each frame of an input surgical video, and When the object is detected, an indexing unit that generates index information including a frame number of a frame in which the object is detected and a time code associated with the frame may be included.

In addition, the scene division unit calculates a distance difference between adjacent frames of the surgery video based on a histogram for each frame corresponding to each of a plurality of preset features, and compares the calculated distance difference with a preset threshold to perform the surgery A scene boundary frame may be determined among frames included in the video.

In addition, the object detection unit may include a text detection unit that detects text with respect to a frame included in the in-vitro photographing scene and the in-vitro photographing scene, and a person detection unit that detects a person with respect to a frame included in the in-vitro photographing scene. .

In addition, the text detection unit may be provided in the form of a plug-in linked to an optical character reader (OCR), and the person detecting unit may be provided in the form of a plug-in linked to a deep learning-based artificial intelligence model.

In addition, the apparatus for detecting personal identification information for de-identification of a surgery video according to an embodiment of the present application matches the index information with metadata associated with the surgery video to store the surgery video together with the index information It may include a storage unit and a de-identification unit for performing de-identification of at least some of the personal identification information candidates appearing in the operation video based on the index information.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, it automatically detects personally identifiable information that can be included in a surgical video and provides indexing for the detected personally identifiable information, thereby assisting fast and effective decision-making required for de-identification of personal information. It is possible to provide an apparatus and method for detecting personal identification information for de-identification of a surgical video that can be performed.

According to the above-described problem solving means of the present application, it is possible to construct an effective automated inspection system for de-identification elements for large-capacity and large-scale surgical video.

According to the above-described problem solving means of the present application, it is possible to improve the de-identification system of medical images limited to conventional still images and support data research using video required in the medical field.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a schematic configuration diagram of a surgical video de-identification system according to an embodiment of the present application.
2 is a diagram for explaining an in-body photographing scene and an in-vitro photographing scene according to an embodiment of the present application.
3 is a diagram exemplarily illustrating a frame in which a person corresponding to a candidate for personal identification requiring de-identification according to an embodiment of the present application is detected.
4 is a diagram for explaining index information according to an embodiment of the present application.
5 is a conceptual diagram for explaining a surgical video processing method based on a standard plug-in API according to an embodiment of the present application.
6 is a schematic configuration diagram of an apparatus for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.
7 is an operation flowchart of a method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.
8 is a chart showing the indexing processing speed measurement results for a plurality of sample surgical videos as an experimental example linked to a method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when a member is positioned “on”, “on”, “on”, “on”, “under”, “under”, or “under” another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to an apparatus and method for detecting personal identification information for de-identification of a surgical video.

1 is a schematic configuration diagram of a surgical video de-identification system according to an embodiment of the present application.

Referring to Figure 1, the surgical video de-identification system 10 according to an embodiment of the present application, the personal identification information detection device 100 for de-identification of the operation video (hereinafter, 'personal identification information detection device ( 100)'), the integrated management server 200 , the inspector terminal 300 and the user terminal 400 may be included.

The personal identification information detection device 100 , the integrated management server 200 , the inspector terminal 300 , and the user terminal 400 may communicate with each other through the network 20 . The network 20 refers to a connection structure in which information exchange is possible between each node, such as terminals and servers, and an example of such a network 20 includes a 3rd Generation Partnership Project (3GPP) network, a long-term LTE (LTE) network. Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but are not limited thereto.

The inspector terminal 300 and the user terminal 400 are, for example, a smartphone (Smartphone), a smart pad (SmartPad), a tablet PC and the like and PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal) Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may be any kind of wireless communication device such as a terminal.

In the description of the embodiment of the present application, the inspector terminal 300 checks the stored surgical video together with index information about the personal identification information candidate extracted by the personal identification information detection device 100, and to each of the personal identification information candidates It may be a terminal possessed by a subject that de-identifies at least some of the personally identifiable information candidates in the surgery video by reviewing the appropriateness of the information or considering the index information.

In the description of the embodiment of the present application, the user terminal 400 receives the de-identification process for the personal identification information that appears in the surgical video is provided, and the de-identified surgical video or content including the surgical video is provided. It may be a terminal for viewing (playback).

The apparatus 100 for detecting personal identification information may receive the operation video 1 . Here, the operation video 1 received by the apparatus 100 for detecting personal identification information may mean a raw (original) video before de-identification processing is performed. For example, the operation video 1 may be a raw video generated during a diagnosis using an endoscope or laparoscopic surgery.

The apparatus 100 for detecting personal identification information may divide the surgical video 1 including a plurality of frames into a plurality of scenes. That is, the apparatus 100 for detecting personal identification information may divide the received surgical video 1 into a plurality of scenes each including a predetermined frame.

According to an embodiment of the present application, the apparatus 100 for detecting personal identification information includes color information determined based on a preset weight for each RGB channel for each frame of the surgical video 1 and an area determined based on a Kirsch edge detector. The operation video 1 may be divided into a plurality of scenes based on the direction pattern information.

Specifically, according to an embodiment of the present application, the apparatus 100 for detecting personal identification information 100 provides color information of 6 bits selected from each RGB channel according to weights for fast and stable scene segmentation, and the maximum value, the second value according to Kirsch edge response. It may be to apply the CeLDP technique for combining 6-bit local directional pattern information composed of the index of the response value and the third response value. To help understanding, the personal identification information detection apparatus 100 expresses a predetermined number (eg, 4096) of features from every frame of the surgical video 1 by the scene segmentation technique using CeLDP. A histogram may be derived, a distance difference between adjacent frames may be obtained based on the derived histogram, and a scene boundary may be determined according to a set threshold value. Here, the apparatus 100 for detecting personal identification information may operate to integrate a plurality of frames between the determined scene boundaries into one scene.

In other words, the apparatus 100 for detecting personal identification information calculates a distance difference between adjacent frames of the surgical video 1 based on a histogram for each frame corresponding to each of a plurality of preset features, and calculates the difference between the calculated distance and the preset threshold By comparing the values, it is possible to determine a scene boundary frame from among the frames included in the surgical video. Here, the plurality of preset features may include the above-described color information and regional direction pattern information.

In this regard, the personal identification information detection device 100 of the present application quickly detects a personal identification information candidate and generates index information for the detected personal identification information candidate, not the entire frame of the surgical video 1, but a significant minimum In order to perform unit examination (detection), the operation video 1 may be divided into respective scenes that can correspond to the minimum examination unit based on the characteristics of each frame of the operation video 1 . For example, dividing the surgery video 1 into 'scenes' can be understood as dividing all frames in the surgery video 1 into a logical hierarchy of shots and scenes, through which The apparatus 100 for detecting personal identification information may obtain structural information of a surgical video.

In addition, the apparatus 100 for detecting personal identification information may classify the divided plurality of scenes into an in-vitro shooting scene and an external shooting scene.

2 is a diagram for explaining an in-body photographing scene and an in-vitro photographing scene according to an embodiment of the present application.

Referring to FIG. 2, (a) of FIG. 2 exemplarily shows a frame within a scene classified as an in-vitro shooting scene, and FIG. 2 (b) exemplarily shows a frame within a scene classified as an in-vitro shooting scene will be. According to an embodiment of the present application, the apparatus 100 for detecting personal identification information detects each scene based on a color (Hue), saturation (Saturation), and brightness (Intensity) of each of a plurality of divided scenes in vivo and in vitro. It can be decided by any one of the shooting scenes. In other words, the apparatus 100 for detecting personal identification information may apply a knowledge-based detection technique associated with the HSI color model to distinguish between an in-body image and an in-vitro image.

In addition, the personal identification information detecting apparatus 100 may detect an object including a text and a person corresponding to a personal identification information candidate requiring de-identification for each frame of the inputted surgical video 1 . Here, the personally identifiable information candidate means the patient's or medical staff's face and body, including the patient's personal information and medical information in text form included in graphic form (in other words, in a form that can be confirmed visually) within the surgical video 1 It may refer to an element that can directly or indirectly identify the identity of a patient or medical staff visually, such as a part. That is, the apparatus 100 for detecting personal identification information may operate to search for each frame of the operation video 1 to detect a candidate for personal identification information, which is a visual element that is expected to require de-identification.

In addition, according to an embodiment of the present application, the personal identification information detecting apparatus 100 considers the preceding scene segmentation result and scene classification result (in vitro shooting scene or in vitro shooting scene), and the personal identification information candidate search type for each frame can be applied differently. Specifically, the apparatus 100 for detecting personal identification information may detect a text with respect to a frame included in an in-body photographing scene and an in-vitro photographing scene. Also, the apparatus 100 for detecting personal identification information may detect a person with respect to a frame included in an in vitro photographing scene.

In this regard, in the frames in the surgical video (1) recording the surgical process, the camera mounted on the surgical tool may go back and forth between the body and the body depending on the situation at the time of surgery. can However, in the case of a general surgical video 1, it can be said that the possibility that a body part (eg, face, etc.) of a person appears in a frame in which the patient's body is photographed is remarkably low. That is, since it may be inefficient to detect a person in all frames when most of the entire frames of the video are taken inside the body, such as in laparoscopic surgery, the personal identification information detection device 100 of the present application is the purpose and implementation of the personal identification information candidate detection According to the type of surgery performed, processing efficiency can be increased by restricting operation when the current frame, which is the target of object detection, is inside or outside the body. However, the present disclosure is not limited thereto, and according to the embodiment of the present application, personal identification is performed so that the person detection process is performed even for a frame within a scene corresponding to a scene taken inside the body according to the purpose of detecting a candidate for personal identification information, the type of surgery performed, etc. The information detecting apparatus 100 may operate.

Specifically, in relation to the text detection process in a frame, the personal identification information detecting apparatus 100 detects a text string longer than a predetermined length based on an optical character reader (OCR) within a predetermined frame. can be determined to be detected.

In this regard, text-type objects such as patient treatment information that may be displayed in the operation video 1 may appear in various ways according to the characteristics of equipment used for the operation, the operation environment, and the like. Therefore, in the case of applying the regular expression to all text recognized by the method such as OCR, the personal identification information detecting apparatus 100 according to an embodiment of the present application may decrease efficiency or slow down the operation speed, so the open source OCR project The text is detected within the frame by using the Tesseract engine, which is one of them, and it can be determined that meaningful text is included only when characters having a predetermined length or longer are detected. Here, when utilizing the Tesseract engine described above, since an image file, not a buffer of the decoded frame, is required to be applied as an input, the personal identification information detecting apparatus 100 is a frame transmitted to the personal identification information detecting apparatus 100 You can act to save an image file (eg TIFF format) to the buffer and then pass it to the Tesseract engine.

3 is a diagram exemplarily illustrating a frame in which a person corresponding to a candidate for personal identification requiring de-identification according to an embodiment of the present application is detected.

Referring to FIG. 3 , in relation to the process of detecting a person within a frame, the personal identification information detecting apparatus 100 detects a person when a body part of a person including a patient or medical staff is detected based on a deep learning-based artificial intelligence model. can be judged to have been

According to an embodiment of the present application, in order to detect a person (eg, a patient or a medical staff, etc.) in the operation video 1 , the personal identification information detection apparatus 100 detects the face of a predetermined person or excludes the face. Body parts can be detected. However, considering the special environment in which the surgical video 1 is recorded, most of the medical staff wears a mask, and most of the patients are also equipped with a respirator. Detection algorithms can be difficult to apply. Accordingly, when a person appears in the operation video, the apparatus 100 for detecting personal identification information of the present office assumes that at least a part of the head and upper body appear, and utilizes a deep learning-based YOLO (You Look Only Once) model. Thus, the person can be detected. In the case of the aforementioned YOLO model, the person detection performance is equivalent to that of other models and has a high processing speed, so there is an advantage suitable for processing the surgical video 1 including a large number of frames. However, the model (algorithm) used by the personal identification information detection device 100 for person detection is not limited to the above-described YOLO model, and is not limited to R-CNN (Recursive Convolutional Neural Network), Fast R-CNN, SSD (Single). Various artificial intelligence-based object recognition algorithm models that have been previously known or developed in the future, such as Shot multibox Detector), may be applied.

4 is a diagram for explaining index information according to an embodiment of the present application.

Referring to FIG. 4 , when an object is detected, the personal identification information detecting apparatus 100 may generate index information including a frame number of a frame in which the object is detected and a time code associated with the corresponding frame. Here, the index information includes, when text or a person corresponding to the personal identification information candidate is detected, the viewpoint information in the surgical video 1 in which the corresponding personal identification information candidate is detected, information on the type of the detected personal identification information candidate, etc. It may mean information. In addition, the index information may include unique identifier information (eg, ID, unique number, etc.) of each of the detected personal identification information candidates.

Also, FIG. 4 may show a user interface for checking the generated index information. According to an embodiment of the present application, the user interface for checking the index information may be displayed on the inspector terminal 300 associated with the personal identification information detection device 100 . Referring to FIG. 4 , the index information display interface may include a type information tab (A) of the detected personal identification information candidate. In addition, the index information display interface may include a partitioned index information tab (B) for each of the detected personal identification information candidates. Specifically, the index information tab B may include a reduced screen of a representative frame in which the corresponding personal identification information candidate appears, a frame number (Frame), a time code (Time), type information (Error Type), and the like. Here, the type information of the personal identification information candidate may include a person detection type and a text detection type.

The above-described index information display interface allows the user of the inspector terminal 300 such as the person in charge of de-identification processing to re-work (re-detection) or additional work (detection) for the personal identification information candidate and de-identify the task. It can function as a visualization tool to assist in appropriate decision-making, such as the progress of

In addition, the apparatus 100 for detecting personal identification information may match the generated index information with metadata associated with the operation video 1 and store the operation video 1 together with the generated index information in a preset storage. According to an embodiment of the present application, the surgical video 1 in which index information is matched (inserted) as metadata may be stored in a preset storage provided in the integrated management server 200 . In other words, the apparatus 100 for detecting personal identification information may use the generated index information as metadata and archive it together with the operation video 1 in storage.

The integrated management server 200 may acquire the surgical video 1 in which index information is matched (inserted) from the personal identification information detection device 100 and transmit it to the inspector terminal 300 . In addition, the integrated management server 200 acquires the surgical video 1 for which de-identification processing is completed by the inspector terminal 300 from the inspector terminal 300 or the personal identification information detection device 100, and the user terminal 400 can be provided as In addition, according to an embodiment of the present application, the integrated management server 200 is a streaming server (not shown) that provides a preview service through transcoding and web UI for the input surgery video, service management and content inquiry WAS (not shown) that provides web services may be included as sub-elements.

In addition, the personal identification information detection apparatus 100 de-identifies at least some of the personal identification information candidates that appeared in the operation video 1 based on the index information matched (inserted) with respect to the specific surgery video 1 can be done According to an embodiment of the present application, the apparatus 100 for detecting personal identification information receives a user input related to de-identification applied to the inspector terminal 300, and performs de-identification based on the received user input. can do. Here, de-identification is to perform de-identification processing such as scrambling, encryption, defacing, etc. for candidates of personal identification information such as patient faces and medical staff faces that need to be de-identified so that they are not disclosed to third parties, or patient treatment information It can be understood that it refers to predetermined measures for privacy protection, such as performing de-identification processing such as anonymization and masking, for text-type personally identifiable information candidates, such as.

In addition, according to an embodiment of the present application, the apparatus 100 for detecting personal identification information may operate to change and apply the de-identification type as needed according to the purpose of use of the surgical video 1, access rights, and processing standards. there is.

5 is a conceptual diagram for explaining a surgical video processing method based on a standard plug-in API according to an embodiment of the present application.

Referring to FIG. 5 , the apparatus 100 for detecting personal identification information is a standard plug-in for each of the detection algorithms for personal identification information candidates in order to secure interoperability with an external system and ease of extension of an algorithm capable of detecting a personal identification information candidate. It can be mounted based on the API method. In other words, each detection algorithm that performs objectification through detection of personal identification information candidates of the present application may be implemented as a plug-in that is easy to add and remove from the system according to a self-defined standard plug-in API specification.

Specifically, referring to FIG. 5 , job information related to detection of personal identification information may be generated, and plug-in execution management may be performed by a job scheduler. Here, the job scheduler allocates a predetermined surgical video (1) to the personal identification information detecting device 100, decodes the bit stream of the inputted surgical video (1), and the personal identification information detecting device ( 100) It can be delivered through standard API to each of my plugins. Each plug-in that receives the decoded frame returns the processing result using the callback function defined by the standard API, and when all plug-ins have completed processing the current frame, the plug-in execution manager (Plug-In Execution) Manager) may repeat the process of requesting the next frame from the input stream handler to complete processing up to the last frame of the video. Here, each of the detection algorithms mounted based on the standard plug-in API method may be divided into a text detection algorithm and a person detection algorithm according to a detection target. In other words, the apparatus 100 for detecting personal identification information may include a text detection plug-in and a person detection plug-in.

6 is a schematic configuration diagram of an apparatus for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.

Referring to FIG. 6 , the apparatus 100 for detecting personal identification information includes a scene division unit 110 , a scene classification unit 120 , an object detection unit 130 , an indexing unit 140 , a storage unit 150 , and non-identification. A flower unit 160 may be included. Also, referring to FIG. 6 , the object detection unit 130 may include a text detection unit 131 and a person detection unit 132 .

The scene dividing unit 110 may divide the input surgery video 1 into a plurality of scenes, each of which includes a predetermined frame.

The scene classification unit 120 may classify the plurality of divided scenes into an in-vitro scene and an in-vitro shooting scene.

The object detection unit 130 may detect an object including a text and a person corresponding to a personal identification information candidate requiring de-identification for each frame of the inputted surgical video 1 .

Specifically, the text detection unit 131 may detect a text with respect to a frame included in an in-body image capture scene and an in-vitro image capture scene. Also, according to an exemplary embodiment of the present disclosure, the text detection unit 131 may be provided in the form of a plug-in linked to an optical character reader (OCR). Also, the text detection unit 131 may determine that the text is detected when a character string having a predetermined length or longer is detected based on an optical character reader (OCR).

Also, the person detection unit 132 may detect a person with respect to a frame included in an in vitro photographing scene. In addition, according to an embodiment of the present application, the person detection unit 132 may be provided in the form of a plug-in linked to a deep learning-based artificial intelligence model. Also, the person detection unit 132 may determine that the person is detected when a body part of a person including a patient or medical staff is detected based on a deep learning-based artificial intelligence model.

When an object corresponding to the personal identification information candidate is detected by the object detection unit 130, the indexing unit 140 generates index information including a frame number of a frame in which the corresponding object is detected and a time code associated with the corresponding frame. can do.

The storage unit 150 may match the index information with metadata associated with the operation video 1 and store the operation video 1 together with the index information. In addition, according to an embodiment of the present application, the apparatus 100 for detecting personal identification information matches the operation video 1 and the operation video 1 stored in the integrated management server 200 in connection with the storage unit 150 . It may include a transmitter (not shown) for transmitting the index information.

The de-identification unit 160 may de-identify at least some of the personal identification information candidates appearing in the surgery video 1 matched with the index information based on the index information.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

7 is an operation flowchart of a method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.

The method of detecting personal identification information for de-identification of the surgical video shown in FIG. 7 may be performed by the apparatus 100 for detecting personal identification information described above. Therefore, even if omitted below, the description of the personal identification information detecting apparatus 100 may be equally applied to the description of the personal identification information detecting method for de-identification of a surgical video.

Referring to FIG. 7 , in step S710 , the scene dividing unit 110 may divide the input surgery video 1 into a plurality of scenes each including a predetermined frame.

Next, in step S720 , the scene classification unit 120 may classify the plurality of divided scenes into an in-vitro scene and an in-vitro shooting scene.

Next, in step S730, the object detection unit 130 may detect an object including a text and a person corresponding to a personal identification information candidate that needs to be de-identified for each frame of the input operation video 1 .

Specifically, in operation S730 , the text detection unit 131 may detect a text with respect to a frame included in an in-body photographing scene and an in-vitro photographing scene. Also, in step S730 , the person detection unit 132 may detect a person with respect to a frame included in the in vitro photographing scene.

Next, when an object corresponding to the personal identification information candidate is detected in step S740, the indexing unit 140 may generate index information including a frame number of a frame in which the corresponding object is detected and a time code associated with the corresponding frame. there is.

Next, in step S750, the storage unit 150 may match the index information with metadata associated with the operation video 1 and store the operation video 1 together with the index information.

Next, in step S760, the de-identification unit 160 may perform de-identification of at least some of the personal identification information candidates appearing in the operation video 1 matched with the index information based on the index information.

In the above description, steps S710 to S760 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

8 is a chart showing the indexing processing speed measurement results for a plurality of sample surgical videos as an experimental example linked to a method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application.

Referring to FIG. 8 , it can be determined whether the method by visual inspection can be replaced by the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application in consideration of the operation speed. In particular, unlike still images, a surgical video consists of continuous frames of at least several tens of minutes and at most several hours as described above, and it is possible to de-identify and provide such raw surgical video in a timely manner according to user needs. In this regard, the inventor of the present application sequentially input 10 videos in which characters and texts are arbitrarily combined with respect to the m2cai16-tool data set, which is an open medical video data set, and calculates the average required time. Here, the required time is measured in seconds from the first frame of each input until the processing of the last frame is completed, and the processing speed is obtained by dividing the length of the video sample (in other words, playback time) by the required time. It is calculated as the processing speed (double speed). As the calculated processing speed (double speed) has a value greater than 1, it may be determined that the processing is completed at a faster speed than the playback time of each video sample.

Specifically, referring to FIG. 8 , according to the method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application, it takes an average of 217.2 seconds for an input video of 485.4 seconds on average, so an average of 2.18 seconds It can be confirmed that processing can be completed at double speed. In the case of the above-mentioned visual inspection, since the inspector has to check the entire frame one by one, it can be assumed that more than the playback time of the video is required, and this visual inspection method shows a working speed of 1x or less. Therefore, it can be confirmed that, according to the present application, automated detection and indexing of the detection results can be performed very effectively and quickly on the personally identifiable information candidates that are evaluated to be targets of de-identification included in the surgical video.

The method for detecting personal identification information for de-identification of a surgical video according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the method for detecting personal identification information for de-identification of the above-described surgical video may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: Surgical video de-identification system
100: Personal identification information detection device for de-identification of surgical video
110: scene division
120: scene classification unit
130: object detection unit
131: text detection unit
132: person detection unit
140: indexing unit
150: storage
160: de-identification unit
20: network
200: integrated management server
300: inspector terminal
400: user terminal

Claims (15)

In the method for detecting personal identification information for de-identification of a surgical video performed by a personal identification information detection device for de-identification of a surgical video,
detecting an object including text and a person corresponding to a personal identification information candidate requiring de-identification for each frame of the inputted surgical video; and
generating index information including a frame number of a frame in which the object is detected and a time code associated with the frame when the object is detected;
including,
Prior to the step of detecting the object,
dividing the operation video into a plurality of scenes each including a predetermined frame; and
classifying the divided scenes into an in-vitro scene and an in-vitro scene;
further comprising,
Detecting the object comprises:
In consideration of the classification result of the scene, the search type of the personal identification information candidate for each of the frames is applied differently,
The personally identifiable information candidates are:
The method for detecting personal identification information, characterized in that the person is included in the frame classified as the in vitro shot scene, and the person is not included in the frame classified as the in vivo shot scene. delete According to claim 1,
The dividing step is
For each frame of the surgery video, the surgical video is divided based on color information determined based on a weight preset for each RGB channel and regional direction pattern information determined based on a Kirsch edge detector, personal identification information detection Way. 4. The method of claim 3,
The dividing step is
calculating a distance difference between adjacent frames of the surgical video based on a histogram for each frame corresponding to each of a plurality of preset features; and
determining a scene boundary frame among frames included in the surgical video by comparing the calculated distance difference with a preset threshold;
That comprising a, personally identifiable information detection method. According to claim 1,
Detecting the object comprises:
detecting texts with respect to frames included in the in-body imaged scene and the in-vitro imaged scene; and
detecting a person with respect to a frame included in the in vitro photographing scene;
That comprising a, personally identifiable information detection method. 6. The method of claim 5,
Detecting the text includes:
When a character string of a predetermined length or longer is detected based on an optical character reader (OCR), it is determined that the text is detected. 6. The method of claim 5,
The step of detecting the person is
When a body part of a person, including a patient or medical staff, is detected based on a deep learning-based artificial intelligence model, it is determined that the person is detected. According to claim 1,
The step of classifying the scene taken inside the body and the scene taken outside the body,
Determining each of the scenes as one of the in-vitro scene and the in-vitro shooting scene based on the divided hue, saturation, and intensity of each of the plurality of scenes. , methods of detecting personally identifiable information. According to claim 1,
Matching the index information with metadata associated with the surgery video and storing the surgery video together with the index information in a preset storage;
Which will further include, personally identifiable information detection method. 10. The method of claim 9,
The method of detecting personal identification information further comprising the step of de-identifying at least some of the personal identification information candidates that appeared in the operation video based on the index information. In the device for detecting personal identification information for de-identification of surgical video,
a scene division unit for dividing the surgical video into a plurality of scenes each including a predetermined frame;
a scene classification unit for classifying the divided scenes into an in-vitro scene and an in-vitro scene;
an object detection unit for detecting an object including text and a person corresponding to a personal identification information candidate requiring de-identification for each frame of the input surgery video; and
when the object is detected, an indexing unit generating index information including a frame number of a frame in which the object is detected and a time code associated with the frame;
including,
The object detection unit,
Differently apply the search type of the personal identification information candidate for each frame in consideration of the classification result of the scene,
The personally identifiable information candidates are:
The apparatus for detecting personal identification information, wherein the person is included in the frame classified as the in-vitro photographed scene, and the person is not included in the frame classified as the in-body photographed scene. 12. The method of claim 11,
The scene division unit,
A distance difference between adjacent frames of the surgery video is calculated based on a histogram for each frame corresponding to each of a plurality of preset features, and the calculated distance difference is compared with a preset threshold value among the frames included in the surgery video. Determining a scene boundary frame, personally identifiable information detection device. 12. The method of claim 11,
The object detection unit,
a text detection unit configured to detect a text with respect to a frame included in the in-body photographing scene and the in vitro photographing scene; and
a person detection unit for detecting a person with respect to a frame included in the in vitro photographing scene;
That comprising a, personally identifiable information detection device. 14. The method of claim 13,
The text detection unit is provided in the form of a plug-in linked to an optical character reader (OCR),
The person detection unit, the personal identification information detection device, characterized in that provided in the form of a plug-in linked to a deep learning-based artificial intelligence model. 14. The method of claim 13,
a storage unit that matches the index information with metadata associated with the surgery video and stores the surgery video together with the index information; and
A de-identification unit that de-identifies at least some of the personal identification information candidates appearing in the operation video based on the index information;
Which will further include, personally identifiable information detection device.

Images