KR102259797B1 - User event detection method through image analysis and device performing the method - Google Patents

Abstract

A user event detection method through image analysis composed of image frames according to an embodiment of the present invention may include the steps of: receiving a current image frame from a medical imaging apparatus; calculating a difference value between the current image frame and a previous image frame; determining whether a first user event signal for capturing the current image frame is input to the medical imaging apparatus according to whether the difference value exceeds a first threshold; and capturing the current image frame when it is determined that the first user event signal is input. The user event detection method can determine whether an event by a user is generated in the medical imaging apparatus by analyzing change information on an image frame.

Description

User event detection method through image analysis and device performing the method}

The present invention relates to a method for detecting a user event through image analysis and an apparatus for performing the method, and more particularly, to an image analysis that enables detection of whether a user event has occurred in a medical imaging apparatus even with only an image analysis result. It relates to a method for detecting a user event through and an apparatus for performing the method.

There are several types of malignant tumors in the stomach, such as gastric adenocarcinoma, gastric lymphoma, and gastrointestinal stromal tumor, but of these, gastric adenocarcinoma is the most common. In this case, gastric cancer may be classified into early gastric cancer (EGC) and advanced gastric cancer (AGC) according to the degree of tumor progression.

More specifically, early gastric cancer is a case in which tumor cells have invaded the mucosal or submucosal layer of the stomach regardless of whether or not metastasis to the lymph nodes is present. invasion is possible.

Gastric endoscopy is the most important method for diagnosing gastric cancer. More specifically, a medical staff may diagnose gastric cancer by reading a gastroscopic image taken for an object suspected of gastric cancer using an endoscope scope, and provide treatment according to the lesion based thereon. Meanwhile, in a diagnosis system based on an endoscopic image, there may be a deviation in diagnosis according to the skill level of a medical staff. Such deviations in diagnosis may also be related to medical accidents such as misdiagnosis, delay in treatment timing, and improper treatment.

In particular, in the case of early gastric cancer, the lesion site may be very small or it may be difficult to distinguish it from normal tissue, so that the diagnosis deviation according to the skill of the medical staff may be greater.

And, as the accuracy of diagnosis is more demanded for the prevention of medical accidents and improvement of medical services, the development of a new diagnostic method capable of providing the location of the lesion as well as the onset of the disease is continuously required. there is a situation.

Meanwhile, according to the prior art, in the process of receiving an image output from the medical imaging apparatus in real time and outputting it through an output unit in order to detect the location of a lesion, a specific event ( When a probe switch or a foot switch is pressed, etc.), the specific event is detected and an image corresponding thereto is detected through the gateway, so that it can be determined whether a specific event has occurred in the medical imaging apparatus.

However, there are cases where the gateway is not equipped with a system such as firmware that performs functions such as event detection and image storage corresponding thereto. Even in this case, a specific event occurring in the medical imaging device must be properly recognized by the image receiving device. there is a need to

The present invention was derived from the above necessity, and is used to analyze change information on an image frame directly by an image receiving device and to determine whether an event by a user has occurred in a medical imaging apparatus even with only the corresponding result. has its purpose in

According to an embodiment, a method for detecting a user event through image analysis composed of image frames includes: (a) receiving, by an image receiving apparatus, a current image frame from a medical imaging apparatus through a communication unit; (b) calculating, by a controller, a difference value between the current image frame and a previous image frame; (c) determining, by the controller, whether a first user event signal for capturing the current image frame is input to the medical imaging apparatus according to whether the difference value exceeds a first threshold; and (d) when the controller determines that the first user event signal is input, capturing the current image frame.

The detection method may include: (e) determining whether a second user event signal for storing the captured image obtained by performing the capturing of the current image frame is input; and (f) generating an index image corresponding to the captured image when it is determined that the second user event signal is input.

The step (f) may further include storing the captured image as a key image.

The step (b) includes: (b1) calculating, by the controller, a difference in brightness between the current image frame and the previous image frame for each channel; (b2) allocating, by the controller, different binary values to each map configured for each channel according to whether the calculated difference in brightness exceeds a second predetermined threshold; and (b3) calculating, by the controller, the difference value based on the sum of the binary values.

The first threshold may be 0.005 or more and 0.02 or less, and the second threshold may be 15 or more and 40 or less.

The key image is first input among partial regions of the entire queue region for storing the image frame, and the partial region of the entire queue region stores images having the same value as the key index corresponding to the key image. it may be for

The key image is an image having the highest prediction probability value through a model for predicting a lesion among images stored in a partial region of the entire cue region for storing the image frame, and the partial region of the entire cue region is It may be for storing the images having the same value as the key index corresponding to the key image.

According to an embodiment, an apparatus for detecting a user event through image analysis composed of an image frame includes: a communication unit configured to receive a current image frame from a medical imaging apparatus; and calculating a difference value between the current image frame and a previous image frame, and when the difference value exceeds a predetermined threshold, a first user event signal for capturing the current image frame is input to the medical imaging apparatus When a second user event signal for performing a capture on the current image frame and storing a captured image obtained by performing the capturing on the current image frame is inputted, corresponding to the captured image It may include; a control unit for generating an index image.

The apparatus may further include a; a display unit for outputting together the index image and a screen including an image listed in correspondence to the index image.

The controller may store the captured image as a key image.

According to the present invention, it is possible to determine whether an event by a user has occurred in the medical imaging apparatus by directly analyzing change information on an image frame by the image receiving apparatus and using only the corresponding result.

1 is a conceptual diagram illustrating an image processing system according to an embodiment.
FIG. 2 is a flowchart illustrating a method of detecting a user event through image analysis according to an embodiment, and FIG. 3 is a flowchart referenced to describe the method of FIG. 2 .
4 to 8 are diagrams referenced to explain the method of FIGS. 2 and 3 .
9 and 10 illustrate a screen output to the display unit 124 according to an embodiment.
11 is a diagram illustrating a structure of a neural network model 121a according to an embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail sufficient to enable a person skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention in relation to one embodiment. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the present invention. Accordingly, the detailed description to be described below is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed by the claims. Like reference numerals in the drawings refer to the same or similar functions over several aspects.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.

1 is a conceptual diagram illustrating an image processing system according to an embodiment.

1 , the medical imaging apparatus 110 may transmit an image frame obtained by photographing an object to the image receiving apparatus 120 . The image frame according to the embodiment may be an image obtained from a medical device such as a digestive endoscopy equipment, but the scope of the present invention is not limited thereto and may be equally/similarly applied to other types of medical devices such as ultrasound or MRI. .

The image receiving apparatus 120 may include various electronic devices, such as a PC or a laptop computer, capable of running software for analyzing an input image frame by performing lesion prediction through the neural network model 121a.

The image receiving apparatus 120 may include a communication unit 122 , a control unit 121 , a memory 123 , and a display unit 124 .

The communication unit 122 receives the image frame from the medical imaging apparatus 110 and transmits it to the control unit 121 .

The controller 121 may detect a change between received image frames using the image change detection module 121b, and determine whether a user event signal is input to the medical imaging apparatus 110 based on this.

The controller 121 may perform lesion prediction using the neural network model 121a to obtain result values (characteristic values, etc.) of the lesion on the input image frame and predict the lesion based thereon.

The memory 123 stores various types of data processed by the control unit 121 , such as a result value obtained from the control unit 121 .

When a result value or an image stored in the memory 123 is read under the control of the controller 121 , the display unit 124 may output the corresponding data and provide it to the user.

FIG. 2 is a flowchart illustrating a method of detecting a user event through image analysis according to an embodiment, and FIG. 3 is a flowchart referenced to describe the method of FIG. 2 . However, each step of the present invention is not limited only to the corresponding order.

2 and 3 together, the image receiving apparatus 120 may receive a current image frame from the medical imaging apparatus 110 through the communication unit 122 ( S210 ). Specifically, when the image receiving apparatus 120 is driven to start analyzing an image, an image analysis start time may be measured and an image frame may be read (s310 to s312). For reference, in the present invention, the current image frame may be a currently read image frame, and the previous image frame may be an image frame read immediately before and may be an image frame adjacent to the current image frame. In this case, the controller 121 may monitor the operation speed (frame rate) of the neural network model 121a and read image frames from the medical imaging apparatus 110 frame by frame. According to an embodiment, the operation of the plurality of neural network models 121a for processing one image frame may be processed in parallel to enable a plurality of analysis at once or may be processed in series.

The controller 121 may detect a change in an image frame received through the communication unit 122 using the image change detection module 121b. (s313)

In detail, the image change detection module 121b calculates a difference value between the current image frame and the previous image frame, determines whether the calculated difference value exceeds a first predetermined threshold, and sends it to the medical imaging apparatus 110 . It may be determined whether a first user event signal for capturing the current image frame is input. (s220 to s230). That is, when the difference value exceeds the first threshold, it is determined that the first user event signal for capturing the current image frame is input, which is a kind of user's 'Freeze' command to the medical imaging apparatus 110 . This is entered. In particular, the calculation of the difference between the image frames and the comparison with the threshold will be described in detail later with reference to FIG. 6 .

)를 키 프레임 인덱스(

)로 저장할 수 있다(s314). 도 5a를 함께 참조하면, 현재 영상 프레임에 대응하는 인덱스(

)인 11을 키 프레임 인덱스(

)(A1)로 저장할 수 있다. On the other hand, if the image change detection module 121b determines that the first user event signal is input, captures the current image frame as shown in (a) of FIG. 4 ( s240 ), and an index corresponding to the current image frame (

) to the keyframe index (

) can be stored as (s314). 5A together, the index corresponding to the current image frame (

), which is 11, is the key frame index (

)(A1).

Then, the image change detection module 121b determines whether a second user event signal for storing the captured image (FIG. 4 (a)) obtained by performing the capture of the current image frame is input, , when it is determined that the second user event signal is input, an index image corresponding to the captured image is generated ((b) of FIG. 4), and the captured image ((c) of FIG. 4) may be stored as a key image ( s250 to s270). For reference, in the present invention, the index image may be displayed on the edge of the image screen of the display unit 124 in the form of a thumbnail of the captured image.

)를 키 영상(

)으로 저장할 수 있다(s316). 도 5a를 참조하면, 11번째 영상 프레임(

)을 키 영상(

)으로 저장할 수 있다. 이 때, 키 영상(

)은 인덱스 이미지에 대응하여 저장될 수 있다.Specifically, the image change detection module 121b is configured to 'Release' the user's medical imaging apparatus 110 when a second user event signal for storing an image is input while the captured image is output. When a command is input, an index image corresponding to the captured image is output, and the captured image (

) to the key image (

) can be saved as (s316). Referring to Figure 5a, the 11th image frame (

) to the key image (

) can be saved as At this time, the key image (

) may be stored corresponding to the index image.

Meanwhile, the image change detection module 121b may determine whether the index image has been changed, and when it is determined that the index image has been changed, the changed index image may be selected and stored as a key index image (s317 and s318).

:11)에 이어서 다음 영상 프레임(

:12)를 리딩하고(i=i+1), 다음 영상 프레임(

:12)을 현재 영상 프레임으로 간주하여 전술한 과정을 반복 수행할 수 있다.And, the current video frame (

:11) followed by the next video frame (

:12) is read (i=i+1), and the next image frame (

:12) can be regarded as the current image frame and the above-described process can be repeated.

:12)를 리딩(i=i+1)하여 다음 영상 프레임(

:12)을 현재 영상 프레임으로 간주한 상태에서, 영상 변화 감지 모듈(121b)은 제1 사용자 이벤트 신호가 입력되지 않은 것으로 판단하면, 이전 영상 프레임에 대응하는 인덱스(

)인 11을 키 프레임 인덱스(

)(B1)로 저장할 수 있다. Meanwhile, referring to FIG. 5A together, the next image frame (

:12) is read (i=i+1) to the next video frame (

:12) as the current image frame and the image change detection module 121b determines that the first user event signal is not input, the index (

), which is 11, is the key frame index (

) (B1).

For reference, when it is determined that the first user event signal is not input, the difference between the current image frame and the previous image frame is less than a predetermined first threshold, and a separate capture operation by the user is not performed, and the user It may mean a section C1 in which an afterimage is generated due to a previous capture operation by . That is, even when the actual first user event signal is inputted, even if the frame is momentarily stopped for about one frame, since there may be additional weak motions, it may mean this section.

)인 15를 키 프레임 인덱스(

) (A2)로 저장할 수 있다(s314).And, when the first user event signal is input again, with reference to FIG. 5A , the index corresponding to the current image frame (

), which is 15, is the key frame index (

) (A2) (s314).

:16)를 리딩(i=i+1)하여 다음 영상 프레임(

:16)을 현재 영상 프레임으로 간주한 상태에서, 영상 변화 감지 모듈(121b)은 제1 사용자 이벤트 신호가 입력되지 않은 것으로 판단하면, 이전 영상 프레임에 대응하는 인덱스(

)인 15를 키 프레임 인덱스(

)(B2)로 저장할 수 있다. Then, the next video frame (

:16) is read (i=i+1) and the next video frame (

:16) as the current image frame and the image change detection module 121b determines that the first user event signal is not input, the index (

), which is 15, is the key frame index (

) (B2).

For reference, when it is determined that the first user event signal is not input, the difference between the current image frame and the previous image frame is less than a predetermined first threshold, and a separate capture operation by the user is not performed, and the user It may mean a section C2 in which an afterimage occurs due to performing a previous capture operation by .

:11,

:15)은 각각 전체 큐(Queue) 영역의 일부 부분 영역들 중에서 각각 첫번째로 입력된 것이다. 즉, 키 영상(

:11)은 전체 큐 영역의 제1 부분 영역(즉, 키 영상의 키 인덱스 11과 동일한 '11'을 가지는 영상들을 저장하기 위한 영역) 중에서 첫번째로 입력된 것이고, 키 영상(

:15)은 전체 큐 영역의 제2 부분 영역(즉, 키 영상의 키 인덱스 15와 동일한 '15'를 가지는 영상들을 저장하기 위한 영역) 중에서 첫번째로 입력된 것이다. 큐의 각 부분 영역 중에서 첫번째로 입력된 것의 의미는, 본 발명에서 큐는 환형 큐를 이용할 수 있고, 이 때, 큐에 가장 먼저 입력되어 가장 오랫동안 머무른 영상을 키 영상으로 꺼낼 수 있음을 의미한다. On the other hand, in Figure 5a, the key image (

:11,

:15) is input first among some partial areas of the entire queue area, respectively. That is, the key image (

:11) is the first input of the first partial area of the entire cue area (that is, an area for storing images having the same '11' as the key index 11 of the key image), and the key image (

:15) is the first input of the second partial area of the entire cue area (ie, an area for storing images having the same '15' as the key index 15 of the key image). The meaning of the first input among each subregion of the cue means that in the present invention, the cue may use a circular cue, and in this case, the image that is first input to the cue and stayed the longest may be taken out as the key image.

As a result, it is possible to store a stable frame with almost no image change as a key image.

Or, referring to FIG. 5B according to another embodiment, the key image is an image having the highest prediction probability value through a model for predicting a lesion among images stored in a partial region of the entire cue region for storing image frames. can Specifically, the partial region of the entire queue region is for storing the images having the same value as the key index corresponding to the key image, and as shown in FIG. 5A , the first partial region of the entire queue region (ie, the key image) A region for storing images having the same '11' as the key index 11) or a second partial region of the entire cue region (that is, an area for storing images having the same '15' as the key index 15 of the key image) one can And, taking the first partial region as an example, an image having the highest prediction probability value 88 through a model for predicting a lesion, which is one of the neural network models 121a, may be stored as a key image.

)은 영상 변화 감지 모듈(121b)로 입력되면서 병변을 예측하기 위한 모델로도 입력되어 해당 모델을 통한 출력값의 정보를 함께 활용하여 키 영상을 선정할 수 있다. 이 때, 프레임 인덱스 정보, 키 프레임 인덱스 정보, 및 병변을 예측하기 위한 모델을 통한 예측 확률값 중 적어도 하나는 각 영상을 저장하기 위한 큐 영역에 1 대 1 대응하여 각각 저장 및/또는 관리될 수 있다. That is, each input image frame (

) is input to the image change detection module 121b and is also input as a model for predicting a lesion, so that a key image can be selected by using information of an output value through the corresponding model. At this time, at least one of frame index information, key frame index information, and a prediction probability value through a model for predicting a lesion may be stored and/or managed in a one-to-one correspondence with a queue area for storing each image. .

As a result, it is possible to store an accurate frame with a higher lesion prediction probability as a key image by reflecting the output value through the neural network model 121a while being stable because there is little change in the image.

6 and 7 are diagrams referenced to explain a process of determining whether a difference value between a current image frame and a previous image frame is calculated and a predetermined first threshold is exceeded.

,

)에 상이한 이진값을 할당하며, (c) 도 6c와 같이 이진값들의 합을 기초로 차이값(

을 산출할 수 있다. Specifically, (a) as shown in FIG. 6A , the image change detection module 121b calculates the difference in brightness between the current image frame and the previous image frame for each channel, and whether the calculated difference in brightness exceeds a second predetermined threshold According to (b) each map configured for each channel as shown in FIG. 6b (

,

) is assigned a different binary value, and (c) the difference value (

can be calculated.

)를 초과한 경우, 각 맵(

,

)에 '1'을 할당하고, 그 밖의 경우에는 각 맵(

,

)에 '0'을 할당할 수 있다. 이러한 동작은 현재 영상 프레임과 이전 영상 프레임의 각 픽셀별 비교를 통해 수행될 수 있으며, 각 맵(

,

) 역시 픽셀별로 구분되어 상기 할당된 이진값들을 저장할 수 있다. 그리고, (c)를 살펴보면, 모든 픽셀들에 할당된 각 이진값들의 총합을 각 채널에 대응하는 모든 픽셀들의 개수로 나누어 차이값(

을 산출할 수 있다. 이러한 차이값(

은 영상차를 나타내는 것으로, 각 픽셀별 평균 맵 변화율로도 정의될 수 있다. 또한, 차이값(

이 소정의 제1 임계치(

)를 초과한 경우, 영상 변화 감지 모듈(121b)은 의료 영상 장치(110)의 영상 프레임에 대한 캡쳐를 수행하기 위한 제1 사용자 이벤트 신호가 입력되어 영상의 변화가 있는 것으로 판단할 수 있다.First, looking at (a) and (b) together, the image change detection module 121b calculates the absolute value of the difference between the brightness of the current image frame and the previous image frame for each channel (R, G, B), The absolute value of the calculated difference in brightness is set to a predetermined second threshold (

), each map (

,

) to '1', otherwise each map (

,

) can be assigned '0'. This operation can be performed by comparing each pixel of the current image frame and the previous image frame, and each map (

,

) can also be divided for each pixel to store the allocated binary values. And, looking at (c), the difference value (

can be calculated. These differences (

denotes an image difference, and may also be defined as an average map change rate for each pixel. Also, the difference (

This predetermined first threshold (

) is exceeded, the image change detection module 121b may determine that there is a change in the image due to the input of the first user event signal for capturing the image frame of the medical imaging apparatus 110 .

)를 0.005 이상 0.02 이하로, 제2 임계치(

)를 15 이상 40 이하로 설정할 수 있다. 제1 임계치(

)는 영상의 변화를 판단하기 위한 임계치이고, 제2 임계치(

)는 맵 생성을 위한 임계치로 정의될 수 있다. According to the embodiment, preferably, the control unit 121 is a first threshold (

) to 0.005 or more and 0.02 or less, the second threshold (

) can be set to 15 or more and 40 or less. the first threshold (

) is a threshold for determining the change of the image, and the second threshold (

) may be defined as a threshold for map generation.

)이 정해지기 전의 인접한 영상 프레임들 사이의 영상 차이를 나타내며, (a): 제2 임계치(

)를 1로 설정한 경우 맵의 예를 나타내고, (b): 제2 임계치(

)를 15로 설정한 경우 맵의 예를 나타낸다. On the other hand, referring to Figure 7, the key image (

) represents the image difference between adjacent image frames before being determined, (a): the second threshold (

) shows an example of a map when set to 1, (b): the second threshold (

) is set to 15, an example of a map is shown.

)를 1로 설정하면 영상 차이를 식별하여 (a)와 같이 불균일한 영상이 출력되는 반면, 제2 임계치(

)를 15로 설정하면 (b)와 같이 영상 차이를 식별하지 않아 균일한 영상으로 출력될 수 있게 된다. For example, when the image difference between adjacent image frames is output as a value of about '7', the second threshold (

) is set to 1, the image difference is identified and a non-uniform image is output as shown in (a), while the second threshold (

) is set to 15, as in (b), image differences are not identified, so that a uniform image can be output.

)를 15 이상 40 이하로 설정할 수 있다. That is, if the difference of about '7' is not a difference that occurs when there is a user event, the situation as in (a) can only cause noise, so the situation in (b) is more preferable. Therefore, in the present invention, the second threshold (

) can be set to 15 or more and 40 or less.

)이 정해진 후의 키 영상 및 키 영상과 인접한 영상 프레임 사이의 영상 차이를 나타내며, (a): 제2 임계치(

)를 1로 설정한 경우 맵의 예를 나타내고, (b): 제2 임계치(

)를 15로 설정한 경우 맵의 예를 나타낸다. On the other hand, Figure 8 is a key image (

) represents the key image and the image difference between the key image and an adjacent image frame, (a): the second threshold (

) shows an example of a map when set to 1, (b): the second threshold (

) is set to 15, an example of a map is shown.

Also, in the case of FIG. 8, the technical features described above in FIG. 7 may be applied in the same/similar manner.

9 and 10 illustrate a screen output to the display unit 124 according to an embodiment. The user captured image of FIG. 9 corresponds to a lesion prediction region, and the user captured image of FIG. 10 corresponds to an actual lesion occurrence region.

First, referring to FIGS. 9 and 10 (a) , the screen output to the display unit 124 is a result that can be output when an operation is performed through 1) a user captured image, and 2) a neural network model 121a for a lesion. A plurality of item images corresponding to the values (analysis result), 3) heat map image, 4) index image (saved image), 5) other patient information, environment setting information, connection status with the medical imaging apparatus 110, etc. It may be displayed on the screen of the display unit 124 . In addition, each piece of information may be displayed separately in a separate box area on the entire display screen. In relation to 2), the controller 121 may differently display a plurality of items corresponding to the respective result values (Lesion, Mucosa, SubMucosa, Differentiated, and Undifferentiated) in different sizes, patterns, colors, and the like. In relation to 3), the heat map is a map for identifying a lesion region included in an image frame, and the controller 121 generates a heat map based on the output result value, and applies a threshold value to the heat map to generate a heat map. By extracting (or separating) the lesion region from the map, the lesion region included in the image frame may be identified based on the extracted image.

And, referring to FIGS. 9 and 10 (b), the control unit 121 may further display an index image corresponding to the user captured image on the screen of the display unit 124, which is also displayed on the stored image list. can be added to Also, referring to FIGS. 9 and 10 ( c ), a key image corresponding to the index image may be stored in the memory 123 .

11 is a diagram illustrating a structure of a neural network model 121a according to an embodiment.

The neural network model 121a according to the embodiment corresponds to (a) a lesion detection model, (b) a lesion type determination model, (c) a lesion location information determination model on an image frame, and (d) a lesion location information. It may be any one of a model for determining whether to proceed with an examination related to a lesion, (e) a recommended examination time notification model, and (f) a risk notification model for lesion occurrence. However, this is an embodiment, and if it is a model implemented in relation to lesion detection, it may be applied to the present invention in the same/similar manner, and at least one or more of the models may be used in combination with each other.

In FIG. 11, NT classes (T ₁ , T ₂ , ... T _t , ... T _NT ) exist, class T ₁ includes two characteristics, and class T ₂ includes two characteristics. It is assumed that n properties belong to class T _t , and three properties belong to class T _{NT .}

Referring to FIG. 11 , a neural network model 121a according to an embodiment of the present invention includes a convolution layer 10 for performing a convolution operation on an input image xi, and The pooling layer 20 for performing pooling on the output (oconv), and each corresponding to a plurality of classes into which the complex characteristic is classified, the weight w _fc (T _{t ) for each class for the output of the pooling layer 20} )) to output a value multiplied by a plurality of class-specific perfect coupling layers 30 .

The neural network model 121a corresponds to the plurality of class-specific fully coupled layers 30, respectively, and a plurality of class-specific classifiers 40 for calculating the characteristic probability for each class according to the output of each class-specific fully coupled layer 30. may further include.

The convolution layer 10 extracts a feature map by performing a convolution operation on the input image using a plurality of convolution filters. 11 , the convolution operation may be performed multiple times. It is assumed that the output of the convolution layer 10, that is, the feature map oconv, has a height (H), a width (W), and the number of channels (C).

The pooling layer 20 is located between the convolutional layer 10 and the fully coupled layer 30, and reduces the size of the feature map (oconv) to reduce the calculation required in the fully coupled layer 30 to be described later, It serves to prevent overfitting. The pooling layer 20 may perform global average pooling of outputting an average value for each channel of the feature map (oconv).

The class-specific fully coupled layer 30 is the output of the pooling layer 20 for each class weight (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _fc (T _NT )) and output the multiplied value. At this time, each of the weights for each class (w _fc (T ₁ ), w _fc (T ₂ ), ... w _fc (T _t ), ... w _fc (T _NT )) is a plurality of channels corresponding to the number of channels. can be a value.

The classifier 40 for each class corresponds to the fully coupled layer 30 for each class, and calculates the characteristic probability for each class according to the output of the perfectly coupled layer 30 for each class. 11, the sorter for the class T ₁ corresponds to the class T probability corresponding to the characteristic of belonging to the _{1 (P 1 (T 1)} , P 2 (T 1)) computing a, and class T _{2 classifier computes the probability (P 1} (T ₂ ), P ₂ (T ₂ )) corresponding to each feature belonging to class T ₂ , and the classifier corresponding to _{class T t} calculates the probability of each feature belonging to _{class T t .} probability that corresponds to the sorter for calculating the _{(P 1 (T t),} P 2 (T t), ... Pn (T t)) and corresponds to the class T _NT is equivalent to each of the attributes belonging to the class T _NT Calculate the probability (P ₁ (T _NT ), P ₂ (T _NT ), P ₃ (T _NT )). As the classifier 40 for each class, for example, a Softmax function, a Sigmoid function, or the like may be used.

The embodiments described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Examples of the computer-readable recording medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, and a magneto-optical medium such as a floppy disk. optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules for carrying out the processing according to the present invention, and vice versa.

Aspects herein may take the form of a computer program product embodied on one or more computer readable media embodied in hardware entirely, software entirely (including firmware, resident software, microcode, etc.) or computer readable program code. .

Features, structures, effects, etc. described in the above embodiments are included in one embodiment of the present invention, and are not necessarily limited to one embodiment. Furthermore, features, structures, effects, etc. illustrated in each embodiment can be combined or modified for other embodiments by those of ordinary skill in the art to which the embodiments belong. Therefore, contents related to such combinations and modifications should be construed as being included in the scope of the present invention.

Claims (10)

In the method of detecting a user event through image analysis composed of image frames,
(a) receiving, by an image receiving apparatus, a current image frame from a medical imaging apparatus through a communication unit;
(b) calculating, by a controller, a difference value between the current image frame and a previous image frame;
(c) determining, by the controller, whether a first user event signal for capturing the current image frame is input to the medical imaging apparatus according to whether the difference value exceeds 0.005 or more and 0.02 or less; and
(d) when the controller determines that the first user event signal is input, capturing the current image frame;
(e) determining, by the controller, whether a second user event signal for storing the captured image obtained by performing the capturing of the current image frame is input; and
(f) when the control unit determines that the second user event signal is input, generating an index image corresponding to the captured image;
The step (f) is,
Further comprising; storing the captured image as a key image by the control unit;
The key image is an image having the highest prediction probability value through a model for predicting a lesion among images stored in a partial region of the entire cue region for storing the image frame,
The partial region of the entire queue region is for storing the images having the same value as the key index corresponding to the key image,
A method of detecting user events through image analysis composed of image frames.
delete delete The method of claim 1,
The step (b),
(b1) calculating, by the controller, a difference in brightness between the current image frame and the previous image frame for each channel;
(b2) assigning, by a controller, different binary values to each map configured for each channel according to whether the calculated difference in brightness exceeds 15 or more and 40 or less; and
(b3) calculating, by the control unit, the difference value based on the sum of the binary values;
A method of detecting user events through image analysis composed of image frames.
delete The method of claim 1,
The key image is first input among some regions of the entire cue region for storing the image frame,
The partial region of the entire queue region is for storing images having the same value as the key index corresponding to the key image,
A method of detecting user events through image analysis composed of image frames.
delete An apparatus for detecting user events through image analysis composed of image frames,
a communication unit configured to receive a current image frame from the medical imaging apparatus; and
A difference value between the current image frame and the previous image frame is calculated, and when the difference value exceeds 0.005 or more and 0.02 or less, a first user event signal for capturing the current image frame is input to the medical imaging apparatus When a second user event signal for performing the capture of the current image frame and storing the captured image obtained by performing the capturing on the current image frame is inputted, corresponding to the captured image Including; a control unit for generating an index image to
The control unit stores the captured image as a key image,
The key image is an image having the highest prediction probability value through a model for predicting a lesion among images stored in a partial region of the entire cue region for storing the image frame,
The partial area of the entire queue area is for storing the images having the same value as the key index corresponding to the key image,
A device for detecting user events through image analysis composed of image frames.
The method of claim 8,
Further comprising; a display unit for outputting together the index image and a screen including an image listed in correspondence to the index image;
User event detection device through image analysis composed of image frames.
delete

Images