JP2020017932A - Medical video system - Google Patents

Abstract

To provide a simpler medical imaging system than conventional technology from the viewpoint of user friendliness.SOLUTION: A mobile terminal 132 includes specifying means for specifying a specific region being captured in a first video by means of image recognition processing on the first video taken by a celestial sphere camera 111, and determination means for determining whether the position where the operation input is received is an operation input allowable area set in the specific region or in the vicinity thereof when an operation input of the user is received in a case in which the first video is displayed. When the determination by the determination means is affirmative, a second video captured by a fixed-point camera 112 is displayed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a medical imaging system.

In recent years, awareness of problems regarding medical malpractice and medical accidents has been increasing as a whole society, and requests for information disclosure to medical institutions have increased. As a part of an effort to respond to such a request from society, a surveillance camera or the like is arranged in a facility, and a system that leaves images of various events occurring in the facility as evidence (hereinafter referred to as a medical image system). May be introduced by some medical institutions.
As a prior art document that discloses a technology applicable to this type of medical imaging system, the following Patent Document 1 is exemplified.

  Patent Literature 1 discloses a technique in which a part of an image is cut out from an entire circumference monitoring image captured by a monocular camera in accordance with position information specified by a user, and the cut out part is corrected for distortion, thereby displaying a directly-facing image. ing.

JP 2012-244480 A

A video obtained by a surveillance camera or the like aiming to capture a wide area for a long time has a low resolution or is distorted due to wide-angle shooting, and the user cannot confirm details with sufficient accuracy. Sometimes.
By applying the technology disclosed in the above-mentioned prior art document, a certain improvement can be achieved, but it is still not enough.

  The present invention has been made in view of the above problems, and has as its object to provide a medical imaging system that is simpler than the related art from the viewpoint of user friendliness.

  According to the present invention, a first image relating to a treatment, a second image obtained by photographing a part of a photographing range of the first image, input means for inputting, the first image input from the input means and Display means capable of displaying the second image, operation position detection means capable of detecting a position at which a user's operation input to the display means is received, and image capturing of the first image by image recognition processing of the first image A specifying unit that specifies a specific region that has been set; and a case where the operation position detection unit receives a user operation input when the display unit displays the first video; Determining means for determining whether or not the specific area or an operation input allowable area set in the vicinity thereof is determined, and the display means determines whether the determination by the determining means is affirmative. Medical imaging system for displaying the second image is provided.

  According to the above invention, when an operation on the display position of the specific area specified by the image recognition processing in the first video is detected, the second video obtained by locally capturing a part of the first video is displayed. The user can confirm in detail the portion corresponding to the specific area by a simple operation (intuitive operation).

  According to the present invention, a medical imaging system that is simpler than the conventional technology is provided from the viewpoint of user friendliness.

It is a figure showing the medical imaging system concerning this embodiment. It is a perspective view of a semi-celestial sphere camera. It is a figure which shows an example of an expansion | deployment image image | photographed by the celestial sphere camera. FIG. 3 is a diagram illustrating a specific example of an image captured by a fixed-point camera. It is a schematic diagram for explaining the image recognition processing of the mobile terminal. It is a schematic diagram for explaining the image recognition processing of the mobile terminal. It is a schematic diagram for explaining the image recognition processing of the mobile terminal. It is a schematic diagram for explaining the image recognition processing of the mobile terminal.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and description thereof will not be repeated.

<About each component included in the medical imaging system 100>
FIG. 1 is a diagram showing a medical imaging system 100 according to the present embodiment.
Note that arrows shown in FIG. 1 indicate output sources and input destinations of video data transmitted and received between the respective components. Therefore, transmission and reception of data other than video data does not necessarily have to coincide with the transmission / reception direction indicated by each arrow.

  The medical imaging system 100 includes a photographing unit (for example, a semispherical camera 111 and a fixed-point camera 112), a server device 120, and a viewing terminal device (for example, a personal computer terminal 131 and a mobile terminal 132).

The celestial sphere camera 111 is a device that captures a wide-angle image of the operating room including the operation field of the operation.
Here, wide-angle shooting refers to capturing images using a monocular wide-angle lens to obtain a wider image than usual, and directing a plurality of lenses (either a standard lens or a wide-angle lens) in different directions. This means that a plurality of captured images are combined to obtain a wider image than usual.
The semi-celestial sphere camera 111 used in the present embodiment has three wide-angle lenses arranged at intervals of 120 degrees, and combines three images captured using each wide-angle lens by software processing (image processing) into one developed image. Get. Since such processing is performed, the developed image obtained by photographing with the celestial sphere camera 111 has a feature that the horizontal development angle reaches 360 degrees.
Therefore, by installing the hemispherical camera 111 in the operating room, it is possible to photograph the entire view of the operating room without omission, and in addition to the state near the operating field, the operation of each medical personnel moving in the operating room, A screen of a medical device displaying vital signs can be photographed at a time. Since it is difficult to falsify the video taken in this way, it is possible to sufficiently ensure the certainty when using the video as situational evidence regarding a treatment.

FIG. 2 is a perspective view of the semi-celestial sphere camera 111.
The celestial sphere camera 111 includes a pedestal portion 116, a support portion 117, and a main body portion 113. The main body 113 has three wide-angle lenses, of which the lenses 114 and 115 are shown in FIG.
The main body unit 113 is a part having a main function (including a photographing function) of the semi-celestial sphere camera 111, and is coupled to the pedestal unit 116 by the support unit 117. The pedestal portion 116 is preferably installed above the operating field of the operation, may be installed directly on the ceiling of the operating room, or extends a dedicated support (not shown) to a position above the operating field to extend the support. May be installed.

As shown in FIG. 2, the axial direction of each wide-angle lens (lens 114 and lens 115) provided on the main body 113 is opposite to the base 116, that is, the base 116 is positioned above the surgical field. It is inclined downward from the horizontal direction on the premise of installation. With such a structure, the hemisphere camera 111 can capture a hemisphere image (an image in which the horizontal development angle reaches 360 degrees and is taken without omission in the downward direction) in the downward direction. it can. It should be noted that the developed video does not necessarily need to be a semi-celestial sphere image, but may be a full celestial sphere image (an image whose development angle reaches 360 degrees in both the horizontal and vertical directions), or may be a horizontal or vertical The image may have an unfolding angle of less than 360 degrees for at least one of the images.
The hemispherical camera 111 illustrated in FIG. 2 is an example of a unit that captures a developed image used in the present invention, and the capturing unit does not necessarily need to be included as a component of the present invention. Further, even when the present invention is implemented by including the photographing means in the components of the present invention, the photographing means may not have the above-described structure. For example, the type of the lens of the photographing unit need not always be a wide-angle lens, and the number of lenses of the photographing unit may be increased or decreased.

FIG. 3 shows a specific example of a developed image captured by the celestial sphere camera 111.
A display device 201 near the ceiling of the operating room, a guide rail 202 provided for sliding an operating light, and the like are photographed above the developed image. As in the case of the display device 201 and the guide rail 202 illustrated in FIG. 3, there may be a subject that is greatly distorted such that it cannot be easily identified.
In addition, a plurality of medical personnel (the surgeon 204, the assistant 203, and the medical staff 205 to 211) are photographed in the developed video. In the following description, these medical personnel may be collectively referred to as a practitioner.

The fixed-point camera 112 is a device that performs fixed-point imaging of the surgical field from a position facing the surgical field of the surgery. Photographing by the fixed-point camera 112 is sufficient for normal photographing (it is not necessary to be wide-angle photographing).
FIG. 4 shows a specific example of an image captured by the fixed-point camera 112. As is clear from the comparison between FIG. 3 and FIG. 4, the image in FIG. 4 can more clearly confirm the situation of the operation field (the movement of the surgeon 204 and the assistant 203 at hand).

  In the following description, the image related to the treatment among the images captured by the celestial sphere camera 111 may be referred to as a “first image”, and the image of the first image capturing range of the image captured by the fixed-point camera 112 may be referred to. A part of the image may be referred to as a “second image”.

The server device 120 receives the first image from the semi-celestial sphere camera 111 and the second image from the fixed point camera 112 and stores them in a predetermined storage area. Therefore, it functions as the input means of the present invention.
In addition, the image stored in the server device 120 may include an image obtained from a not-shown imaging device or a medical device or the like, and these imaging devices or medical devices may be included as an internal configuration of the medical image system 100. Or an external configuration.

The personal computer terminal 131 and the mobile terminal 132 are computer devices in which application software (viewer) for displaying a video stored in the server device 120 is installed.
The mobile terminal 132 is installed with a viewer mainly intended for use by a medical staff (an anesthesiologist or the like) waiting outside the operating room to check the state of a treatment performed in the operating room. Can be distributed and displayed by live streaming.
The personal computer terminal 131 is installed with a viewer mainly intended for analyzing the contents of the operation after the operation, and in addition to the function related to the reproduction of the image stored in the server device 120, the function of editing the image for data. And so on.
Note that the viewer installed in the personal computer terminal 131 and the mobile terminal 132 may not necessarily be realized by the application software dedicated to the present invention, but may be realized by general-purpose application software or software obtained by improving or changing this.

The personal computer terminal 131 and the mobile terminal 132 are both computer devices having a display device and a pointing device, and the type of each display device or the type of each pointing device is not particularly limited.
The display device according to any one of the personal computer terminal 131 and the mobile terminal 132 can display the first video and the second video input from the server device 120, and can constitute the display unit of the present invention.
The pointing device according to any one of the personal computer terminal 131 and the mobile terminal 132 can detect the position at which the user's operation input to the display device (the first image or the second image displayed on the screen) is received. The operation position detecting means of the invention can be constituted.
Note that it is not necessary that only one of the functions of the personal computer terminal 131 and the function of the mobile terminal 132 described in the present embodiment can be executed, and that some or all of the functions described as one function can be executed by the other. There may be. For example, a part or all of the processing regarding the first video and the second video by the mobile terminal 132, which will be described later, may be similarly realized in the personal computer terminal 131.
In addition, the mobile terminal 132 does not necessarily need to be able to execute part or all of the processing relating to the first video and the second video by the mobile terminal 132, which will be described later. , Image recognition processing) on the moving image files of the first video and the second video, and then output to the mobile terminal 132.

<About display on mobile terminal 132>
Next, the display of the mobile terminal 132 will be described.
The mobile terminal 132 is a touch panel capable of acquiring the first video and the second video stored in the server device 120 from the server device 120 and displaying each of them separately or simultaneously. Here, the touch panel is a device in which the screen of the display device is a pointing device.

The mobile terminal 132 has a function of specifying a specific area photographed in the first video by performing image recognition processing on the first video (hereinafter, referred to as specifying means).
The specific region in the present embodiment will be specifically described as an operation field photographed in the first video. However, the present invention is not limited to this. May be set as the specific area.
Note that the image recognition processing for specifying the operation field captured in the first video will be described later.

When receiving the user's operation input while displaying the first video, the mobile terminal 132 determines whether or not the position where the operation input has been received is an operation input allowable region set in the specific region or its vicinity. (Hereinafter, referred to as determination means).
Here, the operation input allowable area is an area on the screen of the mobile terminal 132 that is set based on the processing by the specifying unit, and may be included in the specific area or a part of the specific area. And the rest may deviate from the specific area, or all of them may be deviated from the specific area and located near the specific area.

  The mobile terminal 132 displays the second video when the determination by the determination unit is affirmed. Here, the mode of the second video displayed by the mobile terminal 132 in this case is not particularly limited as long as the user can visually recognize the second video, and the second video is popped up on an upper layer (layer) of the first video. The second video and the first video may be separately displayed in a display area (window) and displayed, or the second video may be displayed after the first video is erased. It may be a mode of doing.

  As described above, the mobile terminal 132 determines whether the position of the operation input received during the display of the first video image of the relatively large area is at or near the operative field (specific area) specified based on the image recognition processing. In the case of the determination area (operation input allowable area) set in the range of, the second image which is relatively easy to confirm can be displayed. Therefore, the user can obtain necessary information from the video regarding the treatment by intuitive operation.

<Image recognition processing to identify the surgical field captured in the first video>
The above-described image recognition processing of the mobile terminal 132 will be described in detail.
The present inventor has adopted a method of specifying an operation field by image recognition processing for detecting a body part of a practitioner to make the image recognition processing general-purpose. This is because a surgical operation is generally performed in a team of a plurality of practitioners, and there is almost no possibility that there is no subject to be processed in the above-described image recognition processing.
In addition, if it is specialized in a treatment using a specific surgical tool or medical device (including a medical robot), the surgical tool or the medical device is detected by image recognition processing instead of or in addition to the body part of the operator. It is also possible.

In the present embodiment, "detecting the body part of the practitioner" is not limited to the processing of focusing on only the actual body part of the practitioner, and for example, the eyes of the practitioner may be detected by detecting safety glasses. This includes detecting, detecting the head of the practitioner by detecting the surgical cap, and detecting the mouth of the practitioner by detecting the surgical mask.
The method of “image recognition processing for detecting a body part of a practitioner” can be appropriately selected. As a result of trial and error by the inventor, the method of extracting the shape (contour) of the body part has the highest versatility. However, in the treatment using the surgical light, the color and brightness of the surgical field are taken into account. If the body part is extracted in this way, the detection accuracy can be increased. Further, depending on the type of the target body part, the body part may be extracted in consideration of the movement (movement pattern).

A specific example of the image recognition processing of the mobile terminal 132 will be described with reference to FIGS.
5 and 6 are schematic diagrams for explaining the image recognition processing of the mobile terminal 132, and are different from the actually displayed video. In these figures, shaded portions will be described as body parts detected by the image recognition processing by the mobile terminal 132.

In this specific example, the body part detected by the mobile terminal 132 is a practitioner's hand or arm.
For example, it is assumed that the hands and arms of the surgeon 204, the assistant 203, the medical staff 206, and the medical staff 207 have been detected by the image recognition processing on the developed video illustrated in FIG. 3 (see FIG. 5).
Here, regarding the medical staff 205, the medical staff 208, and the medical staff 209, the hands and arms cannot be detected by the image recognition processing because the hands and arms are not sufficiently photographed behind other subjects. In addition, the medical staff 210 and the medical staff 211 are far from the semi-celestial sphere camera 111 and are not photographed with a sufficient size, so that hands and arms cannot be detected by the image recognition processing.
As described above, when a plurality of body parts photographed in the developed image are detected by the image recognition processing, the mobile terminal 132 determines the vicinity of the position where the detected hands and arms (body parts) are densely close to the surgical field. It is specified as an OF (see FIG. 6).
Here, the vicinity of the surgeon 204 and the assistant 203 corresponds to the operation field OF.

Next, another specific example different from the above-described image recognition processing will be described with reference to FIGS.
7 and 8 are schematic diagrams for explaining the image recognition processing of the mobile terminal 132, similar to FIGS. 5 and 6, and are different from the actually displayed video. In these figures, shaded portions will be described as body parts detected by the image recognition processing by the mobile terminal 132.

In this specific example, the body part detected by the mobile terminal 132 is the face (head) of the practitioner, and the detection is performed by image recognition processing using both eyes as feature points.
For example, it is assumed that the faces of the surgeon 204 and the medical staffs 206 to 209 have been detected by the image recognition processing on the developed video illustrated in FIG. 3 (see FIG. 7).
Here, since the face of the assistant 203 is facing sideways and the medical staff 205 is facing backward, both eyes are not photographed, and the face cannot be detected by the image recognition processing. In addition, the medical staff 210 and the medical staff 211 are far from the hemispherical camera 111 and are not photographed with a sufficient size, so that a face cannot be detected by the image recognition processing.
Then, the mobile terminal 132 determines the position and orientation of the practitioner from the detected face and both eyes, and when a plurality of practitioners whose detected positions are close to a predetermined value or less are detected, the directions of the plurality of practitioners are determined. Are identified as the surgical field (see FIG. 8).
Here, the mobile terminal 132 detects the surgeon 204, the medical staff 208, and the medical staff 209 as the practitioners in proximity to each other, and determines the gaze direction V4, the gaze direction V8, and the gaze direction V9 of the practitioner. Respectively. Then, the mobile terminal 132 specifies a region including the position of the intersection IP1 between the line of sight V4 and the line of sight V8 and the position of the intersection IP2 between the line of sight V4 and the line of sight V9 as the surgical field OF. In addition, since the line-of-sight direction V8 and the line-of-sight direction V9 do not intersect, they are not used for specifying the operation field OF.

As is clear from a comparison between FIG. 6 and FIG. 8, even if the image recognition processing is performed on the same developed image, the position of the surgical field OF specified by the adopted image recognition processing method fluctuates. sell.
Therefore, it is conceivable to improve the accuracy of specifying the operative field by appropriately changing or combining the method of the image recognition processing by the mobile terminal 132.

<Regarding Modification of the Present Invention>
Although the present invention has been described with reference to the embodiments described with reference to the drawings, the present invention is not limited to the above-described embodiments, and various modifications may be made as long as the object of the present invention is achieved. It also includes aspects such as modification and improvement.
In the modified example described below, even if the function is described as the function of the personal computer terminal 131 or the function of the mobile terminal 132, it is not always necessary that only one of the functions can be executed. Some or all may be executable by the other.

Although the above embodiment has been described on the premise that the components shown in FIG. 1 are used, each component of the present invention may be formed so as to realize its function. Therefore, each component of the present invention does not need to exist independently, and a plurality of components are formed as one member, and one component is formed of a plurality of members. It is permitted that a certain component is a part of another component, that a part of a certain component overlaps a part of another component, and the like.
For example, the medical moving image processing system according to the present invention may not include a photographing device corresponding to the celestial sphere camera 111, and the present invention may be implemented using a developed image acquired from a photographing device outside the system. Good.

The configuration of the celestial sphere camera 111 and the shooting method by the celestial sphere camera 111 described in the above-described embodiment are specific examples, and the present invention is not limited thereto.
For example, the present invention may be implemented using a photographing device using a monocular wide-angle lens, or a developed image photographed by the photographing device.

  In the above embodiment, the hands, arms, and head are given as specific examples of the practitioner's body part detected by the mobile terminal 132. However, other parts may be detected instead or in addition thereto.

  In the above-described embodiment, the description has been given of the case where the mobile terminal 132 detects the face of the practitioner using both eyes as feature points. However, instead of or in addition to this, other parts (nose, mouth, ears, etc.) are characterized. The face and the orientation of the practitioner may be determined by detecting the face by image recognition processing using points.

In the above-described embodiment, the specific area specified by the image recognition processing by the mobile terminal 132 includes only the operative field, but may have a function of specifying another specific area. For example, when the first image is an image of the operating room including the medical device used for the operation and the second image is an image of the medical device facing the medical device, the mobile terminal 132 (identifying unit) The position of the medical device captured in one image may be specified as the specific region.
At this time, the mobile terminal 132 (identifying unit) detects a plurality of markers captured in the first video by the image recognition process, and identifies the position of the medical device based on the detected positions of the plurality of markers. Good (for example, markers are attached to the four corners of the screen of the medical device before surgery, and a rectangular area surrounded by a plurality of markers is specified as the position of the medical device).
Alternatively, the mobile terminal 132 performs pattern recognition on the shape, color, and the like of one or a plurality of medical devices, and specifies, as a medical device, an object captured in the first video that matches the pattern recognition. Is also good.

This embodiment includes the following technical concept.
(1) Input means for inputting a first image relating to a treatment and a second image obtained by photographing a part of the photographing range of the first image, and the first image and the second image inputted from the input means Display means capable of displaying an image, operation position detecting means capable of detecting a position at which a user's operation input to the display means has been received, and image recognition processing on the first image, whereby the first image has been captured. A specifying unit that specifies a specific area; and, when the operation position detecting unit receives an operation input of a user when the display unit is displaying the first video, the position at which the operation input is received is the identification unit. Determining means for determining whether or not the area is an operation input allowable area set in or near the area, wherein the display means is configured to perform the second operation when the determination by the determining means is affirmed. Medical imaging system to display the image.
(2) The first image is an image of an operating room including an operation field, the second image is an image of the operation field, and the identifying unit includes the first image. The medical imaging system according to (1), wherein the operation field captured in the video is specified as the specific area.
(3) The first image is an image of an operating room including a medical device used for treatment, the second image is an image of the medical device taken directly, and the specifying unit includes: The medical image system according to (1), wherein a position of the medical device captured in the first image is specified as the specific region.
(4) The specifying means detects a plurality of markers photographed in the first video by an image recognition process, and specifies the specific region based on positions of the detected plurality of markers. Medical imaging system.
(5) The first image is an image obtained by combining a plurality of images captured in different directions, and the horizontal development angle reaches 360 degrees. The medical imaging system according to any one of the above.

REFERENCE SIGNS LIST 100 Medical imaging system 111 Semispherical camera 112 Fixed point camera 120 Server device 131 Personal computer terminal 132 Mobile terminal 201 Display device 202 Guide rail 203 Assistant 204 Surgeon 205 to 211 Medical staff

Claims (5)

An input means for inputting a first video relating to the treatment and a second video of a part of the shooting range of the first video,
Display means capable of displaying the first video and the second video input from the input means,
Operation position detection means capable of detecting a position at which a user's operation input to the display means is received,
By image recognition processing for the first video, a specifying unit that specifies a specific region that has been captured in the first video,
When the operation position detection unit receives an operation input of the user while the display unit is displaying the first image, the position at which the operation input is received is set to the specific area or its vicinity. Determining means for determining whether or not the operation input is permitted;
Has,
The medical image system, wherein the display unit displays the second image when the determination by the determination unit is affirmed. The first image is an image of the operating room including the surgical field,
The second video is a video taken directly facing the surgical field,
The medical imaging system according to claim 1, wherein the identification unit identifies the operation field captured in the first video as the specific area. The first image is an image of an operating room including a medical device used for the operation,
The second video is a video taken directly facing the medical device,
The medical image system according to claim 1, wherein the specifying unit specifies a position of the medical device captured in the first image as the specific region. The specifying means includes:
Detecting a plurality of markers captured in the first video by image recognition processing,
The medical imaging system according to claim 3, wherein the specific area is specified based on positions of the plurality of detected markers.   5. The image according to claim 1, wherein the first image is an image obtained by combining a plurality of images captured in different directions, and a horizontal development angle reaches 360 degrees. 6. Medical imaging system according to the item.

Images