JP6355146B1 - Medical safety system - Google Patents

Abstract

A user-friendly medical safety system is provided that is simpler than the prior art. A server apparatus 120 for storing a developed video obtained by wide-angle imaging of an operating room and a partial video that is a part of the developed video are displayed and imaged on the developed video by image recognition processing on the developed video. A mobile terminal 132 for specifying a field. The mobile terminal 132 is characterized in that display position adjustment for adjusting the specified operative field to be included in the partial video is executed according to a predetermined trigger. [Selection] Figure 1

Description

  The present invention relates to a medical safety system.

In recent years, awareness of problems related to medical errors and medical accidents has improved as a whole, and there has been an increasing demand for information disclosure to medical institutions. As part of efforts to respond to such demands from society, a surveillance camera is installed in the facility, and various events that occur in the facility are photographed and left as evidence (hereinafter referred to as a medical safety system). Have been introduced in some medical institutions.
The following Patent Document 1 and Patent Document 2 are exemplified as prior art documents disclosing techniques applicable to this type of medical safety system.

  Patent Document 1 discloses a technique for displaying a face-to-face image by cutting out a part of the video from the all-around monitoring video taken by the monocular camera according to the position information specified by the user and correcting the distortion of the cut out part. ing.

  In Patent Document 2, when moving image data obtained by photographing in a city is encoded and an important part (for example, a pedestrian) photographed therein is specified, the encoding method of the important part is changed. Thus, a technique for highlighting important parts when moving image data is reproduced is disclosed.

JP 2012-244480 A JP-A-2005-260501

When watching video obtained by shooting a wide area for a long time like a surveillance camera, it is difficult for the user to identify the important point that he wants to check, even if the important point is shot in the video The user may not be able to confirm with sufficient accuracy.
Applying the technology disclosed in the above prior art documents can achieve a certain improvement, but it is still not sufficient.

  The present invention has been made in view of the above problems, and provides a simpler medical safety system than the prior art from the viewpoint of user friendliness.

  According to the present invention, the storage means for storing the developed video obtained by photographing the operating room at a wide angle, the partial video display means for displaying the partial video that is a part of the developed video, and the image recognition processing for the developed video, Surgical field specifying means for specifying the surgical field photographed in the unfolded video, and the partial video display means adjusts so that the surgical field specified by the surgical field specifying means is included in the partial video A medical safety system that performs display position adjustment according to a predetermined trigger is provided.

  According to the above invention, since the display position of the partial video is adjusted so as to display the operative field specified by the image recognition processing, the operative field is included even if the user does not adjust the display position while watching the developed video by himself / herself. Partial video can be displayed.

  ADVANTAGE OF THE INVENTION According to this invention, the medical safety system simpler than a prior art is provided from a user friendly viewpoint.

It is a figure which shows the medical safety system which concerns on this embodiment. It is a perspective view of a hemispherical camera. It is a figure which shows an example of the expansion | deployment image | video image | photographed with the hemispherical camera. It is a figure which shows an example of the partial image | video displayed by a mobile terminal. It is a schematic diagram for demonstrating the image recognition process of a mobile terminal. It is a schematic diagram for demonstrating the image recognition process of a mobile terminal. It is a schematic diagram for demonstrating the image recognition process of a mobile terminal. It is a schematic diagram for demonstrating the image recognition process of a mobile terminal. It is a specific example of the display by a personal computer terminal.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same components are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

<About each component included in the medical safety system 100>
FIG. 1 is a diagram showing a medical safety system 100 according to the present embodiment.
Note that the arrows shown in FIG. 1 indicate the output source and input destination of video data exchanged between the components. Therefore, the transmission / reception of data other than video data does not necessarily match the transmission / reception direction indicated by each arrow.

  The medical safety system 100 includes a hemispherical camera 110, a server device 120, and a viewing terminal device (for example, a personal computer terminal 131 and a mobile terminal 132).

The hemispherical camera 110 is a device that performs wide-angle imaging of the operating room.
Here, wide-angle shooting means that a single-lens wide-angle lens is used to obtain a wider range of images than usual, and a plurality of lenses (which may be either standard lenses or wide-angle lenses) are directed in different directions. This is to obtain a wider range of images than usual by combining multiple shot images.
The hemispherical camera 110 used in this embodiment has three wide-angle lenses arranged at intervals of 120 degrees, and three developed images obtained by combining three images captured using each wide-angle lens by software processing (image processing). Get. Since such processing is performed, the unfolded image obtained by photographing with the hemispherical camera 110 has a feature that the unfolded angle in the horizontal direction reaches 360 degrees.
Therefore, by installing the hemispherical camera 110 in the operating room, it is possible to capture the entire view of the operating room without omission, and in addition to the state of the vicinity of the operating field, the operation of each medical person moving in the operating room, The medical device screen displaying the vital signs can be taken at once. Since it is difficult to tamper with the video shot in this way, it is possible to sufficiently ensure its certainty when used as situation evidence related to the treatment.

FIG. 2 is a perspective view of the hemispherical camera 110.
The hemispherical camera 110 includes a pedestal part 111, a support part 112, and a main body part 113. The main body 113 has three wide-angle lenses, of which the lens 114 and the lens 115 are shown in FIG.
The main body portion 113 is a portion having the main function (including a photographing function) of the hemispherical camera 110, and is coupled to the pedestal portion 111 by the support portion 112. The pedestal 111 is preferably installed above the surgical field, and may be installed directly on the ceiling of the operating room, or a dedicated column (not shown) is extended above the surgical field. May be installed.

As shown in FIG. 2, the axial direction of each wide-angle lens (lens 114 or lens 115) provided in the main body 113 is opposite to the pedestal 111, i.e., above the operative field. In the premise of installation, it is inclined downward from the horizontal direction. With such a structure, the hemispherical camera 110 can shoot a hemispherical image in the downward direction (an image in which the horizontal development angle reaches 360 degrees and is taken without omission in the downward direction). it can. The developed image does not necessarily have to be a hemispherical image, and may be a whole celestial sphere image (an image in which the expansion angle reaches 360 degrees both in the horizontal direction and in the vertical direction), or in the horizontal direction or the vertical direction. An image having a development angle of less than 360 degrees for at least one of them may be used.
The hemispherical camera 110 illustrated in FIG. 2 is an example of a unit that captures a developed image used in the present invention, and the imaging unit may not necessarily be included as a component of the present invention. Even when the present invention is implemented by including the photographing means in the constituent elements of the present invention, the photographing means may not have the structure as described above. For example, the type of lens of the photographing unit is not necessarily a wide-angle lens, and the number of lenses of the photographing unit may be increased or decreased.

FIG. 3 shows a specific example of a developed image taken by the hemispherical camera 110.
In the upper part of the developed video, a display device 201 near the ceiling of the operating room, a guide rail 202 provided for sliding the surgical light, and the like are photographed. There may be a subject that is distorted so much that it cannot be easily identified, such as the display device 201 and the guide rail 202 shown in FIG.
In addition, a plurality of medical personnel (surgeon 204, assistant 203, medical staff 205 to 211) are photographed in the developed video. In the following description, these medical personnel may be collectively referred to as a practitioner.

The server device 120 is a device that stores images related to treatment.
The video stored in the server device 120 includes at least a developed video of the operating room taken at a wide angle by the hemispherical camera 110 and functions as a storage unit of the present invention.
In addition, the video stored in the server device 120 may include a video acquired from an imaging device or medical device (not shown), and these imaging device or medical device is an internal configuration of the medical safety system 100. Alternatively, an external configuration may be used.

The personal computer terminal 131 and the mobile terminal 132 are computer devices in which application software (viewer) for displaying video stored in the server device 120 is installed.
The mobile terminal 132 is installed with a viewer that is intended for use by a medical staff (anesthesiologist, etc.) waiting outside the operating room to check the state of the operation being performed in the operating room. The video stored in can be received and displayed by live streaming.
The personal computer terminal 131 has a viewer installed mainly for the purpose of analyzing the content of surgery after surgery, and in addition to a function related to playback of video stored in the server device 120, a function of editing the video for data Etc.
The viewer installed in the personal computer terminal 131 and the mobile terminal 132 is not necessarily realized by application software dedicated to the present invention, and may be realized by general-purpose application software or software obtained by improving or changing the application software.

  Note that only one of the functions of the personal computer terminal 131 and the mobile terminal 132 described in the present embodiment is not necessarily executable, and part or all of the functions described as one function can be executed by the other. There may be.

<Display on mobile terminal 132>
Next, display on the mobile terminal 132 will be described.
Since the display area of the mobile terminal 132 is smaller than the display area of the personal computer terminal 131, it is difficult to view the entire developed image captured by the hemispherical camera 110. Accordingly, the mobile terminal 132 has a function of displaying a part of the developed video (hereinafter referred to as a partial video) in a limited manner. That is, the mobile terminal 132 functions as a partial video display unit according to the present invention.

FIG. 4 is a specific example of a partial video displayed by the mobile terminal 132.
As shown in FIG. 4, it is preferable that the partial video displayed by the mobile terminal 132 is displayed face-to-face after performing distortion correction. This is because it is easy for the user to visually recognize.

The display position of the partial video displayed by the mobile terminal 132 is preferably adjustable according to the user's operation, and can be displayed over the entire circumference at least in the horizontal direction (functions as a so-called panoramic viewer). More preferred.
As a function related to this, the mobile terminal 132 identifies the operative field photographed in the developed video by image recognition processing on the developed video, and adjusts the display position to adjust the specified operative field to be included in the partial video. Is executed according to a predetermined trigger. That is, the mobile terminal 132 functions as an operative field specifying unit according to the present embodiment.
Here, the predetermined trigger is not particularly limited as long as the mobile terminal 132 can recognize the event (event). For example, the mobile terminal 132 may activate a partial video display function, or may be mobile. The terminal 132 may accept a specific operation. However, the specific operation handled as the predetermined opportunity is preferably a simple operation (for example, a single operation) from the viewpoint of user friendliness.
As described above, since the mobile terminal 132 has a function of automatically adjusting the display position to the operative field in accordance with a predetermined opportunity, the user can save time and trouble searching for the operative field while checking the partial video. Since the operative field is one of the locations that are particularly desired to be watched in the unfolded video of the operating room, the above function for adjusting the display position of the mobile terminal 132 to the operative field is extremely useful from the viewpoint of user friendliness.

The above-described image recognition processing of the mobile terminal 132 will be described in detail.
In order to make the image recognition processing generic, the present inventor decided to adopt a method of specifying the surgical field by image recognition processing for detecting the body part of the practitioner. This is because a surgical operation is generally performed as a team of a plurality of practitioners, and there is almost no possibility that a subject to be processed does not exist in the case of the image recognition processing as described above.
In addition, if specializing in a procedure using a specific surgical tool or medical device (including a medical robot), the surgical tool or the medical device is detected by image recognition processing instead of or in addition to the body part of the operator. It is also possible.

In this embodiment, “detecting the body part of the practitioner” is not limited to the process of detecting only the actual body part of the practitioner. For example, the eye of the practitioner is detected by detecting protective glasses. It also includes detecting the surgeon's head by detecting it, detecting the surgical cap, and detecting the surgeon's mouth by detecting the surgical mask.
The method of “image recognition processing for detecting the body part of the practitioner” can be selected as appropriate. As a result of trial and error by the present inventor, the method of extracting the shape (contour) of the body part is the most versatile result, but in the operation using a surgical light, the color and brightness of the operative field are taken into account. If the body part is extracted, the detection accuracy can be increased. In addition, depending on the type of the body part to be targeted, it may be possible to extract the body part in consideration of the movement (motion pattern).

A specific example of the image recognition processing of the mobile terminal 132 will be described with reference to FIGS.
5 and 6 are schematic diagrams for explaining the image recognition processing of the mobile terminal 132, and are different from images actually displayed. A shaded portion in these drawings will be described as a body part detected by image recognition processing by the mobile terminal 132.

In this specific example, the body part detected by the mobile terminal 132 is the practitioner's hand or arm.
For example, it is assumed that the hands and arms of the surgeon 204, the assistant 203, the medical staff 206, and the medical staff 207 are detected by the image recognition process for the developed video illustrated in FIG. 3 (see FIG. 5).
Here, regarding the medical staff 205, the medical staff 208, and the medical staff 209, the hands and arms cannot be detected by the image recognition process because the hands and arms are not sufficiently photographed behind other subjects. Further, since the medical staff 210 and the medical staff 211 are separated from the hemispherical camera 110 and are not photographed with a sufficient size, hands and arms cannot be detected by the image recognition process.
As described above, when a plurality of body parts photographed in the developed video are detected by the image recognition process, the mobile terminal 132 determines the vicinity of the position where the detected hands and arms (body parts) are densely located. It is specified as OF (see FIG. 6).
Here, the vicinity of the surgeon 204 and the assistant 203 corresponds to the surgical field OF.

Next, a specific example different from the image recognition process will be described with reference to FIGS.
7 and 8 are schematic diagrams for explaining the image recognition processing of the mobile terminal 132, similarly to FIGS. 5 and 6, and are different from the actually displayed images. A shaded portion in these drawings will be described as a body part detected by image recognition processing by the mobile terminal 132.

In this specific example, the body part detected by the mobile terminal 132 is a practitioner's face (head), and image recognition processing using both eyes as feature points is performed for the detection.
For example, it is assumed that the faces of the surgeon 204 and the medical staff 206 to 209 are detected by the image recognition process on the developed video illustrated in FIG. 3 (see FIG. 7).
Here, since the face of the assistant 203 is sideways and the medical staff 205 is rearward, both eyes are not photographed, and the face cannot be detected by the image recognition process. Further, since the medical staff 210 and the medical staff 211 are separated from the hemispherical camera 110 and are not photographed at a sufficient size, the face cannot be detected by the image recognition process.
Then, the mobile terminal 132 determines the position and orientation of the practitioner from the detected face and both eyes, and when a plurality of practitioners whose detected positions are close to a predetermined value or less are detected, the directions of the plurality of practitioners The part where the crosses are identified as the surgical field (see FIG. 8).
Here, the mobile terminal 132 detects the surgeon 204, the medical staff 208, and the medical staff 209 as the practitioners that are close to each other, and sets the gaze direction V4, the gaze direction V8, and the gaze direction V9 respectively. Each is determined. Then, the mobile terminal 132 specifies an area including the position of the intersection point IP1 between the line-of-sight direction V4 and the line-of-sight direction V8 and the position of the intersection point IP2 between the line-of-sight direction V4 and the line-of-sight direction V9 as the operative field OF. In addition, since the line-of-sight direction V8 and the line-of-sight direction V9 do not intersect, they are not used for specifying the operative field OF.

As is clear from the comparison between FIG. 6 and FIG. 8, even if the image recognition process is performed on the same developed image, the position of the surgical field OF specified by the method of the image recognition process to be used varies. sell.
Therefore, it is conceivable to improve the accuracy of specifying the operative field by appropriately changing or combining the methods of image recognition processing by the mobile terminal 132.

<Display on PC terminal 131>
Next, display on the personal computer terminal 131 will be described.
Since the display area of the personal computer terminal 131 is larger than the display area of the personal computer terminal 131, the entire display of the developed image photographed by the hemispherical camera 110 is allowed. The personal computer terminal 131 can also display other videos in parallel in addition to the developed image.
Note that the above description does not deny that the personal computer terminal 131 displays a partial video (functions as a panoramic viewer of a developed image) like the mobile terminal 132 described above.

FIG. 9 is a specific example of display by the personal computer terminal 131.
In the display area DA1, the entire developed image of the operating room photographed by the hemispherical camera 110 is displayed.
In the display area DA2, an image related to a heart rate monitor of a patient who has been treated in the operating room is displayed.
In the display area DA3, an image obtained by photographing the surgical field in the treatment by a photographing device (not shown) is displayed.
The personal computer terminal 131 can analyze the operation by comparing these images, comparing the state of the entire operating room, the state of the surgical field, and the change in heart rate.

Further, a timeline related to the developed video is displayed in the display area DA4. That is, the personal computer terminal 131 functions as a timeline display unit according to the present invention.
The cursor C1 displayed in the display area DA4 displays where on the timeline the video currently displayed in the display area DA1 is.
The tag T1 and the tag T2 displayed in the display area DA4 display the timing at which the beep sound is generated according to the display position on the timeline, and display the device that has generated the beep sound according to the display mode (color, etc.). ing. Here, the beep sound is a sound (warning sound or the like) emitted from the device.

The hemispherical camera 110 can record audio data using a microphone (not shown) in parallel with the shooting of the developed video.
The server device 120 stores the recorded audio data in association with the video data related to the developed video. Furthermore, the server device 120 has a function of detecting a beep sound included in the sound data by sound recognition processing for the sound data, specifying the timing thereof, and specifying a device that has generated the detected beep sound. . That is, the server apparatus 120 functions as a beep sound specifying unit according to the present invention.
By displaying the tag T1 and the tag T2, the personal computer terminal 131 can make the user recognize the timing of the beep sound detected by the server device 120 and the device that has generated the beep sound.

Since the beep sound to be generated differs depending on the device or the manufacturer of the device, it is possible to know what incident occurred at which timing by recording and analyzing the beep sound.
However, a medical safety system having the above functions such as recording a beep sound together with video data for analysis, identifying the beep sound by voice recognition processing, and displaying the timing of the beep sound has not been introduced in the medical field in the past. . Since the medical safety system 100 according to the present embodiment has the above functions, it can be used, for example, to investigate the cause when a medical accident occurs.

<Modification of the present invention>
The present invention has been described with reference to the embodiments described with reference to the drawings so far. However, the present invention is not limited to the above-described embodiments, and various modifications can be made as long as the object of the present invention is achieved. It includes aspects such as deformation and improvement.
In the modification described below, even if the function is described as the function of the personal computer terminal 131 or the function of the mobile terminal 132, only one of the functions described above is not necessarily executable. The other or part may be executable.

In the above description of the embodiment, the description has been made on the premise of the components shown in FIG. 1, but each component of the present invention may be formed so as to realize its function. Therefore, each component of the present invention does not have to be individually independent, and a plurality of components are formed as a single member, and a single component is formed of a plurality of members. It is allowed that a certain component is a part of another component, a part of a certain component overlaps a part of another component, and the like.
For example, the medical video processing system according to the present invention may not include an imaging device corresponding to the hemispherical camera 110, and the present invention may be implemented using a developed video acquired from an imaging device outside the system. Good.

The configuration of the hemispherical camera 110 and the photographing method using the hemispherical camera 110 described in the above embodiment are merely specific examples, and the implementation of the present invention is not limited thereto.
For example, the present invention may be implemented using an imaging device that uses a monocular wide-angle lens, or a developed image captured by this imaging device.

  In the above-described embodiment, the hand, arm, and head are given as specific examples of the body part of the practitioner detected by the mobile terminal 132. However, instead of or in addition to this, other parts may be detected.

  In the above-described embodiment, it has been described that the mobile terminal 132 detects the practitioner's face using both eyes as feature points. However, instead of or in addition to this, other parts (nose, mouth, ears, etc.) are characterized. The position and orientation of the practitioner may be determined by detecting a face by image recognition processing using dots.

  In the above-described embodiment, an example in which there is one candidate for the surgical field identified by the image recognition processing by the mobile terminal 132 has been described. However, when there are a plurality of candidate surgical fields to be identified, there are options for the user. And the vicinity of the operative field selected by the user may be displayed. Alternatively, in such a case, the mobile terminal 132 may sequentially switch and display a plurality of candidates each time a specific operation is received.

  In the above-described embodiment, the display position specified by the image recognition processing by the mobile terminal 132 is only the surgical field, but may additionally have a function of specifying another display position. For example, the medical staff such as the surgeon 204 and the assistant 203 may be identified, or the screen (such as a vital sign) of the medical device reflected in the developed image may be identified.

  In the above-described embodiment, the personal computer terminal 131 has been described as displaying the timeline related to the developed video displayed in the display area DA1 in the display area DA4, but is displayed in the display area DA2 and the display area DA3. A timeline relating to the current video may be further displayed in the display area DA4. That is, the personal computer terminal 131 may display timelines relating to a plurality of images that are synchronously displayed.

This embodiment includes the following technical ideas.
(1) A storage means for storing a developed video obtained by wide-angle imaging of the operating room, a partial video display means for displaying a partial video that is a part of the developed video, and an image recognition process for the developed video, A display position adjustment that adjusts the partial video display means to include the surgical field specified by the surgical field specifying means in the partial video; A medical safety system that executes according to a predetermined opportunity.
(2) The medical safety system according to (1), wherein the surgical field specifying means specifies the surgical field by image recognition processing for detecting a body part of a practitioner.
(3) The body part detected by the surgical field specifying means is a practitioner's hand or arm, and the surgical field specifying means detects a plurality of the body parts photographed in the developed image, The medical safety system according to (2), wherein a position where the detected body parts are dense is specified as the surgical field.
(4) The surgical field specifying means determines the position and orientation of the practitioner from the detected body part, and when a plurality of practitioners whose determined positions are close to a predetermined value or less are detected, the plurality The medical safety system according to (2), wherein the vicinity of the position where the directions of the practitioners intersect is specified as the surgical field.
(5) The developed video stored in the storage means is a video obtained by combining a plurality of videos taken in different directions, and a horizontal development angle reaches 360 degrees. The medical safety system according to any one of (1) to (4).
(6) The storage means stores the audio data recorded in parallel with the shooting of the developed video in association with the video data related to the developed video, and the voice data is obtained by voice recognition processing on the voice data. The medical safety system according to any one of (1) to (5), further comprising a beep sound specifying unit that detects a beep sound included in the beep sound and specifies a device that has generated the detected beep sound.
(7) Timeline display means for displaying a timeline relating to the developed video is provided, and the timeline display means displays the timing at which the beep sound detected by the beep sound detection means is generated on the timeline. The medical safety system according to (6), wherein the medical safety system is displayed.

100 medical safety system 110 hemisphere camera 120 server device 131 personal computer terminal 132 mobile terminal 201 display device 202 guide rail 203 assistant 204 surgeon 205-211 medical staff DA1-DA4 display area C1 cursor T1, T2 tag

Claims (7)

Storage means for storing a developed image of the operating room taken at a wide angle;
Partial video display means for displaying a partial video that is a part of the developed video;
An operative field specifying means for specifying the operative field photographed in the developed video by image recognition processing for the developed video,
The said partial video display means is a medical safety system which performs the display position adjustment which adjusts so that the said surgical field specified by the said surgical field specification means may be included in the said partial video according to a predetermined opportunity.   The medical safety system according to claim 1, wherein the surgical field specifying means specifies the surgical field by image recognition processing for detecting a body part of a practitioner. The body part detected by the surgical field specifying means is a practitioner's hand or arm,
3. The medical safety according to claim 2, wherein when a plurality of the body parts photographed in the developed image are detected, the surgical field specifying means specifies, as the surgical field, a position where the detected body parts are concentrated. system. The surgical field specifying means is
Determine the position and orientation of the practitioner from the detected body part,
The medical safety according to claim 2, wherein when a plurality of practitioners whose detected positions are close to a predetermined value or less are detected, a vicinity of a position where the directions of the plurality of practitioners intersect is specified as the surgical field. system.   The developed image stored in the storage means is an image obtained by combining a plurality of images taken in different directions, and has a horizontal development angle of 360 degrees. 5. The medical safety system according to any one of 4 to 4. The storage means stores audio data recorded in parallel with shooting of the developed video in association with video data related to the developed video,
6. The apparatus according to claim 1, further comprising: a beep sound specifying unit configured to detect a beep sound included in the sound data by sound recognition processing on the sound data and to identify a device that has generated the detected beep sound. Medical safety system according to item. Timeline display means for displaying a timeline related to the developed video,
The medical safety system according to claim 6, wherein the timeline display unit displays the timing at which the beep sound detected by the beep sound specifying unit is generated on the timeline.

Images