WO2021220872A1

WO2021220872A1 - System, method, and program for surgery operating information management

Info

Publication number: WO2021220872A1
Application number: PCT/JP2021/015944
Authority: WO
Inventors: 卓也中村; 毅前田; 弘充松浦; 加奈松浦
Original assignee: ソニーグループ株式会社
Priority date: 2020-04-27
Filing date: 2021-04-20
Publication date: 2021-11-04

Abstract

This system for surgery operating management of an embodiment comprises: a data processing unit for obtaining information on a person or an object included in a captured image of a region related to a surgery; and a situation determination unit for, on the basis of the information on the person or the object, estimating a role of the person in the surgery.

Description

Surgical operation information management system, surgical operation information management method and program

This disclosure relates to a surgical operation information management system, a surgical operation information management method and a program.

In order to make an optimal surgery plan, it is necessary to arrange various devices related to surgery and to understand the work contents of each staff member.
It is also hoped that information will be provided to further improve operations.

JP-A-2018-196627

However, the technique described in Patent Document 1 can only determine whether or not the medical staff is in charge of the patient, and cannot determine the role of the medical staff in the operating room, the work content, or the like. rice field.
In addition, the staff in the operating room wear almost the same clothes, and it is not possible to identify an individual (person) with general technology. Furthermore, at present, work records are not kept, so while reducing the work processing burden as much as possible, work information such as work content, work time, and number of work for each role of each staff is collected, and the operation room is operated. It is desired to improve the efficiency of surgery-related work by providing the information necessary for the improvement of the operation.

This technology was made in view of such a situation, and aims to provide a surgery operation information management system, a surgery operation information management method, and a program capable of improving the efficiency of surgery-related work. ..

The surgical operation information management system of the embodiment includes a data processing unit that acquires information related to a person included in a captured image in an operating room or an operating room-related area or information related to an object included in the captured image, and the person. It is provided with a situation determination unit for estimating the role of a person involved in the operation in the operation based on the information related to the item and the information related to the object.

FIG. 1 is a schematic block diagram of the surgical operation information management system of the embodiment. FIG. 2 is a detailed block diagram of a main part of the surgical operation information management system of the first embodiment. FIG. 3 is an explanatory diagram of an example of the determination database. FIG. 4 is a flowchart of the determination database construction process. FIG. 5 is an outline processing flowchart of the process of estimating the role of a person. FIG. 6 is a detailed processing flowchart of a main part of the processing for estimating the role of a person. FIG. 7 is an explanatory diagram of a specific example of role estimation. FIG. 8 is an explanatory diagram (preoperative and intraoperative) of a visualization example of work contents. FIG. 9 is an explanatory diagram (postoperative) of a visualization example of the work content. FIG. 10 is a detailed block diagram of a main part of the surgical operation information management system of the second embodiment. FIG. 11 is an explanatory diagram of an example of data to be added in the second embodiment. FIG. 12 is an explanatory diagram of a configuration example of the determination database of the second embodiment. FIG. 13 is a detailed block diagram of a main part of the surgical operation information management system of the third embodiment. FIG. 14 is an explanatory diagram of an example of acquired data of the data processing unit of the third to fifth embodiments. FIG. 15 is a detailed block diagram of a main part of the surgical operation information management system of the fourth embodiment. FIG. 16 is a detailed block diagram of a main part of the surgical operation information management system according to the fifth embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 is a schematic block diagram of the surgical operation information management system of the embodiment.
The surgical operation information management system 10 performs data processing based on the imaging data and a data collecting device 11 equipped with one or a plurality of cameras that capture images in the operating room before, during, and after surgery and output imaging data. , A management server 14 having a data processing device 12 for determining the operation status of surgery, a determination database (DB) 13 for storing determination data used for determining the operation status of surgery, and the determined operation status of surgery are visualized. It includes an information presentation system 15 for presenting.

Further, the surgical operation information management system 10 is connected to the in-hospital system 20 which can operate in cooperation with each other as needed via a communication network.

In the above configuration, the data processing device 12 processes the imaging data input from the camera, performs various devices, a worker who sets the carry-in of surgical instruments, and a doctor (surgeon, assistant, support) who is actually involved in the surgery. Determine the role of a person such as a doctor) or a nurse.

[1] First Embodiment FIG. 2 is a detailed block diagram of a main part of the surgical operation information management system of the first embodiment.
The data collection device 11 includes a plurality of cameras 11A.
Types of the camera 11A used in the data collection device 11 include a camera having a video function using a CCD / CMOS sensor, a camera having an interval shooting mirror, a camera capable of shooting a moving image or a still image by wire or wirelessly, and the like. Can be mentioned.
It is also possible to use a camera having an IR (near infrared) sensor for distance measurement by TOF (Time Of Flight) or the like.

In this case, the position of the camera in the operating room is such that the operating table in the operating room can be used as the main position for taking a panoramic view of the state of surgery. Although it depends on the size and floor plan of the operating room, the layout of the operating table, and the imaging range of the camera used, it will be installed in a position that does not interfere with the surgery and the flow lines of each staff member.

Specifically, the following positions can be mentioned.
(1) Fixed installation on the ceiling (2) Corner of the operating room (3) Attached to the surgical light or built into the surgical light (4) Attached to the surgical monitor or built into the surgical monitor

It is also possible to use existing cameras such as a surgical field camera, a surgical field camera, an endoscopic camera, and a microscope camera.

Furthermore, it is also possible to use a camera provided outside the operating room to perform preoperative and postoperative monitoring and acquire imaging data.
Such a camera outside the operating room may be installed in a corridor outside the operating room on the floor of the operating unit, an equipment room, or the like.
That is, it can be arbitrarily installed as long as it is possible to image at least the operating room in the area related to the operation in the operating room and outside the operating room.

By the way, when a plurality of cameras are installed, it is necessary to synchronize the plurality of cameras with each other.
The following methods can be mentioned as a method for synchronizing.
(1) Collective camera control by control devices connected to multiple cameras by wire or wireless (2) Camera synchronization operation based on synchronization signals from control devices connected to multiple cameras by wire or wireless (3) Wired or wireless Time code setting for each imaging data by a control device connected to multiple cameras wirelessly, or recording of the time code for imaging data input from the cameras

As the imaging data acquired from the camera, compressed image formats such as PEG, still image data in uncompressed image formats such as BMP, moving image data such as MPEG, still image data cut out from the moving image data, and the like are used.

As shown in FIG. 2, the data processing device 12 includes a data processing unit 12A and a status determination unit 12B.
The data processing unit 12A discriminates an object (person and object) from the captured image corresponding to the captured data acquired by the data collection device 11, and acquires the posture of the person, the position of the person and the object, and the time as basic data.

In this case, the posture of the person and the position of the person and the object are detected by using a general object recognition process by deep learning such as YOLO, a posture / skeleton detection algorithm such as OpenPose, and the like.
The time can be obtained from the time code of the imaging data.

In the data processing unit 12A, examples of objects other than the person to be discriminated include medical devices, medical instruments, medical materials, etc. that are placed in the operating room or carried in the operating room.

Medical devices include monitors, trolleys, endoscopic light sources, endoscopic camera control units (endoscopic CCU), endoscopic camera heads (endoscopic CH), microscopes, ultrasonic diagnostic machines, sphygmomanometers, and pulses. Examples include an oximeter, an electrocardiograph, a thermometer (rectal thermometer, etc.), a urinary catheter, a ventilation mask, and the like.

Examples of instruments include forceps, grasping forceps, needle holders, tweezers, scissors, scalpels, electrosurgical scalpels (bipolar, monopolar), laser scalpels, ultrasonic coagulation and incision devices, staplers, troccers, throat mirrors, and the like.

Examples of medical materials include gauze, needles, tissue adhesive seals, physiological saline, various infusions, blood for transfusion, oxygen cylinders, laughing gas cylinders, and the like.

Further, as the data of the position detected by the data processing unit 12A, the two-dimensional position (for example, (x, y)) obtained by projecting the position of the object onto the floor, wall, ceiling, or the like of the operating room, or the object. 3D positions in the operating room (eg, (x, y, z)) are included.

In this case, as a method of acquiring the three-dimensional position data, information on the two-dimensional position of the object detected by the general object recognition process by deep learning such as YOLO and each camera for the captured images taken by a plurality of cameras respectively. Based on the three-dimensional position information, the angle of view, and the like, a method of calculating from the relationship with the positions of reference points virtually provided in the imaging region (positions provided in the horizontal, vertical, and depth directions) can be mentioned.

Alternatively, a direct distance measuring technique such as TOF or LiDAR (Light Detection And Ringing) may be used.

Here, the configuration of the determination database 13 included in the management server 14 will be described.
The determination database 13 stores the data acquired in the past cases for each case.

FIG. 3 is an explanatory diagram of an example of the determination database.
The determination database 13 includes a plurality of case-specific data 21-1 to 21-n.
Since the case-specific data 21-1 to 21-n have the same configuration, the case-specific data 21-1 will be described below as an example.

A case ID (xxx-yyyy in the example of FIG. 3) for identifying the case is assigned to the case-specific data 21-1, and includes role data 31, work content data 32, and time data 33. There is.

The role data 31 is data representing the role of a person in the operation of the case.
Here, the role is an attribute indicating the position (expected work content or job) assigned to the target person in the surgery.
Examples of the role data 31 include the following roles.
(1) Surgeon (2) Assistant (3) Direct caregiver (instrument delivery)
(4) Indirect caregiving (outside)
(5) Anesthesiologist (6) Peri-anesthesia nurse (7) Clinical Engineer
(8) Operating room staff (carry-in and carry-out)
(9) Contractor (cleaning, disinfection, assistance, etc.)

The work content data 32 is data representing the work content of each person.
The work content data 32 includes, for example, the following roles.

(1) General work contents (1-1) Carrying in equipment (1-2) Carrying out equipment (1-3) Setting equipment (1-4) Accepting patients (1-5) Introducing anesthesia (1-5-1) Securing routes (1-5-2) Sensor mounting (electrocardiogram, sphygmomanometer, pulse oximeter, etc.)
(1-5-3) Tracheal intubation (1-6) Preparation for surgery (1-6-1) Urinary catheterization (1-6-2) Wearing a thermometer (1-6-3) Positioning fixed (1-6-4) ) Heat retention (1-6-5) Instrument deployment (1-6-6) Gauze count (1-7) Recording work (1-8) Call (1-9) Patient condition confirmation (body temperature, blood pressure, urine, compression)
(1-10) Delivery of instruments (1-11) Direct assistance support (delivery of instruments and medical materials with direct caregivers)
(1-12) Awakening of anesthesia (1-12-1) Release of postural fixation (1-12-2) Removal of sensor (1-12-3) Confirmation of arousal (1-12-4) Removal of tracheal intubation tube (1-12) 13) Patient leaving room (1-13-1) Bed transfer (1-14) Cleaning

(2) Procedure Work content (2-1) Grasp (2-2) Hemostasis (2-3) Resection (2-4) Separation (2-5) Suture (2-6) Rope splicing (2-7) Anastomosis

Next, the outline of the situation determination unit 12B will be described.
The situation determination unit 12B compares the data acquired by the data processing unit 12A with the case-specific data 21-1 to 21-n, which are the data of the past cases stored in the determination database 13, and thereby the data processing unit. Each person in the data acquired in 12A estimates the corresponding role and work content. In this case, the role and the work content may be estimated based on the information of the temporal change obtained by using the imaging data of the front and rear frames and the imaging data of a plurality of images.

As the estimation method in the situation determination unit 12B, it is also possible to configure to use machine learning using AI.

In addition to the estimated role and work content of the person, the situation determination unit 12B includes the time from the start of work to the end of work (for example, the time from the start of suturing to the end of suturing) for each role, and key events for surgery. Calculate the relative elapsed time based on (sign-in, timeout, sign-out, etc.).

The information presentation system 15 includes an information presentation processing unit 15A that performs processing for presenting various information, and a display 15B that presents various information under the control of the information presentation processing unit 15A.

Next, a specific processing example of the first embodiment will be described.
First, the process of constructing the judgment database will be described.
FIG. 4 is a flowchart of the determination database construction process.
The database builder first prepares for shooting (step S11).
Specifically, one or more cameras 11A are installed in the target operating room. Then, the position information of the installed camera 11A is acquired.

Subsequently, the actual surgery is performed, imaging is performed from the start to the end of the surgery, that is, before, during, and after the surgery, and the imaging data is collected in the management server 14 (step S12).

At the same time, the data processing unit 12A of the data processing device 12 recognizes a person and an object such as an object from the captured image corresponding to the captured data acquired by the data collecting device 11 by general object recognition processing, and the person by a skeleton detection algorithm or the like. Acquires object recognition information that recognizes the posture, person, and object. In addition, the position information of a person and an object is acquired. Further, time information is acquired as basic data from the time code of the imaging data (step S13).

Specifically, the skeleton recognition result for a person on the screen is acquired for each frame of the imaging data output by each camera 11A, and the person in the three-dimensional coordinate system set in the operating room is obtained from the relationship between the imaging positions of the cameras 11A. Calculate the position, body orientation and hand position.

For example, 18 or more joints are detected as a skeleton, the coordinates of the joints constituting the skeleton in the screen are acquired, and the position of a person in the three-dimensional coordinate system set in the operating room from the relationship between the imaging positions of the cameras 11A. , Calculate the orientation of the body and the position of the hand.

In addition, the object recognition result for the object (person and object) in the screen is acquired for each frame of each camera, and the position of the object in the three-dimensional coordinate system set in the operating room is calculated from the relationship between the imaging positions of the cameras 11A. ..

For example, each object is tagged with a person or an object (an object other than the person: for example, a medical device type), the position of the object (coordinates in the screen of the object) is acquired, and images are taken between the cameras 11A. The position of the object in the three-dimensional coordinate system set in the operating room is calculated from the positional relationship.

Further, the status determination unit 12B of the data processing device 12 sets the relationship between each person and the object between frames of the imaged data based on the orientation and position information of the person and the object by the above-described processing.
For example, set as follows.
(1) When the person and the object maintain the same positional relationship and have the same amount of movement: Set that the person is moving the object.
(2) The object is in the same position and the person is facing the object: Set that the person is involved in the object (for example, performing an operation).
(3) An object is held in the person's hand: Set that the person is holding the object.
(4) Person lying on an object: Set the patient to be lying (sleeping) on a bed or operating table.
(5) The person is facing the patient and the person's hand is in the same position as the patient: Set that the person is involved with the patient.

In addition, the area (area) where the person and the object are located is set based on the position information of the person and the object.
Specifically, the area around the patient is set as a clean area, the area above the sterilized non-woven fabric detected as an object is set as a clean area, and the other areas are set as a dirty area.

Subsequently, the database builder collaborates with an expert to refer to the acquired object recognition information, position information, and time information, and perform tagging and correction (step S14).
More specifically, based on the captured moving images, the work of each staff member (doctor, nurse, etc.), the equipment used by each staff member, the role of each staff member, the cleanliness area, the unclean area, etc. Tag.

For example, when the following situations (1) to (6) are detected, the person A is tagged as "an anesthesiologist performing tracheal intubation".
(1) Position of person A: Head side of patient (2) Position of object: Head side of patient (3) Area: Dirty area (4) Hand of person: Hold object (5) Object: Pharyngeal mirror (6) Posture of person A: facing the patient, leaning forward

Further, if the information automatically acquired by the data processing unit 12A is erroneous information, the erroneous information is corrected.

In this case, by adding the position information of a person or object by the sensor or the voice information by the microphone as supplementary data, it will be used as a reference when referring to the data later.

Similarly, for a plurality of operations, the processes of steps S11 to S14 described above are performed, a data set is constructed for each medical department and procedure, and the determination database 13 is used (step S15).

Next, a process of estimating the role of a person using the determination database 13 will be described.
FIG. 5 is an outline processing flowchart of the process of estimating the role of a person.
The data processing worker first prepares for shooting (step S21).
Specifically, one or more cameras 11A are installed in the target operating room. Then, the position information of the installed camera 11A is acquired.
Alternatively, the acquired position information is acquired and updated for the operating room in which the camera 11A is already installed.

Subsequently, imaging is performed from the start to the end of the surgery, that is, before, during, and after the surgery, and the imaging data is collected in the management server 14 (step S22).

At the same time, the data processing unit 12A of the data processing device 12 recognizes a person and an object such as an object from the captured image corresponding to the captured data acquired by the data collecting device 11 by general object recognition processing, and the person by a skeleton detection algorithm or the like. Acquires object recognition information that recognizes the posture, person, and object. In addition, the position information of a person and an object is acquired. Further, time information is acquired as basic data from the time code of the imaging data (step S23).

FIG. 6 is a detailed processing flowchart of a main part of the processing for estimating the role of a person.
Specifically, the data processing unit 12A performs skeleton recognition for a person on the screen for each frame of the imaging data output by each camera 11A (step S31).
Subsequently, the data processing unit 12A extracts the joint from the skeleton recognition result (step S32).
Next, the data processing unit 12A extracts the joint position and calculates the position of the person, the orientation of the body, and the position of the hand in the three-dimensional coordinate system set in the operating room from the relationship between the imaging positions of the cameras 11A (step S33). ).

For example, the data processing unit 12A extracts 18 or more joints from the skeleton detection result, acquires the coordinates of the joints constituting the skeleton in the screen, and sets them in the operating room based on the relationship between the imaging positions of the cameras 11A. The position of the person, the orientation of the body, and the position of the hand in the three-dimensional coordinate system are calculated (step S34).

In parallel with the processing of steps S31 to S35, the data processing unit 12A recognizes an object other than a person on the screen for each frame of each camera (step S35).

For example, each object is tagged with a person or an object (an object other than the person: for example, a medical device type), the position of the object (coordinates in the screen of the object) is acquired, and images are taken between the cameras 11A. The position of the object in the three-dimensional coordinate system set in the operating room is calculated from the positional relationship (step S36).
Then, the situation determination unit 12B refers to the determination database 13 and tags each object (type of object: for example, setting of type of medical device) (step S37).

Further, the status determination unit 12B of the data processing device 12 calculates the positions of the person and the object in the operating room in a three-dimensional coordinate system based on the orientation and position information of the person and the object by the above-mentioned processing between the frames of the imaging data. (Step S38).

Next, the status determination unit 12B of the data processing device 12 sets the positional relationship between the person and the corresponding object (step S39).
Further, the situation determination unit 12B sets an area (clean area or unclean area) where people and objects exist (step S40).

Subsequently, as shown in FIG. 5, the status determination unit 12B of the data processing device 12 uses the object recognition information, the position information, the time information, and the moving image corresponding to the captured data acquired in the process of step S23 as a determination database. It is compared with the data set stored in 13 and matched by AI (step S24).

Then, the situation determination unit 12B tags each person included in the object recognition information with a role having a high matching rate as the role of the person (surgeon, direct caregiving, anesthesiologist, etc.) (step S25).

Here, the estimation of the role will be specifically described.
FIG. 7 is an explanatory diagram of a specific example of role estimation.
For example, as shown on the left side of FIG. 7, the data processing unit 12A of the data processing device 12 acquires the posture and position of a person, the type and position of equipment as an object, and the shooting time as basic data D0 for each frame. There is.

For example, the frame data No. of FIG. 7 In the case of 1, the posture of the person A (joints 1x, y, ...) And the position of the person A (xx, yy) are stored, and the type A of the equipment M, the product name B, and the position of the equipment M (xw, yy) are stored. doing.

On the other hand, the determination database 13 stores the role, work content, posture, position, equipment used, equipment position, etc. of each person for the past cases as the determination data DR.
For example, as shown on the right side of FIG. 7, for the role: X, two data are stored, one of the data has a work content of F, and the posture of the worker (joints 1x, y, ... ), The position of the worker (ww, yy) is stored, and the type C of the equipment used, the product name D, and the position of the equipment used (xz, yy) are stored. In the other data, the work content is G, and the posture of the worker (joints 1x, y, ...) And the position of the worker (xx, yy) are stored, and the type C of the equipment used, the product name D, and the product name D are stored. The position (xw, yy) of the equipment used is stored.

As a result, the situation determination unit 12B compares the data acquired by the data processing unit 12A with the data stored in the determination database 13, and the work content and role of the person corresponding to the data acquired by the data processing unit 12A. Is estimated, and the time required for each work (working time) and the relative time for the event are calculated.

More specifically, the posture of the person A has a high matching rate with the posture of the role X in the work G, the position of the person A has a high matching rate with the position of the role X in the work G, and the equipment A has a high matching rate of the role X. When the matching rate with the equipment used in the work G is high and the distance between the person A and the equipment A is high and the matching rate between the worker and the equipment used in the work G of the role X is high, the situation determination unit 12B determines. The person A is presumed to be the role X performing the work G.

Then, based on the information obtained by the role estimation process, it is possible to easily grasp the working time, the number of workers (number of workers) in the target surgery (case), and the work required inside and outside the surgery room.

FIG. 8 is an explanatory diagram (preoperative and intraoperative) of a visualization example of work contents.
FIG. 9 is an explanatory diagram (postoperative) of a visualization example of the work content.

As shown in FIGS. 8 and 9, in a predetermined case, equipment delivery, equipment setting, anesthesia introduction, sign-in (safety confirmation when entering the operating room), surgical preparation, time-out (safety confirmation immediately before surgery), procedure, A series of tasks such as indirect caregiving, closure, removal, awakening of anesthesia, sign-out (safety confirmation at the end of surgery), patient leaving, cleaning, etc. can be automatically presented in chronological order in relation to the role of the person. .. The time t1 is the time recognized as the transition time from preoperative to intraoperative, and the time t2 is the time recognized as the transition time from intraoperative to postoperative.

Therefore, since the administrator can easily grasp the work contents and the progress status of the work, it is possible to easily confirm the schedule, the necessity of support, etc., which in turn contributes to the efficiency of the surgery-related work. ..

Therefore, according to the first embodiment, since the progress status in real time, which is the working time related to the patient, can be grasped, it is possible to predict the influence on the schedule of the surgery or the operating room and whether or not support is necessary. ..

It also makes it possible to predict the schedule for similar procedures.
In addition, the difference between team formations in similar cases can be easily grasped, and a more optimal work schedule can be constructed.

Furthermore, it is possible to optimize the work sequence and easily grasp the required number of personnel for each work.
Specifically, by comparing with past data (similar case data in the past), the following (1) to (7) can be examined, predicted, and proposed for improving efficiency.
(1) Examination of workflow,
(2) Examination of optimal personnel allocation for each work,
(3) Examining resource allocation,
(4) Examining team formation,
(5) Predicting the schedule of the entire surgery including the procedure,
(6) Generation of alerts according to the progress of surgery: Supporting personnel, adjusting schedules, presenting the next workflow (7) Predicting and proposing procedures according to cases

[2] Second Embodiment FIG. 10 is a detailed configuration block diagram of a main part of the surgical operation information management system of the second embodiment.
The second embodiment of FIG. 10 differs from the first embodiment of FIG. 2 in that the patient estimation processing unit 12C that estimates the position and posture of the patient and the estimation result (position and posture) of the patient estimation processing unit 12C. It is a point provided with a weighting coefficient addition unit 12D that generates a weighting coefficient based on the surgical procedure information (information on the operation content) acquired from the in-hospital system, adds it to the estimation result of the patient estimation processing unit 12C, and outputs it. ..

By the way, the vector (direction, distance) of the person and the object for which the work content is estimated from the part such as the head side of the patient is almost determined for each surgical procedure.
For example, if there is a person holding a laryngoscope within a few tens of centimeters toward the patient's head, that person is likely to be an anesthesiologist performing a tracheal intubation.

Therefore, for a person who has a laryngoscope within a few tens of centimeters toward the patient's head, the weighting factor of the anesthesiologist is increased as the role of the person, and the weighting coefficient of tracheal intubation is increased as the work content. ..
As a result, it is possible to increase the probability that the estimation in estimating the role and work content of the person is correct.

That is, the position and posture of the patient from the patient estimation processing unit 12C and the surgical procedure information acquired from the in-hospital system 20 are input to the weighting coefficient addition unit 12D, and the estimation result of the patient estimation processing unit 12C to which the weighting coefficient is added is displayed. By inputting to the determination unit 12B, the vector from each part of the patient to the person or the object can be used for estimating the role of the person and the work content.

Here, the data acquired by the data processing unit 12A of the second embodiment in addition to the basic data D0 shown in FIG. 7 will be described.
FIG. 11 is an explanatory diagram of an example of data to be added in the second embodiment.
Examples of the data acquired in addition to the basic data D0 in the second embodiment include the distance data DA1 between the patient and the person, the distance data DA2 between the patient and the equipment, and the vector data DA3 from the patient to the person.

More specifically, in the example of FIG. 11, the distance data DA1 is the distance between the patient and the person A, and xxxcm is stored. Further, the distance data DA2 is the distance between the patient and the equipment C, and yycm is stored.
Further, the vector data DA3 is a vector from the patient to the person C, and the direction is xx ° and the distance is xx cm.

FIG. 12 is an explanatory diagram of a configuration example of the determination database of the second embodiment.
The determination database 13 at this time stores the surgical procedure data DR-A1, the surgeon data DR-A2, and the workflow data DR-A3 as the determination data DR-A for past cases.

More specifically, in the example of FIG. 12, the surgical procedure data DR-A1 stores the surgical procedure xxx, and the surgeon data DR-A2 stores the surgeon yyyy.
Further, as the workflow data DR-A3, in the example of FIG. 12, the preoperative workflow data DR-A31 and the intraoperative workflow data DR-A32 are stored.

The workflow data DR-A31 and the intraoperative workflow data DR-A32 store one or more detailed data for the pattern of arrangement of people and objects (for example, pattern A), respectively.

For example, for pattern A of arrangement of a person and an object, two detailed data are stored, and the role of the person, the work content of the person, the posture of the person, the position of the person, the color of clothes, the equipment used (type and product name). ), The equipment position is stored. Further detailed data stores the distance from the patient or the vector from the patient.
As described above, according to the second embodiment, weighting is performed based on the vector (direction, distance) of the person and the object whose work content is to be estimated from the part such as the head side of the patient, which is almost determined for each surgical procedure. By performing the above to estimate the role of the person, in addition to the effect of the first embodiment, the accuracy of estimating the role of the person and the work content can be improved.

[3] Third Embodiment FIG. 13 is a detailed block diagram of a main part of the surgical operation information management system of the third embodiment.
The third embodiment of FIG. 13 differs from the first embodiment of FIG. 2 in that the weighting coefficient is set based on the surgical procedure information (information on the surgical content) from the operating room system 20A constituting the in-hospital system 20. This is a point provided with a weighting coefficient addition unit 12E that is generated and output.

By the way, the workflow (procedure) and working time are almost fixed for each surgical procedure.
For example, if a certain work is performed 30 minutes after the start of carrying in the equipment and 10 minutes before the time-out (safety confirmation immediately before the operation), it is unlikely that the work is an intraoperative work.

Therefore, by inputting the surgical procedure information from the operating room system 20A into the weighting coefficient addition unit 12E, the weighting of the work content is changed and output to the situation determination unit 12B, so that the situation determination unit 12B can perform the event of each work. The relative time from (timeout in the above example) can be used for estimating the work content, and it is possible to increase the probability that the estimation in estimating the role of the person and the work content is correct.

In addition, the arrangement of people and objects, the equipment used, and the workflow (procedure procedure) are almost determined for each surgical procedure of each surgeon.

Therefore, the surgical procedure information corresponding to the surgeon from the operating room system 20A is input to the weighting coefficient addition unit 12E, the weighting of the work content is changed, and the operation information is output to the situation determination unit 12B. The medical technique can be used to estimate the work content, and it is possible to increase the probability that the estimation in estimating the role of the person and the work content is correct.

FIG. 14 is an explanatory diagram of an example of acquired data of the data processing unit of the third to fifth embodiments.
As shown in FIG. 14, the data processing unit 12A of the data processing device 12 includes basic data D0 (see FIG. 7) including the posture and position of a person, the type and position of equipment as an object, and the shooting time for each frame, for example, as shown in FIG. In addition, surgical procedure data D1, surgeon data D2, equipment information data D3, and color information data D4 have been acquired.
In this case, in the third embodiment, the data processing unit 12A has at least basic data D0, surgical procedure data D1 (surgical procedure xxx in the example of FIG. 12) and surgeon data D2 (in the example of FIG. 12). , Surgeon yyy).

As a result, according to the third embodiment, as in the second embodiment, in addition to the effect of the first embodiment, the accuracy of estimating the role of the person and the work content can be improved.

[4] Fourth Embodiment FIG. 15 is a detailed configuration block diagram of a main part of the surgical operation information management system of the fourth embodiment.
The fourth embodiment of FIG. 15 differs from the first embodiment of FIG. 2 from the surgical procedure information (surgery content information) from the operating room system 20A constituting the in-hospital system 20 and the operating room device 41. It is a point provided with a weighting coefficient addition unit 12F that generates and outputs a weighting coefficient based on the equipment information of the above.

In the fourth embodiment, the data processing unit 12A has at least basic data D0 and equipment information data D3 among the acquired data D0 to D4 shown in FIG. 14 (in the example of FIG. 12, the function related to the equipment A (in the example of FIG. 12) In the example of FIG. 14, functions A and B), operation start time (time xx: ya in the example of FIG. 14), operation end time (time xx: yb in the example of FIG. 14), and functions related to equipment B (FIG. 14). In the example of 14, it suffices to acquire the function C), the operation start time, and the operation end time.

By the way, the equipment used and the operating time of the equipment used are almost determined for each surgical procedure.
For example, if there is a person who is operating a button on the monitor and the operation time (or the time from the start of the operation to the end of the operation) is close to the operation start time of the monitor, the operation is the monitor setting work at the time of loading. be. Further, if the operation time is close to the operation end time, there is a high possibility of the end work at the time of carrying out.

Therefore, by inputting the surgical procedure information from the operating room system 20A and the equipment information from the operating room device 41 into the weighting coefficient addition unit 12F, the weighting of the operating status of the operating room device 41 is changed to the situation determination unit 12B. Output.

Here, as the equipment information, operation information, operating time, setting information, etc. from the operating room equipment 41 can be mentioned.
As a result, the situation determination unit 12B can use the operating status of the operating room device 41 for estimating the work content, and can increase the probability that the estimation in estimating the role of the person and the work content is correct.

As a result, according to the fourth embodiment, as in the second and third embodiments, in addition to the effects of the first embodiment, the accuracy of estimating the role of the person and the work content is improved. be able to.

[5] Fifth Embodiment FIG. 16 is a detailed block diagram of a main part of the surgical operation information management system of the fifth embodiment.
The fifth embodiment of FIG. 16 differs from the first embodiment of FIG. 2 in that the color identification processing unit 12G that outputs color information based on the imaging data and the operating room system 20A that constitutes the in-hospital system 20. It is a point provided with a weighting coefficient addition unit 12H that generates and outputs a weighting coefficient based on the surgical procedure information (information on the operation content) from the above and the color information from the color identification processing unit 12G.
In this case, the color identification processing unit 12G acquires the color in the image together with the position where the color is detected by performing color detection using Open CV (Open Source Computer Vision Library) or the like, and uses it as color information. Output. Examples of the color data format include RGB, YUB, YPbPr (YcbCr), HSV, and the like.

In the fifth embodiment, the data processing unit 12A has at least at least basic data D0 and color information data D4 (in the example of FIG. 12, the color of position A) among the acquired data D0 to D4 shown in FIG. It suffices to acquire A, the color A of the position B, and the color C of the position C).

By the way, depending on the hospital where the surgery is performed, the color of the clothes may be determined by the roles of the staff (doctors, nurses, operating room staff, etc.).
Therefore, by inputting the surgical procedure information from the operating room system 20A and the color information from the color identification processing unit 12G into the weighting coefficient addition unit 12H, the weighting of the role of each person according to the color of the clothes is changed and the situation determination unit 12B Output to.
As a result, the situation determination unit 12B can use the color of the person's clothes for estimating the work content, and can increase the probability that the estimation in estimating the role of the person and the work content is correct.

As a result, according to the fifth embodiment, as in the second to fourth embodiments, in addition to the effect of the first embodiment, the accuracy of estimating the role of the person and the work content is improved. be able to.

[6] Modifications of the Embodiment The embodiment of the present technology is not limited to the above-described embodiment, and various changes can be made without departing from the gist of the present technology.

In the above description, the update of the judgment database 13 has not been described, but it is also possible to additionally register and update the newly obtained data in the judgment database at any time. be.

In the above description, the management server 14 has been described as one device, but it can also be constructed as a cloud server composed of a plurality of devices.

In the above description, a case where a camera is used as a sensor as a data collection device for determining the role of a person (for example, an operator) has been described, but any sensor can be used as long as it can determine the role. It is also possible.

Further, the present technology can also have the following configurations.
(1)
A data processing unit that acquires information about a person or object included in a captured image of an area related to surgery,
A situation determination unit that estimates the role of the person in the surgery based on the information about the person or the object.
Surgical operation information management system equipped with.
(2)
The data processing unit acquires at least one of the position information of the person or the object, the discrimination result of the person or the object, and the posture information of the person as the information about the person or the object.
The surgical operation information management system described in (1).
(3)
A judgment database that stores judgment data including at least one of the position, posture, equipment used, and work content of the person for each role for a plurality of cases is provided.
The situation determination unit estimates the role of the person in the surgery by comparing the information about the person or object with the determination data.
The surgical operation information management system according to (1) or (2).
(4)
The situation determination unit estimates the role based on at least one of the positional relationship between the patient and the person who is the target of the operation and the positional relationship between the person and the object.
The surgical operation information management system according to any one of (1) to (3).
(5)
The situation determination unit estimates the role for each surgical procedure based on the position of the person or the object with respect to the patient who is the target of the surgery.
The surgical operation information management system according to any one of (1) to (4).
(6)
The situation determination unit estimates the role for each surgical procedure based on a vector from the patient who is the target of the surgery to the person or the object.
The surgical operation information management system according to any one of (1) to (4).
(7)
The situation determination unit estimates the role for each surgical procedure based on the relative time from each work step in the surgery.
The surgical operation information management system according to any one of (1) to (4).
(8)
The situation determination unit estimates the role for each operation in consideration of the surgical procedure associated with the surgeon in charge of the operation.
The surgical operation information management system according to any one of (1) to (4).
(9)
The situation determination unit estimates the role based on the operating status of the equipment used for the surgery for each surgical procedure.
The surgical operation information management system according to any one of (1) to (4).
(10)
A data collecting device for acquiring the captured image by imaging the region related to the surgery is provided.
The surgical operation information management system according to any one of (1) to (9).
(11)
The data collection device includes a plurality of cameras that output the captured image.
The surgical operation information management system according to (10).
(12)
A presentation unit that presents information about the role.
The surgical operation information management system according to any one of (1) to (11).
(13)
The person is a surgeon,
The surgical operation information management system according to any one of (1) to (12).
(14)
The process of acquiring information about a person or object contained in a captured image of an area related to surgery,
The process of estimating the role of the person in the surgery based on the information about the person or object, and
Surgical operation information management method equipped with.
(15)
In the process of acquiring the information, at least one of the position information of the person or the object, the discrimination result of the person or the object, and the posture information of the person is acquired as the information about the person or the object.
The surgical operation information management method according to (14).
(16)
The process of estimating the role refers to a judgment database that stores judgment data including at least one of the position, posture, equipment used, and work content of the person for each role for a plurality of cases.
The role of the person in the surgery is estimated by comparing the information about the person or object with the determination data.
The surgical operation information management method according to (14) or (15).
(17)
The process of estimating the role estimates the role based on at least one of the positional relationship between the patient and the person to be operated on and the positional relationship between the person and the object.
The surgical operation information management method according to any one of (14) to (16).
(18)
In the process of estimating the role, the role is estimated based on the position of the person or the object with respect to the patient who is the target of the operation for each surgical procedure.
The surgical operation information management method according to any one of (14) to (17).
(19)
In the process of estimating the role, the role is estimated based on the vector from the patient who is the target of the operation to the person or the object for each surgical procedure.
The surgical operation information management method according to any one of (14) to (17).
(20)
The process of estimating the role estimates the role for each surgical procedure based on the relative time from each work step in the surgery.
The surgical operation information management method according to any one of (14) to (17).
(21)
In the process of estimating the role, the role is estimated for each operation in consideration of the surgical procedure associated with the surgeon in charge of the operation.
The surgical operation information management method according to any one of (14) to (17).
(22)
In the process of estimating the role, the role is estimated based on the operating status of the equipment used for the surgery for each surgical procedure.
The surgical operation information management method according to any one of (14) to (17).
(23)
With the process of presenting information about the role,
The surgical operation information management method according to any one of (14) to (22).
(24)
The person is a surgeon,
The surgical operation information management method according to any one of (14) to (23).
(25)
A program for controlling the surgical operation information management system with a computer.
A means for obtaining information about a person or an object included in a captured image of an area related to surgery using the computer.
Means for estimating the role of the person in the surgery based on information about the person or object, and
A program that works.
(26)
The means for acquiring the information acquires at least one of the position information of the person or the object, the discrimination result of the person or the object, and the posture information of the person as the information about the person or the object.
The program according to (25).
(27)
The surgical operation information management system includes a judgment database that stores judgment data including at least one of the position, posture, equipment used, and work content of the person for each role for a plurality of cases.
The means for estimating the role estimates the role of the person in the surgery by comparing the information about the person or object with the determination data.
The program according to (25) or (26).
(28)
The means for estimating the role estimates the role based on at least one of the positional relationship between the patient and the person to be operated on and the positional relationship between the person and the object.
The program according to any one of (25) to (27).
(29)
The means for estimating the role estimates the role based on the position of the person or the object with respect to the patient who is the target of the operation for each surgical procedure.
The program according to any one of (25) to (28).
(30)
The means for estimating the role estimates the role for each surgical procedure based on a vector from the patient who is the target of the surgery to the person or the object.
The program according to any one of (25) to (28).
(31)
The means for estimating the role estimates the role for each surgical procedure based on the relative time from each work step in the surgery.
The program according to any one of (25) to (28).
(32)
The means for estimating the role is to estimate the role for each operation in consideration of the surgical procedure associated with the surgeon in charge of the operation.
The program according to any one of (25) to (28).
(33)
The means for estimating the role estimates the role based on the operating status of the equipment used for the surgery for each surgical procedure.
The program according to any one of (25) to (28).
(34)
Acting the computer as a means of presenting information about the role.
The program according to any one of (15) to (33).
(35)
The person is a surgeon,
The program according to any one of (25) to (34).

10 Surgical operation information management system 11 Data collection device 11A Camera 12 Data processing device 12A Data processing unit 12B Situation judgment unit 12C Patient estimation processing unit 12D-12F Weighting coefficient addition unit 12G Color identification processing unit 12H Weighting coefficient addition unit 13 Judgment database 14 Management server 15 Information presentation system 15A Information presentation processing unit 15B Display 20 In-hospital system 20A Operating room system 21 Case-specific data 31 Role data 32 Work content data 33 Time data 41 Operating room equipment D0 Basic data D1 Operational data D2 Surgeon data D3 Equipment information data D4 Color information data DA1 Distance data DA2 Distance data DA3 Vector data DR Judgment data

Claims

A data processing unit that acquires information about a person or object included in a captured image of an area related to surgery,
A situation determination unit that estimates the role of the person in the surgery based on the information about the person or the object.
Surgical operation information management system equipped with.
The data processing unit acquires at least one of the position information of the person or the object, the discrimination result of the person or the object, and the posture information of the person as the information about the person or the object.
The surgical operation information management system according to claim 1.
A judgment database that stores judgment data including at least one of the position, posture, equipment used, and work content of the person for each role for a plurality of cases is provided.
The situation determination unit estimates the role of the person in the surgery by comparing the information about the person or object with the determination data.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role based on at least one of the positional relationship between the patient and the person who is the target of the operation and the positional relationship between the person and the object.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role for each surgical procedure based on the position of the person or the object with respect to the patient who is the target of the surgery.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role for each surgical procedure based on a vector from the patient who is the target of the surgery to the person or the object.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role for each surgical procedure based on the relative time from each work step in the surgery.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role in consideration of the surgical procedure associated with the surgeon in charge of the surgery.
The surgical operation information management system according to claim 1.
The situation determination unit estimates the role based on the operating status of the equipment used for the surgery for each surgical procedure.
The surgical operation information management system according to claim 1.
A data collecting device for acquiring the captured image by imaging the region related to the surgery is provided.
The surgical operation information management system according to claim 1.
The data collection device includes a plurality of cameras that output the captured image.
The surgical operation information management system according to claim 10.
A presentation unit that presents information about the role.
The surgical operation information management system according to claim 1.
The person is a surgeon,
The surgical operation information management system according to claim 1.
The process of acquiring information about a person or object contained in a captured image of an area related to surgery,
The process of estimating the role of the person in the surgery based on the information about the person or object, and
Surgical operation information management method equipped with.
A program for controlling the surgical operation information management system with a computer.
A means for obtaining information about a person or an object included in a captured image of an area related to surgery using the computer.
Means for estimating the role of the person in the surgery based on information about the person or object, and
A program that works.