JP7240665B2 - Information integration device - Google Patents

Description

The present disclosure relates to technology for displaying medical information obtained from medical devices used in surgery and treatment.

Patent Literature 1 listed below describes a medical information system that obtains medical information from a plurality of medical devices used in surgery and treatment, integrates the plurality of medical information, and displays the information on a screen.

JP 2015-185125 A

This type of system is used for real-time examination and support during surgery, surgery analysis after surgery, review of conferences and the like, investigation of accidents, educational purposes, and the like. At that time, there is a demand to make it possible to easily share various kinds of information among related parties. The information to be shared includes the details of decisions made during surgery, details of support received during surgery, comments on changes in the settings of each medical device and changes in measurement values, and things noticed during and after surgery, etc., for educational purposes. and various information necessary for preparing conference presentation materials.

One aspect of the present disclosure is to provide a technique for easily sharing and recording user comments on information acquired from a medical device and displayed on the screen.

One aspect of the present disclosure is an information integration device comprising an information storage unit (31), a main image generation unit (53: S210 to S230), a target designation unit (52, 53: S310 to S330), a comment It has an input section (52, 53: S420), an information generation section (53: S340 to S410, S440 to S470), and a comment image generation section (53: S210 to S220, S240).

The information storage unit stores device information obtained from a plurality of medical devices (21) used for treatment of a patient as a first tag containing information representing the time when the device information was obtained and the type of the device information. It is stored in association with information. The main image generation unit extracts one or more types of device information associated with the same time from among a plurality of pieces of device information stored in the information storage unit, and image data for displaying the extracted device information on one screen. to generate the main image data. The target specifying section specifies a comment target in the display image displayed by the image data. The comment input unit inputs comment information that is information associated with the comment target specified by the target specifying unit. The information generation unit stores the comment information input by the comment input unit in the information storage unit in association with second tag information including information for specifying the comment target specified by the target specification unit. The comment image generation unit extracts comment information corresponding to the comment target from the information storage unit when a comment target is included in the main image, which is an image displayed by the main image data, and stores the extracted comment information in the main image. Comment image data, which is image data to be displayed in association with the comment target in the image, is generated.

According to such a configuration, information that cannot be read from the device information itself can be easily left as comment information. Moreover, since the comment information is displayed in association with the corresponding device information, the viewer of the display screen can clearly grasp the target of the comment.

It is a block diagram showing the structure of an information integration apparatus. It is a functional block diagram showing the function of an information integration device. FIG. 4 is an explanatory diagram showing the structure of tagged device information; FIG. 4 is an explanatory diagram showing the structure of tagged comment information; 9 is a flowchart showing device information generation processing; 4 is a flowchart showing display processing; FIG. 4 is an explanatory diagram showing information read out in display processing; FIG. 4 is an explanatory diagram showing one example of a layout of a main image; FIG. 4 is an explanatory diagram showing an event list; It is a flow chart showing comment provision processing. FIG. 10 is an explanatory diagram showing a display example of comment information;

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
[1. overall structure]
The information integration device 1 is a device for integrating, storing, and displaying various types of information in an operating room. The information integration device 1 is applied to real-time examination and support during surgery, surgical analysis after surgery, educational purposes, and the like.

As shown in FIG. 1, the information integration apparatus 1 includes a device group 2, a server 3, and a plurality of terminals 5. FIG.
[2. Devices]
The device group 2 includes a plurality of devices 21A-21H, as shown in FIG. Each of the devices 21A-21H generates data and supplies it to the server 3. FIG. Hereinafter, when referring to an arbitrary device, the device 21 is used without distinguishing between the plurality of devices 21A to 21H.

The device 21 is a medical device, medical equipment, or the like used during surgery or treatment. Specifically, medical equipment includes, for example, biological information monitors, respiratory function monitors, hemodynamic monitors, sedation monitors, anesthesia machines, infusion pumps, syringes, blood purifiers, heart-lung machines, assisted circulation devices, and the like. Examples of medical equipment include surgical navigation systems (hereinafter referred to as surgical navigation), IP cameras, medical gas systems, endoscopes, microscopes, electric scalpels, drills, ultrasonic irradiation devices, and the like. Devices may also include air conditioning systems, isolation systems, and the like.

The data supplied from the device 21 (hereinafter referred to as device information) includes measurement values from the device 21, video and still images output from the device 21, device status including the usage status and errors of the device 21, device 21 It includes setting values of itself and setting values related to parameters handled by the device 21 .

The surgical navigation system is a device that acquires the three-dimensional position and orientation of surgical instruments during surgery. Surgical instruments include electric scalpels and forceps, as well as microscopes and the like. As a method of acquiring the three-dimensional position and orientation of a surgical instrument, for example, a plurality of markers are attached in advance to specific positions of the surgical instrument, and a predetermined reference position in the space where the surgery is performed is compared to a specific position on the surgical instrument. It is conceivable to specify the vector up to the point as the three-dimensional position of the surgical instrument, and to specify the orientation from the positional relationship between a plurality of markers. Also, the focal point of the microscope may be specified as a three-dimensional position. At this time, the three-dimensional position is specified by associating the patient's diseased part with a three-dimensional image taken in advance by CT, MRI, or the like. CT is an abbreviation for Computed Tomography, and MRI is an abbreviation for Magnetic Resonance Imaging.

[3. server]
The server 3 includes a database 31 and a server processing section 32 .
The database 31 is implemented by an HDD or the like, and as shown in FIG. 2, at least a device information area 45, a comment information area 46, and an event information area 47 are secured. The device information area 45 stores tagged device information T_Idv, the comment information area 46 stores tagged comment information T_Icm, and the event information area 47 stores an event list L. Also, the tagged device information T_Idv and the tagged comment information T_Icm are collectively referred to as integrated information. The server 3 corresponds to an information storage unit.

As shown in FIG. 3, tagged device information T_Idv has first tag information T1 and device information Idv.
The device information Idv is information acquired from one of the devices 21 belonging to the device group 2 during surgery or the like, and specifically includes measurement values, setting values, image data, and the like.

The first tag information T1 is information related to device information Idv, and includes at least a time tag T11 and a data type tag T12. The time tag T11 is information representing the time when the device information Idv was acquired (hereinafter referred to as acquisition time). The data type tag T12 includes identification information for specifying the device 21 that generated the device information Idv. When a plurality of device information Idv are acquired from one device 21, identification information for specifying which device information Idv is included.

As shown in FIG. 4, tagged comment information T_Icm has second tag information T2 and comment information Icm.
The comment information Icm is information input via the terminal 5 during or after surgery, and specifically includes text data, voice data, image data, and the like. Note that the image data may include bitmap data representing handwritten characters and drawings.

The second tag information T2 is information related to the comment information Icm, and includes at least a comment target tag T21, a time tag T22, a time tag T23, and a comment type tag T24.

The comment target tag T21 is information for specifying the target of the comment. Comment targets include all or part of the device information Idv identified by the data type tag T12, and all information not specifying the device information Idv (that is, the entire screen on which the device information Idv is displayed). When the device information Idv is a value measured by the device 21 and the measured value is displayed as a number or graph, the part of the device information Idv includes the displayed number, graph, or the like.

The time tag T22 is information representing the acquisition time of the comment target, that is, the time similar to the time tag T11 described above. That is, tagged comment information T_Icm is associated with tagged device information T_Idv by comment target tag T21 and time tag T22.

The time tag T23 is information representing the time when the comment target was specified when the comment information Icm was input. The time tag T23 represents the current time at which the integrated information is being reproduced (hereinafter referred to as reproduction time). However, when the integrated information is reproduced in real time, that is, when the integrated information is generated at the same time, the time tag T23 may have the same value as the time tag T22.

The comment type tag T24 is information representing the format of the comment information Icm, and includes text data, voice data, image data, voice recognition data, motion analysis data, and the like.
As shown in FIG. 9, the event list L includes at least "occurrence time", "event category", "event creator", "event content", and "reproduction time" as information items. "Occurrence time" is the time when the event occurred, and represents the time on the same time axis as the acquisition time. The “event category” represents the category of the event that occurred, and more specifically represents which device the event is related to. “Event Creator” represents a person who has performed an operation to generate an event when the event is generated by an input from the terminal 5 . The “event creator” may be input via the terminal 5 , or information pre-registered as a user of the terminal 5 may be obtained from the terminal 5 . The "content of event" indicates the content of the event that has occurred, particularly in the case of a comment event that occurs when a comment is input, the content of the input comment. "Reproduction time" is the time when an event is additionally registered in the list L when the integrated information is reproduced, and represents the current time at the time of reproduction.

The columns of “event creator” and “event content” are filled in via the terminal 5 as appropriate.
Returning to FIG. 1, the server processing unit 32 includes a microcomputer having a CPU 321 and semiconductor memory such as RAM and ROM (hereinafter referred to as memory 322).

The server processing unit 32 functionally includes a plurality of providers 41A to 41H and middleware 42, as shown in FIG. Hereinafter, when an arbitrary provider is indicated without distinguishing among the plurality of providers 41A to 41H, the provider 41 is used.

One of the devices 21 is connected to each provider 41 . The provider 41 mediates exchange of information with the connected device 21 .
The middleware 42 is a communication interface that provides unified access means together with the provider 41 to various devices 21 connected to the provider 41 regardless of differences in programming languages and communication protocols. Here, ORiN is used as the middleware 42 . ORiN is an abbreviation for Open Resource interface for the Network.

The middleware 42 also has at least a device information generation function 421 that generates tagged device information T_Idv and a clock function 422 that provides the current time upon request.

[3-1. device information processing]
Processing executed by the middleware 42 of the server processing unit 32 to realize the device information generation function 421 (hereinafter referred to as device information processing) will be described with reference to the flowchart of FIG. This process is executed for each provider 41 when the server 3 is powered on.

In S110, the server processing unit 32 determines whether or not the device 21 is connected to the provider 41 to be processed. If the server processing unit 32 determines that the device 21 is not connected, it waits by repeating the same step, and if it determines that the device 21 is connected, the process proceeds to S120.

In S<b>120 , the server processing unit 32 acquires device information Idv from the device 21 via the provider 41 . The device information Idv acquired from one device 21 may be of one type or of multiple types.

In subsequent S130, the server processing unit 32 acquires the current time, generates a time tag T11, and inserts the time tag T11 into the device information Idv acquired in S110. The current time may be obtained from the clock function 422 of the microcomputer, or may be time information representing the acquisition time of the device information Idv provided from the device 21 together with the device information Idv.

In subsequent S140, the server processing unit 32 inserts a data type tag T12 representing the type of the device information Idv into the device information Idv acquired in S120. As the data type tag T12, one prepared in advance according to the device 21 connected to the provider 41 is used.

In subsequent S150, the server processing unit 32 stores the tagged device information T_Idv, which is the device information Idv in which the first tag information T1 including the time tag T11 and the data type tag T12 is inserted in the above-described processing, in the database 31. Store in area 45 .

In subsequent S160, the server processing unit 32 determines whether an event has occurred for the device information Idv. An event occurs when the device information Idv satisfies a preset event condition. For example, when the device information Idv is a measured value, the event condition may be that the measured value or the differential value or integral value of the measured value exceeds a preset threshold. However, the event condition is not limited to this.

If the server processing unit 32 determines that an event has occurred, the process proceeds to S170, and if it determines that the event has not occurred, the process proceeds to S180.
In S170, the server processing unit 32 registers the event occurrence information representing the event occurrence time and the event target information representing the device information Idv that is the target of the event in the event list L stored in the event information area 47. Then, the process proceeds to S180.

In S180, the server processing unit 32 determines whether or not to end generation of device information. Specifically, it is determined whether or not a command indicating the end of processing has been input via the terminal 5 . If the server processing unit 32 determines to continue generating the device information, it returns to S110,
If it is determined to end the generation of device information, the process ends.

[4. terminal]
As shown in FIG. 1 , the terminal 5 includes a display device 51 , an input device 52 and a terminal processing section 53 . A plurality of terminals 5 all have the same configuration.

The display device 51 is, for example, a touch panel type liquid crystal display, and displays three-dimensional images such as MPR images, waveform data and numerical values representing measurement results such as biological information, various operation buttons, setting values of each device, and display setting values. can be displayed. MPR is an abbreviation for Multi Planar Reconstruction.

The input device 52 is, for example, an input device such as a touch panel, keyboard, mouse, microphone, camera, motion sensor, etc. provided on the liquid crystal display described above. The input device 52 outputs text data, position data, voice data, image data, etc. corresponding to input operations through these input devices to the terminal processing unit 53 .

The terminal processing unit 53 includes a microcomputer having a CPU 531 and semiconductor memory such as RAM and ROM (hereinafter referred to as memory 532). The terminal processing unit 53 executes one or more applications (hereinafter referred to as APs) related to information stored in the database 31 of the server 3 . Here, the terminal processing unit 53 executes at least a display AP 61, a voice recognition AP 62, a motion analysis AP 63, a comment addition AP 64, and a layout change AP 65, as shown in FIG.

The voice recognition AP 62 executes voice recognition processing on the user's voice acquired by a microphone, which is one of the input devices that constitute the input device 52, and recognizes a command or text data corresponding to the recognition content. Data is generated and output along with the speech to be recognized.

The motion analysis AP 63 analyzes an image of a user's motion (for example, hand, face, head motion, etc.) photographed by a camera, which is one of the input devices constituting the input device 52, and analyzes a command corresponding to the content of the analysis. Alternatively, motion analysis data, which is text data, is generated and output together with the image to be analyzed.

That is, the terminal 5 is provided with the input device 52, the voice recognition AP 62, and the motion analysis AP 63, thereby realizing non-contact input operations.
[4-1. Display processing]
Processing (hereinafter referred to as display processing) realized by the terminal processing unit 53 executing the display AP 61 will be described with reference to the flowchart of FIG.

The display process is repeatedly executed at a preset display cycle from when a display start instruction is input via the input device 52 until when a display end instruction is input. In the display process, the device information Idv and the comment information Icm corresponding to the time S designated via the input device 52 are displayed on the display screen of the display device 51 based on the integrated information stored in the database 31 . However, in the operation mode for displaying a still image, the time S is changed according to the input from the input device 52, and in the operation mode for displaying a moving image, the time S is changed after the initial value is input from the input device 52. , is automatically updated according to the display cycle.

In S210, the terminal processing unit 53 acquires the time S.
In S220, the terminal processing unit 53 obtains from the database 31 device information Idv and comment information Icm with time tags T11 and T22 indicating the latest times before time S for all device types and all comment types. to get Note that an upper limit of time that can be traced back may be set for searching for the latest time.

For example, as shown in FIG. 7, the database 31 stores four types of device information identified by A to D and two types of comment information identified by a and c. In FIG. 7, when the time S is 9:00 X milliseconds, the hours and minutes are omitted and displayed as "X" in units of milliseconds. That is, the column of numerical values shown in the "A" column in FIG.

When the time S is "100", the device information having the latest time tag T11 before the time S is, as indicated by the circle in FIG. , type B has device information with a time tag T11 of "100", type C has device information with a time tag T11 of "99", and type D has device information with a time tag T11 of "98". .

Also, the comment information having the latest time tag T22 before the time S is the comment information with the time tag T22 of "93" for the type a. Type c means that there is no corresponding comment information within the upper limit time range.

In subsequent S230, the terminal processing unit 53 generates main image data, which is image data for displaying the device information Idv read out in S220 according to the layout set at that time. An image displayed by the main image data is hereinafter referred to as a main image.

A layout is a combination of the following elements. (1) How to divide the display area on the display screen. (2) How many types of device information are to be displayed simultaneously in each of the divided areas. (3) How to display the device information (image, numerical value, graph, table, etc.) in each divided area.

As an example of the layout of the main image, for example, as shown in FIG. 8, it may have a monitor area A1, a video area A2, a navigation image area A3, a time area A4, and the like. The monitor area A1 is an area where a graph or the like for monitoring biological information is displayed. In the monitor area A1, biological information is displayed numerically and graphically. The image area A2 is an area in which a moving image of the treatment target captured by a camera or the like is displayed. The navigation image area A3 is an area where the three-dimensional position detected by surgery navigation is displayed together with a three-dimensional image of the affected area prepared in advance. In the navigation image area, three cross-sectional images including the detection position and showing mutually different cross-sections are displayed in a trihedron format. The time area A4 is an area for expressing the time range in which the time series of the integrated information exists on the time axis and displaying the time S on the time axis. In the time area A4, along with the time axis, a balloon-shaped display representing the event occurrence time may be displayed. The setting and change of the layout are realized by the layout change AP 65 prepared separately.

Returning to FIG. 6, in subsequent S240, the terminal processing unit 53 generates comment image data, which is image data for displaying the comment information Icm read out in S220 together with the main image. An image displayed by comment image data is hereinafter referred to as a comment image. The display position and display format of the comment image on the main image are set in advance according to the comment type of the comment information Icm and the data type of the device information Idv to be commented. Note that there may be a dedicated application for changing the display position and display format of the comment image.

In subsequent S250, the terminal processing unit 53 outputs the main image data generated in S230 and the comment image data generated in S240 to the display device 51, thereby displaying the main image and the comment image on the display device 51. It is displayed on the display screen, and the process is once terminated.

Note that S210 to S230 correspond to the main image generating section, and S210 to S220 and S240 correspond to the comment image generating section.
[4-2. Comment adding process]
A process realized by the terminal processing unit 53 executing the display AP 61 (hereinafter referred to as comment adding process) will be described with reference to the flowchart of FIG. 10 . This process is repeatedly executed while the display process is being executed.

In S310, the terminal processing unit 53, through the input device 52, inputs the entire information displayed on the screen of the display device 51 (hereinafter referred to as the display screen) as a comment target. determine whether or not has been performed. The terminal processing unit 53 shifts the process to S340 when determining that all designation inputs have been performed, and shifts the processing to S320 when determining that all designation inputs have not been performed.

In S340, the terminal processing unit 53 acquires the acquisition time corresponding to the timing at which all designation inputs are performed to generate a time tag T22, acquires the reproduction time corresponding to the timing, and generates a time tag T23. do. Note that the acquisition time is the time S acquired most recently in S210 of the display processing before the timing at which all designation inputs are performed, or the time tag associated with the device information Idv displayed at that timing. This is the time indicated by T11. Also, the playback time is the current time obtained from the clock function 422 at the timing when all designation inputs are performed.

In subsequent S350, the terminal processing unit 53 generates a comment target tag T21 indicating that the comment target is "whole", and advances the process to S420.
In S<b>320 , the terminal processing unit 53 determines whether or not a position input indicating a position on the display screen has been performed via the input device 52 . The position input may be a position input by a touch panel on the display screen, or a position indicated by a cursor displayed on the display screen operated by a mouse or keyboard. Furthermore, when the navigation image area A3 is included in the main image, it may be the three-dimensional position indicated by the navigation image. Specifically, it is the center position of each image where the auxiliary line intersects, which is shown in each of the three views. If the terminal processing unit 53 determines that the position input has been performed, the process proceeds to S360, and if it determines that the position input has not been performed, the process proceeds to S330.

In S360, the terminal processing unit 53 acquires the acquisition time and the reproduction time corresponding to the timing of position input via the input device 52, and generates time tags T22 and T23 in the same manner as in S340. .

In subsequent S370, the terminal processing unit 53 determines whether or not a number or waveform (hereinafter referred to as a waveform, etc.) representing a measured value is displayed at the specified position, which is the position on the display screen specified by the position input. Determines whether or not a waveform or the like is specified by the position input. If the terminal processing unit 53 determines that the waveform or the like is not specified, the process proceeds to S380, and if it determines that the waveform or the like is specified, the process proceeds to S390.

In S380, the terminal processing unit 53 identifies the type of device information Idv displayed at the specified position based on the specified position and the layout of the display at that time. The type of device information Idv is represented by a data type tag T12 associated with the device information Idv. Further, a comment target tag T21 indicating that the identified device information Idv is a comment target is generated, and the process proceeds to S420.

In S390, the terminal processing unit 53 generates a comment target tag T21 indicating that the waveform or the like specified by the position input is a comment target, and advances the process to S420.
In S380 and S390, the comment target tag T21 may include information indicating the specified position as a relative position within the display area assigned to the comment target. This is information for preventing the relationship between the display and the specified position in the display area from changing when the layout is changed.

In S330, the terminal processing unit 53 determines whether or not a comment event input, which is an operation for inputting a comment via the input device 52, has been performed. Specifically, as shown in FIG. 6, when the event addition button Bev displayed in the time area A4 on the screen is operated via the input device 52, it is determined that a comment event has been input. If the terminal processing unit 53 determines that a comment event has been input, the process proceeds to S400, and if it determines that the comment event has not been input, the process returns to S310. When the event addition button Bev is operated, as shown in FIG. 9, an event list L to which a new comment event is added is displayed on the screen.

In S400, the terminal processing unit 53 acquires the acquisition time and the reproduction time corresponding to the timing at which the comment event was input, and generates time tags T22 and T23 in the same manner as in S340. This acquisition time and playback time are also displayed in the event list L. FIG.

In subsequent S410, the terminal processing unit 53 generates a comment target tag T21 indicating that the whole is a comment target, and advances the process to S420.
In S420, the terminal processing unit 53 determines whether or not a comment has been input via the input device 52. If there is a comment input, the process proceeds to S440, and if there is no comment input, the process proceeds to S430.

At S430, it is determined whether or not an instruction to cancel the input of the comment has been input via the input device 52. If the instruction to cancel has been input, the process is terminated. Return to S420.

In S440, the terminal processing unit 53 identifies the type of the input comment from the input device used to input the comment, and generates a comment type tag T24. Types of comments include character input via a keyboard, handwriting input via a touch panel, voice input via a microphone, image input via a camera, and the like.

In subsequent S450, the terminal processing unit 53 generates comment information Icm corresponding to the comment type. Specifically, when the comment type is character input or handwriting input, the input text data or image is used as the comment information Icm as it is. When the comment type is voice input, not only voice data but also voice recognition data generated by executing the voice recognition AP 62 is included in the comment information Icm. Similarly, when the comment type is image input, the comment information Icm includes not only image data but also motion analysis data generated by executing the motion analysis AP 63 .

In subsequent S460, the terminal processing unit 53 adds time tags T22 and T23 generated in S340, S360, and S400, S350, S380, S390, and comment target tag T21 generated in S410 to the comment information Icm generated in S450. , and inserts the comment type tag T24 generated in S440 to generate tagged comment information T_Icm.

In subsequent S470, the terminal processing unit 53 causes the database 31 to store the generated tagged comment information T_Icm, and ends the process.
The contents of the tagged comment information T_Icm stored in the database 31 are immediately reflected in the display by the repeatedly executed display processing.

Note that S310 to S330 and the input device 52 correspond to a target designating section, S420 and the input device 52 correspond to a comment input section, and S340 to S410 and S440 to S470 correspond to an information generating section.

Each function of the server processing unit 32 and the terminal processing unit 53 is realized by the CPUs 321 and 531 executing programs stored in non-transitional physical recording media. In this example, the memories 322 and 532 correspond to non-transitional substantive recording media storing programs. Also, by executing this program, a method corresponding to the program is executed. Note that the server processing unit 32 and the terminal processing unit 53 may be provided with one microcomputer, or may be provided with a plurality of microcomputers.

The method of realizing the function of each part included in the server processing unit 32 and the terminal processing unit 53 is not limited to software, and some or all of the functions may be realized using one or more pieces of hardware. good. For example, when the above functions are realized by an electronic circuit that is hardware, the electronic circuit may be realized by a digital circuit, an analog circuit, or a combination thereof.

[5. Display example]
Here, a display example of the comment information Icm by display processing will be described with reference to FIG. 11 .
When the measured value displayed in the monitor area A1 changes significantly, after inputting the position by touching the part on the screen where the measured value is displayed with a finger, etc., "Change by medication" is entered via a keyboard or the like. Enter comments such as Then, as indicated by reference numeral C1 in FIG. 11, a character comment associated with the parameter to be commented is displayed.

After the position input for designating the image area A2 is performed, a line drawing or characters are input by handwriting on the display screen using an input pen for a touch panel or the like. Then, as indicated by reference numeral C2 in FIG. 11, the handwritten line drawing and characters are superimposed and displayed as they are on the image in the image area A2 to be commented.

When an operation to specify the navigation image area A3 is performed while the MPR image is displayed in the navigation image area A3 in a three-view format, the three-dimensional position specified by the MPR image displayed in the navigation image area A3 is displayed. is assumed to have been entered, and a comment can be entered for the specified three-dimensional position. In this case, the three-dimensional position represented by each central position of the three-view drawing becomes a comment target, and the comment is displayed in the "rice 1" portion indicated by reference numeral C3 in FIG. Also, at another position in the three-view drawing, a comment is displayed in the "rice 2" portion indicated by symbol C4 in FIG. be. Note that the comment may be displayed in a part without an image in the navigation image area A3 instead of being superimposed on the image.

When all specification inputs are performed, although illustration is omitted, for example, a vertically long comment display area is set in the right corner of the display screen, and the comment time and comment content are displayed in chronological order in the comment display area. may be displayed as

[6. effect]
The information integration device 1 described in detail above has the following effects.
(6a) When device information Idv is displayed in real time during surgery or treatment, not only can the status of surgery or treatment be shared in real time with any stakeholders, but also advice received during surgery and equipment used during surgery It is possible to leave comments in real time as memorandums such as the operation of , the intention of the work, and what you noticed during the operation.

For example, if the device information displayed on the display screen of the display device 51 has a characteristic portion that differs from other portions, it is possible to read from the device information such as medication, air conditioning setting change, and patient posture change. You can leave information that cannot be displayed as a comment.

(6b) In addition to input that involves operating the device using a touch panel, keyboard, mouse, etc., it is possible to input comments through a microphone through a camera and non-contact comment input using gestures, making it impossible to operate the device. It is also possible to easily leave a comment of a certain practitioner or the like.

(6c) When the device information Idv and the comment information Icm are displayed after the surgery for the purpose of reviewing or educating the surgery or treatment, the viewer can obtain useful information that cannot be obtained from the device information Idv alone. Information can be obtained from the comment information Icm. At this time, new comments can also be added.

(6d) When entering a comment, individual device information Idv shown on the display screen can be specified, and the comment information Icm is displayed and reproduced in association with the specified device information Idv. , the viewer can clearly grasp the target of the comment.

(6e) Since it is possible to make comments on the "whole", by using this function, viewers can discuss with each other on the display screen, and the content of the discussion can be left as a comment. can.

[7. Other embodiments]
Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be made.

(7a) In the above embodiment, a plurality of applications 61 to 65 are installed in the terminal 5, but the present disclosure is not limited to this. For example, the terminal 5 may be integrated with the server 3 and at least some of the applications 61 to 65 may be installed on the server 3 .

(7b) A plurality of functions possessed by one component in the above embodiment may be realized by a plurality of components, or a function possessed by one component may be realized by a plurality of components. . Also, a plurality of functions possessed by a plurality of components may be realized by a single component, or a function realized by a plurality of components may be realized by a single component. Also, part of the configuration of the above embodiment may be omitted. Moreover, at least part of the configuration of the above embodiment may be added or replaced with respect to the configuration of the other above embodiment.

(7c) In addition to the information integration device described above, a system having the information integration device as a component, a program for making a computer function as the information integration device, a non-transitional actual record such as a semiconductor memory in which this program is recorded The present disclosure can also be implemented in various forms such as media.

DESCRIPTION OF SYMBOLS 1... Information integration apparatus, 2... Device group, 3... Server, 5... Terminal, 21A-21H... Device, 31... Database, 32... Server processing part, 41A-41H... Provider, 42... Middleware, 45... Device information area , 46... comment information area, 47... event information area, 51... display device, 52... input device, 53... terminal processing unit, 61 to 65... application, 321, 531... CPU, 322, 532... memory, 421... device Information generation function, 422... Clock function.

Claims (6)

Device information acquired from a plurality of medical devices (21) used for treatment of a patient is associated with first tag information including information representing the time when the device information was acquired and the type of the device information. an information storage unit (31) stored by
image data for extracting one or more types of device information associated with the same time from among the plurality of device information stored in the information storage unit and displaying the extracted device information on one screen; a main image generator (53: S210 to S230) configured to generate main image data;
a target specifying unit (52, 53: S310 to S330) configured to specify a comment target in the display image displayed by the image data;
a comment input unit (52, 53: S420) configured to input comment information that is information associated with the comment target specified by the target specifying unit;
The comment information input by the comment input unit is stored in the information storage unit in association with second tag information including information for specifying the comment target specified by the target specifying unit. The configured information generation unit (53: S340 to S410, S440 to S470);
When the comment target is included in the main image that is the image displayed by the main image data, the comment information corresponding to the comment target is extracted from the information storage unit, and the extracted comment information is stored in the main image data. a comment image generation unit (53: S210 to S220, S240) configured to generate comment image data, which is image data to be displayed in association with the comment target in the image;
with
Include the entire display image as one of the comment targets
Information integrator. 2. The information integration device according to claim 1, wherein said target specifying unit is configured to specify a position on a screen where said display image is to be displayed,
including the device information or part of the device information displayed at the position specified by the specifying unit as one of the comment targets;
Information integrator. The information integration device according to claim 2,
An information integration apparatus including a control parameter related to the device as one of the device information. The information integration device according to claim 2 or claim 3,
An information integration apparatus including a graph representing a waveform of biometric information as one of the device information. The information integration device according to any one of claims 1 to 4,
One of the device information includes a three-dimensional position of a surgical instrument during surgery,
One of the comment targets includes a navigation image in a surgical navigation system that displays a three-dimensional image of a treatment target corresponding to the three-dimensional position of the surgical instrument,
Information integrator. The information integration device according to any one of claims 1 to 5 ,
The target specifying unit is configured to specify the entire display image as the comment target at the timing when an operation unit (Bev) prepared in advance is operated.
Information integrator.

Images