JP2023176561A - Program, model generation method, medical instrument, information processing method and information processing device - Google Patents

Abstract

To provide a program and the like which can properly specify display information that changes in a time-series manner.SOLUTION: A medical instrument displays measurement information that changes in a time-series manner on a display unit. A computer acquires a photographed image obtained by imaging the display unit of the medical instrument which displays the measurement information that changes in a time-series manner. The computer also acquires the measurement information displayed on the display unit of the medical instrument on the basis of the acquired photographed image. The computer stores the acquired measurement information in a storage unit in association with information about a patient.SELECTED DRAWING: Figure 1

Description

The present invention relates to a program, a model generation method, a medical device, an information processing method, and an information processing device.

Patent Document 1 discloses a technique for specifying a medical device to be imaged and medical data displayed on the display section based on a captured image of the display section of the medical device. With the technology disclosed in Patent Document 1, a measured value displayed on a display screen of a thermometer, sphygmomanometer, pulse meter, blood sugar level measuring device, etc. can be specified based on a captured image.

Japanese Patent Application Publication No. 2017-102797

2. Description of the Related Art In medical settings, hospitalized patients are equipped with measuring instruments that measure various biological information, as well as pumps that administer drugs, infusions, and the like into the body. Measuring instruments regularly measure biological information such as blood sugar level and blood oxygen concentration and display it on the display screen, and pumps measure, for example, the amount of infusion that has been administered into the patient's body and display it on the display screen. is displayed. In such medical equipment, measurement information that changes over time is displayed on the display screen. However, in the technique disclosed in Patent Document 1, the measured value displayed on the display screen is read after the measurement process by the medical device is completed, so it is not assumed that the measured value to be read will change. Therefore, for medical devices in which the information displayed on the display screen changes over time, it is difficult to accurately specify the displayed information based on the captured image of the display screen.

One aspect of the present invention is to provide a program or the like that can appropriately specify display information that changes over time.

A program according to one aspect acquires a photographed image of a display section of a medical device that displays measurement information that changes over time, acquires the measurement information based on the acquired photographic image, and calculates the acquired measurement information. A computer is caused to perform a process of storing information in a storage unit in association with patient information.

In one aspect, display information that changes over time can be appropriately specified.

FIG. 1 is an explanatory diagram showing a configuration example of an information processing system. FIG. 2 is a block diagram showing a configuration example of a server. FIG. 2 is an explanatory diagram showing an example of a record layout of an electronic medical record DB. FIG. 2 is an explanatory diagram showing an example of a record layout of a device DB. FIG. 2 is an explanatory diagram showing a configuration example of a learning model. 3 is a flowchart illustrating an example of a learning model generation process procedure. It is a block diagram showing an example of composition of a staff terminal and a photographing device. FIG. 2 is a block diagram showing an example of the configuration of a medical device. It is a flowchart which shows an example of the code display processing procedure by a 1st medical device. 3 is a flowchart illustrating an example of a medical device registration process procedure. 3 is a flowchart illustrating an example of a medical device registration process procedure. It is an explanatory diagram showing an example of a screen of a staff terminal. 3 is a flowchart illustrating an example of a processing procedure for reading display information of a medical device. 3 is a flowchart illustrating an example of a processing procedure for reading display information of a medical device. It is an explanatory diagram showing an example of a screen of a staff terminal. 12 is a flowchart illustrating another example of a procedure for reading display information of a medical device. It is an explanatory diagram showing an example of an alert screen. FIG. 7 is an explanatory diagram showing the usage state of the medical device in Embodiment 2. FIG. 12 is a flowchart illustrating an example of a processing procedure for reading display information of a medical device according to a second embodiment. It is an explanatory diagram showing an example of a screen of a staff terminal. It is an explanatory diagram showing an example of a screen of a staff terminal. 7 is a flowchart illustrating another example of the processing procedure for reading display information of the medical device according to the second embodiment.

Hereinafter, a program, a model generation method, a medical device, an information processing method, and an information processing device of the present disclosure will be described in detail based on drawings showing embodiments thereof.

(Embodiment 1)
An information processing system that reads device information of a medical device used by a patient in a medical institution or the like and display information displayed on a display screen of the medical device will be described. The medical device in this embodiment is a device that is not provided with a communication function for communicating with other devices. For example, medical devices include syringe pumps and infusion pumps (hereinafter simply referred to as pumps) that are used when administering drugs, infusions, or blood for transfusion into a patient's body, and medical devices that are attached to a patient and are used to monitor blood sugar levels, heart rate, etc. , measuring instruments that measure biological information (vital data) such as blood pressure, respiratory rate, and body temperature. In addition, medical devices, such as ECMO (Extra-Corporeal Membrane Oxygenation) and PCPS (Percutaneous Cardio-Pulmonary Support), require measurement information that changes over time ( This includes devices that require continuous reading of display information (display information). On the display screen of the pump, the set dosing rate, scheduled dose, dose already administered (hereinafter referred to as cumulative dose), etc. are displayed. Further, the measurement values periodically measured by the measuring device are displayed on the display screen of the measuring device. Therefore, the information processing system of this embodiment reads, as display information of the pump, the administration rate and scheduled dose set at the beginning of use, as well as the cumulative dose that changes over time, etc., and reads, as display information of the measuring device, Read measured values that are measured periodically and change over time.

FIG. 1 is an explanatory diagram showing an example of the configuration of an information processing system. The information processing system of this embodiment includes a server 10, a staff terminal 20, a photographing device 30, etc., and each device is communicatively connected via a network N. The network N may be the Internet or may be a LAN (Local Area Network) constructed within a facility such as a medical device in which an information processing system is installed. The server 10 is an information processing device capable of various information processing and transmission/reception of information, and is composed of a server computer, a personal computer, and the like. The server 10 may be a multicomputer consisting of a plurality of computers, or may be a virtual machine virtually constructed using software. Further, the server 10 may be a local server installed within a medical institution, or may be a cloud server communicatively connected via the Internet.

The staff terminal 20 is an information terminal used by medical staff such as doctors and nurses. The staff terminal 20 is an information processing device capable of various information processing and transmission/reception of information, and is composed of a smartphone, a tablet terminal, a personal computer, or the like. The staff terminal 20 is not limited to the above-described device as long as it is an information terminal that has a function of transmitting and receiving information to and from other devices via the network N and a camera.

The photographing device 30 is, for example, a photographing device that is installed on the ceiling or wall of each room of a medical institution and performs photographing processing constantly or periodically. In this embodiment, the photographing device 30 will be described as having a configuration that only performs photographing processing, but it may be configured to perform predetermined image processing on a photographed image and output the processing result.

In this embodiment, the staff terminal 20 and the photographing device 30 photograph the entire medical device including the display screen of the medical device being used while being attached to a patient. Therefore, the photographing device 30 is provided at a position where the medical device can be photographed, and the medical device is arranged at a position where the photographing device 30 can photograph the medical device. Note that settings such as the imaging direction of the imaging device 30, the zoom magnification during imaging, and the focal length may be performed by medical staff when starting to use the medical device, or the imaging device 30 is configured to automatically perform the settings. It's okay.

The medical devices photographed by the staff terminal 20 and the photographing device 30 of this embodiment include a first medical device 40 and a second medical device 50. The first medical device 40 generates a code 40a that includes a device type ID for identifying the device type of the medical device, a device ID for identifying the medical device, and display information displayed on the display screen. It has a configuration to be displayed on a display screen. The second medical device 50 is a conventional medical device that does not have a configuration for generating and displaying a code. A tape or the like on which information (for example, a device ID) is printed is attached. Note that in this embodiment, the device ID is affixed to the outer surface of the second medical device 50, but in addition to the device ID, the device type ID may also be affixed to the outer surface of the second medical device 50. . In this case, it becomes possible to read not only the device ID but also the device type ID from the captured image of the second medical device 50. In FIG. 1, pumps are illustrated as the first medical device 40 and the second medical device 50, but they may also be measuring instruments. Further, even if the devices are of the same type, there may be medical devices that have the function of generating and displaying the code 40a and medical devices that do not have the function, and the former is the first medical device 40 and the latter is the second medical device. Let it be a medical device 50. Further, in this embodiment, the code 40a is a two-dimensional barcode (registered trademark), but it may be a one-dimensional barcode, as long as it can be read by the staff terminal 20.

In the information processing system configured as described above, when the staff terminal 20 photographs the medical devices 40, 50 being used by a patient, the staff terminal 20 collects device information (for example, device ID) and information of the medical devices 40, 50 based on the photographed image. Display information is specified, and each piece of specified information is transmitted to the server 10. The photographing device 30 periodically photographs medical devices 40 and 50 used by a patient, and transmits the acquired photographed images to the server 10. At this time, the server 10 executes a process of specifying device information and display information of the medical devices 40 and 50 based on the photographed image received from the photographing device 30. Note that the imaging device 30 may be configured to perform a process of specifying device information and display information of the medical devices 40 and 50 based on the captured image, and transmit the processing results to the server 10. The server 10 stores electronic medical record data, and by comparing the information received from the staff terminal 20 or the information specified based on the image taken by the imaging device 30 with the electronic medical record data, A process is performed to determine whether the display information of the medical devices 40, 50 is appropriate. For example, when the medical devices 40 and 50 are pumps, the server 10 determines whether the medication information regarding the drug or infusion being administered by the pump matches the content prescribed by the doctor (prescription information). Further, when the medical devices 40 and 50 are measuring instruments, the server 10 determines whether the measured value measured by the measuring instrument is within a normal range. With the above-described configuration, the server 10 can monitor whether or not the state of administration of drugs or infusions by the pump is appropriate, and whether or not the patient's condition measured by the meter is normal. It becomes possible to output an alert and notify medical staff if the condition is not normal or not normal.

FIG. 2 is a block diagram showing an example of the configuration of the server 10. As shown in FIG. The server 10 includes a control section 11, a storage section 12, a communication section 13, an input section 14, a display section 15, a reading section 16, etc., and these sections are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or an AI chip (semiconductor for AI). The control unit 11 performs various information processing, control processing, etc. that should be performed by the server 10 by appropriately executing a program 12P stored in the storage unit 12.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance a program 12P (program product) to be executed by the control unit 11 and various data necessary for executing the program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the program 12P. The storage unit 12 also stores a learning model 12M that has been trained on training data, for example, by machine learning. The learning model 12M, when a captured image of the entire medical device including the display screen of the medical device (second medical device 50) as shown on the left side of FIG. 5 (hereinafter simply referred to as a captured image of the medical device) is input. As shown on the right side of FIG. 5, this is a trained model that has been trained to output a detection result image (label image) indicating the region of the object in the image. When a captured image of a pump is input, the objects detected by the learning model 12M include the device ID affixed to the pump in the image, the administration rate displayed on the pump display screen, the scheduled dose, the cumulative dose, etc. When a captured image of a measuring device is input, the information includes the device ID attached to the measuring device in the image, the measured value displayed on the display screen of the measuring device, etc. The learning model 12M is assumed to be used as a program module that constitutes artificial intelligence software. The learning model 12M performs predetermined calculations on input values and outputs the calculation results, and the storage unit 12 stores data such as coefficients and threshold values of functions that define this calculation as the learning model 12M. be done. The storage unit 12 also stores an electronic medical record DB 12a and an equipment DB 12b. The electronic medical record DB 12a and/or the equipment DB 12b may be stored in another storage device connected to the server 10, or may be stored in another storage device with which the server 10 can communicate (for example, an electronic medical record server). .

The communication unit 13 is a communication module for connecting to the network N by wired communication or wireless communication, and sends and receives information to and from other devices via the network N. The input unit 14 accepts operation input by the user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as one unit.

The reading unit 16 reads information stored in a portable storage medium 10a including a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. read. The program 12P (program product) and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. Further, the program 12P and various data stored in the storage unit 12 may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

In the configuration described above, the input unit 14 and the display unit 15 are not essential, and the server 10 may be configured to accept operations through a connected computer and output information to be displayed to an external display device. Furthermore, the program 12P may be executed on a single computer or on multiple computers interconnected via the network N.

FIG. 3 is an explanatory diagram showing an example of the record layout of the electronic medical record DB 12a. The electronic medical record DB 12a is a database that stores electronic medical record data (medical records) of patients using medical institutions. The electronic medical record DB 12a shown in FIG. 3 includes a patient ID column, a patient information column, a medication information column, a vital data column, and the like. The patient ID column stores identification information (patient ID) for identifying each patient. The patient ID may be, for example, a consultation card number issued by a medical institution. The patient information column stores patient information related to patients in association with patient IDs. The patient information includes, for example, the patient's name, attribute information including age and gender, diagnosis name, medical history, height, weight, and the like. The medication information column stores medication information regarding drugs or infusions that have been administered or are scheduled to be administered to the patient in association with the patient ID. Medication information includes, for example, prescription information indicating the details prescribed by the doctor, the start date and time of medication, device ID of the pump used for medication, drug name (type of drug), scheduled dose, administration speed, cumulative dose, etc. including. The scheduled dose, administration rate, and cumulative dose are numerical values (display information) read from the display screen of the pump, and are stored in association with the date and time of reading. The vital data string stores vital data measured from the patient in association with the patient ID. The vital data includes, for example, blood sugar level, heart rate, blood pressure, respiratory rate, body temperature, etc., and each vital data is stored in association with the device ID of the measuring instrument used for measurement and the measured value. Note that the measured value is a numerical value (display information) read from the display screen of the measuring instrument, and is stored in association with the date and time of reading.

The stored contents of the electronic medical record DB 12a are not limited to the example shown in FIG. 3, but include test information regarding various tests such as blood tests and urine tests conducted on patients, treatment information regarding treatment, surgical information regarding surgery, medication history, and doctors. Various information regarding the patient may be stored, such as comments input by the patient. The examination information includes X-ray images, ultrasound images, CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, and PET (Positron Emission Tomography). ) may include medical images such as images.

FIG. 4 is an explanatory diagram showing an example of the record layout of the device DB 12b. The device DB 12b is a database that stores information regarding medical devices 40 and 50 used in medical institutions. The device DB 12b shown in FIG. 4 includes a device ID column, a device type ID column, a type information column, a device information column, a usage information column, etc., and stores each information regarding the medical devices 40 and 50 in association with the device ID. . The device ID column stores identification information (device ID) assigned to each medical device 40, 50. The device type ID column stores identification information (device type ID) for identifying the type (for example, model) of the medical devices 40, 50, and the type information column stores information regarding the type of the medical devices 40, 50, for example, the device. Stores the type name for identifying the type, the name of the medical device, etc. The device information column stores information regarding the medical devices 40, 50, such as the manufacturer name, model number, and product name of the medical devices 40, 50. The usage information column stores information regarding the usage of the medical devices 40 and 50. For example, if the medical devices 40, 50 are pumps, usable drugs, settable scheduled doses and speeds, measurable cumulative doses, etc. are stored as usage information, and if the medical devices 40, 50 are measuring instruments, the information is stored as usage information. , items to be measured, numerical values in measurable ranges, etc. are stored as usage information. The storage contents of the device DB 12b are not limited to the example shown in FIG. 4, and various information regarding the medical devices 40 and 50 may be stored.

FIG. 5 is an explanatory diagram showing an example of the configuration of the learning model 12M. The learning model 12M is a model that recognizes a predetermined object included in an input image. The learning model 12M may be a model that can classify objects in an image pixel by pixel, for example, by semantic segmentation. The learning model 12M of this embodiment inputs the captured image of the second medical device 50, performs calculations to detect objects included in the captured image based on the input captured image, and outputs the detected results. It's a model. Specifically, when a captured image of a pump is input, the learning model 12M calculates objects such as the device ID of the pump in the captured image, the scheduled dose displayed on the display screen, the administration speed, and the cumulative dose. This is a model that realizes multi-label classification for detection, and outputs a detected captured image (hereinafter referred to as a label image) in which a bounding box is added to each detected object. In addition, the learning model 12M is a model that realizes multi-label classification that detects objects such as the device ID of the measuring instrument in the captured image and the measured value displayed on the display screen when a captured image of a measuring instrument is input. It is. Note that the learning model 12M may include a pump model and a measuring instrument model separately. The learning model 12M can be configured with, for example, U-Net, FCN (Fully Convolutional Network), SegNet, or the like. Note that the learning model 12M may be composed of R-CNN (Regions with Convolution Neural Network), Fast R-CNN, SSD (Single Shot Multibook Detector), Mask R-CNN, YOLO (You Only Look Once), etc. , may be configured by combining multiple algorithms.

The learning model 12M has an input layer, a middle layer, and an output layer. A captured image to be processed is input to the input layer. The intermediate layer includes a convolution layer, a pooling layer, and a deconvolution layer. The convolution layer extracts image features from pixel information of the image input via the input layer to generate a feature map, and the pooling layer compresses the generated feature map. A plurality of convolution layers and pooling layers are repeatedly provided, and feature maps generated by the plurality of convolution layers and pooling layers are output to the deconvolution layer. The deconvolution layer enlarges (maps) the feature map generated by the convolution layer and the pooling layer to the original image size. Note that the deconvolution layer detects which object exists in which position in the image based on the feature amount extracted by the convolution layer, and generates a label image in which each object is shown by a bounding box. The output layer outputs a label image indicating the detection result of the target object based on the calculation result of the intermediate layer. In the example shown in FIG. 5, a label image is output in which the device ID (A1) attached to the pump in the photographed image, the administration speed A2 displayed on the display screen of the pump, and the cumulative administration amount A3 are detected. ing.

With the above-described configuration, when a photographed image of the second medical device 50 is input, the learning model 12M can distinguish between the device ID of the medical device in the photographed image and the display information displayed on the display screen of the medical device. Output the detected label image. The server 10 acquires the device ID and display information of the photographed second medical device 50 by extracting each piece of detected information from the photographed image based on the label image output from the learning model 12M. be able to. Therefore, even if the second medical device 50 does not have a communication function, the device ID and display information can be acquired from the captured image of the second medical device 50.

The learning model 12M includes a photographed image for training and a label image (correct label image) in which data (marks) indicating each object are labeled with respect to each object region in the photographed image. It can be generated by preparing training data and using this training data to perform machine learning on an unlearned learning model. Note that in the training label image, a label indicating the coordinate range corresponding to the region of each target object and the type of each target object is added to the training captured image. A captured image and a label image for training data are generated in advance and stored as one training data, for example, in a training DB (not shown) prepared in the storage unit 12.

The learning model 12M learns to output a label image (correct label image) included in the training data when a captured image included in the training data is input. In the learning process, the learning model 12M performs calculations in the intermediate layer based on the input photographed image, and obtains a detection result of detecting a target object in the photographed image. Specifically, the learning model 12M obtains as output a label image in which each target object region in the photographed image is labeled with a value indicating the type of the classified object. Then, the learning model 12M compares the acquired detection result (label image) with the correct coordinate range of the target object region and the type of target object indicated by the training data, and performs arithmetic processing in the intermediate layer so that the two approximate each other. Optimize the parameters used for The parameter is, for example, a weight (coupling coefficient) between nodes in the intermediate layer. The parameter optimization method is not particularly limited, but steepest descent method, error backpropagation method, etc. can be used. As a result, when a photographed image is input, a learning model 12M that outputs a label image indicating the region of each object in the photographed image is obtained.

The server 10 prepares such a learning model 12M in advance and uses it when acquiring the device information and display information of the second medical device 50 based on the photographed image taken by the photographing device 30. Learning of the learning model 12M may be performed by another learning device. A trained learning model 12M generated by learning performed by another learning device is downloaded from the learning device to the server 10 via the network N or the portable storage medium 10a, and is stored in the storage unit 12, for example. Note that the staff terminal 20 also stores in advance a learning model 22M (see FIG. 7) that is the same as the learning model 12M, and determines the device information of the second medical device 50 based on the photographed image taken by the camera 26 (see FIG. 7). and used when acquiring display information.

Below, a process of learning the above-mentioned training data to generate the learning model 12M will be described. FIG. 6 is a flowchart showing an example of the procedure for generating the learning model 12M. The following processing is performed by the control unit 11 of the server 10 according to the program 12P stored in the storage unit 12, but may be performed by another learning device. In the following processing, it is assumed that training data based on the above-mentioned captured images and label images is stored in the training DB.

The control unit 11 of the server 10 acquires one piece of training data from the training DB (S11). Specifically, the control unit 11 reads one set of photographed images and label images as shown in FIG. 5 from the training DB. Then, the control unit 11 performs a learning process on the learning model 12M using the acquired training data (S12). Here, the control unit 11 inputs the photographed image included in the training data to the learning model 12M, and acquires a label image output from the learning model 12M by inputting the photographed image. Note that the learning model 12M performs calculations based on the input captured image and calculates a label image to be output from the output layer. The control unit 11 compares each pixel value of the label image output from the learning model 12M with each pixel value of the correct label image included in the training data, and causes the learning model 12M to learn so that the two approximate each other. . In the learning process, the control unit 11 optimizes, for example, parameters such as weights (coupling coefficients) between nodes in the intermediate layer using an error backpropagation method that sequentially updates parameters from the output layer to the input layer of the learning model 12M. do.

The control unit 11 determines whether or not there is unprocessed training data that has not been subjected to learning processing among the training data stored in the training DB (S13). If it is determined that there is unprocessed training data (S13: YES), the control unit 11 returns to the process of step S11, and performs the processes of steps S11 to S12 on the training data that has not been subjected to the learning process. If it is determined that there is no unprocessed training data (S13: NO), the control unit 11 ends the series of processes.

Through the learning process described above, the learning model has been trained to output, when a captured image is input, a label image in which the device ID and display information of the second medical device 50 in the captured image are indicated by a bounding box. 12M is generated. Note that the learning model 12M can be further optimized by repeatedly performing the learning process using the training data as described above. Further, the learning model 12M that has already been trained can be retrained by performing the above-described processing, and in this case, a learning model 12M with higher discrimination accuracy can be generated. Further, for example, by learning training data for each medical institution, a learning model 12M suitable for each medical institution may be generated. Since the type or manufacturer of medical equipment used at each medical institution differs slightly from one medical institution to another, by generating the learning model 12M for each medical institution, the device ID and display of the medical equipment used at each medical institution can be created. Information detection accuracy can be improved.

FIG. 7 is a block diagram showing a configuration example of the staff terminal 20 and the photographing device 30. The staff terminal 20 includes a control section 21, a storage section 22, a communication section 23, an input section 24, a display section 25, a camera 26, etc., and these sections are interconnected via a bus. The control unit 21, storage unit 22, communication unit 23, input unit 24, and display unit 25 of the staff terminal 20 are similar to the control unit 11, storage unit 12, communication unit 13, input unit 14, and display unit 15 of the server 10. Since this is just a configuration, the explanation will be omitted. The storage unit 22 of the staff terminal 20 stores, in addition to the program 22P executed by the control unit 21, a medical equipment management program 22AP for realizing processing for managing the medical equipment 40, 50 used for each patient, and a learning program. A learning model 22M having the same configuration as the model 12M is stored. Moreover, it is desirable that the communication unit 23 is a communication module for connecting to the network N by wireless communication.

The camera 26 is an imaging unit having a lens, an image sensor, etc., and performs a photographing process according to a photographing instruction received via the input unit 24, and stores the obtained image data (hereinafter referred to as a photographed image) in the storage unit 12. do. The camera 26 performs photographing processing to acquire one photographed image (still image) in response to one photographing instruction.

The photographing device 30 includes a control section 31, a communication section 32, a camera 33, etc., and these sections are interconnected via a bus. The control unit 31 includes an arithmetic processor such as a CPU or an MPU, a memory, etc., and the arithmetic processor executes a program stored in the memory to execute various processes performed by the photographing device 30. The communication unit 32 and camera 33 have the same configuration as the communication unit 23 and camera 26 of the staff terminal 20. In the photographing device 30 of this embodiment, the camera 33 performs photographing processing according to photographing instructions from the control unit 31 periodically, for example, every 10 seconds, every 30 seconds, every minute, etc., and communicates the obtained photographed images. The unit 32 transmits the information to the server 10 via the network N.

FIG. 8 is a block diagram showing an example of the configuration of the medical devices 40 and 50. The first medical device 40 includes a control section 41, a storage section 42, a measurement section 43, an input section 44, a display section 45, a notification section 46, etc., and these sections are interconnected via a bus. The control unit 41, storage unit 42, input unit 44, and display unit 45 of the first medical device 40 have the same configuration as the control unit 21, storage unit 22, input unit 24, and display unit 25 of the staff terminal 20, so they will be explained. omitted. Note that the storage unit 42 of the first medical device 40 stores a code generation program 42a for generating a code from predetermined information. The code generation program 42a is an application program for generating a code that is linked to the device ID assigned to the first medical device 40 and the display information currently displayed on the display unit 45. trademark). Note that the code generated by the code generation program 42a is not limited to a QR code, but may be a barcode, a predetermined mark, or the like.

The measurement unit 43 executes measurement processing in the first medical device 40. For example, when the first medical device 40 is a pump, the measurement unit 43 executes a process of measuring the cumulative amount of the drug being administered. Further, when the first medical device 40 is a measuring instrument, the measuring unit 43 executes a process of measuring vital data such as the patient's blood sugar level, heart rate, blood pressure, respiratory rate, and body temperature. The notification unit 46 includes a lamp, a buzzer, a speaker, etc., and is configured to notify a person near the first medical device 40 by lighting or blinking a lamp, sounding a buzzer, outputting a warning sound or a message from a speaker, etc. Output an alert to.

The second medical device 50 has the same configuration as the first medical device 40, so a description thereof will be omitted. Note that the storage unit 52 of the second medical device 50 does not store a code generation program, and the second medical device 50 does not have a function of generating a code from predetermined information. When the medical devices 40 and 50 are pumps, in addition to the above-described configuration, the medical devices 40 and 50 have a liquid feeding section that feeds drugs, infusions, blood for transfusion, etc. into the patient's body via a tube. .

The processing performed by the first medical device 40 will be described below. FIG. 9 is a flowchart showing an example of a code display processing procedure by the first medical device 40. The following processing is performed by the control unit 41 of the first medical device 40 according to a program stored in the storage unit 42. For example, the control unit 41 of the first medical device 40 executes a process of displaying the measurement results (measurement information that changes in time series) by the measurement unit 43 on the display unit 45, and the displayed content (display information) It is determined whether a change has occurred (S21). For example, in the first medical device 40 shown in FIG. This is the result (measurement information). Therefore, in this case, the control unit 41 determines whether or not there has been a change in the cumulative dose to be displayed. Further, when the set value of the administration speed is changed, the control unit 41 determines that a change has occurred in the administration speed to be displayed.

If it is determined that the display information has changed (S21: YES), the control unit 41 displays the device ID assigned to the first medical device 40 and the changed display information (in this case, the administration speed and cumulative dose). (S22). Note that the device ID is stored in the storage section 42 in advance. The control unit 41 (generation unit) generates a QR code linked to the acquired device ID and display information according to the code generation program 42a (S23). Then, the control unit 41 (display control unit) displays the generated QR code on the display unit 45 (S24). Thereby, as shown in FIG. 1, the QR code 40a is displayed on the display screen of the first medical device 40. Note that when the control unit 41 determines that there is no change in the display information (S21: NO), the process proceeds to step S25.

The control unit 41 determines whether to end the above-described processing (S25). For example, when the control unit 41 receives an instruction to terminate the operation of the first medical device 40 via the input unit 44, the control unit 41 determines to terminate the process. When the control unit 41 determines that the above-described processing is not to be ended (S25: NO), the control unit 41 returns to the processing of step S21, and executes the processing of steps S22 to S24 each time a change occurs in the display information. If the control unit 41 determines to end the above-described processing (S25: YES), it ends the series of processing. Through the above-described process, the first medical device 40 generates a code linked to the device ID and display information and displays it on the display section 45 every time there is a change in the content displayed on the display section 45. Therefore, by reading the code displayed on the display section 45, the device ID of the first medical device 40 and the information displayed on the display section 45 can be acquired.

In the process described above, the first medical device 40 is configured to generate and display a code including the device ID and the content (display information) displayed on the display unit 45. In addition, the QR code displayed on the display unit 45 of the first medical device 40 not only displays the currently displayed display information, but also displays information from the current time to a predetermined time ago (for example, 30 seconds ago, 1 minute ago, etc.). The display information displayed on the display unit 45 may be included in the display information. That is, the control unit 41 of the first medical device 40 generates a code that is linked to the device ID and the display information displayed on the display unit 45 from a time a predetermined time before the current time to the current time. may be displayed on the display section 45. Specifically, the control unit 41 generates time series data that associates each date and time information from a predetermined time ago to the current time with the display information displayed on the display unit 45 at the date and time indicated by each date and time information. Generate code associated with . Thereby, display information that changes over time can be included in the QR code. With such a configuration, for example, even if the timing at which the first medical device 40 updates the QR code is earlier than the timing at which the imaging device 30 performs imaging, the display section 45 of the first medical device 40 does not display the QR code. All of the information displayed in chronological order can be acquired from images captured by the imaging device 30.

Next, a registration process for linking the medical devices 40, 50 to a patient when using the medical devices 40, 50 will be described. 10 and 11 are flowcharts showing an example of the registration process procedure for the medical devices 40 and 50, and FIG. 12 is an explanatory diagram showing an example of the screen of the staff terminal 20. The following processing is executed by the control unit 21 of the staff terminal 20 according to the program 22P and the medical equipment management program 22AP stored in the storage unit 22, and the control unit 11 of the server 10 executes the Execute according to a certain program 12P. In FIGS. 10 and 11, the left side shows the processing performed by the staff terminal 20, and the right side shows the processing performed by the server 10. Below, processing when the medical devices 40 and 50 are pumps will be explained.

When using the medical devices 40, 50 for a patient, the medical staff performs a registration process to associate the patient with the medical devices 40, 50 using the staff terminal 20. Specifically, the medical staff starts the medical device management program 22AP on the staff terminal 20 and performs the registration process according to the medical device management program 22AP. When the control unit 21 of the staff terminal 20 starts the medical device management program 22AP according to the instruction from the input unit 24, it displays a patient ID reading screen as shown in FIG. 12A on the display unit 25 (S31). The patient ID reading screen has a frame indicating the patient ID reading area and a shooting button. The patient ID can be obtained by, for example, a patient wearing a wristband on which a code that encodes the patient ID is printed, and reading the code on the wristband. Therefore, the medical staff takes a picture with the camera 26 with the code printed on the patient's wristband in the reading area of the reading screen, and the control unit 21 of the staff terminal 20 reads the code taken with the camera 26. The patient ID (identification information assigned to the patient) is obtained by analysis (S32). In addition to the configuration in which the control unit 21 reads the patient ID from the code (photographed image) taken by the camera 26, if a code reader such as a barcode reader or a QR code reader is connected to or built in the staff terminal 20. , the patient ID may be read from the code on the wristband by a code reader.

The control unit 21 acquires the name of this patient based on the acquired patient ID (S33). For example, the control unit 21 transmits the acquired patient ID to the server 10 and acquires from the server 10 the patient name corresponding to the patient ID. Note that the process in step S33 is not essential. After reading the patient ID, the control unit 21 displays a medicine ID reading screen as shown in FIG. 12B on the display unit 25 (S34). The medicine ID reading screen has a frame indicating the medicine ID reading area and a shooting button. For example, a code that encodes the drug ID is printed on a drug package, and the drug ID can be obtained by reading the code on the drug package. Therefore, the medical staff uses the camera 26 to take a picture with the code printed on the drug package in the reading area of the reading screen, and the control unit 21 of the staff terminal 20 analyzes the code taken by the camera 26. By doing so, the drug ID is acquired (S35). Here, the control unit 21 may read the drug ID from the code of the drug package using a code reader, instead of reading the drug ID from the code photographed by the camera 26.

The control unit 21 obtains the name of this drug (drug name) based on the obtained drug ID (S36). For example, the control unit 21 transmits the obtained drug ID to the server 10 and obtains from the server 10 a drug name corresponding to the drug ID. The processing in step S36 is also not essential. Note that if the code printed on the drug package includes the drug name, the control unit 21 may acquire the drug name in addition to the drug ID by reading the code.

Next, the control unit 21 displays the imaging screen of the medical device as shown in FIGS. 12C and 12D on the display unit 25 (S37). The photographing screen of the medical device has a frame indicating the photographing area of the medical devices 40, 50 and a photographing button. The medical staff uses the camera 26 to take a picture with the entire medical equipment 40, 50, including the display screen of the medical equipment 40, 50, in the imaging area on the imaging screen, and the control unit 21 of the staff terminal 20 controls the camera 26. 26 to photograph the medical devices 40 and 50 (S38). Note that FIG. 12C shows a state in which the first medical device 40 is photographed, and FIG. 12D shows a state in which the second medical device 50 is photographed. The imaging area of the medical devices 40, 50 is preferably the entire device, but it is sufficient if the main parts such as the display screen and operation buttons of the medical devices 40, 50 are included, as shown in FIGS. 12C and 12D. Furthermore, part of the medical equipment 40, 50 may not be photographed.

The control unit 21 executes a reading process to read the QR code from the photographed images of the medical devices 40, 50 obtained by photographing (S39), and reads the device ID and display information linked to the QR code. Note that since the QR code is displayed on the display screen of the first medical device 40, the QR code can be read for the captured image of the first medical device 40 as shown in FIG. 12C. On the other hand, since the QR code is not displayed on the display screen of the second medical device 50, the QR code cannot be read in the captured image of the second medical device 50 as shown in FIG. 12D. The control unit 21 determines whether the QR code was read from the captured image (S40), and if it is determined that the QR code was read (S40: YES), the device ID and display information linked to the QR code are displayed. (S41). Note that if the device type ID of the first medical device 40 is also linked to the QR code, the control unit 21 also acquires the device type ID from the captured image in addition to the device ID and display information. Additionally, if display information that changes over time (display information that is time-series data) is linked to the QR code, the control unit 21 acquires the device ID and the display information that changes over time from the captured image. do.

On the other hand, when the control unit 21 determines that the QR code cannot be read (S40: NO), the control unit 21 uses the learning model 22M to acquire the device ID and display information from the captured image of the second medical device 50 (S42). Here, the control unit 21 inputs the photographed image of the second medical device 50 obtained by photographing in step S38 to the learning model 22M, and acquires the label image output from the learning model 22M. The control unit 21 acquires a label image as shown in FIG. 5, acquires the device ID from the device ID area A1 in the label image (photographed image), acquires the administration rate from the administration rate area A2, and performs administration. The cumulative dose is acquired from the cumulative dose area A3. Thereby, the control unit 21 can acquire the device ID and display information of the second medical device 50 from the captured image of the second medical device 50. Note that the process of acquiring the device ID and display information from the photographed image of the second medical device 50 is not limited to the configuration executed by the staff terminal 20, and may be executed by the server 10, for example. In this case, the control unit 21 transmits the photographed image of the second medical device 50 to the server 10, and the server 10 uses the learning model 12M to acquire the device ID and display information from the photographed image of the second medical device 50. can be configured so that the acquired device ID and display information are transmitted from the server 10 to the staff terminal 20.

As shown in FIG. 12E, the control unit 21 displays the patient information acquired in steps S32 to S33, the drug information acquired in steps S35 to S36, and the device ID and display information acquired in steps S38 to S42 on the display unit 25. (S43). As a result, information such as the administration rate and scheduled dose set for the medical devices 40, 50 when the medical devices 40, 50 are used is read from the photographed images of the medical devices 40, 50 and sent to the staff terminal 20. will be displayed. In the example shown in FIG. 12E, the administration speed and cumulative administration amount are displayed as the display information, but the second medical device 50 also displays the drug name, scheduled administration amount, scheduled end time, remaining time, etc. on the display screen. If so, the drug name, scheduled dose, scheduled end time, remaining time, etc. are also acquired from the photographed image and displayed on the screen shown in FIG. 12E.

The screen shown in FIG. 12E has a send button, and the medical staff determines whether each piece of information displayed on the display unit 25 is appropriate, and if appropriate, operates the send button. Note that the screen shown in FIG. 12E may be configured so that each piece of displayed information can be modified. In this case, if the information displayed in FIG. 12E is not appropriate, that is, if a reading error by the staff terminal 20 occurs, it can be corrected by the medical staff. Further, if the information displayed in FIG. 12E is not appropriate, the process may be returned to step S37, for example, and the device ID and display information may be reread from the photographed images of the medical devices 40 and 50. The control unit 21 determines whether the send button has been operated (S44), and if it is determined that the transmit button has not been operated (S44: NO), it waits until it is operated. If it is determined that the send button has been operated (S44: YES), the control unit 21 sends the patient information, drug information, device ID, and display information displayed on the screen shown in FIG. 12E to the server 10 (S45). .

The control unit 11 of the server 10 receives each piece of information transmitted from the staff terminal 20, and determines whether the received information matches the content prescribed by the doctor (prescription information) (S46). For example, the control unit 11 reads prescription information corresponding to the received patient information (patient ID) from the electronic medical record DB 12a, and determines whether the drug name indicated by the received drug information and the administration speed indicated by the display information match the prescription details. to judge. Furthermore, since the cumulative dose here is 0, the control unit 11 determines whether the prescription content is met depending on whether the cumulative dose indicated by the received display information is 0 or not. When the display information includes a scheduled dose, the control unit 11 also determines whether the scheduled dose matches the prescription details. In addition to determining whether each piece of information indicated by the received drug information completely matches the prescription content, the control unit 11 also determines whether each piece of information indicated by the drug information is within a predetermined range with respect to the prescription content. It may be configured to determine whether or not. The predetermined range is a range of differences in prescription contents that do not cause a medical accident. If it is determined that the information does not conform to the prescription information (S46: NO), the control unit 11 sends a display screen to the staff terminal 20 indicating that the information does not conform to the prescription information, thereby indicating that the information is not appropriate. Notify (S47). For example, the control unit 11 generates a screen as shown in FIG. 12F and transmits it to the staff terminal 20. The screen shown in FIG. 12F is a screen for notifying that the administration rate differs from the prescription content, and includes a "read again" button for instructing re-imaging of the medical devices 40 and 50.

The control unit 21 of the staff terminal 20 receives the screen transmitted by the server 10 and displays it on the display unit 25, thereby displaying that the information is not appropriate (S48). The medical staff confirms that the information is not appropriate via the screen shown in FIG. 12F, and operates the "Read Again" button to retake the image of the medical devices 40 and 50. The control unit 21 determines whether or not the "read again" button has been operated (S49), and if it is determined that it has not been operated (S49: NO), it waits until it is operated. If it is determined that the "read again" button has been operated (S49: YES), the control unit 21 returns to the process of step S37 and displays the imaging screen of the medical device as shown in FIGS. 12C and 12D on the display unit 25. (S37), and start over from the medical device imaging process. Note that the control unit 21 may be configured to return to the process of step S31 and start over from reading the patient ID.

If the control unit 11 of the server 10 determines that the prescription information matches the prescription information (S46: YES), the control unit 11 stores the medication information including the drug information, device ID, and display information received from the staff terminal 20 in the electronic medical record DB 12a (S50). . Specifically, the control unit 11 stores the current date and time in the start date and time column, the device ID received from the staff terminal 20 in the device ID column, and the staff terminal 20 as medication information corresponding to the patient information received from the staff terminal 20. The drug name indicated by the drug information received from is stored in the drug name column, and the scheduled dose, administration speed, and cumulative dose included in the display information are stored in the scheduled dose column, dosage speed column, and cumulative dose column, respectively. . Note that the scheduled dose, the rate of administration, and the cumulative dose are each stored in each column in association with the current date and time.

The control unit 11 sends the information received from the staff terminal 20, that is, a display screen displaying a message notifying that the settings set for the medical devices 40 and 50 match the prescription information, to the staff terminal 20. By transmitting, it is notified that the setting contents are appropriate (S51). For example, the control unit 11 generates a screen displaying a message such as "The settings were appropriate" and sends it to the staff terminal 20. The control unit 21 of the staff terminal 20 receives the screen transmitted by the server 10 and displays it on the display unit 25, thereby displaying a message indicating that the settings for the medical devices 40 and 50 are appropriate (S52). , completes the series of processing.

Through the above-described processing, when the medical devices 40, 50 are used on a patient, it is possible to link the medical devices 40, 50 to the patient, and also each information set in the medical devices 40, 50 and displayed on the display screen. can be read by the staff terminal 20. Therefore, the medical staff only has to compare the actual display content of the medical devices 40, 50 with the display content of the medical devices 40, 50 read by the staff terminal 20 and judge whether or not it is appropriate. Conventionally, medical staff have had to visually read the displayed contents of the medical devices 40, 50 and manually input them into the staff terminal 20. In this case, human errors such as reading errors or input errors occur. There was a risk. However, in this embodiment, the staff terminal 20 reads the displayed content from the photographed image, so it is possible to reduce the workload of the medical staff and suppress human errors. Further, when medical staff visually check the display contents of the medical devices 40, 50, it is necessary to get close to the medical devices 40, 50 and the patient, and it is necessary to take sufficient measures against infectious diseases. However, in this embodiment, since it is sufficient to photograph the display screens of the medical devices 40, 50 with the staff terminal 20, there is no need to approach the medical devices 40, 50 and patients, which is also effective in preventing infectious diseases.

Although FIGS. 10 and 11 described above show the process when the medical devices 40 and 50 are pumps, the same process is possible when the medical devices 40 and 50 are measuring instruments. Note that when the medical devices 40 and 50 are measuring instruments, the control unit 21 of the staff terminal 20 does not perform the processing in steps S34 to S36, and performs processing to display patient information, device ID, and display information in step S43. , In step S45, processing for transmitting patient information, device ID, and display information to the server 10 is performed. Further, the control unit 11 of the server 10 performs a process of determining whether the measured value indicated by the display information received from the staff terminal 20 is within a normal range in step S46. For example, numerical values in the normal range are set in advance for each vital data and stored in the storage unit 12, and the control unit 11 controls the medical equipment 40 based on the equipment ID received from the staff terminal 20. , 50, and determines whether the measured value indicated by the display information received from the staff terminal 20 is within the normal range. If the control unit 11 determines that the measured value is not within the normal range, it notifies the staff terminal 20 that the measured value is not within the normal range in step S47, and if it determines that the measured value is within the normal range, the measured value is stored in the vital data string of the electronic medical record DB 12a, and in step S51, the staff terminal 20 is notified that the measured value is within the normal range. As a result, even if the medical devices 40, 50 are measuring instruments, when the medical devices 40, 50 are used on a patient, medical care can be performed based on the images of the medical devices 40, 50 taken by the staff terminal 20. Information on the devices 40, 50 can be acquired, and the medical devices 40, 50 can be linked to the patient.

Next, a process of reading information displayed on the display screen of the medical equipment 40, 50 being used by a patient using the staff terminal 20 will be described. 13 and 14 are flowcharts showing an example of a processing procedure for reading display information on the medical devices 40 and 50, and FIG. 15 is an explanatory diagram showing an example of a screen on the staff terminal 20. In FIGS. 13 and 14, the left side shows the processing performed by the staff terminal 20, and the right side shows the processing performed by the server 10.

When the medical staff is using the medical devices 40, 50 on a patient, for example, periodically or when performing some kind of treatment on the patient, the medical staff determines whether the display information of the medical devices 40, 50 is appropriate at a predetermined timing. The staff terminal 20 is used to determine whether or not there is. Here, the control unit 21 of the staff terminal 20 starts the medical device management program 22AP according to the instruction from the input unit 24, and displays the imaging screen of the medical device on the display unit 25 (S61). The imaging screen of the medical device has the same configuration as the screen shown in FIG. 12C or 12D. Further, the control unit 21 of the staff terminal 20 executes the same processing as steps S38 to S42 in FIG. 10 (S62 to S66). Specifically, the control unit 21 (image acquisition unit) photographs the entire medical equipment 40, 50 including the display screen of the medical equipment 40, 50 using the camera 26 (S62).

Then, the control unit 21 (information acquisition unit) executes a reading process to read the QR code from the photographed images of the medical devices 40 and 50 (S63), and reads the device ID and display information linked to the QR code. The control unit 21 determines whether the QR code can be read from the photographed image (S64), and when determining that the QR code can be read (S64: YES), acquires the device ID and display information linked to the QR code. (S65). On the other hand, when the control unit 21 determines that the QR code cannot be read (S64: NO), the control unit 21 uses the learning model 22M to acquire the device ID and display information from the captured image of the second medical device 50 (S66). Here again, the process of acquiring the device ID and display information from the photographed image of the second medical device 50 is not limited to the configuration executed by the staff terminal 20, and may be executed by the server 10, for example.

Next, the control unit 21 displays the device ID and display information acquired in steps S62 to S66 on the display unit 25, as shown in FIG. 15A (S67). As a result, the information displayed on the medical devices 40 and 50 being used for the patient is read from the photographed images of the medical devices 40 and 50 and displayed on the staff terminal 20. In the example shown in FIG. 15A, the administration rate and cumulative dose are displayed as display information. The screen shown in FIG. 15A has a send button, and the medical staff determines whether each piece of information displayed on the display unit 25 is appropriate, and if appropriate, operates the send button. Here again, the screen shown in FIG. 15A may be configured so that each displayed information can be corrected, and if the displayed information is not appropriate, the screen shown in FIG. The information may be configured to be reread.

The control unit 21 determines whether the send button on the screen shown in FIG. 15A has been operated (S68), and if it is determined that it has not been operated (S68: NO), it waits until it is operated. If it is determined that the send button has been operated (S68: YES), the control unit 21 sends the device ID and display information displayed on the screen shown in FIG. 15A to the server 10 (S69). Note that in the process shown in FIG. 13, the control unit 21 may have a configuration in which steps S67 to S68 are not performed. In this case, after acquiring the device ID and display information in step S65 or S66, the control unit 21 transmits the acquired device ID and display information to the server 10 without displaying them on the display unit 25.

When the control unit 11 of the server 10 receives the device ID and display information from the staff terminal 20, it stores the medication information or measurement value including the received device ID and display information in the electronic medical record DB 12a (S70). Specifically, when the received device ID is a pump device ID, the control unit 11 searches for the received device ID from the device IDs stored in the device ID column of medication information in the electronic medical record DB 12a, In association with the retrieved device ID, the scheduled dose, administration rate, and cumulative dose included in the received display information are stored together with the current date and time in the scheduled dose column, dose speed column, and cumulative dose column, respectively. . In addition, when the received device ID is the device ID of a measuring instrument, the control unit 11 searches for the received device ID from the device IDs stored in the device ID column of vital data in the electronic medical record DB 12a. The measured value included in the received display information is stored in a measured value column along with the current date and time in association with the device ID.

Next, the control unit 11 determines whether the medication information stored in the electronic medical record DB 12a matches the prescription information or whether the measured value is within the normal range (S71). Here, when the medication information is stored in the electronic medical record DB 12a in step S70, the control unit 11 determines whether or not this medication information matches the prescription information. Specifically, the control unit 11 identifies the patient corresponding to the device ID received from the staff terminal 20, reads prescription information of the identified patient from the electronic medical record DB 12a, and sets the dosing rate included in the medication information to the prescription content. Determine suitability. Further, the control unit 11 determines whether the cumulative dose included in the medication information is an appropriate increase in the cumulative dose measured most recently, and whether or not it conforms to the prescription content. do. Note that when the display information includes a scheduled dose, the control unit 11 also determines whether the scheduled dose matches the prescription details. Furthermore, when the measured value is stored in the electronic medical record DB 12a in step S70, the control unit 11 determines whether or not this measured value is within the normal range. Specifically, the control unit 11 identifies the patient's vital data corresponding to the device ID received from the staff terminal 20, and determines whether the identified vital data is within a normal range.

When the control unit 11 determines that the medication information does not match the prescription information, or when it determines that the measured value is not within the normal range (S71: NO), it displays an alert screen as shown in FIG. 15B or 15C, for example. It is generated and transmitted to the staff terminal 20 (S72). The screen shown in FIG. 15B is a screen to notify that the administration rate has been changed to a rate different from the prescription content, and the screen shown in FIG. 15C is a screen to notify that the blood sugar level has fallen outside the normal range. This is the screen. The control unit 21 of the staff terminal 20 receives the alert screen sent by the server 10 and displays it on the display unit 25 (S73). As a result, if the medical devices 40, 50 are pumps and the display information of the medical devices 40, 50 is different from the prescription information, or if the medical devices 40, 50 are measuring instruments, the medical devices 40, 50, When the measured value displayed on 50 is outside the normal range, the medical staff can be notified (warned) by an alert screen displayed on the staff terminal 20. When the medical staff receives a warning regarding the pump on the alert screen, they check the status of the medical devices 40 and 50 and reset the settings to match the prescription information. This makes it possible to early detect and respond to the occurrence of a medical accident in which inappropriate medication is administered, such as when settings are changed to contents that do not match the prescription information due to an operation by the patient. Furthermore, when the medical staff receives a warning regarding the measuring device on the alert screen, they check the patient's condition and, if necessary, consult a doctor. This makes it possible to detect at an early stage that the patient's vital data is outside the normal range, and to take appropriate measures.

When the control unit 11 determines that the medication information matches the prescription information, or when it determines that the measured value is within the normal range (S71: YES), the control unit 11 informs the staff terminal that the medication information or the measured value is appropriate. 20 (S74). Specifically, when the medical devices 40 and 50 are pumps, the control unit 11 sends a message to the staff terminal 20 notifying that the settings for the pumps match the prescription information, and If it is a measuring device, a message is sent to the staff terminal 20 notifying that the measured value by the measuring device is within the normal range. The control unit 21 of the staff terminal 20 receives the message sent by the server 10 and displays it on the display unit 25, thereby displaying a message indicating that the settings for the pump (medication information) or the measured value by the meter are appropriate. is displayed (S75), and the series of processing ends.

Through the above-described processing, each information displayed on the display screen of the medical devices 40, 50 can be read by photographing the medical devices 40, 50 being used by a patient with the staff terminal 20. Further, when the medical devices 40 and 50 are pumps, it can be determined whether the display information (medication information) read by the staff terminal 20 matches the prescription information registered in the electronic medical record DB 12a. Furthermore, when the medical devices 40 and 50 are measuring instruments, it can be determined whether the displayed information (measured value) read by the staff terminal 20 is within a normal range. Conventionally, medical staff visually read the display contents of the medical devices 40 and 50 to determine whether the contents are appropriate (normal contents), which reduces human errors such as reading errors or judgment errors. was difficult. However, in this embodiment, the staff terminal 20 reads the displayed content from the photographed image and compares it with the registered content in the electronic medical record DB 12a, so that the workload of the medical staff can be reduced and human error can be suppressed. Furthermore, in this embodiment, since it is sufficient to photograph the display screens of the medical devices 40 and 50 with the staff terminal 20, there is no need for medical staff to approach the medical devices 40 and 50 and patients, and it is possible to take measures against infectious diseases.

Next, a process of reading display information of the medical devices 40, 50 being used on a patient from an image photographed by the photographing device 30 will be described. FIG. 16 is a flowchart showing another example of the processing procedure for reading display information of the medical devices 40, 50, and FIG. 17 is an explanatory diagram showing an example of an alert screen. In FIG. 16, the left side shows the processing performed by the imaging device 30, and the right side shows the processing performed by the server 10.

The photographing device 30 performs photographing processing of the medical devices 40, 50 within the photographing range periodically or when a change occurs in the display content of the medical devices 40, 50 to be photographed. When photographing the medical devices 40, 50 when the display contents change, for example, the photographing device 30 constantly performs photographing processing and changes the display contents of the medical devices 40, 50 based on sequentially acquired photographed images. Monitor whether or not this has occurred. In addition, the imaging device 30 constantly performs imaging processing, monitors whether or not the medical staff has performed any treatment on the patient based on the sequentially acquired imaging images, and when any processing is performed, the imaging device 30 performs imaging processing. It may also be configured to perform photographing processing for the devices 40 and 50. Furthermore, the imaging device 30 may be configured to perform the imaging process of the medical devices 40, 50 when receiving an instruction to perform the imaging process of the medical devices 40, 50 from the server 10 based on the stored contents of the electronic medical record DB 12a. . The imaging timing based on the stored content of the electronic medical record DB 12a may be, for example, when the prescription information stored in the electronic medical record DB 12a is updated or when new prescription information is stored.

The control unit 31 of the photographing device 30 determines whether the photographing timing has arrived (S91). When the control unit 31 determines that the photographing timing has not arrived (S91: NO), it waits until the photographing timing arrives. When the control unit 31 determines that the imaging timing has arrived (S91: YES), the camera 33 photographs the entire medical equipment 40, 50 including the display screen of the medical equipment 40, 50 (S92). The control unit 31 transmits the captured image obtained by the camera 33 from the communication unit 32 to the server 10 (S93). The control unit 31 returns to the process of step S91, performs the photographing process of the medical devices 40 and 50 each time the photographing timing arrives, and transmits the obtained photographed image to the server 10.

The control unit 11 (image acquisition unit) of the server 10 receives the photographed image transmitted by the photographing device 30, and performs the steps executed by the control unit 21 of the staff terminal 20 in the process shown in FIG. 13 on the received photographed image. Processing similar to S63 to S66 is executed (S94 to S97). That is, the control unit 11 (information acquisition unit) executes a reading process of reading the QR code from the captured images of the medical devices 40 and 50 received from the imaging device 30 (S94), and reads the device ID linked to the QR code. and read display information. Then, the control unit 11 determines whether the QR code can be read from the captured image (S95), and if it is determined that the QR code can be read (S95: YES), the device ID and display information linked to the QR code (S96). On the other hand, when the control unit 21 determines that the QR code cannot be read (S95: NO), the control unit 21 uses the learning model 22M to acquire the device ID and display information from the captured image of the second medical device 50 (S97).

Thereafter, the control unit 11 executes the same processes as steps S70 to S71 executed by the control unit 11 of the server 10 in the process shown in FIG. 14 (S98 to S99). That is, the control unit 11 stores the medication information or measurement value including the device ID and display information acquired from the photographed image in the electronic medical record DB 12a (S98). Then, the control unit 11 determines whether the medication information stored in the electronic medical record DB 12a matches the prescription information or whether the measured value is within the normal range (S99). If the control unit 11 determines that the medication information does not match the prescription information, or if it determines that the measured value is not within the normal range (S99: NO), it displays an alert screen as shown in FIG. 17A or 17B, for example. Generate and output (S100). In addition to the configuration of the screen shown in FIG. 15B, the screen shown in FIG. 17A displays patient information (for example, patient ID) of a patient whose medication information (administration rate) does not match the prescription information. In addition to the configuration of the screen shown in FIG. 15C, the shown screen displays patient information of a patient whose measured value (blood sugar level) is determined to be outside the normal range. Note that the control unit 11 may be configured to display the generated alert screen on the display unit 15 to notify (warn) the user of the server 10, or may be configured to transmit the generated alert screen to the staff terminal 20 to notify (warn) the medical staff. good.

If the control unit 11 determines that the medication information matches the prescription information or that the measured value is within the normal range (S99: YES), it skips the process of step S100 and ends the series of processes. do. Note that the control unit 11 may display on the display unit 15 or transmit to the staff terminal 20 a message notifying that the medication information matches the prescription information or that the measured value is within the normal range. Through the above-described processing, the display information of the medical devices 40 and 50 can be read based on the photographed image taken by the photographing device 30, and it can be determined whether the read display information is appropriate. Therefore, if medication is administered that does not match the prescription information, or if the patient's vital data falls outside the normal range, it is possible to detect this at an early stage and take appropriate action. In addition, when reading the display information of the medical devices 40, 50 based on the photographed image taken by the photographing device 30, the work of the medical staff visually confirming the display information of the medical devices 40, 50 or the display of the medical devices 40, 50 is performed. There is no need to take a picture of the information using the staff terminal 20. Therefore, the workload of the medical staff can be reduced, and since there is no need for the medical staff to get close to the medical devices 40, 50 and patients, it is also possible to take measures against infectious diseases.

In this embodiment, it is possible to read display information that changes over time and is displayed on the display screens of the medical devices 40 and 50 based on captured images captured by the staff terminal 20 or the imaging device 30. In addition, in the case of a medical device such as the first medical device 40 that can display a QR code that encodes the device ID and display information on the display section 45, the device ID and the Display information that changes over time can be obtained. Furthermore, in the case of a medical device that cannot display a QR code, such as the second medical device 50, by using the learning models 12M and 22M, it is possible to obtain the device ID and display information of the medical device from the captured image of the medical device. . Therefore, even if the medical devices 40 and 50 are not provided with a communication function, information displayed on the display screens of the medical devices 40 and 50 can be transmitted to the server 10. Further, even in a situation where the first medical device 40 and the second medical device 50 are used together, display information of both the first medical device 40 and the second medical device 50 can be read.

(Embodiment 2)
In the first embodiment described above, the staff terminal 20 or the photographing device 30 is configured to photograph one medical device 40, 50 and read the display information of the medical device 40, 50 from the obtained photographed image. In the present embodiment, the staff terminal 20 or the photographing device 30 performs information processing to photograph a plurality of medical devices 40, 50 in one photograph, and reads display information of each medical device 40, 50 from the obtained photographed image. Explain the system. Since the information processing system of this embodiment has the same devices as the information processing system of Embodiment 1, description of the configuration of each device will be omitted.

FIG. 18 is an explanatory diagram showing how the medical devices 40 and 50 are used in the second embodiment. As shown in FIG. 18, in a medical field, a plurality of medical devices 40, 50 may be used for one patient, and the first medical device 40 and the second medical device 50 may be used in a mixed manner. Ru. In the information processing system of this embodiment, the first medical device 40 is capable of executing a process similar to the process shown in FIG. 9 . Therefore, in the first medical device 40, a QR code generated from the device ID and display information that changes over time, which is measured by the measurement unit 43 and displayed on the display unit 45, is displayed on the display unit 45. .

Furthermore, in the information processing system of this embodiment, the staff terminal 20 and the server 10 are capable of executing processes similar to those shown in FIGS. 10 and 11. Therefore, when using the medical devices 40 and 50 for a patient, the staff terminal 20 is used to acquire the patient ID, drug ID, device ID of the medical devices 40 and 50, and display information and send them to the server 10. This allows the server 10 to associate the medical devices 40, 50 with the patient. Note that the registration process for linking the medical devices 40, 50 and patients may be performed for each medical device 40, 50, and may be performed based on captured images of multiple medical devices 40, 50 in one imaging process. may be performed.

FIG. 19 is a flowchart showing an example of a processing procedure for reading display information of the medical devices 40 and 50 according to the second embodiment, and FIGS. 20 and 21 are explanatory diagrams showing examples of screens of the staff terminal 20. The process shown in FIG. 19 is obtained by adding step S111 between steps S67 and S68 and adding steps S112 to S113 instead of steps S69 to S70 in the process shown in FIGS. 13 and 14. Description of the same steps as in FIGS. 13 and 14 will be omitted. Note that in FIG. 19, illustration of steps S71 to S75 in FIG. 14 is omitted.

The control unit 21 of the staff terminal 20 of this embodiment performs the same processing as steps S61 to S67 in FIG. 13. Accordingly, in this embodiment as well, the staff terminal 20 photographs the medical devices 40, 50 being used by a patient, and acquires the device ID and display information of the medical devices 40, 50 based on the obtained photographed images. and is displayed on the display section 25. In this embodiment, as shown in FIG. 20, for example, the control unit 21 of the staff terminal 20 displays the photographed images of the medical devices 40 and 50 taken in step S62 on the display unit 25, and displays each medical device in the photographed image. The device ID and display information of each medical device 40, 50 obtained in steps S62 to S66 are displayed in association with the devices 40, 50.

After the process in step S67, the control unit 21 determines whether or not the device ID and display information reading process has been completed for all medical devices 40, 50 in the photographed image taken in step S62 (S111). . In the present embodiment, the control unit 21 reads (scans) the information of the medical devices 40, 50 sequentially from the top of the photographed image, and when the reading process is completed to the bottom of the photographed image, all the medical devices 40, 50 It is determined that the reading process for has been completed. If it is determined that the reading process for all the medical devices 40, 50 has not been completed (S111: NO), the control unit 21 returns to the process of step S63, and reads the reading process for all the medical devices 40, 50 in the photographed image. The processes of steps S63 to S67 are executed for the medical devices 40 and 50 that have not undergone the reading process. Thereby, as shown in FIG. 20, the device ID and display information of each medical device 40, 50 can be displayed in association with a plurality of medical devices 40, 50 photographed in one photographed image. The screen shown in FIG. 20 has a send button, and the medical staff can judge whether or not the information of each medical device 40, 50 displayed on the display unit 25 is appropriate, and if it is appropriate, Operate the send button.

If it is determined that the reading process for all medical devices 40, 50 in the captured image has been completed (S111: YES), the control unit 21 determines whether the send button on the screen shown in FIG. 20 has been operated. (S68). If the control unit 21 determines that the send button has been operated (S68: YES), it sends the device ID and display information of each medical device 40, 50 displayed on the screen shown in FIG. 20 to the server 10. (S112). When the control unit 11 of the server 10 receives the device ID and display information of each medical device 40, 50 from the staff terminal 20, it transmits medication information or measured values including the device ID and display information for each medical device 40, 50. It is stored in the electronic medical record DB 12a (S113).

After that, the control unit 11 executes the processing from step S71 onwards. In addition, in step S71, the control unit 11 determines, for each medical device 40, 50, whether the medication information stored in the electronic medical record DB 12a matches the prescription information and whether the measured value is within the normal range. to decide. If the control unit 11 determines that even one medication information does not match the prescription information, or if it determines that the measured value is not within the normal range (S71: NO), it transmits an alert screen to the staff terminal 20. (S72). For example, as shown in FIG. 21, the alert screen here shows the measurement of pumps whose medication information does not match the prescription information, or the measurement of vital data whose measured values are not within the normal range, in images taken of multiple medical devices 40, 50. Display a warning message in association with the device. On the other hand, if the control unit 11 determines that all the medication information matches the prescription information and all the measured values are within the normal range (S71: YES), the control unit 11 determines that the medication information and measured values are appropriate. This is notified to the staff terminal 20 (S74).

FIG. 22 is a flowchart showing another example of the processing procedure for reading display information of the medical devices 40 and 50 according to the second embodiment. The process shown in FIG. 22 is the process shown in FIG. 16 with step S121 added between steps S98 and S99. Description of the same steps as in FIG. 16 will be omitted. Note that in FIG. 22, illustration of steps S91 to S92 in FIG. 16 is omitted.

The control section 31 of the photographing device 30 and the control section 11 of the server 10 of this embodiment perform the same processing as steps S91 to S98 in FIG. 16. Accordingly, in this embodiment as well, the server 10 acquires the device ID and display information of the medical devices 40 and 50 based on the photographed image photographed by the photographing device 30, and includes the acquired device ID and display information. Medication information or measured values are stored in the electronic medical record DB 12a. After the process in step S98, the control unit 11 determines whether or not the device ID and display information reading process has been completed for all medical devices 40 and 50 in the photographed image received from the photographing device 30 (S121 ). If it is determined that the reading process for all the medical devices 40, 50 has not been completed (S121: NO), the control unit 11 returns to the process of step S94, and reads out the reading process for all the medical devices 40, 50 in the photographed image. The processes of steps S94 to S98 are executed for the medical devices 40 and 50 that have not undergone the reading process.

If it is determined that the reading process for all medical devices 40, 50 in the photographed image has been completed (S121: YES), the control unit 11 executes the processes from step S99 onwards. In addition, in step S99, the control unit 11 determines, for each medical device 40, 50, whether the medication information stored in the electronic medical record DB 12a matches the prescription information and whether the measured value is within the normal range. to decide. Then, if the control unit 11 determines that even one of the medication information does not match the prescription information, or if it determines that the measured value is not within the normal range (S99: NO), it outputs an alert screen (S100). ). Here, the control unit 11 generates and outputs an alert screen displaying patient information (eg, patient ID, patient name, etc.) on the screen shown in FIG. 21, for example.

Through the above-described processing, in this embodiment, by photographing a plurality of medical devices 40, 50 being used by a patient in one photographing process, each of the medical devices 40, 50 displayed on the display screen of each medical device 40, 50 can be photographed. Can read information. In addition, for medical devices 40 and 50 that are pumps, it can be determined whether the read display information (medication information) matches the prescription information, and for medical devices 40 and 50 that are measuring instruments, it can be determined whether the read display information (medication information) matches the prescription information. It can be determined whether the (measured value) is within the normal range.

In this embodiment, the same effects as in the first embodiment described above can be obtained. Further, in this embodiment, the staff terminal 20 or the imaging device 30 photographs a plurality of medical devices 40, 50 in one photographing process, and displays display information of the plurality of medical devices 40, 50 based on the obtained photographed image. Can be read. Therefore, for example, when a plurality of medical devices 40, 50 are used by one patient, it is possible to efficiently read the display information of each medical device 40, 50. Further, in this embodiment as well, even in a situation where the first medical device 40 and the second medical device 50 are used together, the first medical device Display information on both device 40 and second medical device 50 can be read. Also in this embodiment, the modifications described in the above-described first embodiment can be applied.

In the first and second embodiments described above, the configuration for reading medication information or measurement values displayed on the display screens of the medical devices 40 and 50 has been described. In addition, for example, the first medical device 40 may be configured to detect the operating state of its own device and display a QR code containing the detected operating information on the display unit 45. For example, when the first medical device 40 is in a power off state, a QR code containing information indicating the power off state may be displayed on the display unit 45, and when an error related to the operation of the first medical device is detected, , a QR code including the type of error may be displayed. In this case, the server 10 can early detect the operating state of the first medical device 40 or the occurrence of an error based on the photographed image taken by the staff terminal 20 or the photographing device 30. Note that if the second medical device 50 is configured to notify the operating state by lighting or blinking the lamp, for example, the lighting or blinking state of the lamp can be detected from the photographed image of the second medical device 50. , the operating state of the second medical device 50 or the occurrence of an error can be detected based on the photographed image. By being able to grasp the operating status of the medical devices 40, 50, it is possible to know that a new medical device 40, 50 has been added to the patient (start of use), and that the medical device 40, 50 has been removed (end of use). can also be detected.

Items described in each embodiment can be combined with each other. Moreover, the independent claims and dependent claims recited in the claims may be combined with each other in any and all combinations, regardless of the form in which they are cited. Furthermore, although the claims are written in a format that refers to two or more other claims (multi-claim format), the invention is not limited to this format. It may also be written using a multi-claim format that cites at least one multi-claim.

The embodiments disclosed herein are illustrative in all respects and should be considered not to be restrictive. The scope of the present invention is indicated by the claims rather than the above-mentioned meaning, and is intended to include meanings equivalent to the claims and all changes within the scope.

10 Server 11 Control unit 12 Storage unit 13 Communication unit 20 Staff terminal 21 Control unit 22 Storage unit 23 Communication unit 26 Camera 30 Imaging device 32 Communication unit 33 Camera 40 First medical device 43 Measurement unit 45 Display unit 50 Second medical device 53 Measurement section 55 Display section 12a Electronic medical record DB
12M learning model 22M learning model

Claims (12)

Obtain an image of the display of a medical device that displays measurement information that changes over time.
Obtaining the measurement information based on the obtained photographed image,
A program that causes a computer to associate acquired measurement information with patient information and store it in the storage unit. The display unit displays device information regarding the medical device and a code linked to the measurement information,
The program according to claim 1, which causes the computer to execute a process of acquiring device information regarding the medical device and the measurement information based on a code in the acquired photographed image. The photographed image is an image of the entire medical device including the display section,
When a photographed image of the entire medical device including the display section of the medical device is input, the acquired photographed image is input to a learning model that outputs device information regarding the medical device in the photographed image, The program according to claim 1 or 2, which causes the computer to execute a process of acquiring device information regarding a medical device in an image. The photographed image is a photographed image of display sections of a plurality of the medical devices taken at once,
Based on the captured image, obtain device information regarding each medical device in the captured image;
The program according to claim 1 or 2, which causes the computer to execute a process of acquiring the measurement information based on the captured image for each of the medical devices. When the photographing area of each medical device in the photographed image includes a code linked to the device information regarding the medical device and the measurement information, the code in the photographing area of the medical device in the photographed image includes acquiring device information and measurement information regarding the medical device based on the
If the photographing area of each medical device in the photographed image does not include the device information regarding the medical device and the code linked to the measurement information, the photographed image of the entire medical device including the display section of the medical device A process of inputting the photographing area of the medical device in the acquired photographic image to a learning model that outputs device information regarding the medical device in the photographic image when the input is input, and acquiring device information regarding the medical device. The program according to claim 4, which causes the computer to execute. obtaining electronic medical record data of the patient;
The program according to claim 1 or 2, which causes the computer to execute a process of determining whether the acquired measurement information is compatible with information included in the electronic medical record data. obtaining identification information assigned to the patient;
The program according to claim 1 or 2, which causes the computer to execute a process of storing measurement information acquired based on the captured image in association with acquired identification information. The program according to claim 1 or 2, wherein the medical device includes a pump that administers an infusion into a patient's body, and a measuring instrument that measures biological information of the patient. Obtaining training data including a photographed image of the entire medical device including a display section of the medical device and device information regarding the medical device;
A model generation method in which a computer executes a process of generating a learning model that outputs device information regarding a medical device in a captured image when a captured image of the entire medical device is input using acquired training data. In a medical device equipped with a display section that displays measurement information that changes over time,
a generation unit that generates a code linked to device information regarding the medical device and measurement information displayed on the display unit;
A display control unit that displays the generated code on the display unit. Obtain an image of the display of a medical device that displays measurement information that changes over time.
Obtaining the measurement information based on the obtained photographed image,
An information processing method in which a computer stores acquired measurement information in a storage unit in association with patient information. an image acquisition unit that acquires a captured image of a display unit of a medical device that displays measurement information that changes over time;
an information acquisition unit that acquires the measurement information based on the acquired captured image;
An information processing device comprising: a storage unit that stores acquired measurement information in association with patient information.

Images