JP2024021900A - Intervention detection apparatus, intervention detection method, intervention detection program, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an intervention detection apparatus, an intervention detection method, an intervention detection program, and an information processing system for analyzing biological information with higher accuracy.

SOLUTION: In an information processing system 1 in which an intervention detection apparatus 100, a camera 140, and an analysis apparatus 150 are connected via a network N, the intervention detection apparatus 100 comprises: an acquisition unit 110 for acquiring an image obtained by capturing a space in which a subject is accommodated, wherein the image is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image; and an intervention determination unit 114 for inputting the image into a pre-trained intervention determination model and determining whether or not there is intervention in the space in the image.

EFFECT: The biological information can be analyzed with higher accuracy by inputting the image into the intervention determination model to determine whether or not there is intervention and excluding and analyzing images that are likely to indicate intervention.

SELECTED DRAWING: Figure 3

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present disclosure relates to an intervention detection device, an intervention detection method, an intervention detection program, and an information processing system.

There is a technology related to detecting objects using thermal images.

For example, there is a technology related to a human detection device that can improve human detection accuracy compared to the conventional technology by using a thermal image captured in a target space (see Patent Document 1).

Japanese Patent Application Publication No. 2021-033516

Here, for example, in a medical field where an incubator is used, there is a need to monitor and analyze biological information of a child housed in an incubator and utilize the information in the medical field. As a means to meet such needs, it is assumed that, for example, an image such as a thermal image of a child housed in an incubator is photographed and the photographed image is used to analyze the biological information of the child.

However, in actual medical settings, there may be cases in which it is not possible to photograph a baby housed in an incubator alone. For example, there may be cases where a part of the body of a medical worker such as a doctor or nurse or the child's family member (hereinafter referred to as "medical worker, etc.") is shown together with the child in order to care for the child. There are cases where parts of medical equipment such as surgical robots (excluding respiratory equipment such as breathing tubes, feeding tubes, etc.) are shown together with the child.

In these cases, a part of the body of a medical worker or a part of a medical device such as a surgical robot appears in the image in such a way that it overlaps with the child. It is expected that it will be difficult to analyze the child's biological information with high accuracy. This problem is thought to occur not only in medical settings where incubators are used, but also in situations where biological information of adults, etc. is analyzed in operating rooms or hospital rooms, for example.

An object of the present disclosure is to provide an intervention detection device, an intervention detection method, an intervention detection program, and an information processing system that make it possible to analyze biological information with higher accuracy.

An intervention detection device according to a first aspect of the present disclosure captures an image of a space in which a subject is accommodated, the image being selected from at least one of a thermal image, a visible image, a distance image, and an infrared image. The image forming apparatus includes an acquisition unit that acquires the image, and an intervention determination unit that inputs the image to an intervention determination model learned in advance and determines whether or not there is an intervention in the space in the image. This makes it possible to analyze biological information with higher accuracy.

Further, the intervention detection device according to a second aspect is the intervention detection device according to the first aspect, in which the acquisition unit acquires a plurality of the images in chronological order, and a plurality of images acquired in the chronological order. The apparatus further includes a difference calculation section that calculates a difference image based on each image, and the intervention determination section inputs the image and the difference image into the intervention determination model, and determines whether or not the intervention is necessary. This makes it possible to analyze biological information with even higher accuracy.

Further, the intervention detection device according to a third aspect is the intervention detection device according to the first or second aspect, wherein the intervention determination model is a difference image between the images for learning and the images acquired in chronological order. is input, and a predetermined machine learning method is used to output a determination result indicating the presence or absence of the intervention. This makes it possible to analyze biological information with higher precision and with relative ease.

In addition, in the intervention determination method according to the fourth aspect of the present disclosure, the processor uses at least one of a thermal image, a visible image, a distance image, and an infrared image, the image taken of the space in which the subject is accommodated. A computer is caused to execute a process of acquiring a selected image, and a process of inputting the image into an intervention determination model learned in advance and determining whether or not there is an intervention in the space in the image.

Further, in the intervention determination program according to the fifth aspect of the present disclosure, the processor selects an image taken of a space in which a subject is accommodated, from at least one of a thermal image, a visible image, a distance image, and an infrared image. A computer is caused to execute a process of acquiring a selected image, and a process of inputting the image into an intervention determination model learned in advance and determining whether or not there is an intervention in the space in the image.

Further, in the information processing system according to the sixth aspect of the present disclosure, the image taken of the space in which the subject is accommodated is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image. The intervention detection device includes an acquisition unit that acquires an image, and an intervention determination unit that inputs the image to an intervention determination model learned in advance and determines whether or not there is an intervention in the space in the image.

Further, an information processing system according to a seventh aspect of the present disclosure is the information processing system according to the sixth aspect, wherein the acquisition unit acquires the plurality of images in chronological order, and the intervention detection device acquires the plurality of images in chronological order. The intervention determination unit further includes a difference calculation unit that calculates a difference image based on each of the plurality of images acquired in the above, and the intervention determination unit inputs the image and the difference image to the intervention determination model, and The presence or absence of an intervention is determined, the intervention detection device outputs the image and the determination result of the presence or absence of the intervention to an analysis device, and the analysis device detects the intervention based on the image and the determination result of the presence or absence of the intervention. The biological information of the subject is analyzed using at least the image without the intervention, using the determination result of the presence or absence of the intervention as a label. Thereby, highly accurate analysis of biological information can be performed using the determination result of the presence or absence of intervention.

Further, an information processing system according to an eighth aspect of the present disclosure is an information processing system according to a seventh aspect, in which the information processing system according to the seventh aspect excludes the image determined to involve intervention from among the plurality of images, and Analyze biological information. Thereby, images with intervention can be excluded using the determination result of presence or absence of intervention, and biometric information can be analyzed with high accuracy.

According to the intervention detection device, intervention detection method, intervention detection program, and information processing system of the present disclosure, it is possible to obtain the effect that biological information can be analyzed with higher precision.

FIG. 3 is a diagram illustrating an example of the presence or absence of intervention in an image. FIG. 2 is a block diagram showing the hardware configuration of the intervention detection device. FIG. 1 is a block diagram showing the configuration of an information processing system including an intervention detection device. FIG. 2 is an image diagram of input and output of an intervention determination model. This is an example comparing a case without intervention and a case with intervention regarding core body temperature estimation. It is a flowchart which shows the flow of intervention detection processing by an intervention detection device.

An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In addition, the same reference numerals are given to the same or equivalent components and parts in each drawing. Furthermore, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

An overview of an embodiment of the present disclosure will be described. In this embodiment, the space in which the subject is accommodated is an infant storage room of an incubator (the space above the bed bed), and a thermal image is taken with the subject as a child. The format method is not limited to this. The method of this embodiment assumes that a person different from the subject or another object is reflected in the space, and can be applied to cases where various spaces are photographed. Furthermore, it is sufficient that the space in which the subject is accommodated is photographed, and it is possible that an image may be photographed in which the reflection of the subject cannot be confirmed. Further, when the target person is assumed to be someone other than a child, for example, the present invention can be applied to cases where the space is an operating room and the target person is a patient, or when the space is a work site and the target person is a worker. Moreover, the image to be photographed is not limited to a thermal image, but may be a visible image, a distance image, or an infrared image. A distance image is photographed by a distance image camera such as a TOF camera, and includes distance information between the distance image camera and the object to be photographed for each pixel. Note that the method of selecting and acquiring one image from these images may be, for example, a method of capturing one image (for example, a thermal image) using one camera and acquiring the image, or a method of capturing one image (for example, a thermal image) using one camera, or You may select and acquire from a plurality of images. When selecting and acquiring multiple images of different types, for example, multiple images (for example, two images, a thermal image and a visible image) are simultaneously captured using one or multiple cameras, and one of the captured images is One image (for example, a thermal image) may be selectively acquired. In addition, multiple images (for example, a thermal image and a visible image) are simultaneously captured using one or multiple cameras, and all of the captured images are acquired and input into the intervention determination model. Only one image (for example, a thermal image) selected from among them may be displayed. Further, in the following explanation, a case will be explained in which the biological information of a child to be analyzed is core body temperature, but the present invention is not limited to this, and the target may be skin temperature (peripheral temperature) or the like.

FIG. 1 is a diagram illustrating an example of the presence or absence of intervention in an image. Images (a1) and (b1) in FIG. 1 are thermal images taken of a child in an incubator, where (a1) is a thermal image without intervention, and (b1) is a thermal image with intervention. In (a1) and (b1), the child is shown lying on his back with his head on the right side of the drawing and his feet on the left side of the drawing. In addition, (a1) shows the child's entire body except for the toes, while (b1) shows the hand of a medical worker extending from the top of the image and overlapping with the child's. Half of the child's body is hidden. (a2) and (b2) are schematic images that visualize the portions where the temperature of the thermograph is equal to or higher than a predetermined temperature for each of the thermal images (a1) and (b1). If there is an intervention, half of the child's body will be hidden by the reflected hand, as shown in the range bv in (b2), and this thermal image will show that the child's body is in the area hidden by the hand of the medical worker. Not only is it impossible to identify the temperature, but the image ends up showing the temperature of an object (ie, the hand of a medical worker, etc.) rather than the temperature of the child in the hidden area. In order to accurately measure the baby's body temperature, each part of the baby can be an important area, but if the temperature of these parts cannot be identified from the captured thermal image, it will be difficult to analyze information such as biological information. Accuracy is compromised. In other words, if there is an intervention as shown in (b1) and (b2), the reflected object obstructs identification of the baby's body temperature, making it difficult to analyze the baby's biological information with high accuracy.

By the way, when trying to learn an analytical model for analyzing biological information such as a child's core body temperature, if images with intervention are used, both learning and evaluation will be affected, and the accuracy of the analytical model will decrease. There is a concern that this may occur. Therefore, in this embodiment, a learned intervention determination model is used to realize a method of determining whether or not there is intervention in a space in an image. This makes it possible to analyze the child's biological information with higher precision and with relative ease. In addition, intervention in space refers to objects that are different from the target person (parts of the body of medical workers, etc., parts of medical equipment such as surgical robots (excluding breathing equipment such as breathing tubes, feeding tubes, etc.), etc.) means entering the space.

FIG. 2 is a block diagram showing the hardware configuration of the intervention detection device 100.

As shown in FIG. 2, the intervention detection device 100 includes a CPU (Central Processing Unit) 11 which is a processor, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, and a display unit 16. and a communication interface (I/F) 17. Each configuration is communicably connected to each other via a bus 19.

The CPU 11 is a central processing unit that executes various programs and controls various parts. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic operations according to programs stored in the ROM 12 or the storage 14. In this embodiment, a prediction program is stored in the ROM 12 or the storage 14.

The ROM 12 stores various programs and data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is constituted by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may function as the input unit 15 by employing, for example, a touch panel system (including non-contact operation systems such as hover input).

The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

FIG. 3 is a block diagram showing the configuration of an information processing system 1 including the intervention detection device 100 of this embodiment. In the information processing system 1, an intervention detection device 100, a camera 140, and an analysis device 150 are connected via a network N. Although each device is connected wirelessly, for example, it may be connected by wire.

The intervention detection device 100 performs intervention determination by inputting a thermal image into an intervention determination model that has been learned in advance, and determines the presence or absence of intervention. The camera 140 takes a thermal image of the child storage room in the incubator. The analysis device 150 inputs the thermal image into an analysis model that has been trained in advance so as to be able to output biological information, using the presence or absence of intervention determined for the thermal image as a flag, and analyzes the infant held in the infant storage room. Analyze biological information. Note that the analysis device 150 may acquire a thermal image from the camera 140 or may acquire a thermal image from the intervention detection device 100. Further, the hardware configuration of the analysis device 150 may be the same as that of the intervention detection device 100.

Each functional configuration of the intervention detection device 100 will be explained. Each functional configuration is realized by the CPU 11 reading out a learning program stored in the ROM 12 or the storage 14, loading it into the RAM 13, and executing it. As shown in FIG. 3, the intervention detection device 100 is functionally configured to include an acquisition section 110, a difference calculation section 112, an intervention determination section 114, an image storage section 120, and a model storage section 122. .

The acquisition unit 110 acquires a plurality of thermal images in time series from the camera 140 and stores them in the image storage unit 120. In addition, although the following explanation explains the aspect which uses the thermal image acquired chronologically, it is not limited to the case where thermal images are acquired chronologically, but it is also possible to acquire one thermal image and perform processing.

The image storage unit 120 stores thermal images acquired in chronological order. In the process described below, it is assumed that the process is performed with reference to the thermal image in the image storage unit 120, so the description of the image storage unit 120 will be omitted in the description.

The difference calculation unit 112 calculates a difference image based on each of the plurality of thermal images acquired in time series. The aspect of the difference image will be explained. For example, assuming that the thermal images are A1 to An, the difference calculation unit 112 calculates a difference image by pairing the thermal image of interest and the next (subsequent) thermal image in time series. For example, if the thermal image of interest is A1, then B1, which is a difference image, is calculated for the set A1-A2 combined with A2, which is the next thermal image in time series. For example, when targeting thermal images A1 to A5, sets of A1-A2, A2-A3, A3-A4, and A4-A5 are obtained, and difference images of B1 to B4 are calculated. In addition, difference images are calculated for the next and subsequent pairs in the time series, that is, not only for the next pair, but also for the pairs that are two pairs ahead (for example, A1-A3 pair), three pairs ahead (for example, A1-A4 pair), etc. Good too. What set of difference images should be used to calculate the difference images may be set as appropriate depending on the specifications of the set of difference images used when learning the intervention determination model.

The model storage unit 122 stores intervention determination models learned in advance. FIG. 4 is an image diagram of the input and output of the intervention determination model. The intervention determination model uses, for example, ResNet-18 with two input systems as the network configuration of the neural network. A thermal image and a differential image are each input using two input systems. The number of input images is, for example, five for thermal images and four for difference images. As preprocessing, a thermal image for learning and a difference image for learning are each standardized and input. In learning the intervention determination model, a correct label indicating the presence or absence of intervention may be manually added to the learning thermal image. The learning specifications are integration using an arbitrary activation function (Linear Unit), Flip for data augmentation, and NLL (Negative Log Likelihood) + Center loss + Flooding for the loss function. Note that using a neural network as an intervention determination model is one example, and any machine learning method that can output binary values for determining the presence or absence of intervention from an image and a difference image can be applied. Alternatively, the network may be one system, and the intervention determination model may be learned using only thermal images without using difference images (in this case, the difference calculation unit 112 is not required). In this way, the intervention determination model inputs a learning image and a differential image of the image acquired in chronological order, uses a predetermined machine learning method, and outputs a determination result indicating the presence or absence of intervention. This is how it is learned.

The intervention determination unit 114 inputs the thermal image and the difference image into an intervention determination model, and determines the presence or absence of intervention. The determination result of the presence or absence of intervention determined for each thermal image is output to the analysis device 150. The presence or absence of intervention can be detected based on the determination result of presence or absence of intervention.

The analysis device 150 uses the determination result of the presence or absence of intervention as a label, and analyzes the child's biological information using a thermal image. For example, the analysis device 150 excludes thermal images determined to require intervention based on the label, and uses the excluded thermal images to analyze the biological information of the child. Furthermore, the analysis device 150 analyzes the child's biological information using the thermal image without considering the label. In this way, by performing the analysis with and without excluding images with intervention, it is possible to compare the analysis results of biological information from the viewpoint of the presence or absence of intervention. Furthermore, for example, the time series may be divided into sections of predetermined time, and images may be excluded in units of sections. When a predetermined number or more of labels with intervention are included in a section, images of labels with intervention in the section are excluded. With this, it is possible to exclude only images that are highly likely to show intervention, taking into consideration the case where the intervention detection is a false detection.

An example of analysis of biological information is estimation of core body temperature. FIG. 5 is an example comparing a case without intervention and a case with intervention regarding core body temperature estimation. In FIG. 5, the vertical axis represents temperature (core body temperature), and the horizontal axis represents time in chronological order. In the case of intervention, images with intervention were excluded and input into the analysis model for analysis. "Estimate" shown by the broken line in Figure 5 is the result of body temperature calculated by simple image processing of thermal images, and "AI" shown by the light solid line is the analysis result of the estimated body temperature, shown by the solid black line. "Temperature" is the measured body temperature. As is clear from FIG. 5, it can be seen that the variance of the estimated body temperature is smaller in the case with intervention than in the case without intervention, and the accuracy of AI is improved.

The analysis device 150 may output the estimated body temperature by segmenting the head, trunk, arms, and legs.

According to the analysis of the analyzer 150, it is also possible to analyze medical information for detecting necrotizing enterocolitis from unevenness in abdominal temperature as described in Reference 1, for example. Peripheral (foot) temperature measurements as described in Ref. 2 are also possible.
[Reference 1] Knobel RB, Guenther BD, Rice HE. Thermoregulation and thermography in neonatal physiology and disease. Biol Res Nurs. 2011;13:274-282.
[Reference 2] Knobel-Dail RB, Holditch-Davis D, Sloane R, Guenther BD, Katz LM. Body temperature in premature infants during the first week of life: exploration using infrared thermal imaging. J Therm Biol. 2017;69: 118-123.

(Processing flow)
Next, the operation of the intervention detection device 100 will be explained. FIG. 6 is a flowchart showing the flow of intervention detection processing by the intervention detection device 100. The intervention detection process is performed by the CPU 11 reading the intervention detection program from the ROM 12 or the storage 14, expanding it to the RAM 13, and executing it. The CPU 11 functions as each part of the intervention detection device 100 to execute the following processing.

In step S<b>100 , the CPU 11 acquires a plurality of thermal images in time series from the camera 140 and stores them in the image storage unit 120 .

In step S102, the CPU 11 calculates a difference image based on each of the plurality of thermal images acquired in time series.

In step S104, the CPU 11 inputs the thermal image and the difference image into the intervention determination model, and determines whether intervention is required. Thereby, the presence or absence of intervention is determined for each of the plurality of thermal images.

In step S106, the CPU 11 outputs the intervention determination results for each of the plurality of thermal images to the analysis device 150.

In the information processing system 1, the analysis device 150 receives the intervention determination result, uses the intervention determination result as a label, and analyzes the child's biological information using a thermal image.

As described above, the intervention detection device 100 according to the present embodiment detects an intervention and makes it possible to analyze biological information with higher accuracy.

Note that the technology of the present disclosure is not limited to the embodiments described above, and various modifications and applications are possible without departing from the gist of the present invention.

Furthermore, although the present specification has been described as an embodiment in which a program is installed in advance, it is also possible to provide the program by storing it in a computer-readable recording medium.

Further, although the above embodiment has been described using an incubator as an example, the present invention is not limited to this, and for example, the space in which the subject is accommodated is defined as "the space above the bed bed in the infant warmer", and The present invention can also be applied to the case of photographing a thermal image of a baby placed on a child's body.

1 Information processing system 100 Intervention detection device 110 Acquisition unit 112 Difference calculation unit 114 Intervention determination unit 120 Image storage unit 122 Model storage unit 140 Camera 150 Analysis device

Claims (8)

an acquisition unit that acquires an image of a space in which the subject is accommodated, which is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image;
an intervention determination unit that inputs the image into a pre-trained intervention determination model and determines whether or not there is an intervention in the space in the image;
an intervention detection device including; The acquisition unit acquires the plurality of images in chronological order,
further comprising a difference calculation unit that calculates a difference image based on each of the plurality of images acquired in time series,
The intervention determination unit inputs the image and the difference image into the intervention determination model, and determines the presence or absence of the intervention.
The intervention detection device according to claim 1. The intervention determination model inputs the images for learning and the difference images of the images acquired in time series, and uses a predetermined machine learning method to output a determination result indicating the presence or absence of the intervention. The intervention detection device according to claim 1, wherein the intervention detection device is trained to perform learning. The processor
A process of acquiring an image of a space in which the subject is accommodated, which is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image;
inputting the image into a pre-trained intervention determination model and determining whether or not there is an intervention in the space in the image;
An intervention detection method that causes a computer to perform The processor
A process of acquiring an image of a space in which the subject is accommodated, which is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image;
inputting the image into a pre-trained intervention determination model and determining whether or not there is an intervention in the space in the image;
An intervention detection program that causes a computer to execute. an acquisition unit that acquires an image of a space in which a subject is accommodated, which is selected from at least one of a thermal image, a visible image, a distance image, and an infrared image; and a pre-trained intervention determination model. an intervention detection device including an intervention determination unit that inputs the image into the image and determines whether or not there is an intervention in the space in the image;
An information processing system equipped with. The acquisition unit acquires a plurality of the images in chronological order,
The intervention detection device further includes a difference calculation unit that calculates a difference image based on each of the plurality of images acquired in time series,
The intervention determination unit inputs the image and the difference image into the intervention determination model, and determines the presence or absence of the intervention;
outputting the image and the determination result of the presence or absence of the intervention from the intervention detection device to an analysis device;
In the analysis device, based on the image and the determination result of the presence or absence of the intervention, the determination result of the presence or absence of the intervention is used as a label, and the biological information of the subject is analyzed using at least the image without the intervention. The information processing system according to claim 6. The information processing system according to claim 7, wherein the biological information of the subject is analyzed by excluding the image determined to involve intervention from among the plurality of images.