JP2023003917A - Estimation device, learning device, estimation system, estimation method, and program - Google Patents

Abstract

To accurately estimate a state of a watching object person.SOLUTION: An estimation device 120 includes an image acquisition section 121, an estimation section 122, and an output section 123. The image acquisition section 121 acquires a photographed image from an imaging device 200 for photographing a room of a watching object person. The estimation section 122 estimates a state of the watching object person from the photographed image acquired by the image acquisition section 121 with the use of a learnt model acquired by machine learning to estimate the state of the watching object person from the photographed image. The output section 123 outputs an estimation result by the estimation section 122.SELECTED DRAWING: Figure 4

Description

The present disclosure relates to an estimation device, a learning device, an estimation system, an estimation method, and a program.

Currently, there is known a watching system that estimates the state of a person being watched over from a captured image obtained by capturing an image of the room of the person being watched over. The person to be watched over is a resident of a care facility, an inpatient of a hospital, or the like. For example, Patent Literature 1 describes a nursing care system that processes an image captured by an infrared camera to detect an abnormal condition of a care recipient.

The care system described in Patent Literature 1 detects an abnormal state such as a state in which the care recipient is wandering, a state in which the care recipient is stationary for a long time, and the like, and when the abnormal state is detected, Outputs a detection signal that notifies of an abnormality. The nursing care system described in Patent Literature 1 performs image processing on the image captured by the infrared camera to identify the motion of the person being cared for, and determines whether or not the identified motion corresponds to an abnormal state.

JP 2003-260095 A

However, in the care system described in Patent Literature 1, it is necessary to appropriately define an action corresponding to an abnormal state and appropriately determine whether or not an action specified by image processing is an action corresponding to an abnormal state. is not likely to be easy. Therefore, in the care system described in Patent Literature 1, it is not easy to accurately determine whether or not the care recipient is in an abnormal state. Therefore, in order to appropriately support the watching over of the person being watched over, a technique for accurately estimating the state of the person being watched over is desired.

The present disclosure has been made in view of the above problems, and aims to provide an estimating device, a learning device, an estimating system, an estimating method, and a program for accurately estimating the state of a person being watched over.

In order to achieve the above object, the estimation device according to the present disclosure includes:
an image acquisition means for acquiring a captured image from an imaging device that captures an image of the room of the person being watched over;
Estimation means for estimating the state of the person being watched over from the imaged image acquired by the image acquisition means, using a trained model acquired by machine learning for estimating the state of the person being watched over from the imaged image. When,
and output means for outputting an estimation result by the estimation means.

In the present disclosure, the state of the person being watched over is estimated from the captured image using a trained model acquired by machine learning for estimating the state of the person being watched over from the captured image. Therefore, according to the present disclosure, it is possible to accurately estimate the state of the person being watched over.

Configuration diagram of a monitoring system according to Embodiment 1 Configuration diagram of a learning device according to Embodiment 1 Configuration diagram of an estimation device according to Embodiment 1 Functional configuration diagram of estimation system according to Embodiment 1 Explanatory diagram of resident state model according to Embodiment 1 A diagram showing a captured image group presumed to be in a state of falling The figure which shows the captured image group estimated to be a wandering state. The figure which shows the captured image group estimated to be a fever state. A diagram showing a captured image estimated to be in a dense state 4 is a flowchart showing learning processing executed by the learning device according to Embodiment 1; 4 is a flowchart showing estimation processing executed by the estimation device according to Embodiment 1; Functional configuration diagram of an estimation system according to Embodiment 2 Explanatory diagram of resident state model according to Embodiment 2 10 is a flowchart showing learning processing executed by the learning device according to the second embodiment; Flowchart showing estimation processing executed by an estimation device according to Embodiment 2 Functional configuration diagram of an estimation system according to Embodiment 3 Flowchart showing estimation processing executed by an estimation device according to Embodiment 3

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The same reference numerals are given to the same or corresponding parts in the drawings.

(Embodiment 1)
FIG. 1 is a diagram showing the configuration of a watching system 1000 according to Embodiment 1. As shown in FIG. The watching system 1000 is a system for watching over a person to be watched over, and is a system for assisting the administrator in watching over the person to be watched over. A person to be watched over is a person to be watched over and a person to be watched over by the administrator. The person to be watched over is a resident of a care facility, an inpatient in a hospital, a person to be cared for at home, or the like. The manager is a person who watches over the person being watched over and who cares about the health and safety of the person being watched over. Managers are staff of nursing homes, staff of hospitals, staff of monitoring service providers, and the like. The watching system 1000 includes an estimation system 100 , an imaging device 200 and a display device 300 .

The estimation system 100 estimates the state of the person being watched over based on the captured image, which is the image captured by the imaging device 200 . The estimation system 100 includes a learning device 110 that executes learning processing, an estimating device 120 that executes estimation processing, and a storage device 130 that stores trained models. The learning process is a process of generating a trained model used in the estimation process. The estimation process is a process of estimating the state of the person being watched over from the captured image using the learned model. The state of the person being watched over is roughly divided into two states, a normal state and an abnormal state. In this embodiment, the abnormal state includes a falling state, a wandering state, a fever state, and a crowded state.

A tipped state is a tipped state. The loitering state is the state of walking around inside or outside the house. The exothermic state is a state in which heat is generated, and the amount of heat generated is higher than normal. A crowded state is a state in which a plurality of people are gathered in a short distance. A normal state is a state other than an abnormal state. The estimation system 100 outputs an estimation result obtained by the estimation process to the display device 300 .

In the learning stage, the learning device 110 performs machine learning using learning data including normal captured images and abnormal captured images, and generates a trained model for estimating the state of the person being watched over from the captured images. A normally captured image is an image obtained by capturing an image of the room of the watching target when the watching target is in a normal state. The abnormal captured image is an image obtained by capturing an image of the room of the watching target when the watching target is in an abnormal state. In the present embodiment, learning device 110 generates one learned model for one person being watched over. As shown in FIG. 2 , learning device 110 includes control unit 11 , storage unit 12 , display unit 13 , operation reception unit 14 , first communication unit 15 , and second communication unit 16 .

The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), RTC (Real Time Clock), and the like. The CPU is also called a central processing unit, a central processing unit, a processor, a microprocessor, a microcomputer, a DSP (Digital Signal Processor), or the like, and functions as a central processing unit that executes processing and calculations related to control of the learning device 110 . In the control unit 11 , the CPU reads programs and data stored in the ROM, uses the RAM as a work area, and controls the learning device 110 in an integrated manner. The RTC is, for example, an integrated circuit with a timer function. Note that the CPU can specify the current date and time from the time information read from the RTC.

The storage unit 12 includes a non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), etc., and serves as a so-called auxiliary storage device. The storage unit 12 stores programs and data used by the control unit 11 to execute various processes. The storage unit 12 also stores data generated or acquired by the control unit 11 executing various processes.

The display unit 13 displays various images under the control of the control unit 11 . For example, the display unit 13 displays a screen for accepting various operations from the user. The display unit 13 includes a touch screen, a liquid crystal display, and the like. The operation accepting unit 14 accepts various operations from the user and supplies the control unit 11 with information indicating the contents of the accepted operations. The operation reception unit 14 includes a touch screen, buttons, levers, and the like.

The first communication section 15 communicates with the imaging device 200 under the control of the control section 11 . For example, the first communication unit 15 receives captured images from the imaging device 200 at a predetermined cycle. The first communication unit 15 communicates with the imaging device 200 according to a known wired communication standard or a known wireless communication standard. Well-known wired communication standards include USB (Universal Serial Bus, registered trademark), Thunderbolt (registered trademark), and the like. Well-known wireless communication standards include Wi-Fi (registered trademark), Bluetooth (registered trademark), Zigbee (registered trademark), and the like. The first communication unit 15 has a communication interface conforming to various communication standards.

The second communication section 16 communicates with the storage device 130 under the control of the control section 11 . For example, the second communication unit 16 reads a trained model from the storage device 130 and writes an updated trained model to the storage device 130 . The second communication unit 16 communicates with the storage device 130 according to a known wired communication standard or a known wireless communication standard. The second communication unit 16 has a communication interface conforming to various communication standards.

In the estimation stage, the estimation device 120 estimates the state of the person being watched over from the captured image, which is the estimation data, using the trained model. As shown in FIG. 3, the estimation device 120 includes a control unit 21, a storage unit 22, a display unit 23, an operation receiving unit 24, a first communication unit 25, a second communication unit 26, a third communication unit a portion 27;

The control unit 21 includes a CPU, ROM, RAM, RTC, and the like. The CPU is also called a central processing unit, a central processing unit, a processor, a microprocessor, a microcomputer, a DSP, or the like, and functions as a central processing unit that executes processing and calculations related to control of the estimating device 120 . In the control unit 21 , the CPU reads programs and data stored in the ROM, uses the RAM as a work area, and centrally controls the estimation device 120 . The RTC is, for example, an integrated circuit with a timer function. Note that the CPU can specify the current date and time from the time information read from the RTC.

The storage unit 22 includes a non-volatile semiconductor memory such as flash memory, EPROM (Erasable Programmable ROM), and EEPROM, and serves as a so-called auxiliary storage device. The storage unit 22 stores programs and data used by the control unit 21 to execute various processes. The storage unit 22 also stores data generated or acquired by the control unit 21 executing various processes.

The display unit 23 displays various images under the control of the control unit 21 . For example, the display unit 23 displays a screen for accepting various operations from the user. The display unit 23 includes a touch screen, a liquid crystal display, and the like. The operation accepting unit 24 accepts various operations from the user and supplies information indicating the contents of the accepted operations to the control unit 21 . The operation reception unit 24 includes a touch screen, buttons, levers, and the like.

The first communication section 25 communicates with the imaging device 200 under the control of the control section 21 . For example, the first communication unit 25 receives captured images from the imaging device 200 at a predetermined cycle. The first communication unit 25 communicates with the imaging device 200 according to a known wired communication standard or a known wireless communication standard. The first communication unit 25 has a communication interface conforming to various communication standards.

The second communication section 26 communicates with the storage device 130 under the control of the control section 21 . For example, the second communication unit 26 reads the learned model from the storage device 130 under the control of the control unit 21 . The second communication unit 26 communicates with the storage device 130 according to a known wired communication standard or a known wireless communication standard. The second communication unit 26 has a communication interface conforming to various communication standards.

The third communication section 27 communicates with the display device 300 under the control of the control section 21 . For example, the third communication unit 27 transmits information indicating the estimation result received from the control unit 21 to the display device 300 under the control of the control unit 21 . The third communication unit 27 communicates with the display device 300 according to a known wired communication standard or a known wireless communication standard. The third communication unit 27 has a communication interface conforming to various communication standards.

The storage device 130 stores the trained model generated by the learning device 110 . A trained model stored in the storage device 130 is updated by the learning device 110 . Also, the trained model stored in the storage device 130 is used by the estimation device 120 . The storage device 130 appropriately stores captured images generated by the imaging device 200 . The storage device 130 includes an HHD (Hard Disk Drive), an SSD (Solid State Drive), and the like.

The imaging device 200 captures an image of the surroundings of the person being watched over and generates a captured image. The imaging device 200 is basically installed so that the resident 210 is within the imaging range when the resident 210 who is the watching target is in the living room 250 . The imaging device 200 is installed, for example, on the ceiling of the living room 250 where the resident 210 lives. In this embodiment, imaging device 200 is an infrared camera. Therefore, the captured image generated by the imaging device 200 is an infrared image obtained by two-dimensionally visualizing the infrared rays emitted from the object. The imaging device 200 includes an infrared image sensor in which imaging elements that detect infrared rays are arranged two-dimensionally. An infrared image is also called a thermal image.

A captured image generated by the imaging device 200 may be a still image or a moving image. In the present embodiment, the captured image generated by the imaging device 200 is a still image, and the imaging device 200 captures the living room 250 at a predetermined cycle to generate the captured image. The imaging device 200 transmits the generated captured image to the estimation system 100 . The captured image is used as learning data used by the learning device 110 in learning processing. The captured image is also used as estimation data used by the estimation device 120 in estimation processing. The imaging device 200 is connected with the estimation system 100 by a well-known communication standard.

The display device 300 displays the estimation result by the estimation system 100. FIG. The display device 300 receives information indicating an estimation result from the estimation system 100 and displays an image indicating the estimation result indicated by the received information. Hereinafter, information indicating an estimation result is simply referred to as an estimation result as appropriate. The estimation results are roughly divided into an estimation result that the state of resident 210 is normal and an estimation result that the state of resident 210 is abnormal. The estimation result that the state of the resident 210 is abnormal is the estimation result that the state of the resident 210 is the overturned state, the estimation result that the state of the resident 210 is the wandering state, and the state of the resident 210. includes at least one of an estimation result that the resident 210 is in a fever state and an estimation result that the state of the occupants 210 is in a crowded state.

The display device 300 is placed in a location where the administrator 310 can see the screen of the display device 300 . For example, the display device 300 is placed in an administration room 350 where an administrator 310 is present. The display device 300 may display the captured image in addition to the image showing the estimation result. When the display device 300 informs the administrator 310 that the state of the resident 210 is abnormal, the administrator 310 can promptly take measures according to the abnormal state. Display device 300 is connected to estimation system 100 by a well-known communication standard. The display device 300 includes, for example, a control unit (not shown), a storage unit (not shown), a display unit (not shown), an operation reception unit (not shown), and a communication unit (not shown). ).

Next, with reference to FIG. 4, functions of the estimation system 100 will be described. The learning device 110 functionally includes an image acquisition unit 111 , a model generation unit 112 , and a label reception unit 113 . The estimation device 120 functionally includes an image acquisition unit 121 , an estimation unit 122 , and an output unit 123 . Each of these functions is implemented by software, firmware, or a combination of software and firmware. Software and firmware are written as programs and stored in the ROM or storage units 12 and 22 . These functions are realized by the CPU executing programs stored in the ROM or the storage units 12 and 22 .

The image acquisition unit 111 acquires the normal captured image and the abnormal captured image from the imaging device 200 or the storage device 130 . The normal captured image and the abnormal captured image acquired by the image acquisition unit 111 are used as learning data for machine learning. A captured image used in one learning may be a single captured image or a group of a series of captured images. A series of captured images is a plurality of captured images arranged in time series. be. In this embodiment, the captured images used in one learning are a series of captured images. The image acquisition unit 111 acquires a large number of a series of normal captured images and a large number of a series of abnormal captured images as learning data.

A series of normal captured images is a captured image group obtained by capturing images of the watching target person's room at regular time intervals when the watching target person is in a normal state. A series of abnormal captured image groups are captured image groups obtained by capturing images of the watching target person's room at regular time intervals when the watching target person is in an abnormal state. The abnormal captured image group includes a falling captured image group, a wandering captured image group, a fever captured image group, and a densely captured image group. The fall captured image group is a captured image group obtained when the person being watched over is in a fall state. The wandering captured image group is a captured image group obtained when the person being watched over falls. The fever captured image group is a captured image group obtained when the person being watched over has a fever. A crowd-captured image group is a captured-image group obtained when the persons being watched over are in a crowded state.

The control unit 11 functioning as the image acquisition unit 111 acquires a series of captured images from the imaging device 200 via the first communication unit 15 . Alternatively, the control unit 11 functioning as the image acquisition unit 111 acquires a series of captured images from the storage device 130 via the second communication unit 16 . This series of captured image groups is either a normal captured image group or an abnormal captured image group. The image acquisition unit 111 is an example of an image acquisition unit and a first image acquisition unit included in the learning device.

The model generating unit 112 generates a learned model for estimating the state of the person being watched over from the captured images by machine learning using the normal captured images and the abnormal captured images obtained by the image obtaining unit 111 . This trained model determines whether or not the person being watched over has fallen, whether or not the person being watched over is wandering, whether the person being watched over has a fever, and whether the person being watched over is crowded, based on the captured image. This is a model for estimating whether or not Various algorithms can be adopted as the learning algorithm used by the model generation unit 112 .

In this embodiment, an example in which k-means, which is unsupervised learning, is adopted as a learning algorithm will be described. Unsupervised learning is a method of learning the features inherent in the learning data by giving the learning data that does not contain the label, which is the result to be output, to the learning device. The k-means method is a non-hierarchical clustering algorithm, and is a method of classifying a given number of clusters into k clusters using the average of clusters. The flow of processing of the k-means method will be described below.

First, clusters are randomly assigned to each data xi. Next, each Vj that is the center of each cluster is calculated based on the allocated data. Next, find the distance between each xi and each Vj and reassign xi to the closest central cluster. Then, if the allocation of all xi clusters does not change in the above-described process, or if the amount of change falls below a preset threshold, it is determined that convergence has occurred, and the process ends.

In the present embodiment, the model generation unit 112 learns the state of the resident by unsupervised learning using learning data including a large number of normal captured image groups and abnormal captured image groups. Here, labeling for each cluster is done by a human user. This labeling assigns to each cluster whether the resident's status is normal or not, and if the resident's status is abnormal, what abnormal status the resident is. labeling for

The model generation unit 112 classifies the captured image group acquired by the image acquisition unit 111 into one of a plurality of groups by clustering. Note that the clusters described above correspond to groups. The model generating unit 112 labels each of the plurality of groups according to instructions from the user. The model generation unit 112 basically updates an existing trained model using a series of newly acquired captured images. In other words, the model generation unit 112 generates a new trained model from the series of newly acquired captured images and the trained model stored in the storage device 130 . The model generation unit 112 stores the generated new trained model in the storage device 130 . The model generation unit 112 is an example of model generation means.

The label accepting unit 113 accepts a label from the user. As described above, in this embodiment machine learning is unsupervised data. Therefore, the user performs labeling for each of the classified groups. The label accepting unit 113 accepts an operation associated with labeling from the user. By this labeling, each of the plurality of groups is set as a group indicating normal state or a group indicating abnormal state. The group indicating the abnormal state is any one of the group indicating the overturned state, the group indicating the wandering state, the group indicating the fever state, and the group indicating the crowded state. Labeling completes the trained model.

The image acquisition unit 121 acquires a captured image from the imaging device 200 that captures the room of the person being watched over. The captured image that the image acquisition unit 121 acquires from the imaging device 200 is estimation data used for estimating the state of the person being watched over. The estimation data may be a single captured image or a group of captured images. In this embodiment, the estimation data is a group of captured images. That is, the control unit 21 functioning as the image acquisition unit 121 acquires a series of captured images from the imaging device 200 via the first communication unit 25 . This series of captured image groups is either a normal captured image group or an abnormal captured image group. The image acquisition unit 121 is an example of an image acquisition unit included in the estimation device and a second image acquisition unit. The captured image acquired by the image acquisition unit 121 is an example of a second captured image.

The estimating unit 122 estimates the state of the person being watched over from the captured image acquired by the image acquiring unit 121 using a learned model acquired by machine learning for estimating the state of the person being watched over from the captured image. This trained model is generated by the model generation unit 112 and stored in the storage device 130 . The estimating unit 122 estimates whether the state of the person being watched over is normal or abnormal, based on the learned model and the acquired series of captured images. The estimation unit 122 supplies the estimation result to the output unit 123 . The estimating unit 122 is an example of estimating means.

The output unit 123 outputs the result of estimation by the estimation unit 122 . The output destination of the output unit 123 can be adjusted as appropriate. In this embodiment, the output destination of the output unit 123 is the display device 300 . That is, the control section 21 functioning as the output section 123 transmits information indicating the estimation result to the display device 300 via the third communication section 27 . The output unit 123 is an example of output means.

Next, a trained model generated by the model generation unit 112 will be described with reference to FIG. In this embodiment, the learned model is a resident state model for estimating the state of the resident from a series of captured images. Therefore, the input of the occupant condition model is a series of captured images. Also, the output of the resident state model is the resident 210 state. The state of the resident 210 includes a normal state, a falling state, a wandering state, a fever state, a crowded state, and the like.

Next, with reference to FIGS. 6, 7, 8, and 9, a captured image group in which the state of the resident 210 is estimated to be abnormal in the resident state model will be described. In the captured image, the hatched portion corresponds to the heat source that radiates heat, and corresponds to the resident 210 . Also, in the captured image, the darkly hatched portion corresponds to the heat source that strongly radiates heat, and is the portion where the body temperature of the resident 210 is high. Note that the position and movement of the resident 210 can be identified from the position and movement of the heat source.

FIG. 6 is a diagram showing a captured image group in which it is estimated that the resident 210 is in a state of falling. When the resident 210 falls over in the living room 250, the resident 210 is considered to lie down from a standing position in a position different from the sleeping position. In particular, if the resident 210 falls due to a fainting, it is highly likely that the resident 210 will lie motionless for a long period of time. Fainting is a sudden loss of consciousness due to cerebral anemia, cerebral hemorrhage, or the like. Therefore, it is assumed that a group of captured images showing that the resident 210 stands up in a position different from the sleeping position, then lies down, and then lies down for a long period of time is a group of captured images representing the overturned state. , a resident state model is generated. That is, in the unsupervised learning, the user assigns a label representing the falling state to a group to which the captured image group representing such a falling state belongs.

The captured image group shown in FIG. 6 is a captured image group captured in the order of the image 20AA, the image 20AB, and the image 20AC. Image 20AA is a captured image captured when resident 210 is standing in a position different from the usual sleeping position. Image 20AB and image 20AC are captured images captured when resident 210 is lying in a position different from the usual sleeping position. Note that the image 20A is a captured image captured when the resident 210 is lying in a normal sleeping position. The usual sleeping position is, for example, the position where the resident 210 is detected most of the time during the night time period.

FIG. 7 is a diagram showing a captured image group in which the resident 210 is estimated to be in a wandering state. When the resident 210 wanders, the resident 210 is often absent from the living room 250 for an extended period of time after the preliminary behavior suggestive of loitering. Pre-behaviors suggestive of loitering are unusual nocturnal behaviors performed prior to loitering. An antecedent behavior suggesting wandering is, for example, a behavior of moving around in the living room 250, a behavior of going in and out of the living room 250, and the like. Therefore, as it is estimated that the group of captured images showing that the resident 210 is absent from the living room 250 for a long time after the preliminary behavior suggesting wandering is the group of captured images representing the state of wandering, the resident A state model is generated. That is, in the unsupervised learning, the user assigns a label representing the wandering state to a group to which the captured image group representing such a wandering state belongs.

The captured image group shown in FIG. 7 is a captured image group captured in the order of image 20BA, image 20BB, image 20BC, image 20BD, and image 20BE. The image 20BA, the image 20BB, and the image 20BC are captured images captured when the resident 210 is taking a preliminary action suggesting wandering, and captured when the resident 210 is moving around in the living room 250. It is a captured image. The image 20BD and the image 20BE are captured images captured when the resident 210 is away from the living room 250 .

FIG. 8 is a diagram showing a captured image group in which it is estimated that the resident 210 is in a fever state. When the resident 210 is generating heat, the state in which the amount of heat generated by the resident 210 in the captured image is greater than the usual amount of heat continues for a long period of time. Therefore, the resident state model is designed so that the group of captured images showing that the amount of heat generated by the resident 210 is greater than the usual amount of heat continues for a long period of time is assumed to be the group of captured images representing the heat generation state. generated. In other words, in the unsupervised learning, the user assigns a label representing the heat generation state to the group to which the captured image group representing such a heat generation state belongs.

The captured image group shown in FIG. 8 is a captured image group captured in the order of the image 20CA, the image 20CB, and the image 20CC. An image 20CA, an image 20CB, and an image 20CC are captured images captured when the resident 210 continues to emit strong heat radiation compared to normal heat radiation. Note that the image 20C is a captured image captured when the resident 210 is normally radiating heat. Normal thermal radiation is, for example, the thermal radiation of occupants 210 that is detected most of the time.

FIG. 9 is a diagram showing a captured image in which the state of residents 210 is estimated to be in a dense state. When a plurality of people including the resident 210 are concentrated, a plurality of heat sources are close to each other on the captured image. Therefore, a resident state model is generated so that a captured image showing that a plurality of heat sources are close to each other is estimated to be a captured image representing a dense state. That is, in unsupervised learning, the user assigns a label representing the state of congestion to a group to which captured images representing such a state of congestion belong. Note that the congestion state can be estimated from one captured image. Therefore, when estimating the congestion state, the input to the occupant state model may be one captured image. An image 20D shown in FIG. 9 is a captured image captured when a plurality of people including the resident 210 are crowded together in the living room 250 .

Next, the learning process executed by the learning device 110 will be described with reference to the flowchart of FIG.

The control unit 11 included in the learning device 110 acquires learning data (step S101). The learning data includes a large number of normal captured image groups and a large number of abnormal captured image groups. A series of captured images acquired by the imaging device 200 may be directly supplied to the learning device 110 as learning data, or may be stored in the storage device 130 as learning data and supplied to the learning device 110 as appropriate. good too. That is, the control unit 11 acquires learning data from the imaging device 200 or the storage device 130 .

After completing the process of step S101, the control unit 11 executes unsupervised learning (step S102). Specifically, the control unit 11 performs unsupervised learning using a large number of captured image groups included in the acquired learning data, and classifies the large number of captured image groups into a plurality of groups. After completing the process of step S102, the control unit 11 performs labeling (step S103). The control unit 11 assigns a label to each of the plurality of groups according to the operation received from the user by the operation receiving unit 14 . This labeling completes the trained model.

After completing the process of step S103, the control unit 11 saves the learned model (step S104). The control unit 11 stores the trained model completed by labeling in the storage device 130 . After completing the process of step S104, the control unit 11 completes the learning process.

Next, the estimation processing performed by the estimation device 120 will be described with reference to the flowchart of FIG. 11 .

The control unit 21 included in the estimation device 120 acquires a captured image group from the imaging device 200 (step S201). After completing the process of step S201, the control unit 21 inputs the acquired captured image group to the learned model (step S202). The trained model estimates the state of the resident 210 from the input group of captured images. The status of the resident 210 is either normal or abnormal. The abnormal state is any one of a falling state, a wandering state, a heat generating state, and a crowded state.

After completing the process of step S202, the control unit 21 acquires an estimation result from the learned model (step S203). After completing the process of step S203, the control unit 21 outputs the estimation result to the display device 300 (step S204). Note that the display device 300 displays an image indicating the estimation result supplied from the estimation device 120 . After completing the process of step S204, the control unit 21 completes the estimation process.

In the present embodiment, a trained model is generated by machine learning using normal captured images and abnormal captured images, and the state of the person being watched over is estimated from the captured images using the generated trained model. According to the present embodiment, since the state of the person being watched over is estimated from the trained model corresponding to the person being watched over, the state of the person being watched over can be accurately estimated.

In addition, in the present embodiment, a learned model capable of estimating a falling state, a wandering state, a fever state, and a crowded state is used to determine whether the person being watched over is in a normal state, a falling state, a wandering state, a fever state, and a crowded state. is estimated to be in any of the states of Therefore, according to the present embodiment, it is possible to accurately estimate whether the person being watched over is in any of the normal state, the overturned state, the wandering state, the fever state, and the crowded state.

(Embodiment 2)
Embodiment 1 has described an example in which a trained model is prepared for each watching target person. Embodiment 2 will explain an example in which one trained model is prepared in common for a plurality of persons being watched over. Note that descriptions of configurations and functions similar to those of the first embodiment will be omitted or simplified as appropriate.

FIG. 12 shows a functional configuration of estimation system 100 according to this embodiment. Learning device 110 according to the present embodiment functionally includes image acquisition unit 111 , model generation unit 112 , label reception unit 113 , and parameter acquisition unit 114 . Estimation apparatus 120 according to the present embodiment functionally includes image acquisition section 121 , estimation section 122 , output section 123 , and parameter acquisition section 124 .

The image acquisition unit 111 acquires a captured image obtained by capturing an image of each room of a plurality of persons being watched over. The image acquisition unit 111 acquires a series of captured images as learning data from each of the plurality of imaging devices 200 provided in the rooms of the plurality of persons being watched over. This series of captured image groups is either a normal captured image group or an abnormal captured image group.

The parameter acquisition unit 114 acquires parameters related to each of a plurality of persons to be watched over. When the image acquisition unit 111 acquires a series of captured images, the parameter acquisition unit 114 acquires parameters corresponding to the series of captured images as learning data. This parameter is a parameter that indicates the attributes of the person being watched over. The parameters are, for example, the age, gender, physical characteristics, and degree of health of the person being watched over. Physical characteristics include height, weight, normal body temperature, average heart rate, and the like. The degree of health is the need for nursing care, the presence or absence of illness, and the like.

For example, the parameter acquisition unit 114 acquires the first parameter, which is the parameter of the first watching target person, when acquiring a series of captured images from the imaging device 200 that captures images of the first watching target person's room. For example, it is assumed that the storage device 130 stores parameter information in which the identification information of the person being watched over, the identification information of the imaging device 200, and the parameters are associated with each other. In this case, the parameter acquisition unit 114 can specify and acquire parameters related to the person being watched over captured by the imaging device 200 from which the group of captured images is acquired, from the parameter information stored in the storage device 130 . .

The model generation unit 112 performs machine learning using learning data including the captured image acquired by the image acquisition unit 111 and the parameters acquired by the parameter acquisition unit 114 to estimate the state of the person being watched over from the captured image. Generate a trained model. For example, the model generation unit 112 classifies a combination of a series of captured images and parameters into one of a plurality of groups. In addition, the model generation unit 112 labels each of the plurality of groups according to instructions from the user.

The model generation unit 112 stores the trained model completed by labeling in the storage device 130 . This trained model is a model acquired by machine learning using captured images acquired for each of the plurality of watching targets and parameters for each of the plurality of watching targets. This learned model is a common learned model for a plurality of persons being watched over.

The label accepting unit 113 accepts a label from the user. The user performs labeling for each of the classified groups. It should be noted that the user can perform labeling in consideration of a series of captured images and parameters. For example, a first group to which a combination of a series of captured images showing a fever state of 37 degrees and a parameter indicating that the normal temperature is 36 degrees belongs, and a group of captured images showing a fever state of 37 degrees and the normal temperature. Assume that there is a second group to which the combination with the parameter indicating 36.5 degrees belongs. In this case, the user may label the first group with a label indicating a fever state and label the second group with a label indicating a normal state.

The image acquisition unit 121 acquires a first captured image from a first imaging device that captures an image of the room of a first person being watched over among the plurality of persons being watched over. The first imaging device is the imaging device 200 that captures the room of the first person being watched over among the plurality of imaging devices 200 . The parameter acquisition unit 124 acquires a first parameter that is a parameter related to a first person being watched over among the plurality of persons being watched over. For example, the parameter acquisition unit 124 can specify and acquire parameters related to the first person being watched over captured by the first imaging device from the parameter information stored in the storage device 130 . The parameter acquisition unit 124 is an example of parameter acquisition means.

The estimation unit 122 estimates the state of the first person being watched over from the first captured image acquired by the image acquisition unit 121 and the first parameters acquired by the parameter acquisition unit 124 using the learned model. The output unit 123 outputs the result of estimation by the estimation unit 122 .

Here, the learned model generated by the model generation unit 112 will be described with reference to FIG. 13 . In the present embodiment, the trained model is a resident state model for estimating the state of the resident from a series of captured images and parameters. Therefore, the input of the occupant condition model is a series of captured images and parameters. Also, the output of the resident state model is the resident 210 state. The state of the resident 210 includes a normal state, a falling state, a wandering state, a fever state, a crowded state, and the like.

Next, the learning process executed by the learning device 110 will be described with reference to the flowchart of FIG. 14 .

The control unit 11 included in the learning device 110 acquires learning data including a captured image group and parameters (step S301). For example, the control unit 11 acquires a series of captured images corresponding to each resident 210 from the multiple imaging devices 200 . Alternatively, the control unit 11 may acquire a series of captured image groups corresponding to each resident 210 from the storage device 130 that stores a series of captured image groups acquired by the plurality of imaging devices 200 . Also, the control unit 11 acquires parameters corresponding to each resident 210 from the storage device 130 . That is, the control unit 11 acquires learning data from the multiple imaging devices 200 and the storage device 130 .

After completing the process of step S301, the control unit 11 executes unsupervised learning (step S302). Specifically, the control unit 11 performs unsupervised learning using a large number of combinations included in the acquired learning data, and classifies the large number of combinations into a plurality of groups. This combination is a combination of captured image groups and parameters. After completing the process of step S302, the control unit 11 performs labeling (step S303). The control unit 11 assigns a label to each of the plurality of groups according to the operation received from the user by the operation receiving unit 14 . This labeling produces a trained model.

After completing the process of step S303, the control unit 11 saves the learned model (step S304). The control unit 11 stores the trained model completed by labeling in the storage device 130 . After completing the process of step S304, the control unit 11 completes the learning process.

Next, the estimation processing executed by the estimation device 120 will be described with reference to the flowchart of FIG. 15 .

The control unit 21 included in the estimation device 120 acquires the first captured image group from the first imaging device (step S401). After completing the process of step S401, the control unit 21 acquires the first parameter (step S402). The control unit 21 acquires from the storage device 130 a first parameter, which is a parameter related to the first resident who lives in the room imaged by the first imaging device. After completing the process of step S402, the control unit 21 inputs the first captured image group and the first parameter to the learned model (step S403). The trained model estimates the state of the resident 210 from the input first captured image group and first parameters.

After completing the process of step S403, the control unit 21 acquires an estimation result from the learned model (step S404). After completing the process of step S404, the control unit 21 outputs the estimation result to the display device 300 (step S405). After completing the process of step S405, the control unit 21 completes the estimation process.

In the present embodiment, a trained model obtained by machine learning using a captured image obtained by capturing an image of each room of a plurality of watching targets and parameters related to each of the plurality of watching targets is used. , the state of the first watching target is estimated from the first captured image obtained from the first imaging device that captures the room of the first watching target and the first parameter related to the first watching target. According to the present embodiment, since the state of the first person being watched over is appropriately estimated according to the first parameter from a trained model common to a plurality of persons being watched over, the state of the person being watched over can be accurately estimated. can do. That is, according to the present embodiment, even when using a trained model common to a plurality of persons being watched over, since the attributes of the persons being watched over are taken into consideration, appropriate estimation according to the attributes is possible.

(Embodiment 3)
Embodiment 1 has described an example in which the output of a trained model itself is treated as an estimation result. Embodiment 3 describes an example in which an estimation result is obtained based on the output of a trained model. Specifically, in the third embodiment, estimation materials for estimating the state of the person being watched over are obtained using the trained model, and the state of the person being watched over is estimated based on the obtained estimation materials. It should be noted that descriptions of configurations and functions similar to those of the first and second embodiments will be omitted or simplified as appropriate.

FIG. 16 shows a functional configuration of estimation system 100 according to this embodiment. Learning device 110 according to the present embodiment functionally includes image acquisition unit 111 , model generation unit 112 , and label reception unit 113 . Estimating apparatus 120 according to the present embodiment functionally includes image acquiring section 121 , estimating section 122 including state determining section 125 and state estimating section 126 , and output section 123 .

The state determination unit 125 determines whether or not the state of the person being watched over is in a suspected abnormal state from the captured image using the learned model. A suspected abnormal state is a state in which an abnormality is suspected, and is a state that can be regarded as an abnormal state. In the present embodiment, the learned model is a model for determining whether or not the state of the person being watched over is in a suspected abnormal state. When a suspected abnormal condition is determined, other conditions are taken into account to estimate whether an abnormal condition exists. On the other hand, if it is determined that it is not in a suspected abnormal state, it is estimated that it is in a normal state. In other words, the suspected abnormal state is a necessary condition for presuming an abnormal state. The state determination unit 125 is an example of state determination means.

The state estimating unit 126 estimates the state of the person being watched over based on the determination result of the state determining unit 125 . For example, when the state determination unit 125 determines that the state of the person being watched over is in a suspected abnormal state, the state estimation unit 126 estimates that the state of the person being watched over is in an abnormal state if another condition is satisfied. do. On the other hand, when the state determination unit 125 determines that the state of the person being watched over is not the suspected abnormal state, the state estimation unit 126 estimates that the state of the person being watched over is the normal state. Note that the output unit 123 outputs the result of estimation by the state estimation unit 126 . The state estimation unit 126 is an example of state estimation means.

Suspicious abnormal conditions and other conditions can be adjusted accordingly. For example, when the state determination unit 125 determines that the state of the person being watched over is in the suspected abnormal state at a rate equal to or greater than the threshold within the unit time, the state estimation unit 126 estimates that the state of the person being watched over is in the abnormal state. can do. In this case, the suspected abnormal state may be any state. In this case, another condition is that the state of the person being watched over is in the suspected abnormal state at a rate equal to or greater than the threshold within the unit time.

For example, assume a case where the person being watched over is considered to have a fever when a heat source of 37° C. or higher is detected at a rate of 80% or more in 10 minutes. In this case, the suspected abnormal state is a state of having a body temperature of 37 degrees or higher. In this case, another condition is that the state of the person being watched over is in the suspected abnormal state at a rate of 80% or more in 10 minutes. In this case, the trained model may be a model that can determine whether or not the person being watched over has a body temperature of 37 degrees or higher from the captured image. That is, in this case, the trained model does not have to be able to determine the percentage of time when the watching target person is in the suspected abnormal state.

As another example, assuming that the surroundings of the person being watched over are in a crowded state, the suspected abnormal state is a state in which a plurality of people including the person being watched over are gathered. This state includes a state in which the watching target is receiving medical treatment, medical treatment, or nursing care service from a medical worker, or a state in which the watching targets pass each other while passing each other. In this case, another condition is that the state of the person being watched over continues as a suspected abnormal state for a certain period of time (for example, 15 minutes) or more. In this case, the trained model may be a model that can determine whether or not the people being watched over are in a crowded state from the captured image.

As another example, it is assumed that the person being watched over is in a fainted state when it is detected that the person is lying still for five minutes in a position other than the sleeping position. In this case, the suspected abnormal condition is lying in a position different from the sleeping position. In this case, another condition is that the state of the person being watched over continues for five minutes and is in the suspected abnormal state. In this case, the learned model may be a model that can determine from the captured image whether or not the person being watched over is lying in a position different from the sleeping position. That is, in this case, the trained model does not have to be able to determine the length of time the watching target person is in the suspected abnormal state.

In addition, the suspected abnormal state includes the first suspected abnormal state and the second suspected abnormal state, and the state estimation unit 126 determines that the state of the person being watched over is the first suspected abnormal state. continues, if the state determination unit 125 continues to determine that the state of the person being watched over is in the second suspected abnormal state, it may be estimated that the state of the person being watched over is in the abnormal state.

For example, assume a case where the watching target person is assumed to be in a wandering state when the watching target person's absence continues for 10 minutes or more after the preliminary behavior suggesting wandering continues for 10 minutes or more. In this case, the suspected abnormal state includes a first suspected abnormal state in which the subject performs an advance action suggesting loitering, and a second suspected abnormal state in which the user is absent from the living room 250 . In this case, another condition is that the second suspected abnormal state continues for 10 minutes or longer after the first suspected abnormal state continues for 10 minutes or longer. In this case, the trained model may be a model that can determine from the captured image whether or not there is a prior behavior suggesting wandering, and from the captured image whether or not the person to be watched over is absent. In other words, in this case, the trained model may not be able to distinguish the duration of the first suspected abnormal state and the second suspected abnormal state, the order of the first suspected abnormal state and the second suspected abnormal state, etc. .

As another example, after the state of low-grade fever continues for 20 minutes or more, when the person being watched over remains motionless for 5 minutes or more, it is assumed that the person being watched over has fallen down due to poor physical condition. Suppose. In this case, the suspected abnormal state includes a first suspected abnormal state suggesting a low-grade fever due to poor physical condition and a second suspected abnormal state suggesting that the patient's physical condition deteriorates and cannot move in the living room 250 and aid is required. In this case, another condition is that the second suspected abnormal state continues for 5 minutes or longer after the first suspected abnormal state continues for 20 minutes or longer. In this case, the trained model can determine whether there is a first suspected abnormal state suggesting a slight fever from the captured image, and whether there is a second suspected abnormal state from the captured image where the person being watched over falls and does not move. model. In other words, in this case, the trained model may not be able to distinguish the duration of the first suspected abnormal state and the second suspected abnormal state, the order of the first suspected abnormal state and the second suspected abnormal state, etc. .

The first suspected abnormal state is the state in which the person being watched over is standing, and the second suspected abnormal state is the state in which the person being watched over is lying in a position different from the sleeping position. For example, after the first suspected abnormal state continues for 20 minutes or longer , when the second suspected abnormal state continues for two minutes or longer, the person being watched over is assumed to be in a state of collapse. In this case, the suspected abnormal state includes a first suspected abnormal state suggesting a standing state and a second suspected abnormal state suggesting a state of not moving in a position different from the normal sleeping position. In this case, another condition is that the second suspected abnormal state continues for 2 minutes or longer after the first suspected abnormal state continues for 20 minutes or longer. In this case, the trained model determines whether or not there is a first quasi-abnormal state suggesting a standing position from the captured image, and a second quasi-abnormal state in which the person being watched over lies in a position different from the sleeping position and does not move, based on the captured image. Any model can be used as long as it can determine whether or not there is In other words, in this case, the trained model may not be able to distinguish the duration of the first suspected abnormal state and the second suspected abnormal state, the order of the first suspected abnormal state and the second suspected abnormal state, etc. .

Further, when the state determination unit 125 determines that the state of the person being watched over is in the suspected abnormal state in the first time period, the state estimation unit 126 estimates that the state of the person being watched over is in the abnormal state, and determines the state of the person being watched over. When the unit 125 determines that the state of the person being watched over is in the suspected abnormal state in the second time period different from the first time period, it may be estimated that the state of the person being watched over is in the normal state.

For example, it is assumed that the person being watched over is in a wandering state when the person being watched over has left the living room 250 after moving around in the living room 250 at night. In this case, the suspected abnormal state is a state in which the person has left the living room 250 after moving around in the living room 250 . In this case, the other condition is that it is nighttime. In this case, the learned model may be any model that can determine from the captured image that the person being watched over has left the living room 250 after moving around in the living room 250 . That is, in this case, the trained model does not have to be able to determine the time period during which the person being watched over is in the suspected abnormal state.

Also, if the person being watched over is in a fever condition during a time slot other than the meal time slot, it may be assumed that the person being watched over is in a fever condition. In this case, the suspected abnormal state is a state in which the person being watched over has a fever. During a meal, it is difficult to determine whether the person being watched over has a fever due to eating a hot meal. In this case, the trained model may be a model that can determine whether or not the person being watched over is in a fevered state from the captured image. That is, in this case, the trained model does not have to be able to determine the time period during which the person being watched over is in the suspected abnormal state.

As another example, if the person being watched over is lying down and does not move after standing in a time period other than the sleeping time period, it may be assumed that the person being watched over is in a fainted state. In this case, the suspected abnormal state is a state in which the person being watched over lies down from the standing state and stands still. In this case, another condition is the condition that it is daytime. In this case, the trained model may be a model that can determine from the captured image that the person being watched over will be in a lying down state without going through a sitting state after being in a standing state. That is, in this case, the trained model does not have to be able to determine the time period during which the person being watched over is in the suspected abnormal state.

Next, the estimation processing executed by the estimation device 120 will be described with reference to the flowchart of FIG. 17 .

The control unit 21 included in the estimation device 120 acquires a captured image group from the imaging device 200 (step S501). After completing the process of step S501, the control unit 21 inputs the captured image group to the trained model (step S502). The learned model determines the state of the resident 210 from the input captured image group.

After completing the process of step S502, the control unit 21 acquires the discrimination result from the learned model (step S503). This determination result is a determination result as to whether or not the state of the resident 210 is a suspected abnormal state. After completing the process of step S503, the control unit 21 determines whether or not the state of the resident 210 is a suspected abnormal state (step S504). When the control unit 21 determines that the state of the resident 210 is not the suspected abnormal state (step S504: NO), the control unit 21 estimates that the state of the resident 210 is normal (step S505). When the control unit 21 determines that the state of the resident 210 is the suspected abnormal state (step S504: YES), the control unit 21 estimates the state of the resident 210 based on other conditions (step S506).

For example, when the suspected abnormal state is a state of having a body temperature equal to or higher than the threshold, the control unit 21 determines whether the body temperature of the resident 210 is equal to or higher than the threshold at a rate equal to or higher than the threshold within a unit time. In the case where the body temperature of the resident 210 is 37 degrees or higher at the present time, the control unit 21 determines whether the resident 210 has a fever when the body temperature of the resident 210 has been 37 degrees or higher at a rate of 80% or more in the past 10 minutes. state. When the body temperature of the resident 210 is 37 degrees or more at the present time, the control unit 21 determines that the resident 210 is normal when the body temperature of the resident 210 is not 37 degrees or more at a rate of 80% or more in the past 10 minutes. state.

After completing the process of step S505 or step S506, the control unit 21 outputs the estimation result to the display device 300 (step S507). After completing the process of step S507, the control unit 21 completes the estimation process.

In the present embodiment, it is determined whether or not the state of the person being watched over is in a suspected abnormal state from the captured image using a learned model, and the state of the person being watched over is estimated based on the result of this determination. In the present embodiment, for example, determination of whether the state of the person being watched over is in a suspected abnormal state, which is easier determination than determination of whether the state of the person being watched over is in an abnormal state, is performed by learning. This is done using the pre-defined model. As described above, in the present embodiment, it is possible to simplify the content to be determined using the trained model. As a result, according to the present embodiment, it is possible to estimate the state of the person being watched over more accurately.

According to the present embodiment, for example, only the determination of the temporary state of the person being watched over is performed by the trained model, and the combination of the multiple determination results, the order of the multiple determination results, the duration of the same determination result, and the , time zone, attributes of the person being watched over, etc. can be determined without using a trained model. According to this configuration, it is possible to improve the accuracy of determination using the learned model, and flexibly adjust the estimation method of the state of the person being watched over without changing the learned model.

(Modification)
Although the embodiment has been described above, various modifications and applications are possible. It is arbitrary to adopt which part of the configuration, function, and operation described in the above embodiment. Also, in addition to the configurations, functions, and operations described above, additional configurations, functions, and operations may be employed. Also, the configurations, functions, and operations described in the above embodiments can be freely combined.

In Embodiment 1, an example was described in which estimation system 100 includes three devices: learning device 110 , estimation device 120 , and storage device 130 . The configuration of estimation system 100 is not limited to this configuration. For example, if one device has the function of the learning device 110, the function of the estimating device 120, and the function of the storage device 130, the estimating system 100 may include this one device. Moreover, the place where the learning device 110, the estimation device 120, and the storage device 130 are provided may be anywhere. For example, the learning device 110, the estimation device 120, and the storage device 130 may be provided in one facility, or at least one of the learning device 110, the estimation device 120, and the storage device 130 may be installed on the cloud. may be provided.

Embodiment 1 has described an example in which display device 300 is provided separately from learning device 110 and estimation device 120 . For example, if the learning device 110 or the estimation device 120 has the function of the display device 300, the display device 300 may not be provided. In Embodiment 1, an example in which the output unit 123 outputs the estimation result to the display device 300 has been described. The output unit 123 may output the estimation result to an audio output device. In this case, the voice output device notifies the administrator 310 of the estimation result by voice.

In the first embodiment, an example has been described in which there are four abnormal states, namely, the overturned state, the wandering state, the heat generating state, and the crowded state. It is possible to appropriately adjust what kind of state the abnormal state should be. For example, the abnormal condition may be at least one of the above four conditions, may include at least one of the above four conditions and another condition, or may be one of the above four conditions. It may not contain any state and may contain other states.

Also, the method of estimating the above four states can be adjusted as appropriate. For example, in Embodiment 1, an example has been described in which it is estimated that the state of the person being watched over is in a crowded state when a plurality of heat sources are close to each other in the captured image. In the case where multiple heat sources are close to each other in the captured image, if at least one heat source has a calorific value equal to or greater than the threshold, the state of the occupants 210 may be estimated to be in a risky crowded state.

Embodiment 2 has described an example in which the parameter used for learning is the attribute of the person being watched over. The parameters to be used for learning can be adjusted as appropriate. For example, parameters used for learning may be time of day, season, weather, and the like.

In the first embodiment, one trained model is prepared for one person being watched over, and in the second embodiment, one trained model is prepared for a plurality of persons being watched over. . The correspondence relationship between the person being watched over and the learned model can be adjusted as appropriate. For example, multiple watching targets may be classified into multiple groups from various points of view, and one trained model may be prepared for each group. In this case, for example, a plurality of persons to be watched over may be classified into a plurality of groups according to age, sex, physical characteristics, degree of health, place of residence, and the like. With such a configuration, a trained model is appropriately generated according to the attributes of the person being watched over, so it can be expected that the state of the person being watched over can be estimated with high accuracy.

Note that a trained model generated using a certain watching target person's captured image may be used to estimate the state of another watching target person, or may be updated using another watching target person's captured image. may Similarly, a trained model generated using captured images of a watching target person belonging to a certain group may be used to estimate the state of a watching target person belonging to another group, or may be used to estimate the state of a watching target person belonging to another group. You may update using a subject's captured image.

Also, the learning algorithm adopted by learning device 110 is not limited to the example shown in the first embodiment. For example, deep learning that extracts the feature amount itself may be employed, or other known learning algorithms may be employed. Moreover, when realizing unsupervised learning, clustering may be performed by a method other than non-hierarchical clustering by the k-means method. For example, hierarchical clustering by the shortest distance method may be employed, or clustering by other known methods may be employed. Also, as a learning algorithm, supervised learning, semi-supervised learning, reinforcement learning, etc. can be adopted instead of unsupervised learning.

In the above embodiments, the CPUs of the control units 11 and 21 function as the units shown in FIGS. However, in the present disclosure, the control units 11 and 21 may be dedicated hardware. Dedicated hardware is, for example, a single circuit, a composite circuit, a programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. When the control units 11 and 21 are dedicated hardware, each function of each unit may be realized by separate hardware, or the functions of each unit may be collectively realized by single hardware. Also, some of the functions of each unit may be realized by dedicated hardware, and other parts may be realized by software or firmware. In this way, the control unit 11 can implement each of the functions described above using hardware, software, firmware, or a combination thereof.

By applying an operation program that defines the operation of the learning device 110 and the estimating device 120 according to the present disclosure to a computer such as an existing personal computer or an information terminal device, the computer can be used as the learning device 110 and the estimating device according to the present disclosure. It is also possible to function as 120. Any method of distributing such programs may be used. For example, a CD-ROM (Compact Disk ROM), a DVD (Digital Versatile Disk), an MO (Magneto Optical Disk), or a computer-readable recording medium such as a memory card may be used. It may be stored in a medium and distributed, or may be distributed via a communication network such as the Internet.

This disclosure is capable of various embodiments and modifications without departing from the broader spirit and scope of this disclosure. In addition, the embodiments described above are for explaining the present disclosure, and do not limit the scope of the present disclosure. That is, the scope of the present disclosure is indicated by the claims rather than the embodiments. Various modifications made within the scope of the claims and within the scope of equivalent disclosure are considered to be within the scope of the present disclosure.

The present disclosure is applicable to watching systems.

11, 21 control unit 12, 22 storage unit 13, 23 display unit 14, 24 operation reception unit 15, 25 first communication unit 16, 26 second communication unit 20A, 20AA, 20AB, 20AC, 20BA , 20BB, 20BC, 20BD, 20BE, 20C, 20CA, 20CB, 20CC, 20D image, 27 third communication unit, 100 estimation system, 110 learning device, 111, 121 image acquisition unit, 112 model generation unit, 113 label reception unit , 114, 124 parameter acquisition unit, 120 estimation device, 122 estimation unit, 123 output unit, 125 state determination unit, 126 state estimation unit, 200 imaging device, 210 resident, 250 living room, 300 display device, 310 administrator, 350 Central control room, 1000 monitoring system

Claims (14)

an image acquisition means for acquiring a captured image from an imaging device that captures an image of the room of the person being watched over;
Estimation means for estimating the state of the person being watched over from the imaged image acquired by the image acquisition means, using a trained model acquired by machine learning for estimating the state of the person being watched over from the imaged image. When,
and an output means for outputting an estimation result by the estimation means,
estimation device. The learned model is a model for estimating whether or not the person being watched over has fallen from the captured image,
The estimation means uses the learned model to estimate whether or not the person being watched over has fallen from the captured image acquired by the image acquisition means.
The estimating device according to claim 1. The trained model is a model for estimating whether or not the person being watched over is wandering from the captured image,
The estimation means uses the learned model to estimate whether the person being watched over is wandering from the captured image acquired by the image acquisition means.
The estimation device according to claim 1 or 2. The learned model is a model for estimating whether or not the person being watched over has a fever from the captured image,
The estimating means estimates whether the person being watched over has a fever from the captured image acquired by the image acquiring means, using the learned model.
The estimation device according to any one of claims 1 to 3. The learned model is a model for estimating from the captured image whether or not a plurality of people including the person to be watched over is crowded,
The estimating means uses the learned model to estimate whether the plurality of people are concentrated from the captured image acquired by the image acquiring means.
The estimation device according to any one of claims 1 to 4. The trained model is a model acquired by machine learning using captured images obtained by capturing images of the rooms of each of the plurality of watching targets and parameters related to each of the plurality of watching targets,
further comprising parameter acquisition means for acquiring a first parameter that is a parameter related to a first person being watched over among the plurality of persons being watched over;
The image acquisition means acquires a first captured image from a first imaging device that captures an image of the room of the first person being watched over,
The estimation means estimates the state of the first watching target person from the first captured image acquired by the image acquisition means and the first parameter acquired by the parameter acquisition means, using the learned model. ,
The estimation device according to any one of claims 1 to 5. The estimation means is
state determination means for determining whether or not the state of the person being watched over is in a suspected abnormal state from the captured image obtained by the image obtaining means using the learned model;
state estimation means for estimating the state of the person being watched over based on the determination result of the state determination means;
The estimation device according to any one of claims 1 to 6. The state estimation means determines that the state of the person being watched over is the abnormal state when the state determination means determines that the state of the person being watched over is the suspected abnormal state at a rate equal to or greater than a threshold within a unit time. presume,
The estimation device according to claim 7. The suspected abnormal state includes a first suspected abnormal state and a second suspected abnormal state,
The state estimating means determines that the state of the person being watched over is the second suspected abnormal state after a period during which the state determination means determines that the state of the person being watched over continues to be the first suspected abnormal state. estimating that the state of the person being watched over is in an abnormal state when the period during which it is determined to be in a suspected abnormal state continues;
The estimation device according to claim 7 or 8. The state estimation means estimates that the state of the person being watched over is in an abnormal state when the state determination means determines that the state of the person being watched over is the suspected abnormal state in a first time period.
The estimation device according to any one of claims 7 to 9. A normal captured image obtained by imaging the room of the watching target when the condition of the watching target is normal, and the room of the watching target when the condition of the watching target is abnormal. an image acquisition means for acquiring an abnormal captured image obtained by imaging;
To estimate the state of the person being watched over from the captured image obtained by capturing the room of the person being watched over by machine learning using the normal captured image and the abnormal captured image obtained by the image obtaining means. a model generating means for generating a trained model of
learning device. An estimation system comprising a learning device and an estimation device,
The learning device
A normal captured image obtained by imaging the room of the watching target when the condition of the watching target is normal, and the room of the watching target when the condition of the watching target is abnormal. a first image acquisition means for acquiring an abnormal captured image obtained by imaging;
By machine learning using the normal captured image and the abnormal captured image obtained by the first image obtaining means, the state of the watching target is estimated from the captured image obtained by capturing the room of the watching target. a model generating means for generating a trained model for
The estimation device is
a second image acquiring means for acquiring the captured image from an imaging device that captures an image of the room of the person being watched over;
estimating means for estimating the state of the person being watched over from the captured image acquired by the second image acquiring means, using the learned model generated by the model generating means;
and an output means for outputting an estimation result by the estimation means,
estimation system. Acquiring a captured image from an imaging device that captures the room of the person being watched over,
estimating the state of the person being watched over from the captured image using a trained model acquired by machine learning for estimating the state of the person being watched over from the captured image;
estimation method. the computer,
Image acquisition means for acquiring a captured image from an imaging device that captures an image of the room of the person being watched over;
Estimation means for estimating the state of the person being watched over from the imaged image acquired by the image acquisition means, using a trained model acquired by machine learning for estimating the state of the person being watched over from the imaged image. ,
function as an output means for outputting the estimation result by the estimation means;
program.

Images