JP2022039815A - System, electronic apparatus, method for controlling electronic apparatus, and program - Google Patents

Abstract

To provide a system, an electronic apparatus, a method for controlling the electronic apparatus, and a program that can contribute to the safety of an object to be monitored.SOLUTION: A system 1 comprises: an imaging unit 20; and an electronic apparatus 10 including a recognition unit 11 and a controller 15. The imaging unit 20 picks up an image. The recognition unit 11 recognizes an object to be monitored and a wheelchair related to the object to be monitored in an image picked up by the imaging unit 20. Based on timing information indicating a start point and an end point of the movement of the object to be monitored leaving the wheelchair in sequential images picked up by the imaging unit 20, the controller 15 performs machine learning about the relationship between the positions of the object to be monitored and the wheelchair between the start point and the end point of the movement of leaving the wheelchair and the start of the movement of the object to be monitored leaving the wheelchair.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to systems, electronic devices, control methods for electronic devices, and programs.

For example, in a field such as a long-term care facility, a device for monitoring the behavior of a monitored person such as a nurse or a care recipient has been proposed. For example, Patent Document 1 discloses a monitored person system that detects a predetermined behavior in a monitored person based on an image obtained by an image pickup device. Patent Document 2 discloses a fall prevention system that prevents a subject from falling while walking by attaching a detection device to the foot of the subject. Further, Patent Document 3 discloses a monitoring support device for determining the body position of a human body by detecting a temperature distribution. Cited Document 4 also discloses a medical system for monitoring elderly psychotic patients at home or in elderly housing with care or long-term care facilities.

Japanese Unexamined Patent Publication No. 2017-91552 Japanese Unexamined Patent Publication No. 2017-221502 Japanese Unexamined Patent Publication No. 2014-106636 Japanese Patent Application Laid-Open No. 2003-91790

It would be beneficial if the monitored object could be safely monitored by monitoring it.

An object of the present disclosure is to provide a system, an electronic device, a control method for the electronic device, and a program that can be safely provided to the monitored object.

The system according to one embodiment is
An image pickup unit that captures an image and
A recognition unit that recognizes a monitored object and a wheelchair related to the monitored object in the image captured by the imaging unit, and a recognition unit.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the image pickup unit. A controller that machine-learns the relationship between the positions of the monitored object and the wheelchair and the start of the operation in which the monitored object leaves the wheelchair.
To prepare for.

In addition, the system according to one embodiment is
An image pickup unit that captures an image and
A recognition unit that recognizes a monitored object and a wheelchair related to the monitored object in the image captured by the imaging unit, and a recognition unit.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, the controller that estimates the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition unit.
To prepare for.

The electronic device according to one embodiment is
A recognition unit that recognizes the monitored object and the wheelchair related to the monitored object in the captured image, and
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the image pickup unit. A controller that machine-learns the relationship between the positions of the monitored object and the wheelchair and the start of the operation in which the monitored object leaves the wheelchair.
To prepare for.

In addition, the electronic device according to one embodiment is
A recognition unit that recognizes the monitored object and the wheelchair related to the monitored object in the captured image, and
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, the controller that estimates the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition unit.
To prepare for.

The method for controlling an electronic device according to an embodiment is as follows.
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the imaging step. A machine learning step for machine learning the relationship between the position of the monitored object and the wheelchair and the start of the operation of the monitored object leaving the wheelchair.
including.

Further, the control method of the electronic device according to the embodiment is as follows.
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, an estimation step of estimating the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition step, and an estimation step.
including.

The program according to one embodiment is
On the computer
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the imaging step. A machine learning step for machine learning the relationship between the position of the monitored object and the wheelchair and the start of the operation of the monitored object leaving the wheelchair.
To execute.

In addition, the program according to one embodiment is
On the computer
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, an estimation step of estimating the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition step, and an estimation step.
To execute.

According to one embodiment, it is possible to provide a system, an electronic device, a control method for the electronic device, and a program that can be safely provided to the monitored object.

It is a functional block diagram which shows the schematic structure of the system which concerns on one Embodiment. It is a flowchart explaining the operation of the learning phase by the system which concerns on one Embodiment. It is a flowchart which shows the example of the image imaged by the system which concerns on one Embodiment. It is a flowchart which shows the example of the image imaged by the system which concerns on one Embodiment. It is a figure which shows the example of the image recognized by the system which concerns on one Embodiment. It is a figure which shows the example of the image recognized by the system which concerns on one Embodiment. It is a figure explaining the timing information acquired by the system which concerns on one Embodiment. It is a flowchart explaining the operation of the estimation phase by the system which concerns on one Embodiment. It is a figure explaining the estimation by the system which concerns on one Embodiment. It is a figure explaining the recognition of an image and the determination of coordinates by the system which concerns on one Embodiment. It is a conceptual diagram explaining the operation of the controller by the system which concerns on one Embodiment. It is a conceptual diagram explaining the operation of the controller by the system which concerns on one Embodiment.

In the present disclosure, the "electronic device" may be a device driven by electric power. Further, the "system" may include a device driven by electric power. Further, the "user" may be a person (typically a human being) who uses the system and / or the electronic device according to the embodiment. The user may include a person who monitors the monitored object by using the system and / or the electronic device according to the embodiment. Further, the “monitored target” may be a person (for example, a human or an animal) to be monitored by the system and / or the electronic device according to the embodiment. Further, the user may include a monitored object.

It is assumed that the system according to one embodiment is used in a specific situation used by a person engaged in social activities such as a company, a hospital, an elderly home, a school, a sports gym, and a long-term care facility. It may be a facility. For example, in the case of a company, it is extremely important to understand and / or manage the health status of employees. Similarly, it is extremely important to understand and / or manage the health status of patients and medical staff in hospitals, and residents and staff in elderly housing with care. The scene in which the system according to the embodiment is used is not limited to the above-mentioned facilities such as companies, hospitals, and elderly housings, but any facility where it is desired to grasp and / or manage the health condition of the monitored object. May be. Any facility may also include non-commercial facilities, such as the user's home. Further, the scene in which the system according to one embodiment is used may be, for example, a moving body such as a train, a bus, or an airplane, a station, a platform, or the like.

The system according to one embodiment may be used for monitoring the behavior of a monitored object such as a nurse-requiring person or a nursing-requiring person in a nursing care facility or the like. The system according to one embodiment can monitor the operation of standing up from a wheelchair used by a monitored object such as a nurse or a care recipient and leaving the wheelchair unattended.

In particular, in the system according to one embodiment, when a monitored person such as a nurse or a care recipient is seated in a wheelchair, before the operation of leaving the wheelchair by the monitored person is completed, or A predetermined warning can be issued after the operation of leaving the wheelchair is completed. Therefore, according to the system according to one embodiment, a staff member such as a care facility is about to leave the wheelchair before a monitored person such as a nurse or a care recipient leaves the wheelchair. You can recognize that. Further, according to the system according to one embodiment, the staff of a nursing care facility, for example, recognizes that the wheelchair was left unattended after the monitored person such as a nurse requiring nursing care or a person requiring long-term care left the wheelchair. Can be.

Hereinafter, the system according to the embodiment will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a system according to an embodiment. As shown in FIG. 1, the system 1 according to an embodiment may be configured to include an electronic device 10 and an image pickup unit 20. The electronic device 10 and the image pickup unit 20 may be connected by wire or wireless, or a combination of wire and wireless. The system 1 according to one embodiment may not include a part of the functional unit shown in FIG. 1, or may include a functional unit other than that shown in FIG. For example, the system 1 according to the embodiment may not include at least one of the warning unit 17 and the communication unit 19. Further, for example, the system 1 according to the embodiment may include a display capable of displaying an image and / or a slot into which a storage such as a memory card can be inserted.

The image pickup unit 20 shown in FIG. 1 may include an image sensor that electronically captures an image, such as a digital camera. The image pickup unit 20 may include an image pickup element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging unit 20 may image the monitored target T, for example, as shown in FIG. Here, the monitored target T may be, for example, a human being. The image pickup unit 20 may convert the captured image into a signal and transmit it to the electronic device 10. For example, the image pickup unit 20 may transmit a signal based on the captured image to the extraction unit 11, the storage unit 13, and / or the controller 15 of the electronic device 10. The image pickup unit 20 is not limited to an image pickup device such as a digital camera as long as it captures the monitored target T, and may be any device.

In one embodiment, the image pickup unit 20 may take an image of the monitored target T, for example, as a still image at predetermined time intervals (for example, 15 frames per second). Further, in one embodiment, the imaging unit 20 may image, for example, the monitored target T as a continuous moving image. Thus, in one embodiment, the image pickup unit 20 captures an image.

As shown in FIG. 1, the electronic device 10 according to the embodiment may include a recognition unit 11, a storage unit 13, a controller 15, a warning unit 17, and a communication unit 19. The electronic device 10 according to the embodiment may not include a part of the functional unit shown in FIG. 1, or may include a functional unit other than that shown in FIG. For example, the electronic device 10 according to the embodiment may include machine learning data 132, which will be described later, stored in the storage unit 13. For example, in the electronic device 10 according to the embodiment, at least a part of the machine learning data 132 described later may be stored in an external device such as an external server.

The recognition unit 11 may have a function of recognizing a predetermined object from the image captured by the image pickup unit 20. For example, the recognition unit 11 may recognize the monitored target T from the image captured by the image pickup unit 20. Further, for example, the recognition unit 11 may recognize the wheelchair from the image captured by the image pickup unit 20. Further, for example, the recognition unit 11 may recognize the wheelchair related to the monitored target T from the image captured by the image pickup unit 20. Here, the wheelchair related to the monitored target T may be, for example, a wheelchair used by the monitored target T, a wheelchair used by the monitored target T, or the like. The recognition unit 11 may recognize the monitored target T and / or the wheelchair from the image captured by the image pickup unit 20, for example, using the existing image recognition technique. In one embodiment, the recognition unit 11 determines the position of each part such as the head, trunk, limbs, and / or joints of the monitored target T from the image of the monitored target T captured by the imaging unit 20. You may recognize it. The recognition unit 11 may be configured as dedicated hardware, may be configured to include software at least in part, or may be configured entirely by software. In this way, the recognition unit 11 may recognize the monitored object T and the wheelchair related to the monitored object T in the image captured by the imaging unit 20.

The storage unit 13 may have a function as a memory for storing various types of information. The storage unit 13 may store, for example, a program executed by the controller 15, the result of processing executed by the controller 15, and the like. Further, the storage unit 13 may function as a work memory of the controller 15. The storage unit 13 can be configured by, for example, a semiconductor memory or the like, but is not limited to this, and can be any storage device. For example, the storage unit 13 may be a storage medium such as a memory card inserted in the electronic device 10 according to the embodiment. Further, the storage unit 13 may be the internal memory of the CPU used as the controller 15 described later, or may be connected to the controller 15 as a separate body.

As shown in FIG. 1, the storage unit 13 may store, for example, machine learning data 132. Here, the machine learning data 132 may be data generated by machine learning. Further, machine learning may be based on AI (Artificial Intelligence) technology that enables a specific task to be executed by training. More specifically, machine learning may be a technique in which an information processing device such as a computer learns a large amount of data and automatically constructs an algorithm or model for performing tasks such as classification and / or prediction. In the present specification, a part of AI (Artificial Intelligence) may include machine learning. In the present specification, machine learning may include supervised learning that learns the features or rules of input data based on correct answer data. Further, the machine learning may include unsupervised learning in which the features or rules of the input data are learned in the absence of correct answer data. Further, the machine learning may include reinforcement learning for learning the features or rules of the input data by giving a reward or a punishment. Further, in the present specification, machine learning may be any combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of machine learning data 132 of the present embodiment may include an algorithm that outputs a predetermined inference (estimation) result by using an algorithm learned for input data. In this embodiment, as this algorithm, for example, a linear regression that predicts the relationship between a dependent variable and an independent variable, a neural network (NN) that mathematically models a human brain nervous system neuron, and a least squares that squares an error are calculated. Other appropriate algorithms such as multiplication, decision trees that make problem solving into a tree structure, and regularization that transforms data in a predetermined way can be used. In this embodiment, deep learning, which is a kind of neural network, may be used. Deep learning is a type of neural network, and a neural network with a deep network hierarchy is called deep learning.

In particular, in one embodiment, the machine learning data 132 may be data obtained by machine learning the movements of the monitored target T and the wheelchair when the monitored target T performs an operation of leaving the wheelchair unattended. Hereinafter, the operation in which the monitored object T leaves the wheelchair related to the monitored object T may be simply referred to as a “leaving operation”. Further, the machine learning data 132 includes the movement of the specific person and the wheelchair when the specific person (for example, a specific person requiring care) leaves the wheelchair related to the specific person as the monitored target T. May be used as machine-learned data. The machine learning data 132 according to the embodiment will be further described later.

The controller 15 controls and / or manages the entire electronic device 10 including each functional unit constituting the electronic device 10. The controller 15 may include at least one processor, such as a CPU (Central Processing Unit), to provide control and processing power to perform various functions. The controller 15 may be realized collectively by one processor, by several processors, or by individual processors. The processor may be realized as a single integrated circuit. The integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be realized as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented on the basis of various other known techniques.

In one embodiment, the controller 15 may be configured as, for example, a CPU and a program executed by the CPU. The program executed by the controller 15, the result of the process executed by the controller 15, and the like may be stored in, for example, the storage unit 13. The controller 15 may appropriately include a memory necessary for the operation of the controller 15. The operation of the controller 15 of the electronic device 10 according to the embodiment will be further described later.

The warning unit 17 may issue a predetermined warning for calling attention to the user of the system 1 or the electronic device 10 based on the predetermined warning signal output from the controller 15. As a predetermined warning, the warning unit 17 may be any functional unit that stimulates at least one of the user's auditory sense, visual sense, and tactile sense, such as sound, voice, light, characters, video, and vibration. Specifically, the warning unit 17 may be at least one of, for example, an audio output unit such as a buzzer or a speaker, a light emitting unit such as an LED, a display unit such as an LCD, and a tactile presentation unit such as a vibrator. .. In this way, the warning unit 17 may issue a predetermined warning based on a predetermined warning signal output from the controller 15. In one embodiment, the warning unit 17 may issue a predetermined alarm as information acting on at least one of auditory, visual, and tactile sensations.

In one embodiment, the warning unit 17 may issue a warning that there is a risk that the monitored target T will perform the neglected operation, for example, before the monitored target T starts the neglected operation. Further, in one embodiment, the warning unit 17 may issue a warning that there is a risk that the monitored target T will perform the neglected operation before the monitored target T completes the neglected operation. For example, in one embodiment, when the warning unit 17 that outputs visual information detects that there is a risk that the monitored target T may perform a neglected operation, the warning unit 17 warns the user by emitting light or a predetermined display to that effect. It's okay. Further, in one embodiment, when the warning unit 17 that outputs auditory information detects that there is a risk that the monitored target T may perform a neglected operation, the warning unit 17 warns the user by a predetermined sound or voice to that effect. It's okay. In the present embodiment, the warning may be a combination of light emission or a predetermined display, and a predetermined sound or voice.

The electronic device 10 shown in FIG. 1 has a built-in warning unit 17. However, in one embodiment, in the system 1, the warning unit 17 may be provided outside the electronic device 10. In this case, the warning unit 17 and the electronic device 10 may be connected by wire or wireless, or a combination of wire and wireless.

The communication unit 19 has an interface function for communicating by wire or wirelessly. The communication method performed by the communication unit 19 of one embodiment may be a wireless communication standard. For example, wireless communication standards include communication standards for cellular phones such as 2G, 3G, 4G, and 5G. For example, the communication standards for cellular phones are LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS. (Personal Handy-phone System) etc. are included. For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), 802.11, WiFi, Bluetooth®, IrDA (Infrared Data Association), NFC (Near Field Communication) and the like. The communication unit 19 can support one or more of the above communication standards. The communication unit 19 may be configured to include, for example, an antenna for transmitting and receiving radio waves, an appropriate RF unit, and the like. Further, the communication unit 19 may be configured as an interface such as a connector for making a wired connection to the outside. Since the communication unit 19 can be configured by a known technique for performing wireless communication, more detailed description of hardware and the like will be omitted.

Various types of information received by the communication unit 19 may be supplied to, for example, the storage unit 13 and / or the controller 15. Various information received by the communication unit 19 may be stored in, for example, a memory built in the storage unit 13 and / or the controller 15. Further, the communication unit 19 may transmit, for example, the processing result by the controller 15, the extraction result by the recognition unit 11, and / or the information stored in the storage unit 13 to the outside.

As shown in FIG. 1, at least a part of each functional unit constituting the electronic device 10 according to the embodiment may be configured by a specific means in which software and hardware resources cooperate.

Next, the operation of the system 1 according to the embodiment will be described.

The operation of the system 1 according to one embodiment can be typically divided into a "learning phase" and an "estimation phase". In the learning phase, for example, an operation of machine learning the relationship between the position (coordinates) of a person and a wheelchair in the operation of leaving a wheelchair related to the person, such as the monitored target T, and the timing of the operation of leaving the person. May be done. Further, in the estimation phase, based on the result of machine learning in the learning phase, an operation of estimating the start of the neglected operation by the monitored target T from the positions (coordinates) of the monitored target T and the wheelchair may be performed. Hereinafter, the operation in each of the above-mentioned learning phase and estimation phase will be described in more detail. First, the operation in the learning phase will be described.

In the technique of the present disclosure, it may be assumed that there is generally a certain relationship between the motion a of the body of the monitored object and the motion result A of the monitored object generated from this motion a. The operation result here may include the operation of the monitored object, the operation start time of the monitored object, an accident, an incident or other event generated from the operation of the monitored object. For example, it is assumed that the movement a of the body of the monitored target is performed, and the movement result A of the monitored target is generated from this movement a. Further, it is assumed that the movement b of the body to be monitored is performed, and the movement result B of the monitored target is generated from this movement b. The technique of the present disclosure may accumulate the relationship between the operation a and the operation result A, the operation b and the operation result B, and other operations and the operation result as machine learning data. Then, in the technique of the present disclosure, when the motion x is extracted, the motion result X related to the motion x may be estimated using the machine learning data.

FIG. 2 is a flowchart showing an example of the operation in the learning phase of the system 1 according to the embodiment. FIG. 2 may be a flowchart focusing on the operation of the electronic device 10 included in the system 1 according to the embodiment in the learning phase.

For example, a person suspected of developing dementia (for example, a person requiring nursing care or a person requiring long-term care) may get up from the wheelchair and start walking while moving in a wheelchair. In such a case, the person cannot remember the existence of the wheelchair, and there is a risk that the wheelchair is left unattended. If the wheelchair is left unattended, those who originally used the wheelchair are at risk of going missing. Also, if the wheelchair is left unattended, there is a risk that the person who originally used the wheelchair will go to an unsafe place. Furthermore, if the wheelchair is left unattended, depending on the environment, the abandoned wheelchair may not only interfere with others but also create a dangerous situation. Therefore, warning others that the wheelchair is about to be left or that the wheelchair has been left can not only provide the safety of the person using the wheelchair (the subject to be monitored), but also the wheelchair. It can also be used for the safety of the surrounding environment. In the learning phase of the system 1 according to the embodiment, as described above, the position (coordinates) of the monitored object T and the wheelchair in the operation in which a person such as the monitored object T leaves the wheelchair, and the leaving operation. Machine learning may be performed on the relationship with the start (timing). Hereinafter, such an operation will be described in more detail.

At the time when the operation shown in FIG. 2 starts, the imaging unit 20 of the system 1 may start imaging of a person such as the monitored target T and / or a wheelchair related to the person. The time point at which the operation shown in FIG. 2 starts may be the time when the image pickup unit 20 starts image pickup of a person such as the monitored target T and / or a wheelchair related to the person. Further, at the time when the operation shown in FIG. 2 starts, after the imaging unit 20 starts imaging, a person such as the monitored target T and / or a wheelchair related to the person is in the imaging range of the imaging unit 20. It may be the time of entry.

When the operation shown in FIG. 2 starts, the controller 15 of the electronic device 10 acquires the image captured by the image pickup unit 20 (step S11).

FIG. 3 is a diagram showing an example of an image acquired by the controller 15 in step S11 shown in FIG. 2, that is, an image captured by the imaging unit 20.

As shown in FIG. 3, the image pickup unit 20 may take an image of a state in which a person such as a monitored object T is seated in a wheelchair W. FIG. 3 schematically shows how the monitored object T is seated in the wheelchair W. In FIG. 3, the monitored object T and / or the wheelchair W may be moving or stationary. In this case, in step S11, the controller 15 acquires an image of a person such as the monitored target T sitting on the wheelchair W. As will be described later, the image pickup unit 20 may capture an image of a state other than the state in which a person such as the monitored target T is seated in the wheelchair W. In this case, in step S11, the controller 15 may acquire an image of a state other than the state in which a person such as the monitored target T is seated in the wheelchair W.

The image pickup unit 20 may capture each image of a predetermined number of frames per second. Here, the image captured by the image pickup unit 20 may be a still image of continuous frames or a moving image. For example, the image pickup unit 20 may capture an image of 15 frames per second. In step S11, the controller 15 may acquire images of a predetermined number of frames per second captured by the image pickup unit 20.

FIG. 4 is a diagram showing another example of the image acquired by the controller 15 in step S11 shown in FIG. 2, that is, the image captured by the imaging unit 20.

As shown in FIG. 4, the image pickup unit 20 may take an image of a state in which a person such as a monitored object T stands up from the wheelchair W and starts walking. FIG. 4 schematically shows how the monitored object T stands up from the wheelchair W and starts walking. In FIG. 4, the monitored object T and / or the wheelchair W may be moving or stationary. In this case, in step S11, the controller 15 acquires an image including a person such as the monitored target T and the wheelchair W. As will be described later, the image pickup unit 20 may capture an image of a state other than the state in which a person such as the monitored target T stands up from the wheelchair W and starts walking. In this case, in step S11, the controller 15 may acquire an image of a state other than the state in which a person such as the monitored target T stands up from the wheelchair W and starts walking.

As shown in FIG. 2, when the image captured in step S11 is acquired, the recognition unit 11 recognizes the monitored target T and the wheelchair W related to the monitored target T (step S12). The operation in step S12 may be performed by the controller 15 instead of the recognition unit 11.

FIG. 5 is a diagram showing an example of the monitored object T and the wheelchair W recognized in step S12. FIG. 5 shows how the recognition unit 11 recognizes the monitored target T and the wheelchair W in the image shown in FIG.

In step S12, the recognition unit 11 may recognize the monitored object T and the wheelchair W as shown in FIG. 5, for example. In FIG. 5, the monitored object T is recognized in the range Tc, and the wheelchair W is recognized in the range Wc. The recognition unit 11 may recognize the monitored object T and the wheelchair W by any technique such as image recognition. Since various techniques for detecting and / or recognizing a predetermined object in the captured image are known, a more detailed description thereof will be omitted.

FIG. 6 is a diagram showing another example of the monitored object T and the wheelchair W recognized in step S12. FIG. 6 shows how the recognition unit 11 recognizes the monitored target T and the wheelchair W in the image shown in FIG.

In step S12, the recognition unit 11 may recognize the monitored object T and the wheelchair W as shown in FIG. 6, for example. Also in FIG. 6, similarly to FIG. 5, the monitored target T is recognized in the range Tc, and the wheelchair W is recognized in the range Wc.

In step S12, the recognition unit 11 may recognize predetermined feature points in the monitored object T and the wheelchair W as shown in FIGS. 5 and 6. Further, the recognition unit 11 may two-dimensionally extract the monitored object T and the wheelchair W shown in FIGS. 5 and 6 according to the coordinate axes shown in FIGS. 5 and 6. Further, the recognition unit 11 may extract the positions (for example, coordinates) of the monitored object T and the wheelchair W in the image when recognizing the monitored object T and the wheelchair W.

Here, when the image pickup unit 20 captures each image of a predetermined number of frames per second, the recognition unit 11 may recognize the monitored target T and the wheelchair W in the predetermined number of frames per second, as shown in FIG. .. Further, even when the controller 15 acquires images of a predetermined number of frames per second, the recognition unit 11 may recognize the monitored target T and the wheelchair W in the predetermined number of frames per second. As an example, the recognition unit 11 may recognize the monitored object T and the wheelchair W at 15 frames per second.

As described above, in one embodiment, the recognition unit 11 two-dimensionally recognizes the positions of the monitored target T and the wheelchair W for each image of a predetermined number of frames per second imaged by the image pickup unit 20. May be good.

When the monitored target T and the wheelchair W are recognized in step S12, the controller 15 determines the coordinates of the monitored target T and the wheelchair W (step S13).

In step S13, the controller 15 may determine the center of the range recognized as the monitored target T and the center of the range recognized as the wheelchair W as the respective coordinates. Further, in step S13, the controller 15 may determine the center of gravity of the range recognized as the monitored target T and the center of gravity of the range recognized as the wheelchair W as the respective coordinates. Further, the controller 15 may determine the ground contact portion in the range recognized as the monitored target T and the ground contact portion in the range recognized as the wheelchair W as the respective coordinates. Further, the controller 15 may determine the lower end portion of the range recognized as the monitored target T and the lower end portion of the range recognized as the wheelchair W as the respective coordinates. Further, the controller 15 may determine the center of the lower end portion of the range recognized as the monitored target T and the center of the lower end portion of the range recognized as the wheelchair W as the respective coordinates. Further, the controller 15 may arbitrarily combine any of the above as the coordinates of the monitored object T and the wheelchair W.

In FIG. 5, the center of the lower end portion of the range Tc recognized as the monitored target T is indicated by the center line Tm. Further, the center of the lower end portion of the range Wc recognized as the wheelchair W is indicated by the center line Wm. In this case, the controller 15 may determine, for example, the coordinates of the monitored object T and the wheelchair W as arbitrary positions on the center line Tm and the center line Wm, respectively. In the case shown in FIG. 5, since the monitored object T is seated in the wheelchair W, the distance D1 between the monitored object T and the wheelchair W becomes a very small value. Here, the distance D1 between the monitored object T and the wheelchair W is, for example, the center of the lower end portion of the range Tc recognized as the monitored target T and the center of the lower end portion of the range Wc recognized as the wheelchair W. It may be the distance between them.

Also in FIG. 6, similarly to FIG. 5, the center of the lower end portion of the range Tc recognized as the monitored target T is indicated by the center line Tm. Further, the center of the lower end portion of the range Wc recognized as the wheelchair W is indicated by the center line Wm. In this case, the controller 15 may determine, for example, the coordinates of the monitored target T and the wheelchair W as arbitrary positions on the center line Tm and the center line Wm, respectively. In the case shown in FIG. 6, since the monitored object T stands up from the wheelchair W and starts walking, the distance D2 between the monitored object T and the wheelchair W becomes a larger value than the distance D1 shown in FIG. Become. Here, the distance D2 between the monitored object T and the wheelchair W is, for example, the center of the lower end portion of the range Tc recognized as the monitored target T and the center of the lower end portion of the range Wc recognized as the wheelchair W. It may be the distance between them.

In one embodiment, the controller 15 may determine (estimate), for example, the distance between the monitored object T and the wheelchair W. For the distance between the monitored object T and the wheelchair W measured by the controller 15, for example, the distance in the image may be calculated, or the actual distance may be determined (estimated) based on the image. When the distance between the monitored object T and the wheelchair W is calculated in the image, the distance is also caused by, for example, the distance between the imaging unit 20 and the monitored object T, and the direction from the imaging unit 20 toward the monitored object T. It is expected that it will vary. On the other hand, if the actual distance between the monitored object T and the wheelchair W is determined (estimated) based on the image, the variation due to the imaging environment can be suppressed.

Here, the operation of the controller 15 will be further described with reference to FIGS. 11 and 12. For example, when an image is captured by an image pickup unit 20 such as a camera, the appearance of the actual distance in the actual environment and the distance in the captured image may differ. Therefore, in one embodiment, the controller 15 may solve the difference in appearance between the actual distance in the actual environment and the distance in the captured image by learning.

For example, an example in which an image as shown in FIG. 12 is captured by an image pickup unit 20 installed in an actual environment as shown in FIG. 11 will be described.

As shown in FIG. 11, it is assumed that an image pickup unit 20 such as a camera is installed on the ceiling of an environment such as a room or a passage. It is assumed that the image pickup unit 20 is installed in a direction parallel to the depth direction of the room or passage shown in FIG. 11 (hereinafter, the direction in which the image pickup unit 20 faces is referred to as "imaging direction"). In such an environment, as shown in FIG. 11, it is assumed that the human being (monitored target T1) initially present on the front side moves 5 m to the back side along the imaging direction (monitored target T1'). Here, as shown in FIG. 11, it is assumed that the predetermined distance D1 at the position of the monitored target T1 and the predetermined distance D1'at the position of the monitored target T1'indicate the same length.

FIG. 12 shows an example of an image captured by the imaging unit 20 in an actual environment as shown in FIG. That is, the monitored target T1 in the actual environment shown in FIG. 11 indicates the monitored target T1 in the captured image shown in FIG. 12. Similarly, the monitored object T1'in the actual environment shown in FIG. 11 indicates the monitored object T1'in the captured image shown in FIG. 12. In FIG. 12, a predetermined distance D1 at the position of the monitored target T1 and a predetermined distance D1'at the position of the monitored target T1'actually indicate distances of the same length.

At a location relatively close to the image pickup unit 20, such as the position of the monitored object T1 shown in FIGS. 11 and 12, for example, when the monitored object T1 moves, the movement of coordinates in the image captured by the image pickup unit 20 (For example, the distance per pixel) becomes large. On the other hand, in a place relatively far from the image pickup unit 20, such as the position of the monitored target T1'shown in FIGS. 11 and 12, for example, when the image moves to the back side along the image pickup direction, the image is captured by the image pickup unit 20. The movement of coordinates (for example, the distance per pixel) in the image is also small.

Based on the above principle, in one embodiment, the controller 15 can solve the difference in appearance between the actual distance in the actual environment and the distance in the captured image by learning. ..

Further, when the controller 15 determines the coordinates of the monitored target T and the wheelchair W, the controller 15 may extract arbitrary points or feature points in each of the monitored target T and the wheelchair W. The controller 15 may determine each of the monitored object T and the wheelchair W as the coordinates of one point (for example, the center or the center of gravity). Further, the controller 15 may determine each of the monitored target T and the wheelchair W as a point cloud including a plurality of points. When the controller 15 determines each of the monitored target T and the wheelchair W as a point cloud including a plurality of points, the controller 15 may determine the center or the center of gravity of each point cloud as the coordinates of one point.

When the coordinates of the monitored target T and the wheelchair W are determined in step S13, the controller 15 acquires timing information (step S14). In order for the controller 15 to acquire the timing information in step S14, the timing information needs to be added (set) in advance to the image acquired in step S11 or the image recognized in step S12. Further, timing information may be given (set) in advance at the coordinates determined in step S13 instead of the image recognized in step S12. Hereinafter, such timing information will be further described.

In one embodiment, as data for the electronic device 10 to perform machine learning, it is necessary to prepare information indicating the start time point and the end time point of the operation of leaving the wheelchair W on which the human being is seated, for example, the monitored object T. There is. If there is information (timing information) indicating the start time and end time of the operation of leaving the wheelchair W on which the monitored target T is seated, the electronic device 10 performs machine learning using this timing information as, for example, teacher data. Can be done. As described above, the information indicating the start time point and the end time point of the operation of leaving the wheelchair W on which the monitored object T is seated is also referred to as "timing information". That is, the "timing information" may be information indicating the start time point and the end time point of the neglected operation in the time-dependent image captured by the image pickup unit 20.

Such timing information may be given (set) by personnel such as staff. That is, for example, an ethology expert or a staff member of a long-term care facility observes (visually recognizes) the images of the monitored object T and the wheelchair W captured by the imaging unit 20 and indicates the start time and end time of the neglected operation. Timing information may be added. In addition, by setting a predetermined standard for giving such timing information in advance, it is possible to give timing information even to general personnel who are not ethology specialists or staff of long-term care facilities. can. As described above, the timing information may be added (set) in the image data acquired in step S11. Further, the timing information may be added (set) in the image data recognized in step S12. Further, the timing information may be added (set) in the coordinate data determined in step S13.

FIG. 7 is a diagram illustrating the setting of timing information. FIG. 7 conceptually shows how the image data is continuous with the passage of time for each frame. That is, in FIG. 7, it is shown that the image data is continuous like frame 1, frame 2, frame 3, ... With the passage of time. Further, FIG. 7 shows an example of adding (setting) timing information to the image data.

At the time of "entry" shown in FIG. 7, for example, the state in which the monitored object T and the wheelchair W enter the room in which the image pickup unit 20 is installed is shown in the image data (image data) captured by the image pickup unit 20. It is assumed that it has been done.
It is assumed that the image data shows that the wheelchair W on which the monitored object T is seated has started the operation of leaving the wheelchair W at the time of "starting the leaving operation" shown in FIG. 7.
It is assumed that the image data shows that the wheelchair W on which the monitored object T was seated has finished the operation of leaving the wheelchair W at the time of "end of the leaving operation" shown in FIG. 7.
At the time of "leaving the room" shown in FIG. 7, the image data shows that the monitored target T and the wheelchair W, which are seated in the wheelchair W again with the assistance of the care staff, for example, leave the room in which the image pickup unit 20 is installed. Suppose it was shown in. Further, at the time of "leaving the room" shown in FIG. 7, for example, the image data may show that the monitored target T, in which the wheelchair W is left unattended, leaves the room in which the image pickup unit 20 is installed. ..

In the image data captured as described above, at least the time point of "start of neglected operation" and the time point of "end of neglected operation" are determined by, for example, a behavioral expert, a staff member of a nursing facility, or other general personnel. The indicated timing information may be given (set).

Here, the time point of "start of the neglected operation" is determined to be that the monitored target T has left the wheelchair W while the monitored target T is seated in the wheelchair W, for example, as shown in FIG. 3 or FIG. It may be a possible timing. FIG. 3 or FIG. 5 shows a state in which the monitored object T is seated in the wheelchair W, which shows the state from the time of “entering the room” to the time of “start of the neglected operation” shown in FIG. It may be a wheelchair.

Generally, when the monitored object T starts the neglected operation from the state shown in FIG. 3 or FIG. 5, the monitored object T is separated from the wheelchair W. Therefore, in one embodiment, the staff or the like obtains the timing information by setting the time when the monitored target T starts the transition from the state where the wheelchair W is not separated to the state where the monitored object T is separated from the wheelchair W as the time when the “leaving operation starts”. It may be given (set). For example, in FIG. 5, a staff member or the like assigns (sets) timing information to the time when the distance D1 between the monitored target T and the wheelchair W changes to a predetermined threshold value or more as the time of “start of neglected operation”. You can do it. Further, for example, in FIG. 5, when the range Tc of the monitored object T and the range Wc of the wheelchair W change from the state of having an overlap to the state of having no overlap, the personnel such as the staff "start the neglected operation". , Timing information may be added (set).

As described above, in one embodiment, the start time of the neglected operation in the timing information may indicate the timing when the monitored target T is separated from the wheelchair W in the time-dependent image captured by the image pickup unit 20.

Further, the time point of "end of the neglected operation" may be a timing at which it can be determined that the monitored object T is separated from the wheelchair W to some extent (for example, 10 m) as shown in FIG. 4 or FIG. 6, for example. Therefore, in one embodiment, the staff or the like assigns (sets) timing information to the time when the monitored target T is separated from the wheelchair W by a predetermined distance, for example, 10 m, as the time point of "end of the neglected operation". good. For example, in FIG. 6, a staff member or the like sets the timing information when the distance D2 between the monitored target T and the wheelchair W becomes equal to or higher than a predetermined threshold value (for example, 10 m) as the time point of “end of neglected operation”. It may be given (set). Further, for example, in FIG. 6, when the shortest distance between the range Tc of the monitored object T and the range Wc of the wheelchair W is separated from a predetermined threshold value (for example, 10 m) or more, the personnel such as staff "end the neglected operation". , Timing information may be added (set).

As described above, in one embodiment, the end point of the neglected operation in the timing information may indicate the timing at which the monitored target T is separated from the wheelchair W by a predetermined distance in the temporal image captured by the imaging unit 20. ..

In the timing information, the time points of the above-mentioned "start of the neglected operation" and the time point of the "end of the neglected operation" do not necessarily have to be given (set) in this order. That is, in the timing information, the time point of "the end of the neglected operation" may be set first, and then the time point of "the start of the neglected operation" may be set.

For example, it is not always easy for personnel such as staff to determine the time point of "start of neglected operation" when reproducing (visually recognizing) the image of the monitored target T captured by the imaging unit 20. Is also assumed. In such a case, first, when a staff member or the like reproduces (visually recognizes) the image of the monitored target T imaged by the image pickup unit 20, the timing of the "end of the neglected operation" is determined. It may be set as information. Next, a staff member or the like may reproduce (visually recognize) the image of the monitored object T captured by the image pickup unit 20 in the reverse time from the time of "the end of the neglected operation". Here, when the image of the monitored target T captured by the image pickup unit 20 is reproduced in reverse in time, the speed of the reverse reproduction may be reduced to some extent. In this way, personnel such as staff can easily set the time point of "start of the neglected operation" which is the time point before the time point of the "end of the neglected operation" in the timing information. For example, a staff member or the like may find a time point at which the monitored target T leaves the wheelchair W while visually recognizing the image data reproduced in the reverse time, and set the time point as timing information.

Further, the time point of "the end of the neglected operation" is, for example, as shown in FIG. 4 or FIG. This may be the timing at which it can be determined that the wheelchair has passed. Therefore, in one embodiment, the staff or the like obtains the timing information by setting the time when the monitored target T is separated from the wheelchair W by a predetermined distance, for example, 10 m, as the time point of "the end of the neglected operation". It may be given (set).

As described above, in one embodiment, at the end of the neglected operation in the timing information, a predetermined time elapses in a state where the monitored target T is separated from the wheelchair W by a predetermined distance in the temporal image captured by the imaging unit 20. The timing may be shown.

When the timing information is added (set) in the image data (or coordinate data) as described above, the controller 15 can acquire the timing information in step S14. In step S14, the controller 15 of the electronic device 10 may acquire the timing information as described above from an external network or the like via, for example, the communication unit 19. Further, in step S14, the controller 15 may acquire the timing information as described above from a slot of an electronic device 10 into which a storage such as a memory card can be inserted. Further, when the system 1 includes a display capable of displaying an image, image data or the like may be displayed on the display. In this case, the timing information set by a person such as a staff member while visually recognizing the display may be input via, for example, an operation unit of the electronic device 10.

After acquiring the timing information in step S14, the controller 15 machine-learns the relationship between the positions (coordinates) of the monitored target T and the wheelchair W and the start of the neglected operation based on the timing information (step S15). In step S15, the controller 15 learns the relationship between the positions (coordinates) of the monitored target T and the wheelchair W between the start time and the end time of the neglected operation and the start of the neglected operation based on the timing information. It may be a thing. Here, the controller 15 may perform machine learning based on the image data (or coordinate data) and the timing information set in the image data (or coordinate data). Hereinafter, the data generated as a result of machine learning in step S15 may be referred to as "machine learning data".

By performing the machine learning as described above, the electronic device 10 determines the timing at which the neglected operation starts and the movement of the positions or coordinates of the monitored target T and the wheelchair W between the start time and the end time of the neglected operation. You can understand the relationship between. Therefore, according to the electronic device 10, in the estimation phase described later, the start time of the neglected operation can be estimated based on the positions or coordinates of the monitored target T and the wheelchair W.

As described above, in one embodiment, the controller 15 determines the relationship between the positions of the monitored target T and the wheelchair W between the start time and the end time of the neglected operation and the start of the neglected operation based on the timing information. Machine learning. Here, the timing information may be information indicating the start time point and the end time point of the neglected operation in the time-dependent image captured by the image pickup unit 20 as described above.

Even if machine learning is performed on the data of a relatively large number of samples (for example, a human such as a monitored target T) in order to improve the accuracy of estimating the start time of the neglected operation of the electronic device 10 in the estimation phase described later. good. By increasing the amount of sample data when performing machine learning, it can be expected that the accuracy when the electronic device 10 estimates the start time of the neglected operation in the estimation phase described later will be improved. Therefore, by increasing the sample data when performing machine learning, it is possible to reduce the occurrence of false alarms or false alarms when the electronic device 10 estimates the start of the neglected operation and issues a warning. obtain.

In the above-mentioned machine learning, the timing information is information indicating the start time point and the end time point of the neglected operation in the time-dependent image captured by the image pickup unit 20. In one embodiment, the timing information may include information indicating a time point other than the start time point and the end time point of the neglected operation in the time-dependent image captured by the image pickup unit 20.

For example, the image data in the section of the learning data (4) shown in FIG. 7 exists between the start time point and the end time point of the neglected operation. Therefore, the learning data (4) may be machine-learned by the controller 15 as data classified into a dangerous (that is, a high risk of neglected operation) class. The image data in the section of the learning data (4) shown in FIG. 7 starts at substantially the same time as the start time of the neglected operation. On the other hand, the image data in the section of the learning data (4) shown in FIG. 7 ends at a time different from the end time of the neglected operation.

On the other hand, the image data of the sections of the learning data (1) to (3) and the learning data (5) shown in FIG. 7 do not exist between the start time and the end time of the neglected operation. Therefore, these learning data may be machine-learned by the controller 15 as data classified into a normal class (that is, a low risk (low risk of neglected operation)). The image data in the section of these training data starts at a time different from the start time of the neglected operation and ends at a time different from the end time of the neglected operation.

As described above, in one embodiment, the timing information may include information indicating a time point other than the start time point and the end time point of the neglected operation in the time-dependent image captured by the image pickup unit 20. Based on such timing information, the controller 15 may machine-learn the relationship between the coordinates of the predetermined portion of the monitored target T between the start time and the end time of the neglected operation and the start of the neglected operation. ..

As shown in FIG. 7, the controller 15 has not only the learning data (4) classified into the dangerous class, but also the learning data (1) to (3) and the learning data (5) classified into the normal class. Machine learning may be performed based on the learning data such as. The controller 15 can improve the accuracy of estimating the start of the neglected operation by performing machine learning based on the learning data classified into the dangerous class and the learning data classified into the normal class.

Next, the operation in the estimation phase will be described.

FIG. 8 is a flowchart showing an example of the operation in the estimation phase of the system 1 according to the embodiment. FIG. 8 may be a flowchart focusing on the operation in the estimation phase of the electronic device 10 included in the system 1 according to the embodiment.

As mentioned above, for example, a person suspected of developing dementia (for example, a person requiring nursing care or a person requiring long-term care) may get up from the wheelchair and start walking while moving in a wheelchair. There is. In such a case, the person cannot remember the existence of the wheelchair, and there is a risk that the wheelchair is left unattended. In the estimation phase of the system 1 according to one embodiment, the start of the neglected operation is estimated from the positions (coordinates) of the monitored target T and the wheelchair W by using the machine learning data obtained in the learning phase. good. Hereinafter, such an operation will be described in more detail. By monitoring the monitored target T and the wheelchair W by the system 1, for example, a staff member of a nursing facility or a hospital recognizes that the wheelchair W is about to be left unattended before the monitored target T completes the neglected operation. can do. Further, by monitoring the monitored target T and the wheelchair W by the system 1, for example, a staff member of a nursing facility or a hospital recognizes that the wheelchair W has been left unattended after the monitored target T has left the wheelchair W. Can be done.

When the operation shown in FIG. 8 starts, the controller 15 of the electronic device 10 acquires machine learning data (step S21). The machine learning data acquired in step S21 may be data generated as a result of machine learning in step S15 shown in FIG. That is, the machine learning data is the position (coordinates) of the person and the wheelchair between the start time and the end time of the operation in which the person leaves the wheelchair related to the person, and the operation in which the person leaves the wheelchair. The relationship with the start may be machine-learned data.

If the machine learning data has already been acquired at the time when the operation shown in FIG. 8 starts, the controller 15 does not have to acquire the machine learning data again in step S21. Further, the machine learning data acquired in step S21 is used in step S25 described later. Therefore, in one embodiment, the acquisition of machine learning data does not necessarily have to be performed in step S21, and may be performed at any timing up to step 25.

When the machine learning data is acquired in step S21, the controller 15 acquires the image captured by the imaging unit 20 (step S22). The operation in step S22 may be performed in the same manner as the operation in step S11 shown in FIG.

When the image captured in step 22 is acquired, the recognition unit 11 recognizes the monitored target T and the wheelchair W related to the monitored target T (step S23). The operation in step S23 may be performed in the same manner as the operation in step S12 shown in FIG. That is, for example, the recognition unit 11 may two-dimensionally recognize the positions of the monitored target T and the wheelchair W for each image of a predetermined number of frames per second captured by the image pickup unit 20.

When the monitored target T and the wheelchair W are recognized in step S23, the controller 15 determines the coordinates of the monitored target T and the wheelchair W (step S24). The operation in step S24 may be performed in the same manner as the operation in step S13 shown in FIG.

When the coordinates of the monitored object T and the wheelchair W are determined in step S24, the controller 15 performs the operation shown in the next step S25. That is, in step S25, the controller 15 determines whether or not the start of the neglected operation is estimated from the coordinates determined in step S24 based on the machine learning data acquired in step S21.

When the start of the neglected operation is estimated in step S25, that is, when the risk of the neglected operation starting from now on is increased, or when it is assumed that the neglected operation has started, the controller 15 sends a predetermined warning signal. Output (step S26). In step S26, the controller 15 may output a predetermined warning signal to the warning unit 17. As a result, the warning unit 17 can issue a predetermined warning.

On the other hand, when the start of the neglected operation is not estimated in step S25, that is, when the risk of the neglected operation starting from now on is not increased, the controller 15 skips the operation of step S26 and ends the operation shown in FIG. It's okay. Further, even when the start of the neglected operation is not estimated in step S25, that is, when the neglected operation is not assumed to have started, the controller 15 skips the operation of step S26 and ends the operation shown in FIG. It's okay. When the operation shown in FIG. 8 is completed, the controller 15 may start the operation shown in FIG. 8 again. For example, the controller 15 may repeat the operation shown in FIG. 8 every time the coordinates are extracted from the image data. That is, for example, when the recognition unit 11 recognizes the monitored target T and the wheelchair W from the image data of 15 frames per second, the controller 15 may estimate the start of the neglected operation in step S25 15 times per second.

As described above, in one embodiment, the controller 15 may estimate the start of the neglected operation from the positions (coordinates) of the monitored object T and the wheelchair W recognized by the recognition unit 11 based on the machine learning data. .. Further, the recognition unit 11 may recognize the positions of the monitored target T and the wheelchair W from the images of a predetermined number of frames per unit time captured by the image pickup unit 20. In this case, the controller 15 may estimate the start of the neglected operation from the positions of the monitored object T and the wheelchair W extracted by the recognition unit 11.

As shown in FIG. 8, the controller 15 may output a predetermined warning signal in step S26 immediately after estimating the start of the neglected operation in step S25. Therefore, the controller 15 may output a predetermined warning signal before the actual neglected operation is completed. As described above, after estimating the start of the neglected operation, the controller 15 may output a predetermined warning signal before the completion of the neglected operation. Further, if possible, the controller 15 may output a predetermined warning signal before the actual neglected operation starts. As described above, after estimating the start of the neglected operation, the controller 15 may output a predetermined warning signal before the start of the neglected operation.

According to the system 1 according to the embodiment, the relationship between the positions of the human and the wheelchair W in the operation of leaving the wheelchair W on which the human being is seated, such as the monitored object T, and the timing of the leaving operation is machined. You can learn. Further, according to the system 1 according to the embodiment, based on the result of machine learning, the wheelchair W on which the monitored object T is seated is left unattended from the positions of the monitored object T and the wheelchair W. The start of can be estimated. Therefore, according to the system 1 according to the embodiment, when the monitored object T intends to leave the wheelchair W, or when the wheelchair W is left unattended, a predetermined warning is issued before or after the unattended operation is completed. Can be done. Therefore, according to the system 1 according to the embodiment, the staff of a nursing care facility, for example, is a person to be monitored before or after the person to be monitored, such as a nurse or a person requiring nursing care, leaves the wheelchair. Can recognize that he is about to leave the wheelchair or has left it. Therefore, according to the system 1 according to the embodiment, the monitored target T can be safely provided.

FIG. 9 is a diagram further explaining the estimation process described in step S25 of FIG. FIG. 9 shows the state of the monitored target T captured in the image data acquired in step S22 of FIG. 8, for example, in the left column. As shown in FIG. 9, in the captured image data, the monitored target T enters the room in which the imaging unit 20 is installed, sits in the wheelchair W, then stands, and then the wheelchair. It is assumed that the user is away from W. From the moment when the wheelchair W is separated from the wheelchair W and the neglected operation starts, the controller 15 shall perform the operation after step S22 shown in FIG. 8 as a process of 15 frames per second. That is, in the system 1, the image pickup unit 20 may capture an image of 15 frames per second. Further, in the system 1, the controller 15 may acquire an image of 15 frames per second. Further, in the system 1, the recognition unit 11 may recognize the monitored target T and the wheelchair W from the image of 15 frames per second. Further, in the system 1, the controller 15 may determine the coordinates of the monitored object T and the wheelchair W recognized from the image of 15 frames per second. Further, in the system 1, the controller 15 may estimate the start of the neglected operation from the coordinates of the monitored object T and the wheelchair W at 15 frames per second based on the machine learning data.

In the central column of FIG. 9, a state in which the controller 15 continuously acquires frames of image data is conceptually shown. Here, the image data of each frame may be the position or coordinates extracted from the image data. Further, in the central column of FIG. 9, the hatched image data may represent a frame for 1 second after the start of the neglected operation.

In such a situation, the controller 15 may estimate the start of the neglected operation at that time based on the frame (frames 1 to 15) for 1 second after the start of the neglected operation (shown in FIG. 9). Estimate 1). Next, the controller 15 may estimate the start of the neglected operation at that time based on the frames from frame 2 to frame 16 (estimation 2 shown in FIG. 9). By repeating the above operation, the controller 15 estimates the start of the neglected operation 15 times per second. Therefore, according to the system 1 according to the embodiment, even if the start of the neglected operation that should be originally estimated is not estimated in the estimation 1 and the estimation 2 shown in FIG. 9, for some reason, the estimation is performed 15 times per second. The risk of misreporting can be reduced.

In the above-described embodiment, the system 1 has described an example of processing 15 frames per second. However, in one embodiment, the system 1 or each functional unit constituting the system 1 may process more than 15 frames per second or may process less than 15 frames per second. In one embodiment, the number of joint points handled by the system 1 and / or the number of frames processed may be adjusted so that the estimation of the start of the neglected operation gives a reasonable result.

Further, the controller 15 of the electronic device of the present embodiment normalizes the X-direction component and the Y-direction component of the determined coordinates, so that the determined coordinates can be used for machine learning in general. You may do so.

In this case, for example, the determined X and Y coordinates may be normalized based on the maximum and minimum values of the X and Y coordinates determined in 15 frames per second. Here, the maximum value of the X coordinate determined in step S13 is Xmax, and the minimum value of the determined X coordinate is Xmin. Further, the maximum value of the X coordinate after normalization is set to X'max. In this case, the X coordinate (X) before normalization can be converted into the X coordinate (X') after normalization by using the following equation (1).

X'= ((X-Xmin) / (Xmax-Xmin)) · X'max (1)

Similarly, the maximum value of the determined Y coordinate is Ymax, and the minimum value of the determined Y coordinate is Ymin. Further, the maximum value of the Y coordinate after normalization is set to Y'max. In this case, the Y coordinate (Y) before normalization can be converted into the Y coordinate (Y') after normalization by using the following equation (2).

Y'= ((Y-Ymin) / (Ymax-Ymin)) · Y'max (2)

According to the above equations (1) and (2), the X-direction component and the Y-direction component of the coordinates determined for the monitored target T and the wheelchair W can be normalized, respectively. As a result, it can be expected that the influence of the environment in which the image pickup unit 20 has imaged the monitored target T and the wheelchair W on machine learning is reduced.

As described above, in one embodiment, the controller 15 determines each direction component of the coordinates of the positions of the monitored object T and the wheelchair W determined two-dimensionally based on the maximum value and the minimum value of each direction component. It may be normalized. Further, such an operation may be performed by the recognition unit 11 instead of the controller 15.

As described above, in one embodiment, the controller 15 may determine the number of the frames so that the validity of the estimation of the start of the neglected operation becomes more than a predetermined value.

In the above embodiment, since the captured image data is used, visible light is used as a detection target for monitoring. However, the present disclosure is not limited to such cases, and other detection targets such as arbitrary electromagnetic waves, sound waves, temperatures, and vibrations may be arbitrarily used.

Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or modifications based on the present disclosure. It should be noted, therefore, that these modifications or modifications are within the scope of this disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically inconsistent, and a plurality of components or steps can be combined or divided into one. Is. Although the embodiment according to the present disclosure has been mainly described with respect to the apparatus, the embodiment according to the present disclosure can also be realized as a method including steps executed by each component of the apparatus. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, which is executed by a processor included in the apparatus. It should be understood that these are also included in the scope of this disclosure.

The above-described embodiment is not limited to the implementation as the system 1. For example, the above-described embodiment may be implemented as an electronic device 10 included in the system 1. Further, the above-described embodiment may be implemented as a monitoring method using a device such as an electronic device 10. Further, the above-described embodiment may be implemented as a program executed by, for example, a device such as an electronic device 10 or an information processing device (for example, a computer).

1 System 10 Electronic equipment 11 Recognition unit 13 Storage unit 132 Machine learning data 15 Controller 17 Warning unit 19 Communication unit 20 Imaging unit

Claims (17)

An image pickup unit that captures an image and
A recognition unit that recognizes a monitored object and a wheelchair related to the monitored object in the image captured by the imaging unit, and a recognition unit.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the image pickup unit. A controller that machine-learns the relationship between the positions of the monitored object and the wheelchair and the start of the operation in which the monitored object leaves the wheelchair.
A system equipped with. The controller has the start time point and the end time point based on the timing information including the time point other than the start time point and the end time point of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the image pickup unit. The system according to claim 1, wherein the system learns the relationship between the positions of the monitored object and the wheelchair with respect to the time point and the start of the operation in which the monitored object leaves the wheelchair. The system according to claim 1 or 2, wherein the recognition unit two-dimensionally recognizes the positions of the monitored object and the wheelchair for each image of a predetermined number of frames per second captured by the image pickup unit. .. In the timing information, the end point of the operation in which the monitored object leaves the wheelchair indicates the timing when the monitored object is separated from the wheelchair by a predetermined distance in the temporal image captured by the imaging unit. The system according to any one of claims 1 to 3. In the timing information, the end point of the operation in which the monitored object leaves the wheelchair is a predetermined time in a state where the monitored object is separated from the wheelchair by a predetermined distance in the temporal image captured by the imaging unit. The system according to claim 1, indicating the timing at which is elapsed. In the timing information, the time when the monitored object starts the operation of leaving the wheelchair unattended indicates the timing when the monitored object leaves the wheelchair in the temporal image captured by the imaging unit, claim 1. The system described in. An image pickup unit that captures an image and
A recognition unit that recognizes a monitored object and a wheelchair related to the monitored object in the image captured by the imaging unit, and a recognition unit.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, the controller that estimates the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition unit.
A system equipped with. The recognition unit recognizes the positions of the monitored object and the wheelchair from the images of a predetermined number of frames per unit time captured by the image pickup unit.
The system according to claim 7, wherein the controller estimates the start of an operation in which the monitored object leaves the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition unit. The eighth aspect of the present invention, wherein the controller outputs a predetermined warning signal after the monitored object estimates the start of the operation of leaving the wheelchair, and before the end of the operation of the monitored object leaving the wheelchair. System. The system according to claim 8, wherein the recognition unit two-dimensionally recognizes the positions of the monitored object and the wheelchair for each image of a predetermined number of frames per second captured by the image pickup unit. The system according to claim 8, wherein the controller determines the number of the frames so that the validity of the estimation of the start of the operation in which the monitored object leaves the wheelchair is more than a predetermined value. A recognition unit that recognizes the monitored object and the wheelchair related to the monitored object in the captured image, and
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the image pickup unit. A controller that machine-learns the relationship between the positions of the monitored object and the wheelchair and the start of the operation in which the monitored object leaves the wheelchair.
Electronic equipment equipped with. A recognition unit that recognizes the monitored object and the wheelchair related to the monitored object in the captured image, and
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, the controller that estimates the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition unit.
Electronic equipment equipped with. An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the imaging step. A machine learning step for machine learning the relationship between the position of the monitored object and the wheelchair and the start of the operation of the monitored object leaving the wheelchair.
How to control electronic devices, including. An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, an estimation step of estimating the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition step, and an estimation step.
How to control electronic devices, including. On the computer
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
The subject between the start time and the end time based on the timing information indicating the start time and the end time of the operation in which the monitored object leaves the wheelchair in the time-dependent image captured by the imaging step. A machine learning step for machine learning the relationship between the position of the monitored object and the wheelchair and the start of the operation of the monitored object leaving the wheelchair.
A program that runs. On the computer
An imaging step to capture an image and
A recognition step for recognizing a monitored object and a wheelchair related to the monitored object in the image captured by the imaging step.
Machine learning was performed on the relationship between the positions of the human and the wheelchair between the start and end points of the movement in which the human-related wheelchair is left unattended, and the start of the movement in which the human leaves the wheelchair. Based on the machine learning data, an estimation step of estimating the start of the operation of the monitored object leaving the wheelchair from the positions of the monitored object and the wheelchair recognized by the recognition step, and an estimation step.
A program that runs.

Images