JP2022050268A - System, electronic apparatus, method for controlling electronic apparatus, and program - Google Patents

Abstract

To provide a system, an electronic apparatus, a method for controlling an electronic apparatus, and a program that can contribute to the safety of the surroundings of a subject to be monitored.SOLUTION: A system comprises: an imaging unit; an extraction unit; and a controller. The imaging unit picks up images of a subject to be monitored and the other person or an object related to the other person. The extraction unit extracts the coordinates of predetermined parts of the subject to be monitored and the coordinates of predetermined parts of the other person or the coordinates of the object related to the other person from the images picked up by the imaging unit. The controller, based on timing information indicating a starting point and an ending point of the behavior of the subject to be monitored of using violence on the other person in a sequential image picked up by the imaging unit, performs machine learning of the relationship between the coordinates of the predetermined parts of the subject to be monitored and the coordinates of the predetermined parts of the other person or the coordinates of the object related to the other person between the starting point and the ending point of the behavior of using violence and the start of the behavior of the subject to be monitored of using violence on the other person.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to systems, electronic devices, control methods for electronic devices, and programs.

For example, in a field such as a long-term care facility, a device for monitoring the behavior of a monitored person such as a nurse or a care recipient has been proposed. For example, Patent Document 1 discloses a monitored person system that detects a predetermined behavior in a monitored person based on an image obtained by an image pickup device. Patent Document 2 discloses a fall prevention system that prevents a subject from falling while walking by attaching a detection device to the foot of the subject. Further, Patent Document 3 discloses a monitoring support device for determining the body position of a human body by detecting a temperature distribution. Cited Document 4 also discloses a medical system for monitoring elderly psychotic patients at home or in elderly housing with care or long-term care facilities.

Japanese Unexamined Patent Publication No. 2017-91552 Japanese Unexamined Patent Publication No. 2017-221502 Japanese Unexamined Patent Publication No. 2014-106636 Japanese Patent Application Laid-Open No. 2003-91790

It would be beneficial if the monitored object could be monitored to ensure the safety of the surroundings of the monitored object.

An object of the present disclosure is to provide a system, an electronic device, a control method for the electronic device, and a program that can be safely provided around the monitored object.

The system according to one embodiment includes an imaging unit, an extraction unit, and a controller. The imaging unit captures a monitored object and another person or an object related to the other person. The extraction unit extracts the coordinates of the predetermined portion of the monitored object, the coordinates of the predetermined portion of the other person, or the coordinates of the object related to the other person from the image captured by the imaging unit. The controller has the start time and the start time based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the image pickup unit. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person between the end point and the time point, and the monitored object with respect to the other person. Machine learning the relationship with the start of violent movements.

In addition, the system according to one embodiment is
An image pickup unit that captures an image of a monitored object and another person or an object related to the other person.
An extraction unit that extracts the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of another person, or the coordinates of an object related to the other person from the image captured by the imaging unit.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined portion of the monitored object extracted by the extraction unit based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. A controller that estimates the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
To prepare for.

The electronic device according to the embodiment is
From the image captured including the monitored object and another person or an object related to the other person, the coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or related to the other person. An extraction unit that extracts the coordinates of the object to be used,
The start time and the above are based on the timing information indicating the start time and the end time of the action in which the monitored object violently acts against the other person in the time-dependent image captured including the monitored object. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person between the end time point and the time point of the end, and the monitored object with respect to the other person. A controller that machine-learns the relationship with the initiation of violent movements,
To prepare for.

Further, the electronic device according to the embodiment is
From the image captured including the monitored object and another person or an object related to the other person, the coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or related to the other person. An extraction unit that extracts the coordinates of the object to be used,
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined portion of the monitored object extracted by the extraction unit based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. A controller that estimates the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
To prepare for.

The method for controlling an electronic device according to an embodiment is as follows.
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The start time and the end time are based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the imaging step. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person, and the monitored object violently attack the other person. Machine learning steps to machine learn the relationship with the start of motion,
including.

Further, the control method of the electronic device according to the embodiment is as follows.
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined part of the monitored object extracted by the extraction step based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. An estimation step of estimating the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
including.

The program according to one embodiment is
On the computer
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The start time and the end time are based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the imaging step. The coordinates of the predetermined part and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person, and the start of the action of the monitored object to violently attack the other person. Machine learning steps to machine learn the relationship between
To execute.

In addition, the program according to one embodiment is
On the computer
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined part of the monitored object extracted by the extraction step based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. An estimation step of estimating the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
To execute.

According to one embodiment, it is possible to provide a system, an electronic device, a control method for the electronic device, and a program that can be safely provided around the monitored object.

It is a functional block diagram which shows the schematic structure of the system which concerns on one Embodiment. It is a flowchart explaining the operation of the learning phase by the system which concerns on one Embodiment. It is a flowchart which shows the example of the image imaged by the system which concerns on one Embodiment. It is a flowchart which shows the example of the image imaged by the system which concerns on one Embodiment. It is a figure which shows the example of the predetermined part extracted by the system which concerns on one Embodiment. It is a figure which shows the example of the predetermined part extracted from the image imaged by the system which concerns on one Embodiment. It is a figure explaining the coordinates of the predetermined part extracted by the system which concerns on one Embodiment. It is a figure which shows the example of the predetermined part extracted from the image imaged by the system which concerns on one Embodiment. It is a figure which shows the example of the predetermined part extracted from the image imaged by the system which concerns on one Embodiment. It is a figure explaining the timing information acquired by the system which concerns on one Embodiment. It is a flowchart explaining the operation of the estimation phase by the system which concerns on one Embodiment. It is a figure explaining the estimation by the system which concerns on one Embodiment.

In the present disclosure, the "electronic device" may be a device driven by electric power. Further, the "system" may include a device driven by electric power. Further, the "user" may be a person (typically a human being) who uses the system and / or the electronic device according to the embodiment. The user may include a person who monitors the monitored object by using the system and / or the electronic device according to the embodiment. Further, the “monitored target” may be a person (for example, a human or an animal) to be monitored by the system and / or the electronic device according to the embodiment. Further, the user may include a monitored object.

It is assumed that the system according to one embodiment is used in a specific situation used by a person engaged in social activities such as a company, a hospital, an elderly home, a school, a sports gym, and a long-term care facility. It may be a facility. For example, in the case of a company, it is extremely important to understand and / or manage the health status of employees. Similarly, it is extremely important to understand and / or manage the health status of patients and medical staff in hospitals, and residents and staff in elderly housing with care. The scene in which the system according to the embodiment is used is not limited to the above-mentioned facilities such as companies, hospitals, and elderly housings, but any facility where it is desired to grasp and / or manage the health condition of the monitored object. May be. Any facility may also include non-commercial facilities, such as the user's home. Further, the scene in which the system according to one embodiment is used may be, for example, a moving body such as a train, a bus, or an airplane, a station, a platform, or the like.

The system according to one embodiment may be used for monitoring the behavior of a monitored object such as a nurse-requiring person or a nursing-requiring person in a nursing care facility or the like. The system according to one embodiment monitors the behavior of a monitored object, such as a person requiring nursing care or a person requiring long-term care, to violently attack another person such as a staff member who cares for or cares for the monitored object. can do. The act of wielding violence may include the act of hitting, hitting, pushing, kicking, ramming, or biting another person. The act of wielding violence may include the act of pushing or rocking a chair or wheelchair on which another person is sitting. The violent movement is not limited to these, and may include various other movements.

In particular, in the system according to one embodiment, when a monitored target such as a nurse or a care recipient is approaching another person, the monitored target violently acts against the other person. A predetermined warning can be issued before the operation starts or ends. Therefore, according to the system according to one embodiment, the staff of a nursing care facility, for example, a monitored object such as a nurse or a person requiring long-term care violently or further violently acts against another person. Before continuing, it is possible to recognize that the monitored subject is or is trying to violence against another person.

Hereinafter, the system according to the embodiment will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing a schematic configuration of a system according to an embodiment. As shown in FIG. 1, the system 1 according to an embodiment may be configured to include an electronic device 10 and an image pickup unit 20. The electronic device 10 and the image pickup unit 20 may be connected by wire or wireless, or a combination of wire and wireless. The system 1 according to one embodiment may not include a part of the functional unit shown in FIG. 1, or may include a functional unit other than that shown in FIG. For example, the system 1 according to the embodiment may not include at least one of the warning unit 17 and the communication unit 19. Further, for example, the system 1 according to the embodiment may include a display capable of displaying an image and / or a slot into which a storage such as a memory card can be inserted.

The image pickup unit 20 shown in FIG. 1 may include an image sensor that electronically captures an image, such as a digital camera. The image pickup unit 20 may include an image pickup element that performs photoelectric conversion, such as a CCD (Charge Coupled Device Image Sensor) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image pickup unit 20 may take an image of the monitored object T, for example, as shown in FIG. Here, the monitored target T may be, for example, a human being. The image pickup unit 20 may convert the captured image into a signal and transmit it to the electronic device 10. For example, the image pickup unit 20 may transmit a signal based on the captured image to the extraction unit 11, the storage unit 13, and / or the controller 15 of the electronic device 10. The image pickup unit 20 is not limited to an image pickup device such as a digital camera as long as it captures the monitored target T, and may be any device. The image pickup unit 20 is connected to the electronic device 10 by wire, wirelessly, or any combination thereof.

In one embodiment, the image pickup unit 20 may take an image of the monitored target T, for example, as a still image at predetermined time intervals (for example, 15 frames per second). Further, in one embodiment, the imaging unit 20 may image, for example, the monitored target T as a continuous moving image.

As shown in FIG. 1, the electronic device 10 according to the embodiment may include an extraction unit 11, a storage unit 13, a controller 15, a warning unit 17, and a communication unit 19. The electronic device 10 according to the embodiment may not include a part of the functional unit shown in FIG. 1, or may include a functional unit other than that shown in FIG. For example, the electronic device 10 according to the embodiment may include machine learning data 132, which will be described later, stored in the storage unit 13. For example, in the electronic device 10 according to the embodiment, at least a part of the machine learning data 132 described later may be stored in an external device such as an external server.

The extraction unit 11 may have a function of extracting a predetermined feature point from the image captured by the image pickup unit 20. For example, the extraction unit 11 may extract the coordinates of a feature point such as a predetermined portion of the body of the monitored object T from the image of the monitored object T captured by the imaging unit 20. Here, the feature points will be further described later. In one embodiment, the extraction unit 11 obtains the coordinates of each part such as the head, trunk, limbs, and / or joints of the monitored object T from the image of the monitored object T captured by the imaging unit 20. It may be extracted. The extraction unit 11 may be configured as dedicated hardware, may be configured to include software at least in part, or may be configured entirely by software. In this way, the extraction unit 11 may extract the coordinates of the predetermined portion of the monitored target T from the image captured by the image pickup unit 20. The extraction unit 11 may extract the coordinates of a feature point such as a predetermined part in the body of the other person W from the image of the other person W (see FIG. 3 or the like). The extraction unit 11 may extract the coordinates of a predetermined portion of another person W from the image captured by the image pickup unit 20. The extraction unit 11 may recognize a chair or a wheelchair on which another person W is sitting from the image captured by the image pickup unit 20. A chair or wheelchair on which another person W sits is also referred to as an object related to the other person W.

The storage unit 13 may have a function as a memory for storing various types of information. The storage unit 13 may store, for example, a program executed by the controller 15, the result of processing executed by the controller 15, and the like. Further, the storage unit 13 may function as a work memory of the controller 15. The storage unit 13 can be configured by, for example, a semiconductor memory or the like, but is not limited to this, and can be any storage device. For example, the storage unit 13 may be a storage medium such as a memory card inserted in the electronic device 10 according to the embodiment. Further, the storage unit 13 may be the internal memory of the CPU used as the controller 15 described later, or may be connected to the controller 15 as a separate body.

As shown in FIG. 1, the storage unit 13 may store, for example, machine learning data 132. Here, the machine learning data 132 may be data generated by machine learning. Further, machine learning may be based on AI (Artificial Intelligence) technology that enables a specific task to be executed by training. More specifically, machine learning may be a technique in which an information processing device such as a computer learns a large amount of data and automatically constructs an algorithm or model for performing tasks such as classification and / or prediction. In the present specification, a part of AI (Artificial Intelligence) may include machine learning. In the present specification, machine learning may include supervised learning that learns the features or rules of input data based on correct answer data. Further, the machine learning may include unsupervised learning in which the features or rules of the input data are learned in the absence of correct answer data. Further, the machine learning may include reinforcement learning for learning the features or rules of the input data by giving a reward or a punishment. Further, in the present specification, machine learning may be any combination of supervised learning, unsupervised learning, and reinforcement learning. The concept of machine learning data 132 of the present embodiment may include an algorithm that outputs a predetermined inference (estimation) result by using an algorithm learned for input data. In this embodiment, as this algorithm, for example, linear regression that predicts the relationship between the dependent variable and the independent variable, a neural network (NN) that mathematically models a human brain nervous system neuron, and a least squares that squares an error are calculated. Other appropriate algorithms such as multiplication, decision trees that make problem solving a tree structure, and regularization that transforms data in a predetermined way can be used. In this embodiment, deep learning, which is a kind of neural network, may be used. Deep learning is a type of neural network, and a neural network with a deep network hierarchy is called deep learning.

In the technique of the present disclosure, it may be assumed that there is generally a certain relationship between the movement a of the body of the monitored target T and the movement result A of the monitored target T generated from this movement a. The operation result here may include the operation of the monitored target T, the operation start time of the monitored target T, an accident, an incident or other event generated from the operation of the monitored target T. For example, it is assumed that the body movement a of the monitored target T is performed, and the movement result A of the monitored target T is generated from this movement a. Further, it is assumed that the body movement b of the monitored target T is performed, and the movement result B of the monitored target T is generated from this movement b. The technique of the present disclosure accumulates the relationship between the operation a and the operation result A, the operation b and the operation result B, and other operations and the operation result as machine learning data. Then, in the technique of the present disclosure, when the motion x is extracted, the motion result X related to the motion x may be estimated by using the machine learning data.

In particular, in one embodiment, the machine learning data 132 may be machine-learned data on the movement of feature points when the monitored target T behaves violently against another person W. Hereinafter, the action in which the monitored target T violently acts against another person W may be referred to as "violent action". Further, the machine learning data 132 is data obtained by machine learning the movement of a feature point when a specific person (for example, a specific person requiring nursing care) acts violently against another person W as a monitored target T. May be. The machine learning data 132 according to the embodiment will be further described later.

The controller 15 controls and / or manages the entire electronic device 10 including each functional unit constituting the electronic device 10. The controller 15 may include at least one processor, such as a CPU (Central Processing Unit), to provide control and processing power to perform various functions. The controller 15 may be realized collectively by one processor, by several processors, or by individual processors. The processor may be realized as a single integrated circuit. The integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be realized as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented on the basis of various other known techniques.

In one embodiment, the controller 15 may be configured as, for example, a CPU and a program executed by the CPU. The program executed by the controller 15, the result of the process executed by the controller 15, and the like may be stored in, for example, the storage unit 13. The controller 15 may appropriately include a memory necessary for the operation of the controller 15. The operation of the controller 15 of the electronic device 10 according to the embodiment will be further described later.

The warning unit 17 may issue a predetermined warning for calling attention to the user of the system 1 or the electronic device 10 based on the predetermined warning signal output from the controller 15. As a predetermined warning, the warning unit 17 may be any functional unit that stimulates at least one of the user's auditory sense, visual sense, and tactile sense, such as sound, voice, light, characters, video, and vibration. Specifically, the warning unit 17 may be at least one of, for example, an audio output unit such as a buzzer or a speaker, a light emitting unit such as an LED, a display unit such as an LCD, and a tactile presentation unit such as a vibrator. .. In this way, the warning unit 17 may issue a predetermined warning based on a predetermined warning signal output from the controller 15. In one embodiment, the warning unit 17 may issue a predetermined alarm as information acting on at least one of auditory, visual, and tactile sensations.

In one embodiment, the warning unit 17 has a risk that the monitored target T starts to violently attack the other person W, for example, before the monitored target T starts to violently attack the other person W. You may issue a warning that there is. Further, in one embodiment, the warning unit 17 continues to violence against the other person W before the monitored object T finishes the action of violently acting against the other person W. You may issue a warning that there is a risk. For example, in one embodiment, when the warning unit 17 that outputs visual information detects that the monitored target T has a risk of starting or continuing to violence against another person W, The user may be warned by light emission or a predetermined display to that effect. Further, in one embodiment, when the warning unit 17 that outputs auditory information detects that the monitored target T has a risk of starting or continuing to violence against another person W, The user may be warned to that effect by a predetermined sound or voice. In the present embodiment, the warning may be a combination of light emission or a predetermined display, and a predetermined sound or voice.

The electronic device 10 shown in FIG. 1 has a built-in warning unit 17. However, in one embodiment, in the system 1, the warning unit 17 may be provided outside the electronic device 10. In this case, the warning unit 17 and the electronic device 10 may be connected by wire or wireless, or a combination of wire and wireless.

The communication unit 19 has an interface function for communicating by wire or wirelessly. The communication method performed by the communication unit 19 of one embodiment may be a wireless communication standard. For example, wireless communication standards include communication standards for cellular phones such as 2G, 3G, 4G, and 5G. For example, the communication standards for cellular phones are LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), CDMA2000, PDC (Personal Digital Cellular), GSM (Registered Trademark) (Global System for Mobile communications), and PHS. (Personal Handy-phone System) etc. are included. For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), 802.11, WiFi, Bluetooth®, IrDA (Infrared Data Association), NFC (Near Field Communication) and the like. The communication unit 19 can support one or more of the above communication standards. The communication unit 19 may be configured to include, for example, an antenna for transmitting and receiving radio waves, an appropriate RF unit, and the like. Further, the communication unit 19 may be configured as an interface such as a connector for making a wired connection to the outside. Since the communication unit 19 can be configured by a known technique for performing wireless communication, more detailed description of hardware and the like will be omitted.

Various types of information received by the communication unit 19 may be supplied to, for example, the storage unit 13 and / or the controller 15. Various information received by the communication unit 19 may be stored in, for example, a memory built in the storage unit 13 and / or the controller 15. Further, the communication unit 19 may transmit, for example, the processing result by the controller 15, the extraction result by the extraction unit 11, and / or the information stored in the storage unit 13 to the outside.

As shown in FIG. 1, at least a part of each functional unit constituting the electronic device 10 according to the embodiment may be configured by a specific means in which software and hardware resources cooperate.

Next, the operation of the system 1 according to the embodiment will be described.

The operation of the system 1 according to one embodiment can be typically divided into a "learning phase" and an "estimation phase". In the learning phase, for example, the position (coordinates) of each part of the body in the action of a human being violently acting against another person, such as the monitored target T, and the position (coordinates) of each part of the body of another person W. Alternatively, the motion of machine learning the relationship between the position (coordinates) of the object related to the other person W and the timing of the violent motion may be performed. Further, in the estimation phase, based on the result of machine learning in the learning phase, the position (coordinates) of each part of the body of the monitored target T in the movement in which the monitored target T violently acts against another person W, and , The movement of estimating the start of the violent movement may be performed from the position (coordinates) of each part of the body of the other person W or the position (coordinates) of the object related to the other person W. Hereinafter, the operation in each of the above-mentioned learning phase and estimation phase will be described in more detail. First, the operation in the learning phase will be described. In the present disclosure, for example, there may be a plurality of systems 1 shown in FIG. 1, and the systems that carry out the "learning phase" and the "estimation phase" may be different.

FIG. 2 is a flowchart showing an example of the operation in the learning phase of the system 1 according to the embodiment. FIG. 2 may be a flowchart focusing on the operation of the electronic device 10 included in the system 1 according to the embodiment in the learning phase.

For example, a person suspected of developing dementia (for example, a nurse or a care recipient) may violence against another person W such as a nurse or a caregiver. In such a case, another person W may be injured. In addition, another person W feels uncomfortable or dangerous. That is, there is a risk that the safety of another person W will be threatened. Therefore, to warn another person that violence is about to be violent against another person W or that violence has been violent against another person W is to warn others around the monitored target T. Not only can it be used for the safety of the person W, but it can also be used for the safety of the surrounding environment of the monitored target T. In the learning phase of the system 1 according to the embodiment, as described above, the position of each part of the body of the monitored object T in the action of a human being such as the monitored object T violently acting against another person W ( Coordinates) and the position (coordinates) of each part of the body of the other person W or the position (coordinates) of the object related to the other person W may be machine-learned about the relationship between the timing of the violent movement. Hereinafter, such an operation will be described in more detail.

At the time when the operation shown in FIG. 2 starts, the imaging unit 20 of the system 1 may start imaging a human such as the monitored target T. The time point at which the operation shown in FIG. 2 starts may be the time when the image pickup unit 20 starts image pickup of a human such as the monitored target T. Further, the time point at which the operation shown in FIG. 2 starts may be the time when a person such as the monitored target T enters the image pickup range of the image pickup unit 20 after the image pickup unit 20 starts image pickup.

When the operation shown in FIG. 2 starts, the controller 15 of the electronic device 10 acquires the image captured by the image pickup unit 20 (step S11).

FIG. 3 is a diagram showing an example of an image acquired by the controller 15 in step S11 shown in FIG. 2, that is, an image captured by the imaging unit 20.

As shown in FIG. 3, the image pickup unit 20 may take an image of a state in which a person such as a monitored target T is approaching another person W. FIG. 3 schematically shows how a person such as the monitored target T is approaching another person W. In this case, in step S11, the controller 15 acquires an image in which a person such as the monitored target T is approaching another person W. As will be described later, the image pickup unit 20 may capture an image in a state other than the state in which a person such as the monitored target T is approaching another person W. In this case, in step S11, the controller 15 may acquire an image of a state other than the state in which a person such as the monitored target T is approaching another person W.

FIG. 4 is a diagram showing another example of the image acquired by the controller 15 in step S11 shown in FIG. 2, that is, the image captured by the imaging unit 20. As shown in FIG. 4, the image pickup unit 20 may take an image of a state in which a person such as a monitored target T is hitting another person W. FIG. 4 schematically shows how the monitored target T is hitting another person W. In FIG. 4, the monitored target T and / or another person W may be moving or stationary. In this case, in step S11, the controller 15 acquires an image including a human being such as the monitored target T and another person W. As will be described later, the image pickup unit 20 may capture an image in a state other than the state in which a person such as the monitored target T is hitting another person W. In this case, in step S11, the controller 15 may acquire an image of a state other than the state in which a person such as the monitored target T is hitting another person W.

The image pickup unit 20 may capture each image of a predetermined number of frames per second. Here, the image captured by the image pickup unit 20 may be a still image of continuous frames or a moving image. For example, the image pickup unit 20 may capture an image of 15 frames per second. In step S11, the controller 15 may acquire images of a predetermined number of frames per second captured by the image pickup unit 20.

As shown in FIG. 2, when the image captured in step S11 is acquired, the extraction unit 11 extracts the coordinates of a predetermined portion of the body of the monitored target T (step S12). The operation in step S12 may be performed by the controller 15 instead of the extraction unit 11. The extraction unit 11 further extracts not only the coordinates of the predetermined portion of the body of the monitored target T but also the coordinates of the predetermined portion of the body of the other person W. The extraction unit 11 may further extract the coordinates of the object related to the other person W.

FIG. 5 is a diagram showing an example of a predetermined portion of the body of the monitored target T extracted in step S12.

In step S12, the extraction unit 11 may extract the coordinates of a predetermined portion of the body of the monitored target T as shown in FIG. 5, for example. As shown in FIG. 5, the predetermined portion from which the coordinates are extracted in step S12 may include, for example, the neck, left shoulder, left elbow, left wrist, right shoulder, right elbow, and right wrist in the body of the monitored object T. .. Further, as shown in FIG. 5, the predetermined portion for which the coordinates are extracted in step S12 further includes, for example, the left hip, left knee, left ankle, right hip, right knee, and right ankle in the body of the monitored target T. It's fine. As described above, the coordinates of the predetermined portion extracted in step S12 may be the coordinates of the predetermined joint point in the body of the monitored target T or the like.

FIG. 6 is a diagram showing an example of coordinates extracted as a predetermined part of the body of the monitored target T in the image shown in FIG. The image of the monitored object T shown in FIG. 6 is the same as the image of the monitored object T shown in FIG. FIG. 6 shows the coordinates extracted as a predetermined part of the body of the monitored object T shown in FIG. 5 in the image of the monitored object T shown in FIG.

In step S12, the extraction unit 11 extracts the coordinates of the plurality of dots shown in FIG. 6 as predetermined parts of the body of the monitored target T shown in FIG. For example, the extraction unit 11 may two-dimensionally extract the coordinates of the plurality of dots shown in FIG. 6 according to the coordinate axes shown in FIG. That is, in the extraction unit 11, the lower left end portion of the image pickup range of the image captured by the image pickup unit 20 may indicate the origin of the coordinate axis shown in FIG. For example, the extraction unit 11 acquires the coordinates of the position of the neck of the monitored target T shown in FIG. 6 according to the coordinate axes shown in FIG.

Here, when the imaging unit 20 captures each image of a predetermined number of frames per second, the extraction unit 11 may extract the image as a predetermined portion of the body of the monitored target T in the predetermined number of frames per second. Further, even when the controller 15 acquires images of a predetermined number of frames per second, the extraction unit 11 may extract the images of the monitored target T as a predetermined part in the body in the predetermined number of frames per second. As an example, the extraction unit 11 may extract a predetermined portion of the body of the monitored target T at 15 frames per second.

FIG. 7 is a diagram showing the coordinates of predetermined parts extracted in the body of the monitored target T collectively, for example, in 15 frames per second. As shown in FIG. 7, in step S12, the controller 15 (or the extraction unit 11) may arrange the coordinates of the predetermined parts extracted in the body of the monitored target T side by side for each frame. As shown in FIG. 7, the extraction unit 11 may extract the coordinates of a predetermined portion two-dimensionally (as X and Y coordinates) in the body of the monitored target T for each frame. In the table shown in FIG. 7, each row schematically shows how a predetermined part of the body of the monitored target T is extracted as X and Y coordinates in each frame. Also, in the table shown in FIG. 7, the rows showing each frame are shown from top to bottom over time. The coordinates of the 15 frames shown in FIG. 7 may be, for example, the coordinates of one second in the image (or moving image) as shown in FIG. 6 tracked. Further, even after the 15 frames shown in FIG. 7, the coordinates of a predetermined portion in the body of the monitored target T may be sequentially extracted.

As described above, in one embodiment, the extraction unit 11 two-dimensionally obtains the coordinates of a predetermined number of joint points in the body of the monitored target T from the images of the predetermined number of frames per second captured by the image pickup unit 20. It may be extracted.

The coordinates of the predetermined portion in the body of the other person W may be extracted in the same manner as the coordinates of the predetermined portion in the body of the monitored target T. The coordinates of the object related to the other person W may be further extracted. Further, as shown in FIG. 6, not only the coordinates when the violence in the mode in which the monitored target T is hitting the other person W is being performed, but also the violence in the case where the violence in another mode is being performed. Coordinates may be extracted.

FIG. 8 is a diagram showing another example of the monitored target T and another person W recognized in step S12. FIG. 8 is a diagram schematically showing how the monitored target T is biting the other person W, and the feature points obtained as a result of the extraction unit 11 recognizing the monitored target T and the other person W. It is the figure which superposed and displayed.

FIG. 9 is a diagram showing another example of the wheelchair C in which the monitored target T and another person W recognized in step S12, and the other person W are sitting. FIG. 9 is a diagram schematically showing how the monitored target T strongly pushes the wheelchair C on which the other person W is sitting, and the extraction unit 11 shows the monitored target T, the other person W, and the wheelchair. It is a figure which superposed and displayed the feature points obtained as a result of recognizing C.

In step S12, the extraction unit 11 may recognize not only FIG. 6 but also predetermined feature points in the monitored target T and the other person W as shown in FIGS. 8 and 9. Further, the extraction unit 11 may two-dimensionally extract the monitored target T and another person W shown in FIGS. 8 and 9 according to the coordinate axes shown in FIGS. 8 and 9. Further, the extraction unit 11 may extract the positions (for example, coordinates) of the monitored target T and the other person W in the image when recognizing the monitored target T and the other person W. Further, the extraction unit 11 may extract the wheelchair C as an object related to the other person W, and extract the coordinates of the wheelchair C or the coordinates of the feature point of the wheelchair C.

After the coordinates of the predetermined portion are extracted in step S12, the extraction unit 11 extracts the coordinates according to the maximum and minimum values of the coordinates (X, Y) in the extracted predetermined number of frames (for example, 15 frames per second). Normalize (step S13).

It is assumed that the coordinates of the predetermined portion extracted in step S12 vary due to, for example, the size of the body of the monitored target T. Further, it is assumed that the coordinates of the predetermined portion extracted in step S12 vary due to, for example, the distance between the image pickup unit 20 and the monitored target T, the direction from the image pickup unit 20 toward the monitored target T, and the like. Will be done. Therefore, in one embodiment, by normalizing the X-direction component and the Y-direction component of the coordinates extracted in step S12, the extracted coordinates can be used for machine learning in general.

In this case, for example, the extracted X and Y coordinates may be normalized based on the maximum and minimum values of the extracted X and Y coordinates in 15 frames per second. Here, the maximum value of the X coordinate extracted in step S12 is Xmax, and the minimum value of the X coordinate extracted in step S12 is Xmin. Further, the maximum value of the X coordinate after normalization is set to X'max. In this case, the X coordinate (X) before normalization can be converted into the X coordinate (X') after normalization by using the following equation (1).

X'= ((X-Xmin) / (Xmax-Xmin)) · X'max (1)

Similarly, the maximum value of the Y coordinate extracted in step S12 is Ymax, and the minimum value of the Y coordinate extracted in step S12 is Ymin. Further, the maximum value of the Y coordinate after normalization is set to Y'max. In this case, the Y coordinate (Y) before normalization can be converted into the Y coordinate (Y') after normalization by using the following equation (2).

Y'= ((Y-Ymin) / (Ymax-Ymin)) · Y'max (2)

By normalizing the X-direction component and the Y-direction component of the extracted coordinates according to the above equations (1) and (2), the individual difference of the monitored target T and the imaging unit 20 determine the monitored target T. It can be expected to reduce the influence of the imaged environment on machine learning.

As described above, in one embodiment, the extraction unit 11 sets each direction component of the coordinates of a predetermined number of joint points in the body of the monitored target T two-dimensionally extracted to the maximum value and the minimum value of each direction component. It may be normalized based on the value. Further, such an operation may be performed by the controller 15 instead of the extraction unit 11.

By normalizing the coordinates shown in step S13, each of the coordinates (X, Y) shown in FIG. 7 is normalized to the coordinates (X', Y'), respectively.

The coordinates of the predetermined part in the body of the other person W may be normalized in the same manner as the coordinates of the predetermined part in the body of the monitored object T. The coordinates of the object associated with the other person W may be further normalized.

When the coordinates are normalized in step S13, the controller 15 acquires the timing information (step S14). In order for the controller 15 to acquire the timing information in step S14, the timing information needs to be added (set) in advance to the image acquired in step S11 or the coordinates extracted in step S12. Further, timing information may be given (set) in advance at the coordinates normalized in step S13 instead of the coordinates extracted in step S12. Hereinafter, such timing information will be further described.

In one embodiment, as data for the electronic device 10 to perform machine learning, information indicating the start time point and the end time point of an action in which a human being, such as a monitored target T, violently acts against another person W is prepared. There is a need. If there is information (timing information) indicating the start time and end time of the operation in which the monitored target T violently acts against another person W, the electronic device 10 performs machine learning using this timing information as, for example, teacher data. be able to. As described above, the information indicating the start time point and the end time point of the action in which the monitored target T violently acts against the other person W is also referred to as "timing information". That is, the "timing information" may be information indicating the start time point and the end time point of the violent movement in the time-dependent image captured by the image pickup unit 20.

Such timing information may be given (set) by personnel such as staff. That is, for example, an ethology expert or a staff member of a long-term care facility observes (visually recognizes) the image of the monitored target T captured by the imaging unit 20, and obtains timing information indicating the start time and end time of the violent movement. May be given. In addition, by setting a predetermined standard for giving such timing information in advance, it is possible to give timing information even to general personnel who are not ethology specialists or staff of long-term care facilities. can. As described above, the timing information may be added (set) in the image data acquired in step S11. Further, the timing information may be added (set) in the coordinate data extracted in step S12. Further, the timing information may be given (set) in the coordinate data normalized in step S13.

FIG. 10 is a diagram illustrating the setting of timing information. FIG. 10 conceptually shows how the image data is continuous with the passage of time for each frame. That is, in FIG. 10, it is shown that the image data is continuous like frame 1, frame 2, frame 3, ... With the passage of time. Further, FIG. 10 shows an example of adding (setting) timing information to the image data. However, as described above, the image data shown in FIG. 10 may be replaced with the coordinate data or may be replaced with the normalized coordinate data.

At the time of "entry" shown in FIG. 10, for example, the state in which the monitored target T entered the room in which the image pickup unit 20 was installed was shown in the image data (image data) captured by the image pickup unit 20. do. It is assumed that the image data shows that the monitored target T has started the action of violently acting against the other person W at the time of "start of the violent action" shown in FIG. It is assumed that the image data shows that the monitored target T has finished the action of violently acting against the other person W at the time of "the end of the violent action" shown in FIG. It is assumed that the image data shows that the monitored target T has left the room in which the image pickup unit 20 is installed, for example, at the time of “leaving the room” shown in FIG.

In the image data captured as described above, at least the time of "beginning of violent movement" and the time of "end of violent movement" are determined by, for example, an ethology expert, a staff member of a long-term care facility, or other general personnel. The indicated timing information may be given (set).

Here, the time point of "start of violent movement" is the time point when the monitored target T who has entered the room in which the image pickup unit 20 is installed starts to move closer to another person W as shown in FIG. 3, for example. May be.

Generally, when the monitored target T starts a violent action in a manner of hitting another person W as shown in FIG. 6, the feature point Ta representing the wrist of the monitored target T is at least a part of the other person W. Begin to approach the feature points of. Therefore, in one embodiment, a person such as a staff member "starts a violent action" when the feature point Ta representing the wrist of the monitored target T begins to approach at least a part of the feature points of the other person W. Timing information may be added (set) as the time point of. For example, in FIG. 6, a person such as a staff member "violence" when the distance D0 between the feature point Ta of the wrist of the monitored target T and the feature point Wa representing the elbow of another person W changes below a predetermined threshold value. Timing information may be added (set) as the time point of "start of operation". Further, for example, in FIG. 6, a person such as a staff member has a “start of violent action” at a time when the range in which the monitored target T spreads and the range in which another person W spreads change from a state in which they do not overlap to a state in which they overlap. Timing information may be added (set) as a time point. Further, when the monitored target T starts a violent action in a manner of hitting another person W, the monitored target T may swing his arm up. Personnel such as staff give (set) timing information as the time when the feature point Ta representing the wrist of the monitored target T rises to a position higher than the feature point representing the elbow as the time point of "start of violent movement". You may.

Further, when the monitored target T starts a violent motion in a manner of biting the other person W as shown in FIG. 8, the feature point Ta representing the upper part of the neck or the head of the monitored target T is the other person W. Start approaching at least some feature points. Therefore, in one embodiment, a staff member or the like refers to the time when the feature point Ta representing the upper part of the neck or the head of the monitored target T begins to approach at least a part of the feature points of the other person W. Timing information may be added (set) at the time of "start of violent movement". For example, in FIG. 8, in FIG. 8, the distance D0 between the feature point Ta representing the neck or head of the monitored target T and the feature point Wa representing the wrist of another person W changes to less than a predetermined threshold value. Timing information may be added (set) with the time point as the time point of "start of violent action". Further, for example, in FIG. 8, the time point at which the range in which the monitored target T spreads and the range in which the other person W spreads changes from the non-overlapping state to the overlapping state is defined as the “start of violent movement”. Timing information may be added (set) as a time point. Further, when the monitored target T starts a violent action in a manner of biting another person W, the monitored target T may bend the upper body unnaturally. Even if personnel such as staff members give (set) timing information, the time when the feature point representing the upper body of the monitored target T begins to move to the positional relationship representing forward bending is set as the time point of "start of violent movement". good.

Further, as shown in FIG. 9, when the monitored target T starts a violent movement by pushing or shaking the wheelchair C on which another person W is sitting to give an impact, it represents the wrist of the monitored target T. Feature point Ta begins to approach at least part of wheelchair C on which another person W is sitting. Therefore, in one embodiment, a person such as a staff member "starts a violent movement" when the feature point Ta representing the wrist of the monitored target T comes into contact with at least a part of the wheelchair C in which another person W is sitting. Timing information may be added (set) as the time point of. For example, in FIG. 9, a person such as a staff member “starts violent movement” when the feature point Ta representing the wrist of the monitored target T and the handle Ca of the wheelchair C on which another person W sits come into contact with each other. Timing information may be added (set) as a time point. Further, for example, in FIG. 9, a person such as a staff member changes from a state in which the range in which the monitored target T expands to a state in which the range in which the wheelchair C on which another person W sits expands does not overlap with each other. Timing information may be added (set) at the time of "start of violent movement".

As described above, in one embodiment, at the start time of the violent motion in the timing information, each feature point of the monitored target T is a feature of at least a part of the other person W in the temporal image captured by the image pickup unit 20. It may indicate the timing at which the point or at least a part of the chair or wheelchair on which the other person W is sitting approaches within a predetermined distance. Further, the start time of the violent movement in the timing information may indicate the timing when the position of each feature point of the monitored target T deviates from the normal movement in the time-dependent image captured by the image pickup unit 20.

Further, the time point of "end of violent movement" means that the monitored target T is a chair or a wheelchair on which another person W or another person W is sitting in the time-dependent image captured by the image pickup unit 20 (for example,). It may be a timing that can be judged to be separated (1 m, etc.). Therefore, in one embodiment, a person such as a staff member “violently operates” when the monitored object T is separated from another person W or a chair or wheelchair on which another person W is sitting by a predetermined distance such as 1 m. Timing information may be added (set) as the time point of "end of".

As described above, in one embodiment, the end point of the violent motion in the timing information is the chair in which the monitored target T is another person W or another person W in the time-dependent image captured by the image pickup unit 20. Alternatively, the timing may be indicated by a predetermined distance from the wheelchair.

Further, the time point of "end of violent movement" is a predetermined time (for example, 1 m) in a state where the monitored target T is separated from the chair or wheelchair on which the other person W or the other person W is sitting to some extent (for example, 1 m). It may be a timing when it can be judged that the minute) has passed. Therefore, in one embodiment, the staff or the like has elapsed a predetermined time with the monitored object T separated from the other person W or the chair or wheelchair on which the other person W sits by a predetermined distance, for example, 1 m. Timing information may be added (set) with the time point as the time point of "end of violent action".

As described above, in one embodiment, the end point of the violent motion in the timing information is the chair in which the monitored target T is another person W or another person W in the time-dependent image captured by the image pickup unit 20. Alternatively, it may indicate the timing at which a predetermined time has elapsed while being away from the wheelchair by a predetermined distance.

In the timing information, the time points of the above-mentioned "start of violent movement" and the time point of "end of violent movement" do not necessarily have to be given (set) in this order. That is, in the timing information, the time point of "end of violent action" may be set first, and then the time point of "start of violent action" may be set.

For example, it is not always easy for personnel such as staff to determine the time point of "beginning of violent movement" when reproducing (visually recognizing) the image of the monitored target T captured by the imaging unit 20. Is also assumed. In such a case, first, when the staff or the like reproduces (visually recognizes) the image of the monitored target T imaged by the image pickup unit 20, the timing of the "end of violent movement" is determined. It may be set as information. Next, a staff member or the like may reproduce (visually recognize) the image of the monitored object T captured by the image pickup unit 20 in the reverse time from the time of "the end of the violent movement". Here, when the image of the monitored target T captured by the image pickup unit 20 is reproduced in reverse in time, the speed of the reverse reproduction may be reduced to some extent. In this way, personnel such as staff can easily set the time point of "beginning of violent movement", which is the time point before the time of "end of violent movement", in the timing information. For example, a staff member or the like first moves the left or right buttock of the monitored object T to the front of the monitored object T on the seat surface of the chair while visually recognizing the image data reproduced in the reverse time. A time point may be found and the time point may be set as timing information.

When the timing information is added (set) in the image data (or the coordinate data or the normalized coordinate data) as described above, the controller 15 can acquire the timing information in step S14. In step S14, the controller 15 of the electronic device 10 may acquire the timing information as described above from an external network or the like via, for example, the communication unit 19. Further, in step S14, the controller 15 may acquire the timing information as described above from a slot of an electronic device 10 into which a storage such as a memory card can be inserted. Further, when the system 1 includes a display capable of displaying an image, image data or the like may be displayed on the display. In this case, the timing information set by a staff member or the like while visually recognizing the display may be input via, for example, an operation unit of the electronic device 10.

After acquiring the timing information in step S14, the controller 15 machine-learns the relationship between the coordinates extracted from the image data and the start of the violent movement based on the timing information (step S15). In step S15, the controller 15 determines the coordinates of the predetermined portion (joint point) of the monitored target T between the start time and the end time of the violent movement, and the coordinates of the predetermined portion (joint point) of the other person W. The relationship between the coordinates of the object related to the other person W and the start of the violent movement may be learned based on the timing information. Here, the controller 15 may perform machine learning based on the image data (or coordinate data or normalized coordinate data) and the timing information set in the image data. Hereinafter, the data generated as a result of machine learning in step S15 may be referred to as "machine learning data".

By performing the machine learning as described above, the electronic device 10 has the coordinates of the timing at which the violent movement starts and the coordinates of the predetermined portion (joint point) of the monitored target T between the start time and the end time of the violent movement. It is possible to grasp the relationship with movement. Therefore, according to the electronic device 10, in the estimation phase described later, the start time of the violent movement can be estimated based on the movement of the coordinates of the predetermined portion (joint point) of the monitored target T.

As described above, in one embodiment, the controller 15 has the coordinates of the predetermined portion of the monitored target T between the start time and the end time of the violent movement, and the coordinates of the predetermined portion (joint point) of the other person W. Alternatively, the relationship between the coordinates of the object related to the other person W and the start of the violent movement is machine-learned based on the timing information. Here, the timing information may be information indicating the start time point and the end time point of the violent movement in the time-dependent image captured by the image pickup unit 20 as described above. In the present disclosure, the number of relationships between the coordinates of the predetermined portion of the monitored target T between the start time and the end time of the violent act and the start of the violent act, which the controller 15 machine-learns, is, for example, 1 or more. , 10 or more, 100 or more, 1000 or more, etc., as long as it is at least 1 or more.

In order to improve the accuracy with which the electronic device 10 estimates the start time of violent movement in the estimation phase described later, even if machine learning is performed on the data of a relatively large number of samples (for example, a human such as a monitored target T). good. By increasing the sample data when performing machine learning, it can be expected that the accuracy when the electronic device 10 estimates the start time of the violent movement in the estimation phase described later will be improved. Therefore, by increasing the sample data when performing machine learning, it is possible to reduce the occurrence of false alarms or false alarms when the electronic device 10 estimates the start of violent movement and issues a warning. obtain. In the present disclosure, the number of samples machine-learned by the controller 15 may be at least 1 or more, for example, 1 or more, 10 or more, 100 or more, 1000 or more, and the like.

In the above-mentioned machine learning, the timing information is information indicating the start time point and the end time point of the violent movement in the time-dependent image captured by the image pickup unit 20. In one embodiment, the timing information may include information indicating a time point other than the start time point and the end time point of the violent movement in the time-dependent image captured by the image pickup unit 20.

For example, the image data of the section of the learning data (4) shown in FIG. 10 exists between the start time point and the end time point of the violent movement. Therefore, the learning data (4) may be machine-learned by the controller 15 as data classified into a dangerous class (that is, there is a high possibility of violent behavior). The image data of the section of the learning data (4) shown in FIG. 10 starts at substantially the same time as the start time of the violent movement. On the other hand, the image data in the section of the learning data (4) shown in FIG. 10 ends at a time different from the end time of the violent movement.

On the other hand, the image data of the sections of the learning data (1) to (3) and the learning data (5) shown in FIG. 10 do not exist between the start time and the end time of the violent movement. Therefore, these learning data are machine-learned by the controller 15 as data classified into a normal class (that is, there is little risk (the risk that the monitored target T violently acts against another person W)). You may let me. The image data of the section of these learning data starts at a time different from the start time of the violent movement and ends at a time different from the end time of the violent movement.

As described above, in one embodiment, the timing information may include information indicating a time point other than the start time point and the end time point of the violent movement in the time-dependent image captured by the image pickup unit 20. Based on such timing information, the controller 15 uses the coordinates of the monitored target T between the start time and the end time of the violent movement, the coordinates of the predetermined portion (joint point) of the other person W, or the other person W. Machine learning may be performed on the relationship between the coordinates of a predetermined part of the coordinates of the object related to the above and the start of the violent movement.

As shown in FIG. 10, the controller 15 has not only the learning data (4) classified into the dangerous class, but also the learning data (1) to (3) and the learning data (5) classified into the normal class. Machine learning may be performed based on the learning data such as. The controller 15 can improve the accuracy of estimating the start of violent movement by performing machine learning based on the learning data classified into the dangerous class and the learning data classified into the normal class.

Next, the operation in the estimation phase will be described.

FIG. 11 is a flowchart showing an example of the operation in the estimation phase of the system 1 according to the embodiment. FIG. 11 may be a flowchart focusing on the operation in the estimation phase of the electronic device 10 included in the system 1 according to the embodiment.

As described above, for example, a person suspected of developing dementia (for example, a nurse or a care recipient) may violence against another person W such as a nurse or a caregiver. .. In such a case, another person W may be injured. In addition, another person W feels uncomfortable or dangerous. That is, there is a risk that the safety of another person W will be threatened. In the estimation phase of the system 1 according to one embodiment, the monitored object T in an operation in which the monitored object T violently acts against another person W by using the machine learning data obtained in the learning phase. The start of the violent movement may be estimated from the position (coordinates) of the predetermined part on the body and the position (coordinates) of the predetermined part on the body of the other person W or the position (coordinates) of the object related to the other person W. Hereinafter, such an operation will be described in more detail. By monitoring the monitored target T by the system 1, for example, a staff member of a nursing care facility or a hospital can recognize that the violent movement starts before the monitored target T actually ends the violent movement. ..

When the operation shown in FIG. 11 starts, the controller 15 of the electronic device 10 acquires machine learning data (step S21). The machine learning data acquired in step S21 may be data generated as a result of machine learning in step S15 shown in FIG. That is, the machine learning data is the human body between the start time and the end time of the action in which a human such as the monitored target T violently acts against another human such as another person W. The coordinates of a predetermined part, the coordinates of a predetermined part in another human body, or the coordinates of an object related to another human such as another person W, and the start of an action in which the person violently acts against another person. The relationship may be machine-learned data.

If the machine learning data has already been acquired at the time when the operation shown in FIG. 11 starts, the controller 15 does not have to acquire the machine learning data again in step S21. Further, the machine learning data acquired in step S21 is used in step S25 described later. Therefore, in one embodiment, the acquisition of machine learning data does not necessarily have to be performed in step S21, and may be performed at any timing up to step 25.

When the machine learning data is acquired in step S21, the controller 15 acquires the image captured by the imaging unit 20 (step S22). The operation in step S22 may be performed in the same manner as the operation in step S11 shown in FIG.

When the image captured in step 22 is acquired, the extraction unit 11 extracts the coordinates of a predetermined portion of the body of the monitored target T (step S23). The extraction unit 11 further extracts not only the coordinates of the predetermined portion of the body of the monitored target T but also the coordinates of the predetermined portion of the body of the other person W. The extraction unit 11 may further extract the coordinates of the object related to the other person W. The operation in step S23 may be performed in the same manner as the operation in step S12 shown in FIG. That is, for example, the extraction unit 11 may two-dimensionally extract the coordinates of a predetermined number of joint points in the body of the monitored target T from the images of a predetermined number of frames per second captured by the image pickup unit 20. ..

After the coordinates are extracted in step S23, the extraction unit 11 normalizes the coordinates according to the maximum and minimum values of the coordinates (X, Y) in the extracted predetermined number of frames (for example, 15 frames per second). (Step S24). The operation in step S24 may be performed in the same manner as the operation in step S13 shown in FIG. That is, for example, the extraction unit 11 normalizes each direction component of the coordinates of a predetermined number of joint points in the body of the monitored target T extracted two-dimensionally based on the maximum value and the minimum value of each direction component. It may be transformed into. Further, the extraction unit 11 normalizes each direction component of the coordinates of a predetermined number of joint points in the body of another person W extracted two-dimensionally based on the maximum value and the minimum value of each direction component. May be good.

Before executing the procedure of step S24, the controller 15 keeps the monitored target T within the predetermined range of the other person W based on the coordinates of the predetermined portion of the monitored target T and the coordinates of the predetermined portion of the other person W. You may determine if you are approaching. The controller 15 determines whether the monitored object T is approaching within a predetermined range of the object related to the other person W based on the coordinates of the predetermined part of the monitored object T and the coordinates of the object related to the other person W. You may. The controller 15 may perform an operation to normalize the coordinates shown in step S24 when the monitored object T is approaching within a predetermined range of another person W or an object related to the other person W.

The controller 15 may calculate the distance between the coordinates for at least a part combination of the coordinates of at least a part of the predetermined part of the monitored object T and the coordinates of at least a part of the predetermined part of the other person W. The controller 15 may calculate the distance between the coordinates for all combinations of the coordinates of all the predetermined parts of the monitored target T and the coordinates of all the predetermined parts of the other person W. The controller 15 may determine whether the monitored target T has approached another person W based on the calculation result of the distance between the coordinates. The controller 15 may calculate the distance between the coordinates for the combination of the coordinates of at least a part of the predetermined portion of the monitored target T and the coordinates of the object related to the other person W. The controller 15 may determine whether the monitored object T has approached an object related to another person W based on the calculation result of the distance between the coordinates.

After the coordinates are normalized in step S24, the controller 15 determines whether or not the start of the violent motion is estimated from the coordinates normalized in step S24 based on the machine learning data acquired in step S21. (Step S25).

When the start of the violent action is estimated in step S25, that is, when the risk of the start of the violent action is increased, the controller 15 outputs a predetermined warning signal (step S26). In step S26, the controller 15 may output a predetermined warning signal to the warning unit 17. As a result, the warning unit 17 can issue a predetermined warning.

On the other hand, if the start of the violent motion is not estimated in step S25, that is, if the risk of the violent motion starting from now on is not increased, the controller 15 skips the motion of step S26 and ends the motion shown in FIG. It's okay. When the operation shown in FIG. 11 is completed, the controller 15 may start the operation shown in FIG. 11 again. For example, the controller 15 may repeat the operation shown in FIG. 11 every time the coordinates are extracted from the image data. That is, for example, when the extraction unit 11 extracts the coordinates (X, Y) from the image data of 15 frames per second, the controller 15 may estimate the start of the violent motion in step S25 15 times per second.

As described above, in one embodiment, the controller 15 has the coordinates of the predetermined portion of the monitored target T extracted by the extraction unit 11 based on the machine learning data, the coordinates of the predetermined portion of the other person W, or the like. The start of the violent movement may be estimated from the coordinates of the object related to the person W. Further, the extraction unit 11 has the coordinates of the predetermined portion of the monitored target T, the coordinates of the predetermined portion of the other person W, or the other person W from the images of a predetermined number of frames per unit time captured by the imaging unit 20. You may extract the coordinates of the object related to. In this case, the controller 15 violently operates from the coordinates of the predetermined portion of the monitored target T extracted by the extraction unit 11, the coordinates of the predetermined portion of the other person W, or the coordinates of the object related to the other person W. The start of may be estimated.

As shown in FIG. 11, the controller 15 may output a predetermined warning signal in step S26 immediately after estimating the start of the violent action in step S25. Therefore, the controller 15 may output a predetermined warning signal before the actual violent operation ends. As described above, the controller 15 may output a predetermined warning signal after estimating the start of the violent movement and before the end of the violent movement. Further, if possible, the controller 15 may output a predetermined warning signal before the actual violent operation starts. As described above, after estimating the start of the violent movement, the controller 15 may output a predetermined warning signal before the start of the violent movement.

According to the system 1 according to the embodiment, the coordinates of the joint points of the body of the monitored object T and the other person W in the action of a human being violently acting against the other person W, for example, the monitored object T. Machine learning can be performed on the relationship between the coordinates of the joint points of the body or the coordinates of an object related to another person W and the timing of the action of wielding the violence. Further, according to the system 1 according to the embodiment, the coordinates of the joint points of the body of the monitored object T in the action of the monitored object T violently acting against another person W based on the result of machine learning and the coordinates of the body of the monitored object T. The start of violent movement can be estimated from the coordinates of the joint points of the body of the other person W or the coordinates of the object related to the other person W. Therefore, according to the system 1 according to the embodiment, before the monitored target T begins to violence against another person W, or while the violence continues to be violent, before the violence is finished. A predetermined warning can be issued. Therefore, according to the system 1 according to the embodiment, the staff of a nursing care facility, for example, before the monitored object such as a nurse or a person requiring nursing care begins to violence against another person W. Alternatively, it can be recognized that the monitored target T is trying to violence against another person W or is trying to continue violence while continuing to wield violence. Therefore, according to the system 1 according to the embodiment, it is possible to provide safety around the monitored target T.

FIG. 12 is a diagram further explaining the estimation process described in step S25 of FIG. FIG. 12 shows the state of the monitored target T captured in the image data acquired in step S22 of FIG. 11, for example, in the left column. As shown in FIG. 12, in the captured image data, the monitored target T approaches the other person W after entering the room in which the image pickup unit 20 is installed, reaches for the other person W, and violence. Suppose that the operation is started. From the moment when the violent operation starts, the controller 15 shall perform the operation after step S22 shown in FIG. 11 as a process of 15 frames per second. That is, in the system 1, the image pickup unit 20 may capture an image of 15 frames per second. Further, in the system 1, the controller 15 may acquire an image of 15 frames per second. Further, in the system 1, the extraction unit 11 extracts the coordinates of the body joint point of the monitored target T, the coordinates of the body joint point of the other person W, or an object related to the other person W from the image of 15 frames per second. It may be used to extract the coordinates of. Further, in the system 1, the controller 15 (or the extraction unit 11) may normalize the coordinates extracted from the image of 15 frames per second. Further, in the system 1, the controller 15 may estimate the start of the violent movement from the normalized coordinates of 15 frames per second based on the machine learning data.

In the central column of FIG. 12, the controller 15 conceptually shows how to continuously acquire frames of image data. Here, the image data of each frame may be the coordinates extracted from the image data or may be the normalized coordinates. Further, in the central column of FIG. 12, the hatched image data represents a frame for one second after the violent movement starts.

In such a situation, the controller 15 may estimate the start of the violent movement at that time based on 15 frames per second (frames 1 to 15) after the start of the violent movement (FIG. 12). Estimate shown 1). Next, the controller 15 may estimate the start of the violent movement at that time based on 15 frames per second from frame 2 to frame 16 (estimation 2 shown in FIG. 12). Further, the controller 15 may estimate the start of the throwing operation at that time based on 15 frames per second from the frame (N-14) to the frame N (estimation 3 shown in FIG. 12). By repeating the above operation, the controller 15 estimates the start of the violent operation 15 times per second. Therefore, according to the system 1 according to the embodiment, for example, even if the start of the violent movement that should be originally estimated is not estimated in the estimation 1, the estimation 2 and the estimation 3 shown in FIG. 12, 15 times per second. The risk of misreporting can be reduced by the estimation of.

In the above-described embodiment, for example, as shown in FIG. 5, the extraction unit 11 has described an example in which the extraction unit 11 extracts the coordinates of 13 sites as the joint points of the monitored target T. However, in one embodiment, the extraction unit 11 may extract the coordinates of the parts having more than 13 parts, or may extract the coordinates of the parts having less than 13. Further, in the above-described embodiment, the system 1 has described an example of processing 15 frames per second. However, in one embodiment, the system 1 or each functional unit constituting the system 1 may process more than 15 frames per second or may process less than 15 frames per second. In one embodiment, the number of joint points handled by the system 1 and / or the number of frames processed may be adjusted so that the estimation of the onset of violent movement results in a reasonable result.

Thus, in one embodiment, the controller 15 may determine at least one of the number of the frames and the number of the joint points so that the validity of the estimation of the start of the violent movement becomes more than a predetermined value.

In the above embodiment, since the captured image data is used, visible light is used as a detection target for monitoring. However, the present disclosure is not limited to such cases, and other detection targets such as arbitrary electromagnetic waves, sound waves, temperatures, and vibrations may be arbitrarily used.

Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or modifications based on the present disclosure. It should be noted, therefore, that these modifications or modifications are within the scope of this disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically inconsistent, and a plurality of components or steps can be combined or divided into one. Is. Although the embodiment according to the present disclosure has been mainly described with respect to the apparatus, the embodiment according to the present disclosure can also be realized as a method including steps executed by each component of the apparatus. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, which is executed by a processor included in the apparatus. It should be understood that these are also included in the scope of this disclosure.

The above-described embodiment is not limited to the implementation as the system 1. For example, the above-described embodiment may be implemented as the electronic device 10 included in the system 1. Further, the above-described embodiment may be implemented as a monitoring method using a device such as the electronic device 10. Further, the above-described embodiment may be implemented as a program executed by, for example, a device such as an electronic device 10 or an information processing device (for example, a computer). Further, in the technique of the present disclosure, it is not necessary that all the components of the electronic device 10 shown in FIG. 1 exist in one housing or server. For example, each part such as a controller and a storage part of the components of the electronic device 10 are connected to each other by a network consisting of wired, wireless or a combination thereof, and can be connected to different housings, servers, devices, rooms, buildings, regions, countries, etc. It may be arranged arbitrarily.

1 System 10 Electronic equipment 11 Extraction unit 13 Storage unit 132 Machine learning data 15 Controller 17 Warning unit 19 Communication unit 20 Imaging unit

Claims (20)

An image pickup unit that captures an image of a monitored object and another person or an object related to the other person.
An extraction unit that extracts the coordinates of a predetermined portion of the monitored object, the coordinates of a predetermined portion of the other person, or the coordinates of an object related to the other person from the image captured by the imaging unit.
The start time and the end time are based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the image pickup unit. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person, and the monitored object violently attack the other person. A controller that machine-learns the relationship with the start of operation,
A system equipped with. The controller is based on timing information including a time point other than the start time point and the end time point of the action in which the monitored object violently acts against the other person in the time-dependent image captured by the image pickup unit. The coordinates of the predetermined part of the monitored object between the start time point and the end time point, the coordinates of the predetermined part of the other person, or the coordinates of the object related to the other person, and the monitored object is the other person. The system according to claim 1, wherein the system learns the relationship with the start of a violent movement against a person by machine learning. The extraction unit obtains coordinates of a predetermined number of joint points in the body to be monitored and a predetermined number of joint points in the body of the other person from images of a predetermined number of frames per second captured by the imaging unit. The system according to claim 1 or 2, wherein the coordinates or the coordinates of an object related to the other person are extracted two-dimensionally. The extraction unit is a two-dimensionally extracted coordinate of a predetermined number of joint points in the body of the monitored object, coordinates of a predetermined number of joint points in the body of the other person, or an object related to the other person. The system according to any one of claims 1 to 3, wherein each direction component of the coordinates is normalized based on the maximum value and the minimum value of each direction component. In the timing information, the time point at which the monitored object violently acts against the other person is a predetermined distance from the other person in the time-dependent image captured by the imaging unit. The system according to any one of claims 1 to 4, which indicates the timings separated from each other. In the timing information, the time when the monitored object starts to violently attack the other person is a predetermined distance from the other person in the time-dependent image captured by the imaging unit. The system according to any one of claims 1 to 5, indicating the timing of approaching within. In the timing information, the wrist of the monitored object is the other person or the wrist of the monitored object in the time-dependent image captured by the imaging unit at the start time of the operation in which the monitored object violently acts against the other person. The system according to any one of claims 1 to 6, indicating the timing at which an object related to the other person begins to approach. An image pickup unit that captures an image of a monitored object and another person or an object related to the other person.
An extraction unit that extracts the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of another person, or the coordinates of an object related to the other person from the image captured by the imaging unit.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined portion of the monitored object extracted by the extraction unit based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. A controller that estimates the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
A system equipped with. The extraction unit is an object related to the coordinates of the predetermined portion of the monitored object, the coordinates of the predetermined portion of the other person, or the object related to the other person from the images of a predetermined number of frames per unit time captured by the imaging unit. Extract the coordinates of
In the controller, the monitored object is the other from the coordinates of the predetermined part of the monitored object extracted by the extraction unit, the coordinates of the predetermined part of the other person, or the coordinates of the object related to the other person. 8. The system according to claim 8, which estimates the start of a violent action against a person. When the controller estimates the start of the action of the monitored object to behave violently against the other person, the controller gives a predetermined warning before the start of the action of the monitored object to behave violently against the other person. The system of claim 8 or 9, which outputs a signal. When the controller estimates the start of the action of the monitored object to behave against the other person, a predetermined warning is given before the end of the action of the monitored object to behave against the other person. The system of claim 8 or 9, which outputs a signal. The extraction unit obtains coordinates of a predetermined number of joint points in the body to be monitored and a predetermined number of joint points in the body of the other person from images of a predetermined number of frames per second captured by the imaging unit. The system according to any one of claims 8 to 11, wherein the coordinates or the coordinates of an object related to the other person are extracted two-dimensionally. The extraction unit is a two-dimensionally extracted coordinate of a predetermined number of joint points in the body of the monitored object, coordinates of a predetermined number of joint points in the body of the other person, or an object related to the other person. 12. The system of claim 12, wherein each directional component of the coordinates is normalized based on the maximum and minimum values of the directional component. The controller determines at least one of the number of frames and the number of joint points so that the validity of the estimation of the start of the action of the monitored object to violently attack the other person is equal to or higher than a predetermined value. The system according to claim 12 or 13. From the image captured including the monitored object and another person or an object related to the other person, the coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or related to the other person. An extraction unit that extracts the coordinates of the object to be used,
The start time and the above are based on the timing information indicating the start time and the end time of the action in which the monitored object violently acts against the other person in the time-dependent image captured including the monitored object. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person between the end time point and the time point of the end, and the monitored object with respect to the other person. A controller that machine-learns the relationship with the initiation of violent movements,
Electronic equipment equipped with. From the image captured including the monitored object and another person or an object related to the other person, the coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or related to the other person. An extraction unit that extracts the coordinates of the object to be used,
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined portion of the monitored object extracted by the extraction unit based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. A controller that estimates the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
Electronic equipment equipped with. An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The start time and the end time are based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the imaging step. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person, and the monitored object violently attack the other person. Machine learning steps to machine learn the relationship with the start of motion,
How to control electronic devices, including. An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined part of the monitored object extracted by the extraction step based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. An estimation step of estimating the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
How to control electronic devices, including. On the computer
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The start time and the end time are based on the timing information indicating the start time and the end time of the operation in which the monitored object violently acts against the other person in the time-dependent image captured by the imaging step. The coordinates of the predetermined part of the monitored object and the coordinates of the predetermined part of the other person or the coordinates of the object related to the other person, and the monitored object violently attack the other person. Machine learning steps to machine learn the relationship with the start of motion,
A program that runs. On the computer
An imaging step of imaging a monitored object and another person or an object related to the other person.
An extraction step of extracting the coordinates of a predetermined part of the monitored object, the coordinates of a predetermined part of the other person, or the coordinates of an object related to the other person from the image captured by the imaging step.
The coordinates of the predetermined part of the human and the coordinates of the predetermined part of the other human or the coordinates of the object related to the other human between the start time and the end time of the action of the human being violent against another human. And the coordinates of the predetermined part of the monitored object extracted by the extraction step based on the machine learning data in which the relationship between the human being and the start of the action of violently acting against the other human being is machine-learned. An estimation step of estimating the start of an action in which the monitored object acts violently against the other person from the coordinates of a predetermined part of the other person or the coordinates of an object related to the other person.
A program that runs.

Images