WO2021192398A1

WO2021192398A1 - Behavior recognition server and behavior recognition method

Info

Publication number: WO2021192398A1
Application number: PCT/JP2020/042056
Authority: WO
Inventors: 佐知田中; 健太郎佐野; 大平　昭義; 卓男姚; 浩平京谷; 優佑円谷
Original assignee: 株式会社日立製作所
Priority date: 2020-03-25
Filing date: 2020-11-11
Publication date: 2021-09-30
Also published as: JP2021157274A; CN114424263A; CN114424263B; JP7366820B2

Abstract

A behavior recognition server (1) comprises: a sensor information acquisition unit (11) that acquires, from a group of sensors (2) for detecting a person being observed, sensor information which indicates detection results from each of the sensors (2); an image conversion unit (13) that creates images of the sensor information of each sensor by writing values, which are in an arbitrary range based on the values of sensor information of each sensor, to regions for sensor information of each sensor within images specified by layout data (13L); and a behavior inference unit (14) that, by inputting the images of the sensor information which have been generated by the image conversion unit (13) into an inference model (14m) in which correspondence data associating images of sensor information with behaviors of the person being observed has been prepared in advance, determines the behavior of the person being observed output from the inference model (14m).

Description

Behavior recognition server and behavior recognition method

The present invention relates to a behavior recognition server and a behavior recognition method.

In recent years, high-performance sensors connected to the Internet have become widespread as IоT (Internet of Things) devices. Then, an attempt is made to provide a monitoring service for the observed person by analyzing a large amount of sensor information from a large number of sensors installed in an environment such as a home.

For example, Patent Document 1 describes a system that estimates living behavior by combining an existence detection sensor installed in a home and a motion detection sensor. The degree of abnormality is determined by comparing the estimated current living behavior with the living behavior pattern database prepared in advance.

Japanese Unexamined Patent Publication No. 2002-352352

When providing the observer watching service, it is necessary to recognize with high accuracy whether the current behavior of the observer is normal or abnormal. This is because the quality of the watching service deteriorates if a warning is issued or an abnormal behavior is overlooked even though the behavior is normal. Therefore, in order to improve the recognition accuracy, it is important to have a phase of uniquely identifying the behavior of the observed person from a plurality of types of sensor information measured at the same time.

Note that individual sensor information can only obtain basic information such as "the observer is in the living room" and "the observer is sitting" when used alone. Therefore, it is necessary to identify the detailed behavior of the observed person such as "eating in the living room" and "watching TV in the living room" by combining a plurality of sensor information.

In Patent Document 1, it is necessary to prepare a living behavior database in advance as a process for identifying living behavior from sensor information, and the trouble of preparing the database is a burden on the administrator. For example, in order to construct a living behavior database, statistical parameters and question-and-answer data from the observer must be prepared in advance, which is a burden.

Therefore, the main subject of the present invention is to recognize the behavior of the observed person with high accuracy and low cost.

In order to solve the above problems, the behavior recognition server of the present invention has the following features.
The present invention includes a sensor information acquisition unit that acquires sensor information indicating a detection result for each sensor from a set of sensors that detect an observed person.
An image conversion unit that images each sensor information by writing a value in an arbitrary range based on the value of each sensor information in the area of each sensor information in the image defined by the layout data.
By inputting the image of the sensor information created by the image conversion unit into the inference model in which the correspondence data between the image of the sensor information and the behavior of the observed person is prepared in advance, the inference model can be used. It is characterized by having a behavior inference unit that obtains the output behavior of the observed person.
Other means will be described later.

According to the present invention, the behavior of the observed person can be recognized with high accuracy and at low cost.

It is a block diagram of the behavior recognition system which concerns on one Embodiment of this invention. It is a hardware block diagram of the action recognition system which concerns on one Embodiment of this invention. It is a block diagram which shows the detail of the behavior recognition server which concerns on one Embodiment of this invention. It is a flowchart which shows the process of the action recognition server which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the layout data used for the imaging process which concerns on one Embodiment of this invention. It is a table for explaining the layout data of FIG. 5 concerning one Embodiment of this invention. It is explanatory drawing of the image data of the result of writing the "value" of FIG. 6 with respect to the layout data of FIG. 5 concerning one Embodiment of this invention. It is explanatory drawing of the image data as a result of writing the action different from FIG. 7 with respect to the layout data of FIG. 5 concerning one Embodiment of this invention. It is explanatory drawing which shows an example of the normal behavior pattern stored in the database which concerns on one Embodiment of this invention. It is explanatory drawing which shows an example of the present action pattern stored in the present action accumulation part with respect to one Embodiment of this invention. It is a screen view which shows the display screen when the present behavior pattern about one Embodiment of this invention is normal. It is a screen view which shows the display screen when the present behavior pattern which concerns on one Embodiment of this invention is abnormal.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of an action recognition system.
The behavior recognition system is configured so that the observer 3u remotely watches the behavior pattern of the observer 2u living at home 2h using the observer terminal 3. The "behavior pattern" is a series of patterns in which the "behavior" of the observed person 2u at each time is connected in chronological order.
The behavior recognition server 1 recognizes the behavior pattern of the observed person 2u based on the sensor information acquired from various sensors 2, and notifies the observer terminal 3 of the recognition result. As a result, the observer 3u who sees the display screen of the observer terminal 3 can grasp the behavior pattern of the observer 2u.
The observer 2u is, for example, a care recipient, and the observer 3u is, for example, the family of the care recipient. Alternatively, a behavior recognition system may be introduced in a hospital or a long-term care facility instead of the home 2h, in which case the observer 3u becomes a doctor or a care manager.

At home 2h, various sensors 2 for monitoring the behavior of the observed person 2u are connected to the network. The sensor 2 may be, for example, a sensor incorporated in a home electric appliance such as a refrigerator 2a or an autonomous mobile vacuum cleaner 2b, or a single sensor such as a motion sensor 2c.
It is desirable that the sensor 2 such as the motion sensor 2c is installed in a direction in which the measurement area does not face the entrance of the room. By this installation, it is possible to prevent the motion sensor 2c from erroneously detecting a person different from the observed person 2u passing through the corridor outside the room.

FIG. 2 is a hardware configuration diagram of the behavior recognition system.
The sensor 2 notifies the observer 2u of a communication unit 121 that notifies other devices of sensor information detected by the detection unit 122, a detection unit 122 that detects the observer 2u, and a message from the observer 3u. It has a notification unit 123 and the like. The notification unit 123 may be incorporated in the sensor 2 or may be configured as a device (display, speaker, etc.) different from the sensor 2.
The action recognition server 1 controls the communication unit 111 that receives the sensor information from the sensor 2 and notifies the observer terminal 3 of the recognition result from the sensor information, the control unit 112 that recognizes the behavior pattern of the observer 2u, and the control unit 112. It has a storage unit 113 for storing data used for processing of the unit 112.
The observer terminal 3 inputs a communication unit 131 that receives the recognition result of the observer 2u, a notification unit 132 that notifies the observer 3u of the recognition result of the observer 2u, a message from the observer 2u, and the like. It has an input unit 133 to be operated.

The action recognition server 1 is configured as a computer having a CPU (Central Processing Unit) as an arithmetic unit (control unit 112), a memory as a main storage device, and a hard disk as an external storage device (storage unit 113).
In this computer, the CPU operates a control unit (control means) composed of each processing unit by executing a program (also called an application or an abbreviation for application) read in the memory.

FIG. 3 is a configuration diagram showing details of the action recognition server 1.
The control unit 112 (FIG. 2) of the action recognition server 1 includes a sensor information acquisition unit 11, a time information acquisition unit 12, an image conversion unit 13, an action inference unit 14, a current action identification unit 15, and a current action accumulation. It has a unit 16 and a change detection unit 18.
The storage unit 113 (FIG. 2) of the action recognition server 1 stores the layout data 13L, the inference model 14m, and the database 17.
Hereinafter, the details of the components of FIG. 3 will be described with reference to the flowchart of FIG.

FIG. 4 is a flowchart showing the processing of the action recognition server 1.
The sensor information acquisition unit 11 acquires sensor information from the sensors 2 (refrigerator 2a, vacuum cleaner 2b, motion sensor 2c) installed in the home 2h (S101). The data format of the sensor information may differ depending on the type of the sensor 2.
The time information acquisition unit 12 acquires time information indicating the measurement time of the sensor information acquired in S101 (S102). The time information acquisition unit 12 acquires the time when the sensor 2 includes the time stamp in the sensor information, and acquires the reception time of the sensor information when there is no time stamp.

The image conversion unit 13 images the sensor information acquired in S101 and the time information acquired in S102 as one image (S103).
In the layout data 13L that the image conversion unit 13 refers to when imaging, information regarding the layout in the image, such as which part of the image the sensor information of which sensor 2 is to be arranged, is defined in advance ( 5 and 6).

The behavior inference unit 14 infers the behavior of the observed person 2u in the time information of the sensor information by inputting the image data indicating the sensor information into the inference model 14m (S104). For this inference process, an inference model 14m that outputs the corresponding behavior when image data is input in advance is prepared in advance. The inference model 14m may be trained by a machine learning algorithm such as deep learning, or the equation of the inference model 14m may be established from the estimation contents illustrated below.
・ If all the motion sensors in the house do not respond, it is estimated that you are out.
・ If the motion sensor responds in the bedroom at night and the illuminance is low, it is estimated that you are sleeping.
The current behavior identification unit 15 stores the “current behavior pattern” in which the “behavior” of the observed person 2u at each time inferred by the behavior inference unit 14 is connected in chronological order in the current behavior storage unit 16 (S105).

When the change detection unit 18 detects a change (deviation) from the normal behavior pattern registered in the database 17 in advance (S111, Yes), an abnormality occurs in the behavior pattern of the observed person 2u. Since it has been done, the change (abnormality) is notified to the observer 3u via the observer terminal 3 (S112). Further, the output destination of S112 is not limited to the customer environment (observer terminal 3), and may be output to another system such as a database system or a cloud system.
On the other hand, when the change (deviation) from the normal behavior pattern is not detected (S111, No), the change detection unit 18 registers the current behavior pattern as it is in the database 17 as the normal behavior pattern (S113), and processes the process in S101. return.

Hereinafter, a specific example of the process (S103) of the image conversion unit 13 will be described with reference to FIGS. 5 to 8.
FIG. 5 is an explanatory diagram showing an example of layout data 13L used by the image conversion unit 13 for the imaging process. In the layout data 13L, the data contents to be written at each position in the square image data of 12 squares in the vertical direction and 12 squares in the horizontal direction are arranged as symbols in the figure such as "T" and "ACC1". The "mass" is the smallest unit in which the area in the image to which the data content is written is subdivided, and at least one area is assigned to the sensor information and the time information. Further, the 12 × 12 squares in the image data are examples, and the number of squares may be smaller or larger than that.

FIG. 6 is an explanatory table of the layout data 13L of FIG. For example, the top "T" in FIG. 5 corresponds to the symbol "T" in the figure of the first line "time" in FIG. The image data arranged at the position of the uppermost portion "T" in FIG. 5 is the time data acquired by the time information acquisition unit 12. That is, one image shown in FIG. 6 is visualized by aggregating a set of sensor information measured at the same measurement time (time data of "T") from the sensors 2 arranged at each location. The result.

The layout data 13L of FIG. 6 is an example of arranging the same type of sensor information in close proximity in an image. On the other hand, sensor information having the same sensor installation location (room) may be arranged close to each other in the image.
The types of sensors 2 include, for example, an acceleration sensor, a (door) open / close sensor, and the like that detect the operation of the observed person 2u, a motion sensor, and the like that detect the presence of the observed person 2u, and humidity. Examples include sensors and the like that measure the environment of the home 2 hours.

The third column "number of squares" in the explanation table indicates the size of the area. When the amount of data to be written is smaller than the amount of data that can be expressed in the area, the area is left over. At that time, the image conversion unit 13 fills the number of cells in the image by copying and writing the same data content to a plurality of locations.

The number of cells in the layout data 13L indicates the weight between the information to be written, and the larger the number of cells is assigned, the greater the influence on the behavior. The distribution of the number of squares is determined by, for example, the following policy P1.
・ Since it is customary for humans to take actions depending on the time of day, such as going out during the day and sleeping at night, the time information "T" allocates a larger number of cells (24 cells) than other sensor information. ..
-Since the actions that humans can take are narrowed down to some extent depending on where they are, the sensor information (location information) of the motion sensors "HM1 to HM5" allocates a larger number of cells (12 cells) than other sensor information.
・ Since human life has a habit of taking the same actions depending on the day of the week, such as going to work on weekdays and resting at home on holidays, the day of the week information "DoW" has more cells than the sensor information that measures the environment of home 2h. Allocate (12 squares).
-As sensor information for detecting human movements, the acceleration sensors "ACC1 to ACC4" and the open / close sensors "OC1 to OC3" allocate a larger number of cells (4 cells) than the sensor information for measuring the environment of the home 2h. ..
The above-exemplified policy P1 is an example in which a motion sensor is attached to each room and "where a person is = what is being done" is mainly read in a floor plan having a plurality of rooms.

On the other hand, the policy (necessary sensor and its importance) may be appropriately changed depending on the action to be detected and the type of floor plan.
For example, although the importance of the environmental sensor was low in the policy P1, the policy P2 in which the importance of the environmental sensor is increased may be used as described below.
・ By increasing the sensitivity of the temperature sensor, the ON / OFF of the air conditioner can be read from the temperature sensor.
・ By increasing the sensitivity of the humidity sensor near the bathroom, it is possible to detect the opening and closing of the bathroom door by changing the humidity.
・ If the action and the illuminance are linked (turning off the lights when sleeping at night, opening the curtain in the morning, etc.), the action can be obtained from the illuminance.

As another example, the motion sensor was of high importance in the policy P1, but the policy P3 that obtains the position of a person from a sensor other than the motion sensor may be used as described below.
-If the number of motion sensors is too large to install due to the floor plan of a studio, the operation information of home appliances may be able to read the position of a person in more detail than the motion sensors.
・ You can get information that there is a person in front of the refrigerator while the refrigerator door is being operated.

In this way, the layout data 13L is defined according to policies P1 to P3 and the like. For example, in policy P1, the time information "T" of the sensor information, the sensor information "HM1 to HM5" indicating the whereabouts of the observer, and the sensor information "ACC1 to ACC4" indicating the operation of the observer are displayed in descending order of the area in the image. , The size of the area defined as the sensor information "TMP" that measures the living environment of the observed person is specified.
As a result, the behavior recognition accuracy can be improved by reflecting the empirical rule of the degree of influence of each sensor information on the behavior as the size of the region in the input data of the inference model 14m. This time, we used an empirical rule in which time information and whereabouts information have a greater influence on behavior recognition than other information.

The fourth column "value" of the explanatory table indicates the data content to be written in the area. For example, when the color depth of the image data is 8 bits of gray scale, the amount of data that can be expressed by the area is 2 to the 8th power = 256 numerical values.
Of course, the color depth of the image data is not limited to the gray scale, and a color capable of expressing a value at a more detailed stage may be used. Since the color depth of the image data can be set arbitrarily, the amount of data that can be expressed is not limited to 256. Therefore, for example, 8-bit grayscale and 16-bit color may be converted to different values and accuracy even if the reaction values of the same sensor are used.

In this embodiment, the range of 0.00 to 1.00 is described with an accuracy of 0.01.
For example, the value "0.31" of the time "T" indicates 7:40 am when 0:00 is the value "0.00" and 23:59 is the value "1.00". On the other hand, the day of the week is selected from seven ways when Monday is set to the value "0.00" and Sunday is set to the value "1.00".
The above-mentioned "value" is a value in an arbitrary range based on the value of each sensor information. In addition to the case of referring to the color corresponding to the value of each sensor information as described above, the case of referring to the value of each sensor information as it is is also included. In this way, even if the range of possible values varies depending on the type of sensor, it is normalized as a value in an arbitrary range.

In the row "HUM" of "humidity" in FIG. 6, the number of cells "5 (= 1x5)" means that the size of the area of the humidity sensor is one cell for one sensor and the number of sensors is five. It means a total of 5 squares. The "humidity" value "0.66,0.57,0.64,0.58,0.7" is the value of the first humidity sensor "0.66", the value of the second humidity sensor "0.57", ..., The value of the fifth humidity sensor in order from the left. Indicates "0.7".
Furthermore, although the number of motion sensors "HM1 to HM5" is shown as an example, the type and number of sensors change and increase or decrease depending on the layout of the room and the behavior to be recognized.

FIG. 7 is an explanatory diagram of image data as a result of writing the “value” of FIG. 6 with respect to the layout data 13L of FIG. In FIG. 7, symbols in the figure such as “T” and “ACC1” are also shown for the sake of clarity, but the symbols in the figure are actually omitted from the image.
For example, the image conversion unit 13 writes black indicating the value “0” as a color corresponding to the value of the sensor information in the area of “ACC1”. On the other hand, the image conversion unit 13 writes white indicating the value "1" as a color corresponding to the value of the sensor information in the area of "HM4". That is, the larger the value to be written, the closer to white.
Further, the inference model 14m is defined by associating the image data created by the image conversion unit 13 with the behavior "returning home" of the observer 2u indicating the situation indicated by the image data.

The behavior inference unit 14 refers to the inference model 14m registered in the past, and when image data matching or similar to the image data of the inference model 14m is detected from the current observer 2u, the inference model The corresponding action "returning home" at 14 m is output as an inference result (S104).
In addition, it is preferable that a human such as the observer 3u acts as a teacher and associates action labels such as "go home" and "break" with the image data of the inference model 14 m.

FIG. 8 is an explanatory diagram of image data as a result of writing an action different from that of FIG. 7 with respect to the layout data 13L of FIG. In the inference model 14m of FIG. 8, since the motion sensors "HM1 to HM5" are all white (the observer 2u is absent in each room), the behavior "going out" of the observer 2u is associated with it.
Hereinafter, with reference to FIGS. 9 to 12, a specific example of a comparison process (S111) between the current behavior pattern and the normal behavior pattern and an abnormality notification process (S112) of the current behavior pattern extracted by the comparison processing. Will be explained.

FIG. 9 is an explanatory diagram showing an example of a normal behavior pattern stored in the database 17. The normal behavior pattern is a behavior pattern that connects the normal behaviors of the observed person 2u from "sleeping" at 5 am to "washing" at 6 am, ..., and "sleeping" at 23:00.
FIG. 10 is an explanatory diagram showing an example of the current behavior pattern stored in the current behavior storage unit 16. The current behavior pattern starts from "sleeping" at 5 am, "meal" at 9 am, ..., and behavior that connects the current behavior of the observer 2u until "sleeping" at 23:00. It is a pattern.

Here, the change detection unit 18 compares the current behavior pattern of FIG. 9 with the normal behavior pattern of FIG. 10 for each behavior (S111), and finds the following abnormalities in the current behavior pattern of the observed person 2u. Extract actions.
・ I didn't get up until 9 o'clock and there was no movement indoors.
・ Meal time is different from usual and meal time is long.
・ I am temporarily returning home at an unexpected time (15:00) while I am out.
・ Bathing time is different and bathing time is long.

In this way, the change detection unit 18 detects the abnormality of the observed person 2u as illustrated below.
(1) The normal behavior defined in the normal behavior pattern is not currently performed in the behavior pattern.
(2) Unnatural behaviors that are not in the normal behavior pattern have been performed in the current behavior pattern.
(3) The normal behavior defined in the normal behavior pattern was also performed in the current behavior pattern, but it was different from the time defined in the normal behavior pattern.
(4) The normal behavior defined in the normal behavior pattern was also performed in the current behavior pattern, but the length was different from the length defined in the normal behavior pattern.
(5) The behavioral speed of the observed person 2u in the current behavioral pattern deviated from (greatly lower than) the behavioral speed defined as the normal behavioral pattern.

Then, as S112, the change detection unit 18 notifies the observer 3u of an emergency signal via the observer terminal 3 in response to the serious abnormality of (1) or (2), thereby arranging an ambulance or the like. Encourage the observer 3u to take an urgent response.
On the other hand, the change detection unit 18 transmits a signal requiring observation to the observer 3u via the observer terminal 3 for the minor abnormalities (3) to (5), so that the observer 2u is sick. Notify the observer 3u of the sign. Signs of illness are, for example, a state before the need for long-term care (frail) and mild cognitive impairment (MCI).

In particular, it is difficult to detect the minor abnormalities (3) to (5) from the sensor information of a single unit. On the other hand, since the image conversion unit 13 can extract the action interval by collectively imaging a plurality of sensor information, the detection becomes easy in the present embodiment.
The process of acquiring the behavioral speed of the observed person 2u from the current behavioral pattern in (5) includes a method exemplified below.
-Measure the moving speed between the washroom and the kitchen from the operation interval between the washing machine and the refrigerator 2a (range, IH).
-Measure the reaction speed from changes in the behavior (opening / closing speed, opening / closing interval, opening / closing time, etc.) related to the door of the refrigerator 2a.
・ Measure the action speed from the interval between the opening and closing of the entrance door and the ON / OFF of the lighting.
-Measure the walking speed during cleaning from the acceleration of the vacuum cleaner 2b.

FIG. 11 is a screen view showing a display screen of the observer terminal 3 when the current behavior pattern is normal (S111, No).
The change detection unit 18 informs the observer 3u of the healthy state of the observer 2u by displaying a display screen including the following display contents on the observer terminal 3.
-Message column 221 showing the outline of the current behavior pattern
・ Time chart column 222 of current behavior pattern
・ Statistical data column 223 of this week's life pattern
・ Statistical data column 224 of the difference between this week and last week
-Statistical data column 225 of the difference from the history so far

FIG. 12 is a screen view showing a display screen (S112) of the observer terminal 3 when the current behavior pattern is abnormal (S111, Yes).
In the floor plan column 211 in the center of the screen, the room names such as the bedroom and the living room and the sensor information (temperature, humidity, illuminance) measured by the environmental sensor for each room are displayed for each room of the home 2h.
Here, the estimated position 212 of the observed person 2u and the warning message 213 (emergency signal) indicating the abnormal behavior of the observed person 2u are also displayed in the floor plan column 211. As a result, the observer 3u can easily grasp the details of the abnormality of the observed person 2u such as a person falling down.

In the present embodiment described above, the image conversion unit 13 displays many types of sensor information such as sensor information of connected home appliances such as a refrigerator 2a and a vacuum cleaner 2b, and sensor information of a motion sensor 2c as one image. (Figs. 7 and 8).
As a result, it is possible to detect a large change in the behavior of the observed person 2u such as a fall, which could not be grasped only by the sensor information of a single unit.

Here, the present embodiment is compared with the method of constructing a living behavior database as in Patent Document 1. In the conventional technique such as Patent Document 1, it takes time and effort to manually input the rule of which sensor information affects the behavior and how much.
On the other hand, in the method of machine learning the imaged sensor information of the present embodiment, the rule of which sensor information affects the behavior to what extent is automatically discovered by machine learning, so that a human can handle the rule. You can save the trouble of inputting in the work. Furthermore, even rules that humans did not know are automatically discovered by machine learning, improving the accuracy of behavior recognition.

The present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations.
Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration. Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit.
Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function.

Information such as programs, tables, and files that realize each function can be stored in memory, hard disks, recording devices such as SSDs (Solid State Drives), IC (Integrated Circuit) cards, SD cards, DVDs (Digital Versatile Discs), etc. Can be placed on the recording medium of.
In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.
Further, the communication means for connecting each device is not limited to the wireless LAN, and may be changed to a wired LAN or other communication means.

2 Sensor 2u Observer 3 Observer terminal 3u Observer 1 Behavior recognition server 11 Sensor information acquisition unit 12 Time information acquisition unit 13 Image conversion unit 13L Layout data 14 Behavior inference unit 14m Inference model 15 Current behavior identification unit 16 Current behavior accumulation Department 17 Database 18 Change detection unit

Claims

A sensor information acquisition unit that acquires sensor information indicating the detection result for each sensor from a set of sensors that detect the observed person, and a sensor information acquisition unit.
An image conversion unit that images each sensor information by writing a value in an arbitrary range based on the value of each sensor information in the area of each sensor information in the image defined by the layout data.
By inputting the image of the sensor information created by the image conversion unit into the inference model in which the correspondence data between the image of the sensor information and the behavior of the observed person is prepared in advance, the inference model can be used. An action recognition server having an action inference unit that obtains an output action of the observed person.
The claim is characterized in that the image conversion unit writes in the area a value in an arbitrary range based on the time information of the sensor information in addition to a value in an arbitrary range based on the value of each sensor information. The action recognition server according to 1.
In the layout data, the size of the region in the image is defined according to the type of each sensor information.
The behavior recognition server according to claim 2, wherein the image conversion unit writes a value in an arbitrary range based on the value of each sensor information according to the size of the area defined in the layout data. ..
In the layout data, the time information of the sensor information, the sensor information indicating the whereabouts of the observed person, the sensor information indicating the operation of the observed person, and the observed person are described in descending order of the area in the image. The action recognition server according to claim 3, wherein the size of the region as the sensor information for measuring the living environment of the user is defined.
The action recognition server further includes a current action identification unit, a database, and a change detection unit.
The current behavior specifying unit identifies a current behavior pattern that connects the behaviors of the observed person obtained by the behavior inference unit in chronological order.
The database stores normal behavior patterns that connect normal behaviors in chronological order.
The change detection unit is characterized in that by comparing the current behavior pattern with the normal behavior pattern, the change detection unit detects the current behavior pattern deviating from the normal behavior pattern as an abnormal behavior of the observed person. The behavior recognition server according to claim 1.
The change detection unit
When the normal behavior defined in the normal behavior pattern is not performed in the current behavior pattern, or when an action not in the normal behavior pattern is performed in the current behavior pattern, the observed person Notifies an emergency signal indicating a serious abnormality and
When the normal behavior defined in the normal behavior pattern is also performed in the current behavior pattern but different from the time defined in the normal behavior pattern, or when the normal behavior defined in the normal behavior pattern is the present. The fifth aspect of claim 5, wherein the behavior pattern is also performed, but when the length is different from the length defined in the normal behavior pattern, an observation-required signal indicating a slight abnormality of the observed person is notified. Behavior recognition server.
The change detection unit acquires the behavior speed of the observed person from the interval of the behavior included in the current behavior pattern, and the acquired behavior speed of the observed person deviates from the behavior speed defined as the normal behavior pattern. The behavior recognition server according to claim 5, wherein when the behavior is performed, an observation signal indicating a slight abnormality of the observed person is notified.
The action recognition server has a sensor information acquisition unit, an image conversion unit, and an action inference unit.
The sensor information acquisition unit acquires sensor information indicating a detection result for each sensor from a set of sensors that detect an observed person.
The image conversion unit images each sensor information by writing a value in an arbitrary range based on the value of each sensor information in the area of each sensor information in the image defined by the layout data.
The behavior inference unit inputs an image of the sensor information created by the image conversion unit into an inference model in which correspondence data between the image of the sensor information and the behavior of the observed person is prepared in advance. A behavior recognition method characterized by obtaining the behavior of the observed person output from the inference model.