JP2005258830A - Understanding system on person and action - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide understanding system on a person and action that can be inexpensively constructed in a simple configuration, extracts data on the part region of a human body and a specific region, grasps the action features of the person, and further understands the action of the person with the action feature as a database. <P>SOLUTION: The understanding system on the person and his action comprises an imaging section 2 for detecting an imaging region including a subject person, a part detector 5 for extracting a part region, an action feature detector 6 for detecting the feature of action according to the amount of change in the data at the part region, and an action understanding section 7 for determining the action of the person by combining the features of action detected by the action feature detection section 6. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a system for understanding a person's behavior while monitoring the person with an imaging device such as a television camera, and in particular, a processing unit for extracting person's part area data and specific area data from imaging data. In addition, the present invention relates to a human behavior understanding system having a hierarchical processing structure such as a processing unit for detecting a human behavior feature using the data, and a processing unit for understanding the behavior of the person from the behavior feature.

In recent years, the proportion of elderly people in Japan's population has gradually increased, and it is certain that this trend will become more prominent in the future. In addition, due to late marriage and unmarried marriage, the society is declining and aging. Along with such changes in society, there is an urgent need to develop a safe living environment for elderly people living alone or persons with disabilities to live a healthy and independent life with peace of mind.
On the other hand, progress in IT (information technology) is remarkable, and various information can be easily obtained instantly by construction of the Internet network.
Under such circumstances, there is an increasing demand for an abnormality notification system that promptly notifies a caregiver or family of an abnormal sign regarding an elderly household including a single living alone or a disabled person such as a disabled person. Such needs are great not only in homes, but also in hospitals, nursing homes, nursing homes and other medical institutions, welfare institutions, etc., and a system that is inexpensive, highly accurate, and easy to handle is required.
As such a system, for example, in Patent Document 1, as “behavior monitoring apparatus and method thereof”, an imaging unit provided in a space used by a person to be monitored and an imaging signal captured by the imaging unit are captured. Abstraction processing means for abstracting a person portion to be monitored from the image signal; and image means for compressing and storing an image signal in which the person portion to be monitored is abstracted by the abstraction processing means; A technique including a communication means that traditionally directs a compressed image signal to the outside is disclosed.
The invention disclosed in Patent Document 1 also discloses a technique for performing mosaic processing on an abstracted portion or performing time difference processing of an image.
According to the technique disclosed in Patent Document 1, a person's action is remotely monitored by an image signal, and the person image is monitored while protecting privacy by abstracting the person image by mosaic processing or temporal difference processing. Can help.

In Patent Document 2, as an “intruding object tracking image processing system”, a television camera equipped with a wide-field imaging lens that outputs an image signal used to detect an intruding object, and an intruding object mounted on an electric head are used. Process and compute image signals input from a television camera with a zoomable imaging lens that outputs an image signal for use in tracking and enlarging image monitoring, and a television with an imaging lens with a wide field of view. An image processing device that outputs a rotation control signal for rotating the electric head by a required amount and a zooming control signal for zooming the zoomable imaging lens by a required amount; A system for automatically tracking and automatically zooming in on an intruding object is disclosed.
In this system, an intruding object is automatically detected, and an imaging lens capable of automatically tracking and zooming in an intruding object is controlled by controlling a head that rotates in the vertical and horizontal directions based on the detected and calculated data. The intruding object is enlarged, and the effect of facilitating the observer to confirm the intruding object on the video monitor screen is exhibited.

Further, in Patent Document 3, a “driver's face image processing device” includes a camera that captures the face of a driver of an automobile and an image processing device that processes a face image obtained from the camera. A face model group corresponding to the various face orientations of the driver is generated based on the driver's front face obtained by the camera in Step 1. After completing this generation, the driver's face image is cyclically generated. The driver's face orientation is detected by inputting and selecting the best matching face model in the face model group.
Such an apparatus can detect the orientation of the face of the driver of the automobile, and can monitor the behavior of the driver such as looking aside.

JP 2000-216 A Japanese Patent Laid-Open No. 11-69342 JP 2003-308533 A

However, in the inventions disclosed in Patent Documents 1 and 2 described above, it is possible to monitor the behavior of a person or an intruding object, but it is only human beings who judge the content of the behavior and the like. There was a problem that the person had to grasp the contents. That is, for example, it cannot be monitored unattended, and has a problem that it ultimately requires human judgment.
In the invention disclosed in Patent Document 3, the face direction of a driver of a car is detected to generate a face model group, and the face image obtained from the camera is the most in any of the face model groups. It is different from the inventions disclosed in Patent Documents 1 and 2 in that it is determined whether they match and one face model is selected, so that the selection does not require human judgment.
However, it is only possible to make a simple determination as to whether or not the face is facing the front by comparing the captured face image and the face model group. There is a problem that it is not possible to further determine the action content of the face or the driver himself using the determined face orientation such as the action content when the driver is present.

  The present invention has been made in response to such a conventional situation, can be constructed at a low cost with a simple configuration, while grasping the behavior characteristics of the person while extracting the data of the region of the human body and the specific region, It is another object of the present invention to provide a human behavior understanding system that can understand the behavior of the person based on the behavior characteristics.

  In order to solve the above-described problems, a human behavior understanding system according to claim 1 of the present invention detects an imaging region including a person to be imaged, and a human region imaged by the imaging unit. A part detection unit that extracts part region data included in the action part, a behavior feature detection part that detects a behavior feature based on the amount of change in the part region data extracted by the part detection unit, and a behavior detected by the behavior feature detection part A behavior understanding unit that determines a person's behavior by combining the features of the part, and the part detection unit is a part region candidate of a part region candidate included in the person region imaged by the imaging unit Part region data is extracted by comparing the means for extracting data with the extracted part region candidate data and template data of a preset part region The behavior feature detection unit includes a means for calculating a change amount of the part region data detected by the part detection unit, and the person from the behavior feature data table indicating the behavior feature determined by the change amount of the part region data. The behavior understanding unit includes a behavior understanding data table that associates a combination of the person's behavior features extracted by the behavior feature detection unit with a preset behavior of the person. Is provided.

  Further, the human behavior understanding system according to claim 2 is an image pickup unit that detects an image pickup region that includes a specific region that is defined in advance separately from a person region that includes a person to be imaged, and is picked up by the image pickup unit. A human body part detection unit including a part detection unit for extracting part region data included in the person region and a specific region detection unit for extracting specific region data; and the part region data and the specific region data extracted by the human body part detection unit A behavioral behavior understanding system having a behavioral feature detection unit that detects a behavioral feature based on a change amount of a behavior and a behavioral understanding unit that determines the behavior of a person by combining the behavioral features detected by the behavioral feature detection unit. The part detection unit includes means for extracting part region candidate data of a part region candidate included in the person region imaged by the imaging unit, and the extracted part Means for collating region candidate data with template data of a preset part region and extracting part region data, and the specific region detection unit extracts specific region data included in the specific region imaged by the imaging unit; Means for extracting, the behavior feature detecting unit calculates a change amount of the part region data detected by the part detection unit, and a means for calculating the change amount of the specific region data detected by the specific region detection unit; Means for extracting the behavior feature of the person from the behavior feature data table indicating the behavior feature determined by the change amount of the part region data and the change amount of the specific region data, and the behavior understanding unit is extracted by the behavior feature detection unit The action of the person is extracted from the action understanding data table that associates the combination of the action characteristics of the person with the action of the person set in advance. It is intended to provide means.

  The human behavior understanding system according to claim 3 is the human behavior understanding system according to claim 2, wherein the specific region is a plurality of regions divided in advance, and the behavior feature detection unit is extracted by the part detection unit. In addition, there is provided means for detecting part existence area data indicating in which of the divided specific areas the part area data is present, and the behavior feature table associates the part existence region data with the behavior feature.

  The human behavior understanding system according to claim 4 is the human behavior understanding system according to any one of claims 1 to 3, wherein the part detection unit extracts part region data of a person's head region. At least one of a head detection unit, a hand detection unit that extracts part region data of the hand region, a torso detection unit that extracts part region data of the torso region, and a foot detection unit that extracts part region data of the foot region One detection unit is provided.

  The human behavior understanding system according to claim 5 is the human behavior understanding system according to any one of claims 1 to 4, wherein the change amount of the part region data is the change amount of the data of the part region. It is one of change rate of change amount and change acceleration of change amount.

  The person behavior understanding system according to claim 6 is the person behavior understanding system according to any one of claims 1 to 5, wherein the behavior understanding unit cannot extract the behavior of the person from the behavior understanding data table. In such a case, a signal is transmitted so that re-detection is executed in the behavior feature detection unit or the human body part detection unit.

The human behavior understanding system according to claim 7 is the human behavior understanding system according to any one of claims 1 to 6, wherein the part detection unit is configured such that the extracted part region data is independent. , A task control means for activating the specific area detection unit, the behavior feature detection unit, and the behavior understanding unit. In the present specification, the region region data “single” means that when the number of organs in the region is one, the region region data relating to the organ is one, When the number of organs of a part is two and it is a pair, it means that the part region data regarding the organ is one or a pair.
In the case of a pair of organs such as hands and feet, each of the pair may be detected. However, one of them may be hidden behind a futon, for example, and only the other may be detected. Even one was included in a single concept.

  The human behavior understanding system according to claim 8 is the human behavior understanding system according to any one of claims 1 to 7, wherein the part region data extracted by the part detection unit is alone. Task control means for operating the specific area detection unit, the behavior feature detection unit, and the behavior understanding unit is provided.

  In the human behavior understanding system according to the present invention, in the human behavior understanding system according to the first aspect, the human body part detection unit extracts data of the human part region from the image obtained by the imaging unit, and the behavior feature detection unit It has a hierarchical structure in which the behavioral features of the person are extracted using the amount of change in the extracted part region data, and further, the behavioral understanding unit uses this behavioral feature or a combination thereof to extract the behavior. Since the functions are shared and information is selectively extracted for each, it is possible to grasp the behavior of the person with high accuracy.

  In addition, in the human behavior understanding system according to claim 2, by providing a specific region detection unit in addition to the region detection unit, not only the region region data but also the specific region data is extracted, and the data is based on the data. Therefore, it is possible to more accurately understand the behavior.

  In particular, in the human behavior understanding system according to claim 3, in addition to the effect of the invention according to claim 2, it is possible to detect the region existing region data. It can be easily grasped whether it exists in a specific area, and since it becomes possible to combine a person's part and its existence position, it becomes easier to detect a behavior feature. Therefore, it is possible to understand the behavior of the person more easily and with high accuracy.

  In addition, in the human behavior understanding system according to claim 4, in addition to the effects of the inventions according to claims 1 to 3, a head, hand, torso, and foot that are easy to grasp as human characteristics are detected. Therefore, it is possible to grasp the human behavior with higher accuracy.

  In the human behavior understanding system according to the fifth aspect, in addition to the same effects as the inventions according to the first to fourth aspects, in particular, not only the change amount of the position data but also the change speed as the change amount of the part region data. It is possible to understand the behavior in detail by obtaining more detailed behavioral characteristics using the change acceleration.

  In particular, in the human behavior understanding system according to claim 6, in addition to the effects of the inventions of claims 1 to 5, a feedback function is provided so that correct detection of erroneous detection and miscalculation can be made and more accurate behavior can be obtained. It is possible to promote understanding.

  In the inventions according to claim 7 and claim 8, when executing the process for understanding the behavior, first, only when the correct part is detected in the part detection part, the specific region detection part and the subsequent action feature detection By operating the unit and the behavior understanding unit, it is possible to operate the human behavior understanding system efficiently with energy saving, and it is also possible to increase the accuracy of operation.

A human behavior understanding system according to an embodiment of the present invention will be described below with reference to FIGS.
FIG. 1 is a configuration diagram of a human behavior understanding system according to an embodiment of the present invention. In FIG. 1, the human behavior understanding system 1 includes an imaging unit 2, a system body 3, and an output unit 4, and further includes a first database 17, a second database 18, and a third database 19.
First, the imaging unit 2 captures a person in the background together with the background, and acquires the imaging data 20. This imaging data 20 is stored in the first database 17 connected to the imaging unit 2. The imaging unit 2 is installed in, for example, a medical facility such as a hospital, a nursing facility, or a family with an elderly person or a disabled person. The subject person needs some kind of care such as a patient, an elderly person, or a disabled person. Is assumed. Therefore, for example, the imaging unit 2 is installed on the ceiling near the bed in a hospital room, and the patient is imaged from a high position including around the bed.
The imaging area by the imaging unit 2 includes a person area occupied by the person to be imaged and a specific area of a part or all of the preset imaging area, and the person area is further composed of respective part areas. . Since the specific area is set in advance without a person, a person may enter the specific area. In this case, the specific area may overlap with the person area. In addition, this specific area is, for example, the area around the head position when the patient on the bed gets up, the area 2 around the pillow, the area 3 around the futon, or the areas 1, 2, 3 around the bed It is preferable to divide the area that does not belong to the area 4 or the like.
The imaging unit 2 is assumed to be a single fixed camera, and a plurality of processing units to be described later are applied to a moving image obtained from the imaging unit 2 to ensure high measurement accuracy while being simple and inexpensive. It provides a human behavior understanding system.

The system body 3 includes a hierarchized processing structure including a human body part detection unit 5, a behavior feature detection unit 6, and a behavior understanding unit 7. The human body part detection unit 5 which is a first processing unit includes a part detection unit 8 for extracting a person's part region and a specific region detection unit 9 for extracting the above-mentioned specific region. These processing units distribute and cooperate to understand the behavior of the subject person.
Further, the part detection unit 8 includes a part region candidate data extraction unit 10, a part region data collation unit 11 and a task control unit 13, and the specific region detection unit 9 includes a specific region data extraction unit 12. .
The behavior feature detection unit 6 is connected to the human body part detection unit 5 and includes a part region data change amount calculation unit 14a, a specific region data change amount calculation unit 14b, a part presence region detection unit 14c, and a behavior feature extraction unit 15. The Further, the behavior understanding unit 7 includes a person behavior extraction unit 16.

The part detection unit 8 of the human body part detection unit 5 reads the imaging data 20 directly from the imaging unit 2 or reads the imaging data 20 stored in the first database 17, and the part region candidate data extraction unit 10 performs imaging. Part region candidate data is extracted from the data 20, and the part region data collating unit 11 collates the data related to the part region from the template data 24 stored in the second database 18, True region data 21 is extracted from the above. The extracted part region data 21 is stored in the first database 17.
Note that the imaging data 20 is data transmitted from the imaging unit 2 in a certain time span, and is stored in the first database 17 with each time as a key, and can be read out every time. . Also, the extracted part region data 21 and the specific region data 22 can be stored for each time, and the time is stored using the time as a key, so that it can be read using the time as a key. The fixed time span can be set as appropriate by the user of the present system.

Further, the extraction in the part region candidate data extraction unit 10 takes a difference from the background image data 23 captured in advance (hereinafter, the difference from the background image data may be referred to as a background difference). Alternatively, by taking a difference between the imaging data 20 captured every fixed time span and stored in the first database 17 (hereinafter, the difference between the imaging data 20 may be referred to as an inter-frame difference). Done. The background difference can obtain a change from the initial state, and the inter-frame difference can obtain a motion that changes every moment.
On the other hand, the specific area detection unit 9 extracts the specific area data 22 from the imaging data 20 by the specific area data extraction unit 12 as in the part detection unit 8. The extracted specific area data 22 is stored in the first database 17. The specific area data 22 is used to extract pixel data belonging to an area specified in advance (hereinafter sometimes referred to as an area).
As described above, the human body part detection unit 5 extracts the part area data 21 and the specific area data 22 of the person area included in the imaging data 20 and stores them in the first database 17. Of course, you may transmit to the action characteristic detection part 6, without storing in the 1st database 17. FIG.

The behavior feature detection unit 6 uses the part region data 21 and the specific region data 22 extracted by the human body part detection unit 5 by the part region data change amount calculation unit 14a and the specific region data change amount calculation unit 14b. The change amount of the part region data and the change amount of the specific region data are calculated. Specific examples of the change amount will be described later. For example, there are a part region data change amount 27, a part region data change speed 28, a part region data change acceleration 29, and the like indicating a change amount of the position of the part (particularly the center of gravity position).
For the calculation of these change amounts, the difference may be calculated by comparing with the part region data 21 which is the reference of the part region data 21 if the position change amount of the part, and the difference between frames is taken. Thus, the part change speed 28 is obtained by calculating the amount of movement during the unit time, and the part change acceleration 29 is obtained by calculating the change of the part change speed 28 during the unit time. If the specific area data change amount 31 indicates the change amount of the specific area data, the pixel data of the area included in the specific area data 22 is compared with the pixel data at a time different from the time when the pixel data was captured. Can be obtained. As a result, it is understood that some kind of change occurs in the specific area. Further, the specific area data change rate 32 is obtained by calculating the change in the specific area data change amount 31 during the unit time, and the specific area data change acceleration 33 is obtained by calculating the change of the speed during the unit time. Is obtained.
The obtained part region data change amount 27, part region data change rate 28, part region data change acceleration 29, specific region data change amount 31, specific region data change rate 32, and specific region data change acceleration 33 store calculation results. Is stored in the third database 19 for the purpose.

In addition, the increase or decrease of data in a specific area can detect information such as where the movement of the subject person itself occurs, information on changes in the position of pillows or futons in the specific area, etc. Therefore, it is possible to judge more accurately in understanding the behavior of a person, which is an important judgment material.
Using the part region data change amount and the specific region data change amount obtained in this manner, the behavior feature extraction unit 15 reads and references the behavior feature data table 25 stored in the second database 18 while referring to The behavior feature defined by the region region data change amount and the peripheral region data change amount is extracted.
Note that the specific area detection unit 9, the specific area data change amount calculation unit 14b, and the part presence area detection unit 14c are not essential, but if the specific area data change amount is calculated by these, they are set in the behavior feature data table 25 in advance. It is possible to specify behavior features in detail by increasing variations of behavior feature data.

Using the behavior feature extracted by the behavior feature detection unit 6, the human behavior extraction means 16 of the behavior understanding unit 7 reads out and refers to the behavior understanding data table 26 stored in the second database 18, and this behavior feature. Extract the action defined by. The behavior features extracted by the behavior feature detection unit 6 will be described later with reference to FIG. 20, but the behavior associated with the combination of the two behavior features is always extracted. Under the first behavior feature, if it is the second behavior feature, it is understood as “behavior”. In the present specification, this first behavior feature is referred to as a precursor behavior feature. The behavior feature is called a definite behavior feature.
Here, the task control means 13 included in the part detection unit 8 will be described. As described above, the human behavior understanding system 1 according to the present embodiment is, for example, a system for ensuring safety while imaging a patient's bed around a hospital room and monitoring the behavior of the patient. Function. In such a case, if the patient's family is visiting, or if a doctor or nurse is nearby, it is possible to deal with treatment even if the condition suddenly changes. The system does not have to operate. That is, monitoring is necessary when the patient is alone in the hospital room. Therefore, the behavior feature detection unit 6 that functions as a subsequent stage of the specific region detection unit 9 or the human body part detection unit 5 only when a part considered to be one patient is detected in the imaging data 20 obtained by the imaging unit 2. The behavior understanding unit 7 may be activated.

The task control means 13 has such a function, and when the part region candidate data extraction means 10 and the part region data collation means 11 extract one true part region data 21, there is one or a pair. In this case, that is, only when a part of a person to be imaged is detected alone, a signal is transmitted to the specific area detection unit 9, the behavior feature detection unit 6, and the behavior understanding unit 7 to operate.
Thus, when a plurality of part region data are detected, the subsequent operation is stopped, and the part region candidate data extraction unit 10 and the part region data collating unit 11 are operated again to obtain the part region data from the imaging data 20. By extracting and judging whether the data is single data, it is possible to efficiently understand the behavior, and at the same time, avoid misjudgment due to behavioral features and misperception of the behavior and promote accurate behavior understanding. is there.
In the present embodiment, the task control unit 13 is included in the part detection unit 8, but it is not necessarily included in the part detection unit 8, and the human body part detection unit in which the task control unit 13 forms a hierarchical structure. 5. Needless to say, it may exist independently as a component other than the behavior feature detection unit 6 and the behavior understanding unit 7.

Furthermore, the part presence area | region detection means 14c provided in the action characteristic detection part 6 is demonstrated.
As described above, the specific area set in advance is divided, ID codes are attached to the divided areas to form areas 1 and the like, and the part area data 21 extracted by the part detection unit 8 is divided into ID codes. It is a means to detect as which site | part area | region area | region data 30 it is included in which area | region provided.
The detected part existence region data 30 is stored in the third database 19. This part existence region data 30 is combined with the part region data change amount or the specific region data change amount in the behavior feature extraction means 15 of the behavior feature detection unit 6, and the behavior feature determined for the combination is the behavior feature data table. 25. A specific example will be described later.

Next, the operation until the part detection unit 8 extracts the part region data 21 will be described with reference to FIGS. FIG. 2 is a configuration diagram of a head detection unit 8a which is a kind of the site detection unit 8 and assumes a head as a site.
In FIG. 2, an input image 40 is a name when the imaging data 20 is captured as input information to the system body 3, and is substantially the same as the imaging data 20. The input image 40 input to the head detection unit 8a is read by the head candidate data extraction unit 10a as a kind of part region candidate data extraction means 10, and the skin color region constituting the head candidate data extraction unit 10a The data is transmitted to the extraction unit 44a, the normalized lightness background difference extraction unit 44b, the RGB background difference extraction unit 44c, and the hair color region extraction unit 44d.

First, the RGB background difference extraction unit 44c related to the two common area extraction units 46a and 46b will be described.
The RGB background difference extraction unit 44c extracts a difference from the background image data 23 stored in the second database 18 with respect to the input image 40 having RGB components as shown in FIG. Since the background image data 23 is data acquired from the imaging unit 2 in a state where no subject person exists, by taking the difference from the input image 40, the background image data 23 is further centered on the subject person. Objects in a different state from the original state, such as futons and pillows, are extracted as differences. A binary image 45c is obtained in this way.

  Note that the data shown as the input image 40 in FIG. 3 is a one-dimensional array, and the number of data is 240 pixels and 320 pixels in the vertical and horizontal directions, respectively, and three RGB components are taken into consideration. Therefore, it is three times that. FIG. 4 is a conceptual diagram showing the pixel arrangement of the input image 40. The coordinates are the origin at the upper left in FIG. 4, from which the X axis is 320 pixels rightward and the Y axis is 240 pixels downward, i-th on the X-axis and j-th on the Y-axis, respectively. Is expressed as (xi, yi).

Further, the difference calculation will be described with reference to FIG. FIG. 5 is a flowchart showing the processing contents executed by the RGB background difference extraction unit 44c. 5, in step S11, the lightness R (x, y), G (x, y), B (x, y) of the pixel (x, y) in the input image 40 and the pixel (x in the background image data 23) , Y) brightness R ^B (x, y), G ^B (x, y), B ^B (x, y) are extracted, and the discrepancy D _B is obtained in step S12. In step S13, by comparing the threshold _{S D} for the inconsistency _{D B} and inconsistency, if inconsistency degree _{D B} is larger than the threshold value _{S D} is the value of pixel (x, y) 1 If this is not the case, a binarization process of 0 is performed. As a result, a binary image 45c as shown in FIG. 2 is extracted. The threshold value _SD is a numerical value set in advance as appropriate.

  Next, the skin color area extraction unit 44a shown in FIG. 2 will be described with reference to FIG. The skin color area extraction unit 44a reads the input image 40 directly from the imaging unit 2 or reads it from the first database 17 in the same manner as the RGB background difference extraction unit 44c, etc., and compares the input image 40 with the skin model data 24a to generate a binary image 45a. Extract. Here, the skin model data 24a will be described. The I value indicating the skin color is a technique already known as a numerical value represented by Expression (1) using the RGB component of each pixel, and is generally used.

  As the skin model data 24a, the distribution of the I value of the skin color of the person to be imaged is assumed to be a normal distribution and expressed using the average μ of the I value and the standard deviation σ, and each pixel in the input image 40 is expressed. On the other hand, by using the discriminant represented by the equation (2), a binary image 45a is extracted by performing binarization processing by inputting 1 to a pixel determined to be skin color, and otherwise inputting 0. To do. In other words, the template data 24 includes such a case expressed by an expression.

  Next, the normalized lightness background difference extraction unit 44b extracts a background difference in the same manner as the RGB background difference extraction unit 44c, but the normalized lightness background difference extraction unit 44b extracts a difference in normalized lightness. There are features. This normalized lightness represents the ratio of the lightness of each pixel RGB shown in step S11 of FIG. 5 to the total lightness, and is specifically expressed by the following equation (3). The

It is known that taking the difference in normalized brightness in this way has a characteristic that there is no effect on the normalized brightness value even if the illumination condition changes compared to taking the difference in RGB background. Since the influence of light colors such as white can be reduced, for example, a person's part can be extracted more clearly without extracting a futon or the like.
In the present embodiment, in particular, a patient in a hospital room is assumed as a subject person, and in this background difference, in addition to the person, an object moved from the initial position is also extracted. In order to do this, it is important to eliminate as much as possible white or colored materials such as moving futons and pillows. Therefore, not only the RGB background difference extraction unit 44c but also the normalized lightness background difference extraction unit 44b is added.
Similarly to the RGB background difference extraction unit 44c, the normalized lightness background difference extraction unit 44b performs binarization processing according to the flowchart shown in FIG. 5 and extracts a binary image 45b.

Furthermore, the hair color region extraction unit 44d also reads the hair model data 24b from the second database 18 and performs a binarization process while comparing it with the data extracted from the hair color region extraction unit 44d to obtain a binary image 45d.
The binary images 45a to 45d obtained by the skin color region extraction unit 44a to the hair color region extraction unit 44d in this way are binary values related to the common portion using the common region extraction unit 46a and the common region extraction unit 46b. An image 47a and a binary image 47b are extracted. In this way, binarization processing is performed on the common region to obtain a binary image, whereby a head region candidate with higher accuracy is extracted.

A method of acquiring the binary image 47b by extracting the common region by the common region extraction unit 46b using the binary image 45c and the binary image 45d will be described with reference to FIG. FIG. 7 is a conceptual diagram for explaining that the common area extraction unit 46b extracts a common image from the binary image of the RGB background difference and the hair color.
In FIG. 7, a binary image A is a binary image 45c extracted by the RGB background difference extraction unit 44c, and a binary image B is a binary image 45d extracted by the hair color region extraction unit 44d. These binary images A and B are read by the common area extraction unit 46b, and 0 is input to the common image of the two images except for 1 and the common part to obtain the binary image 47b. In the calculation in the common area extraction unit 46b, the pixel value of the pixel (x, y) on the binary image A is A (x, y), and the pixel value of the pixel (x, y) on the binary image B is calculated. Assuming B (x, y), the output binary image C obtains a value based on an equation as shown in equation (4). That is, if A (x, y) and B (x, y) are each 1, C (x, y) is also 1, but if either is 0, C (x, y) is also 0.

The sign 中 in equation (4) indicates a logical product. In the binary image 47b, a portion painted in black is a common portion, and this portion is a region where pixels having a change extracted by the RGB background difference and pixels extracted as a hair color are mixed. Therefore, it can be said that this region is a candidate for a hair region in the head. Similar to the common area extraction unit 46b, the binary image 47a is obtained by extracting the common area using the binary images 45a to 45c. The binary image 47a can be said to be a face region candidate, whereas the binary image 47b is a hair region candidate.
The head detecting unit 8a extracts two binary images, and the template matching unit 11a further compares the binary images with the head template data 24c stored in the second database 18 in advance to obtain a final head. A region binary image 48a is extracted. That is, at the time of the binary image 47a and the binary image 47b, it is still a head candidate, and there may be a plurality of each of them. The head data of the person to be copied can be narrowed down to one. In detail, the head template data 24c includes two types of face template data and hair template data. The binary image 47a and the binary image 47b are collated with each other to obtain a head region binary image 48a. Extract it.

A method of extracting the head region binary image 48a in the template matching unit 11a will be described with reference to FIGS. FIG. 8 is a conceptual diagram showing a state in which head candidate data is collated with head template data. FIG. 9A schematically shows a state in which a binary image 47a related to a head candidate region is captured as a rectangular border. FIG. 9B is a conceptual diagram for showing a part used for calculation in the head template data 24c.
8 and 9A, the binary image 47a related to the head candidate region extracted by the common region extraction unit 46a is regarded as a rectangle including the same, and the center (x, y) is obtained. With respect to this center, the center of the head template data 24c stored in advance in the second database 18 is obtained as (x, y−h / 4). The center coordinates of the head template data 24c are obtained as a rough outline based on the creation of a human face, and may be modified as appropriate.
Next, as shown in FIG. 9B, the major axis “a” and the minor axis “b” of the head template data 24c are obtained by Expression (5).

By defining in this way, the position of the bangs coincides with the upper side of the previously considered rectangle. Therefore, the upper side of the rectangle is the hair region, and the lower side is the face region. Therefore, the sum T of the number of pixels of the binary image 47a in the face region of the head template data 24c and the number of pixels of the binary image 45c in the hair region of the head template data 24c is obtained, and the equation (6) The precision r expressed by is obtained.
In the head template data 24c stored in the second database 18, the portion used for calculation of the precision is limited by the long axis and the short axis.

Among the candidate areas, a threshold value is provided for the matching rate r that is maximized, or a threshold value that exceeds the threshold value is extracted as a head area binary image 48a.
If it is the maximum, one head is always recognized, but if there are relatives other than the patient or medical personnel in the hospital room as described above, there is a possibility that one person other than the patient will be recognized. For this reason, even if one tries to use the task control means, the task control means will not function if one part is always detected. Therefore, when operating the task control means or the like, a threshold value is provided so that when there are a plurality of persons in the imaging region, a plurality of parts, that is, parts that are not single are detected. desirable.
Needless to say, the formula using h of the major axis a and the minor axis b is an example and may be modified as appropriate.
Further, the extraction of the head region binary image 48a described above in the present embodiment is based on the face template data in the head template data 24c, but the head region binary image 48a is obtained using the hair template data. May be determined. That is, when the binary image 47b is used, the number of pixels of the binary image 47b in the hair region of the head template data 24c and the number of pixels of the binary image 45c in the face region of the head template data 24c. And the matching rate r ′ expressed in the same manner as Equation (6) may be obtained. In this case, the formula (5) may be appropriately modified and used.
Furthermore, the matching rate of each other may be applied to each head region candidate, and the region with the highest matching rate may be set as the head region.

Next, the body detection unit 8b which is a kind of the part detection unit 8 shown in FIG. 1 will be described with reference to FIG. FIG. 10 is a configuration diagram of the trunk detection unit 8b assuming a trunk as a part.
In FIG. 10, an input image 40 input to the body detection unit 8b is read out by a body candidate data extraction unit 10b, which is a kind of part region candidate data extraction means 10, and RGB constituting the body candidate data extraction unit 10b. It is transmitted to the difference extraction unit 50. This RGB difference extraction unit 50 is the same as the RGB background difference extraction unit 44c of the head detection unit 8a shown in FIG. 2, and reads and compares the background image data 23 stored in the second database 18. The background difference is taken.

The background difference area 51 obtained here is sent to the clustering unit 52. The clustering unit 52 performs clustering on the pixels in the background difference area 51. Clustering means grouping or grouping, and the body region is detected more easily and accurately by grouping pixels having similar brightness at similar positions.
Usually, the person to be imaged wears some kind of clothes, and the clothes are, for example, pajamas but are often provided with a pattern. Therefore, the feature of the pattern is stored in advance as template data of the trunk in the template data 24, and clustering is performed in order to collate with the template data of the trunk.

A clustering procedure will be specifically described.
(1) First, J _N pixels in the background difference area 51 are selected at random and set as the initial cluster center. In the initial clusters, each of the coordinates of J _N pixels, the color component as a component of the average vector of each cluster.
(2) Next, the Euclidean distance d (k, p) between the pixel p and the average vector Z _k of the cluster k (k = 1, 2,..., J _N ) is obtained, and the pixel p is set to d (k, Integrate into the cluster with the smallest p).
(3) The process (2) is performed on all the pixels in the background difference area 51.
(4) Recalculate the average vector Z _k of the cluster k reconstructed by (3).
(5) The distance between the average vectors of each cluster is calculated, and the clusters at the minimum distance are integrated.
(6) until the number of clusters is _{J P} (4), and repeats the processing of (5).
Cluster number _{J P} (6) may be set as appropriate. Further, the clustering unit 52 calculates the normalized lightness with respect to the average value of the lightness in each cluster obtained by clustering in this way. The normalized brightness value and preset template data of the trunk are collated by the pajama model matching unit 11b to extract the trunk region binary image 48b. The body template data is, for example, a model of pajama color distribution, and assumes that the pajama color distribution model is a normal distribution and uses its mean μ and standard deviation σ as in the skin color I value distribution. In other words, the cluster having the largest number of pixels satisfying this threshold value is detected as the body cluster from the same threshold value as in the expression (3). However, when the number of pixels included in the pajama color distribution model is equal to or less than a threshold value set separately in advance, it is determined that there is no body in the image.

Next, a hand detection unit 8c, which is a type of the part detection unit 8 shown in FIG. 1, will be described with reference to FIG. FIG. 11 is a configuration diagram of the hand detection unit 8c that assumes a hand as a part.
In FIG. 11, an input image 40 input to the hand detection unit 8c is read by the hand candidate data extraction unit 10c as a kind of part region candidate data extraction means 10, and the skin color region constituting the hand candidate data extraction unit 10c. The data is transmitted to the extraction unit 54a, the interframe difference extraction unit 54b, and the edge binary image extraction unit 54c.
Among these, the skin color area extraction unit 54a has the same operations and effects as the skin color area extraction unit 44a shown in FIG. In order to extract the skin color region binary image 55a, the skin model data of the hand is stored in the template data 24 of the second database 18, and a binary image 45a is obtained by using an equation similar to the equation (1). The same skin color area binary image 55a can be extracted.

On the other hand, the inter-frame difference extraction unit 54b will be described with reference to FIG. FIG. 12 is a conceptual diagram for explaining the operation of the inter-frame difference extraction unit 54b. In FIG. 12, the input image 40 is read out, and the input image 40 a before _Tb frame is read out from the part region data 21 stored in the first database 17. The inter-frame difference binary image 55b is calculated by taking the inter-frame difference regarding the two input images.
A method of calculating the inter-frame difference binary image 55b will be described with reference to FIG. FIG. 13 is a flowchart showing the processing contents executed by the inter-frame difference extraction unit 54b. 13, the brightness R of the pixels in the input image 40 in step S21 (x, y) (x , y), G (x, y), B (x, y) and _{T b} previous frame of the input image 40a in The brightness R ^F (x, y), G ^F (x, y), and B ^F (x, y) of the pixel (x, y) are extracted, and the mismatch degree D _F is obtained in step S22. In step S23, by comparing the threshold _{S F} for that inconsistency _{D F} and inconsistency, if inconsistency degree _{D F} is larger than the threshold value _{S F} is the value of pixel (x, y) 1 If this is not the case, a binarization process of 0 is performed. Thereby, the inter-frame difference binary image 55b as shown in FIG. 11 is extracted. The threshold value _SF is a numerical value set in advance as appropriate.

Next, the edge binary image extraction unit 54c will be described with reference to FIG. FIG. 14 is a conceptual diagram for explaining the operation of the edge binary image extraction unit 54c. In FIG. 14, reading of the input image 40 expressed as step S31 by the edge binary image extraction unit 54c leads to an edge component detection step in step S32. The detection of the edge component is performed as if it were as if for each of the RGB components of four adjacent pixels, as is apparent from the coordinates shown.
The edge component thus obtained is obtained as a sum E of the edge components of the pixels in the input image 40 as shown in step S33, and the sum E of the edge components and the threshold value Se are compared to determine the edge component. If the sum E is large, the pixel value is set to 1, and if not, the pixel value is set to 0 and binarization processing is performed. In this way, the edge binary image 55c is extracted.

Next, returning to FIG. 11, the common area extraction unit 56 will be described. The common area extraction unit 56 extracts a binary image 57 including a hand candidate area by the same method as the common area extraction unit 46b described above with reference to FIG.
The edge matching unit 11c extracts the hand region binary image 48c from the binary image 57 including the hand candidate region and the edge binary image 55c described above. The operation of extracting the hand region binary image 48c by the edge matching unit 11c will be described with reference to FIG. FIG. 15A is a conceptual diagram schematically showing a binary image including a hand candidate area, and FIG. 15B is a conceptual diagram schematically showing an edge binary image. In FIG. 15A, the circles shown indicate the boundary lines of the binary image 57 including the hand candidate area. This boundary line is first extracted, and then the logical product of the binary image and the edge binary image 55c at this boundary line is obtained. This logical product is the one described above as equation (4). In both cases, when the pixel value is 1, the logical product is 1, but the number of pixels having the pixel value of 1 is counted. Then, the ratio of the number of pixels having the pixel value of 1 as the coincidence rate and the number of pixels on the previous boundary line is calculated. Then, an area where the coincidence rate is maximum is taken as a hand area.
As described in the section of the head detection unit 8a, if it is the maximum, the task control means will not function although one or a pair of hands will always be recognized. It is desirable to set a threshold value when operating the camera, and to detect a plurality of parts, that is, parts that are not singular when there are a plurality of persons in the imaging region.

  In the description of the present embodiment, the foot detection unit can be extracted with the same algorithm as the hand detection unit. Specifically, for example, in FIG. 11, an element configured for the hand may be used as a component for the foot. The skin color area extraction unit 54a, the interframe difference extraction unit 54b, the edge binary image extraction unit 54c, and the like may be configured for the foot. The same applies to other components. Of course, it is necessary to store template data for the foot in the second database 18.

Heretofore, the part detection unit 8 has been described. Next, the part area data change amount calculation means 14a and the specific area data change amount calculation means 14b constituting the behavior feature detection unit 6 will be described with reference to FIG.
FIG. 16A is a flowchart for explaining the operation of the specific area data change amount calculating means 14b, and FIG. 16B is a flowchart for explaining the action of the region existing area detecting means 14c. In FIG. 16A, regarding the number of pixels of the RGB background difference area at time t included in each area of the divided specific area extracted by the specific area detection unit 9, the same RGB background difference area before the unit time is obtained. By comparing the number of pixels, for example, it is determined whether or not the difference of the futon area (area 3) has increased. This area 3 has been described above as an example, but is as shown in FIG. Specifically, when the patient on the bed gets up, the area around the head position belongs to area 1, area around the pillow area 2, area 3 of the futon area, and areas 1, 2, and 3 around the bed. The area that does not exist is designated as area 4.
In FIG. 16B, paying attention to the center-of-gravity coordinates of the head extracted by the head detection unit 8a, it is determined whether or not the center-of-gravity coordinates are included in the area 1. Otherwise, binarization processing is performed with 0.

  FIG. 18 is a conceptual diagram expressing behavior features defined by the result of the binarization process using the output from the specific area detection unit 9 executed as shown in FIG. In FIG. 18, for example, the current state is set to “sleeping”, and the specific area detection unit 9 obtains the difference in the number of pixels of the RGB difference of area 1 at time t and time t−1 and increases it. It is judged whether it is equal or decreased.

The result itself is included in a part of the behavior feature data table 25 shown in FIG. FIG. 19 is a conceptual diagram illustrating an example of a behavior feature data table. For example, if it is assumed that the number of pixels of the RGB difference in area 1 in FIG. 18 has increased in area 1, the behavior feature data table 25 in FIG. 19 has the behavior feature indicated by ID = b2. The calculation and determination shown in FIG. 18 are executed by the specific area data change amount calculation means 14b in FIG. Further, the behavior feature data obtained as a result of the execution by the specific area data change amount calculation means 14b is read out by the behavior feature extraction means 15 from the behavior feature data table 25 stored in the second database 18. The specific area data change amount 31, the specific area data change speed 32, and the specific area data change acceleration 33 calculated by the specific area data change amount calculation unit 14 b are stored in the third database 19.
In addition to using the specific area data 22 extracted by the specific area detection unit 9, for example, the part presence area detection means 14c regarding the barycentric coordinates of the head extracted by the head detection unit 8a shown in FIG. 16 is performed, and the behavior feature data table 25 stored in the second database 18 is read by the behavior feature extraction means 15 and the behavior feature indicated by ID = a7 is “head. “Existing in area 1” is extracted. The part existing area data 30 calculated by the part existing area detecting means 14 c is stored in the third database 19.

Although not shown in FIG. 16, the region region data change amount calculation means 14 a similarly calculates the change amount due to the movement of the center of gravity coordinates of the head extracted by the head detection unit 8 a, and the change For example, the action feature indicated by ID = a1 in FIG. 19 “the head moves at a certain speed or higher” is extracted from the quantity. The part region data change amount 27, the part region data change rate 28, and the part region data change acceleration 29 calculated by the part region data change amount calculation unit 14 a are stored in the third database 19.
The behavior features extracted by the behavior feature detection unit 6 using the behavior feature data table 25 in this way are transmitted to the behavior understanding unit 7 shown in FIG. In the behavior understanding unit 7, the human behavior extraction means 16 reads out the behavior understanding data table 26 stored in the second database 18 by combining the behavior features, and extracts the final human behavior.

  As a major feature of the human behavior understanding system 1 according to the present embodiment, there is a behavior understanding process based on the precursor behavior feature and the confirmed behavior feature described above. FIG. 20 schematically shows this process of understanding behavior. In FIG. 20, it is indicated that “behavior” is understood as “behavior” under “behavior feature”, but the first “behavior feature” is a precursor behavior feature and the subsequent “behavior feature” is It becomes a definite behavior feature. In this behavior understanding process, an action is not understood with one behavior characteristic, but is understood with behavior only with two behavior characteristics of a precursor and a confirmation. By providing a two-stage action understanding process in this way, the action understanding is ensured and no misunderstandings are caused. In addition, when there are a plurality of behaviors with a predictive behavior feature, these are considered as OR, and when there are a plurality of behaviors with a definite behavior feature, they are considered as AND. This clarifies the stance of broadly considering the possibilities but narrowing down when they are finalized.

A specific example of the behavior understanding process will be described with reference to FIGS. 21 and 22. FIGS. 21A to 21D are examples of the behavior understanding data table 26 stored in the second database 18. FIG. 22 (a) shows a state transition diagram of the behavior of the person to be imaged, and FIG. 22 (b) further expands a part of the behavior state transition diagram of (a). It is a conceptual diagram which shows the conditions of the transition.
In FIG. 21A, based on the predictive behavior feature “background difference increases in area 3”, the definite behavior feature “head exists in area 1” and “background difference does not exist in area 4”. If it has another deterministic behavior feature, it is understood that the behavior is “being awake because of sleeping”. Since the other FIGS. 21B to 21D are as described, the description thereof is omitted.
In FIG. 22A, reference numerals 60a to 60e indicate candidate human states, and reference numerals 61a to 61g indicate the direction of the state transition. Since there is a possibility of transition between the sleeping state 60a and the waking state 60d, the arrows are pointing to each other, but the sleeping state 60a and the falling state 60b fall from the sleeping state 60a. The arrow is pointing only to the active state 60b.
Looking at the behavior of FIG. 21 (a), it is a transition from “sleeping” to “waking up”, and the arrow from the reference numeral 60a to the reference numeral 60d in FIG.

This transition condition will be described once again with reference to FIG. In FIG. 22 (b), it is considered that there are some behavior features extracted by the behavior feature detection unit 6, and among them, 1. If there is an action feature detection for “Waking up” expressed by “”, in order to proceed to the state transition direction indicated by the reference numeral 61a and become “Waking up” state, 2. The behavioral feature detection for “Waking up” expressed in the above must be made.
This is called the precursor behavior feature and the definite behavior feature described above. Only when these two behavior features are detected is a state transition determined, and the state transition is a behavior.
In the calculation using the behavior understanding data table 26 for understanding the behavior, a logical calculation may be executed by using the ID as shown in FIG. 19 as a key.

The contents understood as actions in this way are displayed on the output unit 4 as shown in FIG.
At that time, various data and calculation results stored in the first database 17 to the third database 19 may be appropriately displayed.

Next, feedback of the human behavior understanding system 1 according to the present embodiment will be described with reference to FIGS.
FIG. 23 is a schematic configuration diagram including feedback of the entire human behavior understanding system 1, FIG. 24 (a) is a conceptual diagram for explaining the action of feedback, and FIG. 24 (b) shows feedback conditions. It is a flowchart for demonstrating.
In FIG. 23, as already described, the input image 40 is read by the human body part detection unit 5, and the human body part detection unit 5 detects a head, a hand, and a torso. After detection, the behavior feature detection unit 6 extracts behavior features based on the information, and the behavior understanding unit 7 performs behavior understanding based on the extracted behavior features. The first feedback executed from the behavior understanding unit 7 to the behavior feature detection unit 6 so as to be able to cooperate with each other using the hierarchical structure of the human body part detection unit 5, the behavior feature detection unit 6, and the behavior understanding unit 7. 65 and a second feedback 66 that is executed from the behavior feature detection unit 6 to the human body part detection unit 5 is provided.

The two feedbacks will be described with reference to FIGS. 24 (a) and 24 (b). In FIG. 24A, it is assumed that the action understanding unit 7 first detects “sleeping”. At that time, the behavior feature detector 6 confirms the presence or absence of contradictory behavior features by the first feedback 65. As a result, as shown in FIG. 24A, it is assumed that the behavior feature “the head is in area 4” is detected. As described above, this area 4 is an area where the head cannot be there when sleeping such as under a bed. Therefore, the behavior feature is inconsistent with the behavior understanding in the behavior understanding unit 7. Therefore, in the human behavior understanding system 1 according to the present embodiment, the behavior feature detection unit 6 performs the second operation on the human body part detection unit 5. The feedback 66 is executed to instruct redetection by the head detection unit.
On the other hand, as shown in FIG. 24 (b), if the behavior feature “the head is in area 4” is not detected by the behavior feature detection unit 6, there is no contradiction, so there is no feedback. It becomes. As described above, the flow of information from the human body part detection unit 5 to the behavior understanding unit 7 and the check function that flows in the opposite direction are provided, and by using the hierarchical structure effectively, more accurate human behavior It is an understanding system.

  As described above, the human behavior understanding system 1 according to the present embodiment has a hierarchical structure of the human body part detection unit 5, the behavior feature detection unit 6, and the behavior understanding unit 7, and processes input images while being distributed and coordinated. And high accuracy can be achieved. In addition to a person's part, assuming a specific area, dividing it into multiple parts and calculating the increase / decrease of the background difference for each area, not only the person to be imaged but also the surrounding changes At the same time, the pixel data to be processed is relatively light because it is not divided into areas, and the surroundings are not analyzed in detail. Therefore, there is a remarkable effect that there is almost no limit on the performance of the arithmetic unit necessary for judgment of behavior understanding while exhibiting high accuracy, and the necessary time can be shortened.

Next, the verification test conducted by the inventor will be described with reference to FIGS.
FIG. 25A is a conceptual diagram schematically showing the state of the verification test, and FIG. 25B shows an example of an image actually captured by a fixed camera.
A demonstration test was conducted simulating a patient lying on a hospital bed. There is one fixed camera and images the patient and the surrounding area from the ceiling of the hospital room.
FIG. 26A shows an example of the input image 40 shown in FIG. 2, for example, FIG. 26B shows a background difference image extracted by the RGB background difference extraction unit 44c, and FIG. 2 is an image extracted by the normalized lightness background difference extraction unit 44b, FIG. 2D is an image extracted by the skin color region extraction unit 44a, and FIG. 2E is an image extracted by the hair color region extraction unit 44d. Finally, (f) shows an image extracted by the inter-frame difference extraction unit 54b shown in FIG. 11, for example. These (b) to (f) are images obtained based on the input image of (a).

FIGS. 27A to 27F show states indicating “sleeping”, “getting up”, “outside the bed”, “outside the imaging region”, “falling”, and “grabbing the fence”, respectively. These are examples of states to be detected by the person behavior understanding system 1.
Finally, FIG. 28 shows an input image and an example detected thereby. (A) is an input image, (b) is a head detection example, (c) is a hand detection example, and (d) is a torso detection example.
Tables 1 and 2 summarize the results of the demonstration tests shown in this way.

  Table 1 shows human body part detection results. From Table 1, the detection rate of the hand is significantly lower than the detection rate of the head and torso, but the level of the head and torso is almost detectable.

  Table 2 shows the behavior understanding results from the initial state of “sleeping”. From Table 2, “sleeping”, “waking up”, “out of bed”, and “leaving out” were detected with a high accuracy of 90% or more. Further, the state of “falling” also has a detection rate of 70% or more, and it can be said that the human behavior understanding system according to the present embodiment is sufficiently practical.

  It can be used widely from general households to medical facilities such as hospitals and nursing care facilities for patients and elderly people who need nursing care. Moreover, even if it is not in such a medical field, it can be applied to monitoring dangerous work and the like by appropriately modifying the behavior feature data table and the behavior understanding data table.

It is a lineblock diagram of a person action understanding system concerning an embodiment of the invention. It is a block diagram of the head detection part which assumes a head as a site | part of the site | part detection part which concerns on this Embodiment. It is a flowchart which shows the effect | action in an RGB background difference extraction part. It is a conceptual diagram which shows pixel arrangement | positioning of an input image. It is a flowchart which shows the processing content performed in the RGB background difference extraction part. It is a flowchart which shows the effect | action in a skin color area | region extraction part. It is a conceptual diagram for demonstrating that a common area extraction part extracts a common image from the binary image of skin color and hair color. It is a conceptual diagram which shows the state which collates head candidate data with head template data. (A) is a conceptual diagram which shows typically a mode that the binary image which concerns on a head candidate area | region is caught as a rectangular border, (b) is a concept for showing the part used for a calculation among head template data. FIG. It is a block diagram of the trunk | drum detection part which assumes a trunk | drum as a site | part. It is a block diagram of the hand detection part which assumes a hand as a site | part. It is a conceptual diagram for demonstrating the effect | action of the difference extraction part between frames. It is a flowchart which shows the processing content performed in the difference extraction part between frames. It is a conceptual diagram for demonstrating the effect | action in an edge binary image extraction part. (A) is a conceptual diagram schematically showing a binary image including a hand candidate region, and (b) is a conceptual diagram schematically showing an edge binary image. (A) is a flowchart for demonstrating the effect | action of a specific area data variation | change_quantity calculating means, (b) is a flowchart for demonstrating the effect | action of a site | part location area | region detection means. It is a conceptual diagram which shows the example of the division area of a specific area. It is a conceptual diagram expressing the action characteristic defined by the result of the binarization process using the output from the specific area | region detection part performed as FIG. 16 shows. It is a conceptual diagram which shows an example of an action characteristic data table. It is the conceptual diagram which expressed the process of behavior understanding typically. (A) thru | or (d) are all examples of the action understanding data table stored in a 2nd database. (a) shows the state transition diagram of the behavior of the person to be imaged, and (b) shows the condition of the transition while enlarging a part of the state transition diagram of the behavior of (a). It is a conceptual diagram. It is a schematic block diagram including the feedback of the whole human action understanding system. FIG. 24A is a conceptual diagram for explaining the action of feedback, and FIG. 24B is a flowchart for explaining conditions for feedback. (A) is the conceptual diagram which expressed the mode of the verification test typically, (b) shows the example of an image actually imaged with the fixed camera. (A) shows an example of the input image shown in FIG. 2, for example. (B) shows the background difference image extracted by the RGB background difference extraction unit, and (c) shows the regular image shown in FIG. (D) is an image extracted by the skin color region extracting unit, (e) is an image extracted by the hair color region extracting unit, (f) Indicates an image extracted by the interframe difference extraction unit shown in FIG. 11, for example. (A) to (f) show states indicating sleeping, getting up, outside the bed, outside the imaging area, falling, and grabbing the fence, and these are the detection targets by the human behavior understanding system 1. It is an example. (A) is an input image, (b) is a head detection example, (c) is a hand detection example, and (d) is a torso detection example.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Person action understanding system 2 ... Imaging part 3 ... System main body 4 ... Output part 5 ... Human body part detection part 6 ... Action feature detection part 7 ... Action understanding part 8 ... Part detection part 8a ... Head detection part 8b ... Body detection Part 8c ... Hand detection part 9 ... Specific area detection part 10 ... Part region candidate data extraction means 10a ... Head candidate data extraction part 10b ... Body candidate data extraction part 10c ... Hand candidate data extraction part 11 ... Part area data collation means 11a ... Template matching unit 11b ... Pajamas model matching unit 11c ... Edge matching unit 12 ... Specific area data extraction means 13 ... Task control means 14a ... Partial area data change amount calculating means 14b ... Specific area data change amount calculating means 14c ... Part existence area Detecting means 15 ... action feature extracting means 16 ... person action extracting means 17 ... first data 18 ... 2nd database 19 ... 3rd database 20 ... Imaging data 21 ... Site region data 22 ... Specific region data 23 ... Background image data 24 ... Template data 24a ... Skin model data 24b ... Hair model data 24c ... Head template Data 25 ... Action feature data table 26 ... Action understanding data table 27 ... Part region data change amount 28 ... Part region data change speed 29 ... Part region data change acceleration 30 ... Part existing region data 31 ... Specific region data change amount 32 ... Specific Region data change speed 33 ... Specific region data change acceleration 40 ... Input image 40a ... Input image 44a ... Skin color region extraction unit 44b ... Normalized lightness background difference extraction unit 44c ... RGB background difference extraction unit 44d ... Hair color region extraction unit 45a ... Binary image 45b ... Value image 45c ... Binary image 45d ... Binary image 46a ... Common region extraction unit 46b ... Common region extraction unit 47a ... Binary image 47b ... Binary image 48a ... Head region binary image 48b ... Body region binary image 48c ... hand region binary image 50 ... RGB difference extraction unit 51 ... background difference region 52 ... clustering unit 54a ... skin color region extraction unit 54b ... interframe difference extraction unit 54c ... edge binary image extraction unit 55a ... skin color region binary image 55b ... Inter-frame difference binary image 55c ... Edge binary image 56 ... Common area extraction unit 57 ... Binary image including hand candidate areas 60a to 60e ... State 61a to 61g ... State transition direction 65 ... First feedback 66 ... First 2 feedback

Claims (8)

An imaging unit that detects an imaging region including a person to be imaged, a part detection unit that extracts part region data included in a person region imaged by the imaging unit, and a part region extracted by the part detection unit A human behavior having a behavior feature detection unit that detects a feature of a behavior based on a change amount of data, and a behavior understanding unit that determines the behavior of the person by combining the behavior features detected by the behavior feature detection unit An understanding system,
The part detection unit includes means for extracting part region candidate data of a part region candidate included in the person region imaged by the imaging unit, and the extracted part region candidate data and the preset part region Means for collating with template data and extracting part region data;
The behavior feature detection unit is configured to calculate a change amount of the part region data detected by the part detection unit and a behavior feature data table indicating a behavior feature determined by the change amount of the part region data. Means for extracting behavioral features,
The behavior understanding unit includes means for extracting the behavior of the person from a behavior understanding data table that associates the combination of the behavior features of the person extracted by the behavior feature detection unit with the behavior of the person set in advance. Human behavior understanding system characterized by that. An imaging unit that detects an imaging region including a specific region that is defined in advance separately from a person region that includes a person to be imaged, and a region that extracts part region data included in the person region imaged by the imaging unit A human body part detection unit comprising a detection unit and a specific area detection unit for extracting the specific area data; and an action for detecting a feature of the action based on the region area data extracted by the human body part detection unit and the amount of change in the specific area data A human behavior understanding system comprising: a feature detection unit; and a behavior understanding unit that determines the behavior of the person by combining the behavior features detected by the behavior feature detection unit,
The part detection unit includes means for extracting part region candidate data of a part region candidate included in the person region imaged by the imaging unit, and the extracted part region candidate data and the preset part region Means for collating with template data and extracting part region data;
The specific area detection unit includes means for extracting specific area data included in the specific area imaged by the imaging unit,
The behavior feature detection unit calculates a change amount of the part region data detected by the part detection unit, and calculates a change amount of the specific region data detected by the specific region detection unit, Means for extracting a behavior feature of the person from a behavior feature data table indicating a behavior feature defined by the amount of change of the part region data and the amount of change of the specific region data;
The behavior understanding unit includes means for extracting the behavior of the person from a behavior understanding data table that associates the combination of the behavior features of the person extracted by the behavior feature detection unit with the behavior of the person set in advance. Human behavior understanding system characterized by that.   The specific region is a plurality of regions divided in advance, and the behavior feature detecting unit determines in which region of the divided specific region the region region data extracted by the region detecting unit exists. 3. The human behavior understanding system according to claim 2, further comprising means for detecting the part presence region data, wherein the behavior feature table associates the part presence region data with the behavior feature.   The part detection unit includes a head detection unit that extracts part region data of a person's head region, a hand detection unit that extracts part region data of a hand region, and a torso detection unit that extracts part region data of a torso region 4. The human behavior understanding system according to claim 1, further comprising: at least one of a foot detection unit that extracts part region data of the foot region. 5.   The change amount of the part region data is any one of a change amount of the data of the part region, a change speed of the change amount, and a change acceleration of the change amount. The human behavior understanding system according to any one of the above.   The behavior understanding unit transmits a signal so that re-detection is performed in the behavior feature detection unit or the human body part detection unit when the behavior of the person cannot be extracted from the behavior understanding data table. The human behavior understanding system according to any one of claims 1 to 5.   The said part detection part is provided with the task control means to operate the said specific area | region detection part, the said action feature detection part, and the said action understanding part, when the extracted said part area | region data is single. The human behavior understanding system according to any one of claims 1 to 6. The task control means for operating the specific region detection unit, the behavior feature detection unit, and the behavior understanding unit when the region region data extracted by the region detection unit is independent. The human behavior understanding system according to any one of claims 1 to 6.