JP2022073882A - Behavior estimation device, behavior estimation method, program and behavior estimation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy of estimating human behavior.

SOLUTION: A behavior estimation device 10 includes an image acquisition unit 131 for acquiring captured image data representing captured images, a person detecting unit 132 for detecting movement of a person contained in a captured image, a subject detecting unit 133 for detecting a type of a subject contained in the captured image, a relation specification unit 135 for specifying a relative relation of the person and the subject induced by the person's movement, and an estimation unit 136 for estimating a content of the person's behavior based on a combination of the type of the subject and the relative relation.

SELECTED DRAWING: Figure 4

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a behavior estimation device, a behavior estimation method, a program, and a behavior estimation system for estimating a person's behavior.

A behavior estimation device that detects a human joint in a time-series image and estimates the behavior of the subject based on the change in the position of the detected joint is known (see, for example, Patent Document 1).

Japanese Patent No. 6525181

Since the conventional behavior estimation device estimates the behavior based only on the movement of the subject, there is a problem that it is difficult to discriminate the behavior having similar movements.

Therefore, the present invention has been made in view of these points, and an object thereof is to improve the accuracy of estimating human behavior.

The behavior estimation device according to the first aspect of the present invention includes an image acquisition unit that acquires captured image data showing an captured image, a human detection unit that detects the movement of a person included in the captured image, and the captured image. The subject detection unit that detects the type of the included subject, the relationship specifying unit that specifies the relative relationship between the person and the subject due to the movement of the person, the type of the subject, and the relative relationship. It has an estimation unit that estimates the behavior content of the person based on the combination.

The relationship specifying unit specifies the distance between the person and the subject as the relative relationship, and the estimation unit uses the reference distance associated with the type of the subject detected by the subject detecting unit and the relationship specifying unit. The action content may be estimated based on the result of comparing the distance between the person and the subject specified by.

The relationship specifying unit specifies the relationship between the direction of the person's line of sight and the position of the subject as the relative relationship, and the estimation unit is the line of sight associated with the type of the subject detected by the subject detecting unit. The action content is based on the result of comparing the reference line-of-sight relationship, which is the relationship between the direction of the subject and the position of the subject, and the relationship between the direction of the line of sight of the person specified by the relationship specifying unit and the position of the subject. You may estimate.

The relationship specifying unit specifies the relationship between the orientation of the person's body and the position of the subject as the relative relationship, and the estimation unit is the body associated with the type of the subject detected by the subject detecting unit. The action content based on the result of comparing the reference body orientation relationship, which is the relationship between the orientation of the subject and the position of the subject, and the relationship between the direction of the line of sight of the person specified by the relationship specifying unit and the position of the subject. May be estimated.

The image acquisition unit acquires a plurality of the captured image data created in time series, the relationship specifying unit identifies a change in the relative relationship, and the estimation unit determines the type of the subject and the relative. The behavioral content of the person may be estimated based on the combination with the change of the relationship.

The estimation unit further determines the action content based on the result of determining whether or not the subject was included in any of the plurality of captured image data before the person detection unit detects the person. You may estimate.

The person detection unit may further detect the attribute of the person, and the estimation unit may estimate the behavior content based on the attribute of the person detected by the person detection unit.

The subject detection unit detects the first subject and the second subject included in the captured image, and the relationship specifying unit is the first relative, which is the relationship between the person and the first subject due to the movement of the person. The relationship, the second relative relationship between the person and the second subject due to the movement of the person, and the third relative relationship between the first subject and the second subject due to the movement of the person. The estimation unit identifies the behavior of the person based on the combination of the type of the subject and at least two relative relationships of the first relative relationship, the second relative relationship, or the third relative relationship. The content may be estimated.

The setting reception unit that accepts the setting of the attribute of the subject included in the captured image may be further provided, and the estimation unit may estimate the action content based on the attribute of the subject.

The relationship specifying unit further specifies the positional relationship between the subject detected by the subject detecting unit and the subject for which the setting receiving unit has received the attribute setting, and the estimation unit further identifies the positional relationship specified by the relationship specifying unit. The action content may be estimated based on the relationship and the attribute of the subject for which the setting receiving unit has received the setting.

Output that outputs notification information when a place specifying unit that specifies a place reflected in the captured image, the action content estimated by the estimation unit, and a place specified by the place specifying unit satisfy predetermined conditions. And may further have.

The output unit may output the notification information of the type corresponding to the combination of the location specified by the location specifying unit and the action content.

Further having a color specifying unit that specifies at least one of the change in the color of the person or the color of the subject, the estimation unit is the change in the color of the person specified by the color specifying unit or the change in the subject. The behavior may be estimated based further on at least one of the color changes.

The image acquisition unit acquires a plurality of the captured image data, the relationship specifying unit identifies a plurality of the relative relationships, and the estimation unit combines each of the plurality of the relative relationships with the type of the subject. The behavior content of the person may be estimated based on the above.

The behavior estimation method according to the second aspect of the present invention includes a step of acquiring captured image data showing a captured image, a step of detecting a movement of a person included in the captured image, and the captured image, which are executed by a computer. Based on the step of detecting the type of the subject included in the above, the step of specifying the relative relationship between the person and the subject due to the movement of the person, and the combination of the type of the subject and the relative relationship. It has a step of estimating the behavior content of the person.

The program according to the third aspect of the present invention includes a step of acquiring captured image data showing a captured image, a step of detecting a movement of a person included in the captured image, and a subject included in the captured image in a computer. Based on the combination of the step of detecting the type of the subject, the step of specifying the relative relationship between the person and the subject due to the movement of the person, and the type of the subject and the relative relationship, the person. To execute the step of estimating the action content of.

The behavior estimation system according to the fourth aspect of the present invention displays a behavior estimation device that estimates a person's behavior content based on an image captured by the image pickup device, and the behavior content of the person estimated by the behavior estimation device. The behavior estimation device includes an image acquisition unit that acquires captured image data indicating the captured image, a person detection unit that detects the movement of a person included in the captured image, and the captured image. A subject detection unit that detects the type of subject included in the above, a relationship specifying unit that specifies a relative relationship between the person and the subject due to the movement of the person, a type of the subject, and the relative relationship. It has an estimation unit that estimates the behavior content of the person based on the combination of the above, and a transmission unit that transmits the behavior content information indicating the behavior content estimated by the estimation unit to the information terminal. The terminal has an information receiving unit that receives the action content information, and a display unit that displays the action content based on the action content information received by the information receiving unit.

According to the present invention, there is an effect of improving the accuracy of estimating human behavior.

It is a figure for demonstrating the outline of the behavior estimation system S which concerns on this embodiment. It is a figure for demonstrating the structure of the behavior estimation system S. It is a figure for demonstrating the structure of the information terminal 2. It is a figure for demonstrating the structure of the behavior estimation apparatus 10. It is a figure which shows an example of the action content database. It is a flowchart which shows an example of the action which the action estimation apparatus 10 estimates a person's action content. It is a figure which shows the structure of the behavior estimation apparatus 10 which estimates the behavior content of a person based on the change of the color of a person or a subject.

[Overview of behavior estimation system S]
FIG. 1 is a diagram for explaining an outline of the behavior estimation system S according to the present embodiment. FIG. 2 is a diagram showing a configuration of the behavior estimation system S. The behavior estimation system S includes an image pickup device 1, an information terminal 2, and a behavior estimation device 10. As shown in FIG. 2, the image pickup device 1, the information terminal 2, and the behavior estimation device 10 are connected via a network N. The network N is, for example, an intranet or the Internet. The network N may include a wireless network.

FIG. 1 (a) is a view of the space R as viewed from above, and FIG. 1 (b) is a view of the space R as viewed from the side. The space R is a space in which an object is placed, such as a store sales floor, a store warehouse, a factory warehouse, or a facility storage space. As shown in FIG. 1, a plurality of shelves T (T1 to T4) are installed in the space R. As shown in FIG. 1 (b), a plurality of objects P (P1 to P3) are respectively placed on the plurality of shelves T (T1 to T3). As shown in FIG. 1, a person M is staying in the space R. The person M has a bag B in which the object P can be put.

In FIG. 1, the behavior estimation system S includes a behavior estimation device 10 that estimates the behavior content of the person M based on the captured images generated by one or more image pickup devices 1 installed in the space R. The image pickup device 1 is, for example, a surveillance camera, and captures an area for estimating a person's behavior. The image pickup device 1 transmits the captured image data generated by taking an image to the behavior estimation device 10. The image pickup apparatus 1 may generate moving image data as captured image data. The moving image data is, for example, H. 264 or H. The data is encoded by a compression standard such as 265.

The behavior estimation device 10 estimates the behavior content of the person M based on the captured image generated by the image pickup device 1, and transmits the behavior content information indicating the estimated behavior content to the information terminal 2. The behavior estimation device 10 is an information processing device that estimates the behavior content of the person M based on the relative relationship between the person M and the subject included in the captured image, and is, for example, a computer. The subject is, for example, a plurality of shelves T, a plurality of objects P, a bag B, or another person. The behavior estimation device 10 may be a single information processing device, or may be composed of a plurality of information processing devices such as a cloud-type server.

The behavior estimation device 10 identifies the relative relationship between the person M and the subject, and estimates the behavior content of the person M based on the combination of the specified relative relationship and the type of the subject. The relative relationship is, for example, the distance between the person M and the object P, the relationship between the direction of the line of sight of the person M and the position of the object P, or the relationship between the orientation of the body of the person M and the position of the object P. The type of subject is represented by information that can identify the subject based on at least one of shape or color, such as canned beverages, knives, and product baskets. In the present embodiment, the type of the subject may be represented by a product name for the sake of simplicity. The subject is an object or a person.

When the object P2 is put into the bag B after the person M picks up the object P2, the behavior estimation device 10 determines that "the person M in contact with the object P2 is in contact with the object P2" based on the distance between the person M's hand and the subject. The hand is in close proximity to bag B. " Subsequently, when the type of the detected object P2 is "canned beverage" and the type of the detected bag B is "handbag", the behavior estimation system S has the specified relative relationship and the canned beverage which is the type of the subject. And by specifying that "person M put the canned beverage in the handbag" based on the combination with the handbag, it is presumed that the action content of person M is a baggage.

On the other hand, even if the specified relative relationship is the same, if the type of the detected object P2 is "canned beverage" and the type of the detected bag B is "commodity basket", the behavior estimation device 10 has a relative relationship. By specifying that "person M put the canned beverage in the product basket" based on the combination of the canned beverage and the product basket, which are the types of the subject, it is presumed that the action content of the person M is shopping.

The behavior estimation device 10 transmits the estimated result to the information terminal 2. The information terminal 2 is, for example, a smartphone, a tablet, or a personal computer. The information terminal 2 is a terminal used by, for example, a store clerk who monitors the sales floor of a store or an employee who manages a warehouse of a factory (hereinafter, referred to as "manager").

The information terminal 2 displays the behavior content of the person estimated by the behavior estimation device 10 on the display. The information terminal 2 acquires behavior content information including, for example, the behavior content of a person estimated by the behavior estimation device 10 and the captured image of the person, and is included in the captured image included in the behavior content information and the captured image. It may be displayed in association with the behavior content of the person.

In this way, the behavior estimation device 10 estimates the behavior content of the human M based on the combination of the relative relationship between the human M and the subject and the type of the subject, thereby discriminating the behaviors having similar movements with high accuracy. be able to. Then, the action estimation device 10 notifies the information terminal 2 that a preset action (for example, shoplifting action) has been detected, so that the administrator using the information terminal 2 performs the preset action. It is possible to grasp that with high accuracy.

The administrator of the information terminal 2 can set various information used by the behavior estimation device 10 to estimate the behavior content of a person. The information terminal 2 transmits setting information, which is information used for estimating the behavior content of a person, to the behavior estimation device 10 in response to the operation of the administrator.

The setting information includes, for example, at least one of subject attribute information or subject type information. The subject attribute information is information in which the position or shape of the subject is associated with the subject attribute. The subject attribute is information indicating the characteristics of the subject such as the owner of the subject and the function of the subject. For example, the attribute information of the shelf T1 shown in FIG. 1 is information in which the position or shape of the shelf T1 is associated with the "store" indicating the attribute of the shelf T1.

The subject type information is information in which the shape of the subject and the type of the subject are associated with each other. For example, when the subject is a canned beverage, the subject type information is information in which the "canned beverage" indicating the type of the subject is associated with the shape of the canned beverage.

[Configuration of information terminal 2]
FIG. 3 is a diagram for explaining the configuration of the information terminal 2. The information terminal 2 has a communication unit 21, a storage unit 22, a control unit 23, a display unit 24, and an operation unit 25. The control unit 23 has an information transmission unit 231 and an information reception unit 232.

The communication unit 21 includes a communication device for transmitting and receiving information via the network N. The communication device is, for example, a LAN (Local Area Network) controller or a wireless LAN controller.

The storage unit 22 has a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an SSD (Solid State Drive). The storage unit 22 stores a program executed by the control unit 23.

The control unit 23 is, for example, a CPU (Central Processing Unit). The control unit 23 functions as an information transmission unit 231 and an information reception unit 232 by executing a program stored in the storage unit 22. The information transmission unit 231 transmits the setting information of the behavior estimation device 10 created by the administrator who uses the information terminal 2 to the behavior estimation device 10 via the communication unit 21. The information receiving unit 232 receives the behavior content information transmitted by the behavior estimation device 10 via the communication unit 21.

The display unit 24 has, for example, a liquid crystal display or a display device such as an organic EL (Electro Luminescence). The display unit 24 displays the action content based on the action content information received by the information receiving unit 232. The display unit 24 extracts, for example, the behavior content of the person included in the behavior content information received by the information receiving unit 232 and the captured image in which the person is captured. The display unit 24 displays the extracted captured image on the display, and displays the behavior content of the person in the captured image in association with the captured image.

The operation unit 25 is a device that accepts the operation of the administrator who uses the information terminal 2, and is a keyboard, a mouse, or a touch panel. The operation unit 25 notifies the information transmission unit 231 of information indicating the operation content of the administrator.

[Structure of behavior estimation device 10]
FIG. 4 is a diagram for explaining the configuration of the behavior estimation device 10. The behavior estimation device 10 has a communication unit 11, a storage unit 12, and a control unit 13. The control unit 13 includes an image acquisition unit 131, a person detection unit 132, a subject detection unit 133, a setting reception unit 134, a relationship identification unit 135, an estimation unit 136, a location identification unit 137, and an output unit 138. Have.

The communication unit 11 includes a communication device for transmitting and receiving information via the network N. The communication device is, for example, a LAN controller or a wireless LAN controller. The storage unit 12 has a storage medium such as a ROM, RAM, and SSD. The storage unit 12 stores a program executed by the control unit 13. The storage unit 12 stores the setting information acquired from the information terminal 2.

The storage unit 12 stores various types of information used by the estimation unit 136 for estimating the behavior content of a person in a database. The storage unit 12 stores, for example, an action content database in which a person's action content, a person's movement, a subject type, and a relative relationship between the person and the subject are associated with each other. The subject is an object or a person.

FIG. 5 is a diagram showing an example of an action content database. In FIG. 5, the category of human behavior content, the behavior content, and a plurality of requirements for specifying the behavior content are associated with each other. The plurality of requirements are the attributes of the person who estimates the behavior, the movement of the person, the types of the plurality of subjects (“subject 1” and “subject 2” shown in FIG. 5), and the relative relationship between the person and the subject. The attribute of a person indicates the position or role of the person in the space for estimating the behavior of the person, for example, a purchaser or an employee.

For example, in the action content "view", the attribute of the person is "purchaser", the movement of the person is "watching", the subject 1 is "product", and the relative relationship is "the distance between the person and the subject 1 is less than the distance A1". And "the direction difference between the person's line of sight and the subject 1 is less than the direction difference A2". The distance A1 is, for example, the maximum value of the distance at which a person can come into contact with the subject. The direction difference A2 is, for example, the direction farthest from the subject among the directions of the line of sight where the person is considered to be gazing at the subject. Although the details will be described later, the estimation unit 136 specifies the behavior content of the subject included in the captured image by specifying the behavior content corresponding to the combination of the requirements included in the behavior content database.

Returning to FIG. 4, the control unit 13 is, for example, a CPU. By executing the program stored in the storage unit 12, the control unit 13 executes an image acquisition unit 131, a person detection unit 132, a subject detection unit 133, a setting reception unit 134, a relationship identification unit 135, an estimation unit 136, and a location. It functions as a specific unit 137 and an output unit 138.

The image acquisition unit 131 acquires captured image data indicating the captured image. The image acquisition unit 131 acquires the captured image data imaged and generated by the image pickup device 1 via the communication unit 11. The image acquisition unit 131 may acquire a plurality of captured image data created in time series.

The human detection unit 132 detects the movement of a person included in the captured image. The movement of a person is, for example, a movement as described in the column of "movement of a person" shown in FIG. The person detection unit 132 detects the position of the person included in the captured image, and further detects the movement of the person who has detected the position. The human detection unit 132 detects the movement of a person included in the captured image, for example, by specifying changes in the positions of the joints of the person and the direction of the line of sight of the person included in the captured image. The person detection unit 132 outputs the person detection information indicating the movement of the detected person to the estimation unit 136.

The person detection unit 132 may further detect a person's attributes. The person detection unit 132 detects a person's attributes by, for example, determining whether or not the employee is an employee based on the pattern or color of the work clothes or uniform worn by the employee. The person detection unit 132 is an employee based on the result of comparing the employee characteristics such as the contour and shape of the employee's face stored in the storage unit 12 with the characteristics of the person included in the captured image. A person's attribute may be detected by determining whether or not it is.

The subject detection unit 133 detects the type of subject included in the captured image. The subject detection unit 133 detects the position of the subject included in the captured image, and further detects the type of the subject for which the position has been detected. The subject detection unit 133 detects a subject included in the captured image based on the shape of the subject included in the setting information stored in the storage unit 12, for example, and is based on the type of the subject associated with the shape of the subject. Detects the type of subject included in the captured image. The subject detection unit 133 outputs subject detection information indicating the position of the detected subject and the type of the subject to the estimation unit 136.

The subject detection unit 133 may detect a plurality of subjects included in the captured image. The subject detection unit 133 may detect the first subject and the second subject included in the captured image. The first subject and the second subject are, for example, articles of the type described in the columns of "subject 1" and "subject 2" shown in FIG. When the subject detection unit 133 detects a plurality of subjects, the estimation unit 136 estimates a person's behavior content based on the type of one or more of the detected plurality of subjects.

The setting receiving unit 134 receives the setting of the attribute of the subject included in the captured image. For example, the setting receiving unit 134 receives the setting information including the subject attribute information and the subject type information transmitted by the information terminal 2, and receives the setting of the subject attribute included in the captured image. The setting reception unit 134 may store the setting information including the setting of the attribute of the received subject in the storage unit 12, or may output it to the subject detection unit 133.

The relationship specifying unit 135 identifies the relative relationship between the person and the subject that accompanies the movement of the person. The relationship specifying unit 135 specifies the relative relationship based on the person detection information output by the person detection unit 132 and the subject detection information output by the subject detection unit 133. The relationship specifying unit 135 specifies, for example, the relative relationship between the portion corresponding to the movement of the person detected by the person detecting unit 132 and the subject. The relationship specifying unit 135 identifies the relationship between the human eye and the subject when, for example, the movement of the person detected by the person detecting unit 132 is "gaze", and when it is "take", the human hand and the subject. Identify the relationship with. The relationship specifying unit 135 notifies the estimation unit 136 of the specified relative relationship. The relative relationship is used by the estimation unit 136 to identify the behavior content of a person. The relationship specifying unit 135 specifies, for example, the distance between a person and a subject as a relative relationship. The relationship specifying unit 135 specifies the distance between the person and the subject based on, for example, the difference between the position of the joint of the person included in the person detection information and the position of the subject included in the subject detection information.

The relationship specifying unit 135 may specify the relationship between the direction of the person's line of sight and the position of the subject as a relative relationship. When the movement of the person is, for example, the movement of the human eye such as "gaze" or "see", the relationship specifying unit 135 indicates the direction of the person's line of sight included in the person detection information and the subject included in the subject detection information. The relationship with the position is specified as a relative relationship. The relationship specifying unit 135 specifies, for example, the difference between the direction of the line of sight when a person directly looks at the subject and the direction of the line of sight included in the person detection information as a relative relationship.

The relationship specifying unit 135 may specify the relationship between the orientation of the human body and the position of the subject as a relative relationship. For example, when the movement of a person is "conversation", the relationship specifying unit 135 indicates the orientation of the body of the person specified based on the position of the joint of the person included in the person detection information and the position of the subject included in the subject detection information. The relationship with is specified as a relative relationship. The relationship specifying unit 135 specifies, for example, the difference between the orientation of the body when the person faces the front of the subject and the orientation of the body specified based on the positions of the joints of the person as a relative relationship.

The relationship specifying unit 135 may specify a change in the relative relationship based on the specified plurality of relative relationships. The relationship specifying unit 135 identifies a plurality of relative relationships based on, for example, a plurality of movements of a person detected based on a plurality of captured images created in time series and a subject, and each of the specified plurality of relative relationships is specified. Identify changes in relative relationships based on.

For example, at time T, the relationship specifying unit 135 specifies the relative relationship as "the distance between the person and the subject is 10 cm", and at time T + 1, the relationship specifying unit 135 specifies the relative relationship as "the distance between the person and the subject is 50 cm". In the case of specifying, the relationship specifying unit 135 specifies that "a person is moving in a direction away from the subject" based on the relative relationship specified at time T and time T + 1. By specifying the change in the relative relationship in this way, the relationship specifying unit 135 can specify the relative relationship with high accuracy.

When the captured image includes a person who estimates the behavior and a plurality of subjects, the relationship specifying unit 135 may specify a plurality of relative relationships. The relationship specifying unit 135 is, for example, a first relative relationship, which is a relationship between a person and a first subject due to the movement of a person, a second relative relationship, which is a relationship between a person and a second subject due to the movement of a person, and a person. The third relative relationship, which is the relationship between the first subject and the second subject due to the movement, may be specified. The first relative relationship is, for example, the relative relationship between a person and the subject 1 shown in FIG. 5, the second relative relationship is the relative relationship between the person and the subject 2 shown in FIG. 5, and the third relative relationship is the subject. It is a relative relationship between 1 and subject 2.

For example, when a purchaser staying in a store's sales floor puts a product in a bag, the purchaser who estimates the behavior acts using the product as the first subject and the bag as the second subject. ing. Therefore, the relationship specifying unit 135 determines the first relative relationship, which is the relationship between the purchaser and the product, the second relative relationship, which is the relationship between the purchaser and the bag, and the third relative relationship, which is the relationship between the product and the bag. Identify. When the relationship specifying unit 135 notifies the estimation unit 136 of such a plurality of relative relationships, the estimation unit 136 can improve the accuracy of specifying the human behavior content.

The relationship specifying unit 135 may specify the positional relationship between the subject for which the attribute is set and another subject among the plurality of subjects included in the captured image. The relationship specifying unit 135 further specifies, for example, the positional relationship between the subject detected by the subject detecting unit 133 and the subject for which the setting receiving unit 134 has received the attribute setting.

For example, a captured image of a store's sales floor includes an image of a fixedly installed product shelf and an image of a product placed on the product shelf. For the purpose of detecting shoplifting, the relationship between the position of the product shelf and the position of the product is effective in presuming that a person is shoplifting. Therefore, the relationship specifying unit 135 specifies a change in the distance between the position of the product that is the subject detected by the subject detecting unit 133 and the position of the product shelf that is the subject that the setting receiving unit 134 has received the setting of the attribute.

The estimation unit 136 estimates the behavior content of a person. The estimation unit 136 estimates the human behavior content based on the combination of the type of the subject and the relative relationship. The estimation unit 136 searches, for example, whether or not the combination of the subject type and the relative relationship is included in the action content database stored in the storage unit 12. When the behavior content database contains similar or matching combinations, the estimation unit 136 estimates the behavior content associated with the similar or matching combination as the human behavior content.

For example, when a person staying in a store's sales floor puts a product in a container, the subject detection unit 133 detects the product as the first subject and the container as the second subject. The relationship specifying unit 135 specifies the relative relationship based on the distance between the human hand and the first subject and the distance between the first subject and the second subject. The relationship specifying unit 135 describes the relative relationship as "the distance between the human hand and the first subject and the distance between the first subject and the second subject are predetermined distances (for example, the distance D1 and the distance D2 shown in FIG. 5). Identify as "less than".

When the type of the second subject is "bag", the estimation unit 136 estimates the human behavior content as shoplifting, and when the second subject type is "product basket", the estimation unit 136 estimates the human behavior content as shopping. do. As described above, the estimation unit 136 can estimate the human behavior with high accuracy by estimating the human behavior content using the type of the subject even when the human behavior is similar.

Further, the estimation unit 136 may estimate the human behavior content based on the fact that the relationship specifying unit 135 further identifies the change in the relative relationship as "the distance between the first subject and the second subject becomes shorter". good. By operating in this way, the estimation unit 136 can estimate the human behavior with higher accuracy.

When the relationship specifying unit 135 specifies the distance between the person and the subject as a relative relationship, the estimation unit 136 sets the reference distance associated with the type of the subject detected by the subject detecting unit 133 and the person specified by the relationship specifying unit 135. The action content may be estimated based on the result of comparing the distance between the subject and the subject. The reference distance is, for example, a distance corresponding to the characteristics of a person's behavior, such as “distance A1 or less” and “distance B1 or less” shown in FIG. The reference distance may be different depending on the type of subject.

For example, when a person staying in a store is gazing at a product that is a subject, the relationship specifying unit 135 specifies the relative relationship between the person and the subject as "the distance between the person and the subject is the distance Y1". In that case, the estimation unit 136 compares the reference distance “less than the distance A1” shown in FIG. 5 with the distance Y1 specified by the relationship specifying unit 135. When the distance Y1 is less than the distance A1, the estimation unit 136 coincides with "the distance between the person and the subject 1 is less than the distance A1" which is one of the relative relationships associated with the action content "looking at the product". Therefore, it is determined that a person may be looking at the product.

When the relationship specifying unit 135 specifies the relationship between the direction of the human line of sight and the position of the subject as a relative relationship, the estimation unit 136 indicates the direction of the line of sight and the subject associated with the type of subject detected by the subject detection unit 133. The action content may be estimated based on the result of comparing the reference line-of-sight relationship, which is the relationship with the position, and the relationship between the direction of the line of sight of the person specified by the relationship specifying unit 135 and the position of the subject. The relationship between the direction of the line of sight and the position of the subject is expressed by the difference between the direction of the line of sight of the person and the direction of the line of sight when the person directly looks at the subject. The reference line-of-sight relationship is a direction difference corresponding to the characteristics of the action content, for example, “direction difference less than A2” shown in FIG.

For example, when a person staying in a store is looking at a product that is a subject, the relationship specifying unit 135 describes the relative relationship between the person and the subject as "the relationship between the direction of the person's line of sight and the position of the subject is a direction difference Y2. To specify. In that case, the estimation unit 136 compares the “direction difference A2 or less”, which is the reference line-of-sight relationship shown in FIG. 5, with the direction difference Y2 specified by the relationship specifying unit 135. When the direction difference Y2 is closer to the subject than the direction difference A2, the estimation unit 136 has the "human eye and subject 1" which is one of the relative relationships associated with the action content "looking at the product". Since the direction difference of is less than the direction difference A2, it is determined that there is a possibility that a person is looking at the product.

The estimation unit 136 estimates that a person's behavior is "looking at a product" when both the distance between the person and the subject and the direction difference between the direction of the person's line of sight and the position of the subject satisfy the criteria. You may. By estimating the behavior content of a person based on a plurality of relative relationships, the estimation unit 136 further improves the estimation accuracy of the behavior content.

When the relationship specifying unit 135 specifies the relationship between the orientation of the human body and the position of the subject as a relative relationship, the estimation unit 136 indicates the orientation of the body and the subject associated with the type of the subject detected by the subject detection unit 133. The action content may be estimated based on the result of comparing the reference body orientation relationship, which is the relationship with the position, and the relationship between the direction of the line of sight of the person specified by the relationship specifying unit 135 and the position of the subject. The relationship between the orientation of the body and the position of the subject is expressed by the difference between the orientation of the body and the orientation of the body when the person faces the direction of the subject. The reference body orientation relationship is a difference in orientation corresponding to the characteristics of the behavioral content, for example, the orientation “less than the angle difference B2” shown in FIG.

For example, when a person (purchaser) staying in a store is talking to an employee who is the subject, the relationship specifying unit 135 describes the relative relationship between the person and the subject as "the orientation of the person's body and the position of the subject". The relationship is specified as "angle difference X1". In that case, the estimation unit 136 compares the “angle difference B2 or less”, which is the reference body orientation relationship shown in FIG. 5, with the angle difference X1 specified by the relationship specifying unit 135. When the angle difference X1 points in a direction closer to the subject than the angle difference B2, the estimation unit 136 is the "human body" which is one of the relative relationships associated with the action content "talking to the employee". Since the angle difference between the orientation and the subject 1 is less than the angle difference B2, it is determined that a person may be talking to the employee.

The estimation unit 136 determines that a person's behavior is "speaking with an employee" when both the distance between the person and the subject and the angle difference between the orientation of the person's body and the position of the subject meet the criteria. You may estimate. By estimating the behavior content of a person based on a plurality of relative relationships, the estimation unit 136 further improves the estimation accuracy of the behavior content.

When the relationship specifying unit 135 identifies a change in the relative relationship using a plurality of relative relationships, the estimation unit 136 estimates the human behavior content based on the combination of the type of subject and the change in the relative relationship. May be good. For example, when a product picked up by a person staying in a store is put in a bag, the relationship identification unit 135 relatives the distance between the product as the first subject and the bag as the second subject at each of a plurality of times. Identify as a relationship. The relationship specifying unit 135 identifies the change in the relative relationship as "the distance between the first subject and the second subject becomes shorter".

The estimation unit 136 determines that it is an action of a person to put a product in a bag based on the combination of the change in the relative relationship specified by the relationship identification unit 135 and the product and the bag which are the types of the subject, and the behavior content of the person. Is estimated to be "shoplifting". In this way, by using the change in the relative relationship, the estimation unit 136 can improve the estimation accuracy of the human behavior content.

When the estimation unit 136 estimates the behavior content of a person based on the combination of the type of the subject and the change in the relative relationship, the estimation unit 136 detects the subject before the person detection unit 132 detects the person in any of the plurality of captured image data. The action content may be estimated based on the result of determining whether or not is included.

For example, when a person staying in a store's sales floor picks up an object placed on a shelf, it is necessary to determine whether or not the type of the object is a product. In this case, the estimation unit 136 detects, for example, an object reflected in the captured image shown by the captured image data created at the time before the store opens, and a person who picks up the object placed on the shelf by the person detection unit 132. It is determined whether or not the object reflected in the captured image indicated by the captured image data created at the same time matches. If it is determined that they match, the estimation unit 136 specifies that the type of the object is a product. The estimation unit 136 estimates that the behavior of a person is "shoplifting" when the person puts the object in a bag and the object is a product.

By operating the estimation unit 136 in this way, for example, when a customer who visits the store misplaces his / her personal belongings on the shelf of the store and moves, and after a certain period of time, he / she returns to pick up the misplaced personal belongings, the estimation unit 136 operates. , It is possible to distinguish between personal items and products taken from the shelves by the purchaser based on the captured image before the store opens. As a result, the estimation unit 136 can suppress the mistaken estimation of the content of an action other than shoplifting as "shoplifting".

When the relationship specifying unit 135 specifies a plurality of relative relationships based on the captured image, the estimation unit 136 may estimate the human behavior content based on the combination of each of the plurality of relative relationships and the type of the subject. ..

For example, when a person staying in the sales floor of a store puts a product placed on a shelf in a bag, the relationship specifying unit 135 is an example of the first relative relationship, that is, the distance between the person and the shelf and the second relative. Identify the distance between a person and a product, which is an example of a relationship. When the distance indicated by the first relative relationship and the distance indicated by the second relative relationship are less than the reference distance, the estimation unit 136 includes the first relative relationship, the second relative relationship, and the product and bag which are the types of the subject. It is estimated that the behavior of a person is shoplifting based on the combination of.

Further, when the relationship specifying unit 135 identifies a change in the second relative relationship based on a plurality of second relative relationships specified at a plurality of times, the estimation unit 136 determines the change in the first relative relationship and the second relative relationship. The content of human behavior may be estimated based on the combination of the subject and the type of the subject. The estimation unit 136 is, for example, a type of subject such as "the distance between the person and the shelf is less than the reference distance" which is the first relative relationship and "the distance between the person and the product becomes shorter" which is the change of the second relative relationship. A person's behavior may be presumed to be shoplifting based on the combination of shelves and goods.

When the relationship specifying unit 135 specifies a plurality of relative relationships based on a person and a plurality of subjects, the estimation unit 136 may estimate the behavior content of the person based on the combination of the plurality of relative relationships. The estimation unit 136 may estimate the behavior content of a person based on, for example, a combination of a type of subject and at least two relative relationships of a first relative relationship, a second relative relationship, or a third relative relationship.

For example, when a person puts a product in a bag at a store sales floor, the subject detection unit 133 detects the product and the bag as subjects. The relationship specifying unit 135 specifies the relationship between the person and the product as the first relative relationship, the relationship between the person and the bag as the second relative relationship, and the relationship between the product and the bag as the third relative relationship. .. The estimation unit 136 is based on the fact that the type of subject is a bag and a product, the distance between the human hand and the bag is less than D1, and the distance between the bag and the product is less than D2. Estimate the action content as "shoplifting".

Here, the case where the estimation unit 136 estimates the human behavior content as "shoplifting" by using the second relative relationship and the third relative relationship is illustrated, but the estimation unit 136 is the first relative relationship ". The content of a person's behavior may be estimated by further using "a person is holding a product".

The estimation unit 136 may estimate the behavior content based on the attributes of the person detected by the person detection unit 132. For example, when a person puts a product in a bag at the sales floor of a store, if the attribute of the person is a purchaser, the estimation unit 136 estimates that the behavior of the person is "manipulation", and the attribute of the person is an employee. For example, the estimation unit 136 estimates the content of a person's behavior as "product arrangement". By operating the estimation unit 136 in this way, the estimation unit 136 is a case where the movements of people are similar, for example, "shoplifting" and "product arrangement", which are actions of picking up a product. However, the behavior content can be estimated correctly.

The estimation unit 136 may further estimate the action content based on the attributes of the subject received by the setting reception unit 134. The subject attribute is information indicating the characteristics of the subject such as the owner of the subject and the function of the subject. For example, when the setting reception unit 134 accepts that the attribute of the object placed on the shelf installed in the warehouse of the store is "the property of the store", the estimation unit 136 moves to pick up the object. It is presumed that the behavior of the employee is "organization of goods or equipment". On the other hand, when the setting reception unit 134 does not have the attribute of the object "property of the store", the estimation unit 136 estimates that the behavior content of the employee who picks up the object is, for example, "moving personal belongings".

Further, when the relationship specifying unit 135 specifies the positional relationship between the subject detected by the subject detecting unit 133 and the subject for which the setting receiving unit 134 has received the attribute setting, the estimation unit 136 specifies the relationship specifying unit 135. The action content may be estimated based on the positional relationship and the attributes of the subject for which the setting receiving unit 134 has received the setting. For example, when the subject specified by the relationship specifying unit 135 is a product shelf and the subject for which the setting receiving unit 134 has received the setting is an object placed on the product shelf, the estimation unit 136 is specified by the relationship specifying unit 135. Based on the positional relationship, it is determined that a person other than the employee is moving to put the object placed on the product shelf into the bag. Subsequently, the estimation unit 136 estimates that the behavior content of a person other than the employee is shoplifting when the attribute of the object is "property of the store".

By estimating the action content based on the attributes of the subject in this way, the estimation unit 136 is a case where a plurality of products and an object (for example, a personal property) different from the plurality of products are placed in the same place. However, it is possible to estimate the behavior of a person by distinguishing between a product and an object different from the product.

The place specifying unit 137 identifies the place shown in the captured image. The location specifying unit 137 identifies a location in the captured image based on, for example, the subject in the captured image. The location specifying unit 137 may specify a location shown in the captured image based on, for example, a GPS (Global Positioning System) signal. The location specifying unit 137 notifies the output unit 138 of information indicating the specified location.

The output unit 138 functions as a transmission unit that transmits the action content information indicating the action content estimated by the estimation unit 136 to the information terminal 2. The output unit 138 may output the notification information when the action content estimated by the estimation unit 136 and the location specified by the location identification unit 137 satisfy a predetermined condition. The predetermined condition is that the user of the information terminal 2 should receive the notification information and take some action. The user of the information terminal 2 is, for example, an employee or a security guard of the store when the image pickup device is installed in the store.

Specifically, when the action content estimated by the estimation unit 136 is "shoplifting" and the place specified by the location identification unit 137 is a "store", the employee or security guard of the store selects the person who shoplifted. It is necessary to secure it. Therefore, the output unit 138 determines that the predetermined condition is satisfied and outputs the notification information.

On the other hand, when the action content estimated by the estimation unit 136 is "product arrangement" and the place specified by the location identification unit 137 is "store", the estimated action content is the normal behavior of the employee, so it is output. Unit 138 determines that the predetermined condition is not satisfied and does not output the notification information. By operating in this way, the output unit 138 can output the information required by the user of the information terminal 2.

The output unit 138 may output the notification information of the type corresponding to the combination of the place specified by the place specifying unit 137 and the action content. For example, if the place specified by the location specifying unit 137 is the "store sales floor" and the behavior content of the person estimated by the estimation unit 136 is "shoplifting", the output unit 138 is shoplifted at the store sales floor. Output notification information indicating that.

[Flowchart of behavior estimation device 10]
FIG. 6 is a flowchart showing an example of an operation in which the behavior estimation device 10 estimates the behavior content of a person. In FIG. 6, the behavior estimation device 10 identifies the change in the relative relationship based on the relative relationship at the time T1 and the relative relationship at the time T2, and estimates the human behavior content using the change in the specified relative relationship. The operation will be explained.

The image acquisition unit 131 acquires the first image capture image data generated by the image pickup apparatus 1 at time T1 (S11). The location specifying unit 137 specifies a location in the captured image indicated by the first captured image data acquired by the image acquisition unit 131. Further, the person detection unit 132 detects the position of a person included in the captured image indicated by the first captured image data (S12). The subject detection unit 133 detects the position of the subject and the type of the subject in the captured image indicated by the first captured image data (S13). The relationship specifying unit 135 specifies the relative relationship between the position of the person, the position of the subject, and the type of the subject at the time T1 (S14).

The image acquisition unit 131 acquires the second captured image data generated by the imaging device 1 at time T2 (S15). The human detection unit 132 detects the position of a person included in the captured image indicated by the second captured image data (S16). The subject detection unit 133 detects the position of the subject and the type of the subject in the captured image indicated by the second captured image data (S17). The relationship specifying unit 135 specifies the relative relationship between the position of the person, the position of the subject, and the type of the subject at the time T2 (S18).

The relationship specifying unit 135 specifies a change in the relative relationship based on the relative relationship at the time T1 specified in S14 and the relative relationship at the time T2 specified in S18 (S19). The estimation unit 136 estimates the human behavior content based on the combination of the change in the relative relationship specified by the relationship specifying unit 135 and the type of the subject detected by the subject detecting unit 133 (S20).

When the operation for terminating the process of estimating the human behavior content is not performed (NO in S21), the behavior estimation device 10 repeats the processes from S11 to S19. When the operation for terminating the process of estimating the human behavior content is performed (YES in S21), the behavior estimation device 10 ends the process.

[Other application examples]
In the above description, the case of estimating the behavior content of the purchaser or the employee in the store is illustrated, but the behavior estimation device 10 estimates various behavior content of the person based on the captured image of another place. You may.

(Estimation based on color)
FIG. 7 is a diagram showing a configuration of a behavior estimation device 10 that estimates a person's behavior content based on a change in the color of the person or the subject. The behavior estimation device 10 shown in FIG. 7 is different from the behavior estimation device 10 shown in FIG. 4 in that it has a color specifying unit 139, and is the same in other respects. The color specifying unit 139 specifies at least one of a change in the color of a person and a change in the color of a subject. The color identification unit 139 changes the color of the person or subject included in the captured image acquired by the image acquisition unit 131 based on the position of the person detected by the person detection unit 132 or the position of the subject detected by the subject detection unit 133. To identify.

The estimation unit 136 may estimate the action content based on at least one of the change in the color of the person specified by the color identification unit 139 or the change in the color of the subject. For example, when a person is using a knife, the subject detection unit 133 detects a knife, which is a type of subject. The relationship specifying unit 135 specifies the relative relationship between the person and the knife, which is the subject, as "the distance between the person and the subject is less than the reference distance". The color identification unit 139 identifies that the human hand and the knife have turned red based on the position of the person and the position of the knife. The estimation unit 136 estimates that the person is injured based on the fact that the person's hand and knife turn red.

In addition, when another person is staying around the person using the knife, the relationship identification unit 135 describes the change in the relative relationship between the knife, which is the subject, and the other person. By specifying that the distance changes greatly with each time, the estimation unit 136 estimates the human behavior as dangerous behavior. Then, the color identification unit 139 identifies that the color of the knife or another person has changed to red, and the estimation unit 136 estimates that the other person is injured.

In this way, the color specifying unit 139 identifies the change in the color of the person or the subject, and the estimation unit 136 identifies the behavior of the person based on the color change, so that the behavior estimation device 10 responds urgently to bloodshed or the like. It is possible to estimate the content of actions to be taken. Further, the behavior estimation device 10 can estimate the behavior content such as spilling paint by detecting that the color has changed from red, for example.

(Detection of dangerous acts)
The behavior estimation device 10 may detect a dangerous behavior that has occurred outdoors, for example, based on an image of a surveillance camera installed outdoors. A dangerous act is an act of destroying an object such as a bench installed in a park. When a person who moves karate and the bench are detected based on the image of the surveillance camera, the relationship specifying unit 135 identifies the relative relationship between the person and the bench which is the subject.

When the relationship specifying unit 135 specifies the relative relationship as "the distance between the person and the subject is equal to or greater than the threshold value", the estimation unit 136 specifies the human behavior content as "karate practice". The threshold value is a distance indicating a range in which a person moves in practice of a martial art such as karate. On the other hand, when the relationship specifying unit 135 specifies the relative relationship as "the distance between the person and the subject is less than the threshold value", the estimation unit 136 estimates that the behavior content of the person "destroys the bench". By operating in this way, the behavior estimation device 10 can monitor the behavior of the device installed outdoors and the behavior of the person around the device.

(Detection of intrusion into prohibited area)
The behavior estimation device 10 may estimate the behavior of a person entering the restricted area based on the image of the surveillance camera. For example, when a fence is installed at the boundary of the restricted area, the subject detection unit 133 detects the fence that is the subject. When a person climbs over the fence and invades, the person detection unit 132 detects the movement of the person over the fence. The relationship specifying unit 135 specifies the relative relationship as "a person is overcoming the subject". The estimation unit 136 estimates that the human behavior content is "overcoming the fence" based on the combination of the fence, which is the type of the subject, and the relative relationship.

When the fence is not installed in the no-entry area, the subject detection unit 133 may detect the no-entry area as the type of the subject. When a person enters the restricted area, the person detection unit 132 detects that the person is moving. The relationship specifying unit 135 specifies the relative relationship as "a person is moving inside the subject" based on the position information indicating the area where the subject is prohibited from entering and the position where the person has moved. The estimation unit 136 estimates that the content of a person's behavior is "invading an area where entry is prohibited".

By operating in this way, the behavior estimation device 10 can detect that a person enters the restricted area. The manager of the no-entry area can notify the information terminal 2 that the behavior estimation device 10 has invaded the no-entry area, thereby knowing that the person has invaded and issuing a warning to the person. can.

[Effect of behavior estimation device 10]
As described above, the behavior estimation device 10 includes a human detection unit 132 that detects the movement of a person included in the captured image, a subject detection unit 133 that detects the type of the subject included in the captured image, and a human movement. It has a relationship specifying unit 135 that specifies a relative relationship that is a relationship between an accompanying person and a subject. Then, the estimation unit 136 estimates the human behavior content based on the type of the subject detected by the subject detection unit 133 and the relative relationship specified by the relationship specifying unit 135.

The behavior estimation device 10 estimates the content of a person's behavior using the type of subject and the relative relationship between the person and the subject in this way, so that even if the movements are similar, the behavior of the person is accurate. The content can be estimated. Further, according to the behavior estimation system S, the information terminal 2 receives the behavior content information indicating the behavior content of the person estimated by the behavior estimation device 10, so that, for example, the employee of the store who is the user of the information terminal 2 can receive the behavior content information. It is possible to accurately know the abnormal behavior of a person. As a result, the user of the information terminal 2 can accurately respond to human behavior.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes can be made within the scope of the gist. be. For example, all or part of the device can be functionally or physically distributed / integrated in any unit. Also included in the embodiments of the present invention are new embodiments resulting from any combination of the plurality of embodiments. The effect of the new embodiment produced by the combination has the effect of the original embodiment together.

1 Image pickup device 2 Information terminal 10 Behavior estimation device 11 Communication unit 12 Storage unit 13 Control unit 21 Communication unit 22 Storage unit 23 Control unit 24 Display unit 25 Operation unit 131 Image acquisition unit 132 Person detection unit 133 Subject detection unit 134 Setting reception unit 135 Relationship identification unit 136 Estimating unit 137 Location identification unit 138 Output unit 139 Color specification unit 231 Information transmission unit 232 Information reception unit

Claims (17)

An image acquisition unit that acquires captured image data indicating an captured image,
A person detection unit that detects the movement of a person included in the captured image,
A subject detection unit that detects the type of subject included in the captured image, and
A relationship specifying unit that specifies the relative relationship between the person and the subject due to the movement of the person, and
An estimation unit that estimates the behavior of the person based on the combination of the type of the subject and the relative relationship.
Behavior estimation device with. The relationship specifying unit specifies the distance between the person and the subject as the relative relationship.
The estimation unit performs the action based on the result of comparing the reference distance associated with the type of the subject detected by the subject detection unit and the distance between the person and the subject specified by the relationship identification unit. Estimate the content,
The behavior estimation device according to claim 1. The relationship specifying unit specifies the relationship between the direction of the person's line of sight and the position of the subject as the relative relationship.
The estimation unit has a reference line-of-sight relationship, which is a relationship between the direction of the line of sight associated with the type of the subject detected by the subject detection unit and the position of the subject, and the direction of the person's line of sight specified by the relationship specifying unit. The behavior content is estimated based on the result of comparing the relationship between the subject and the position of the subject.
The behavior estimation device according to claim 1 or 2. The relationship specifying unit specifies the relationship between the orientation of the person's body and the position of the subject as the relative relationship.
The estimation unit is a reference body orientation relationship, which is a relationship between the body orientation and the position of the subject associated with the type of the subject detected by the subject detection unit, and the line of sight of the person specified by the relationship identification unit. The action content is estimated based on the result of comparing the relationship between the direction and the position of the subject.
The behavior estimation device according to any one of claims 1 to 3. The image acquisition unit acquires a plurality of the captured image data created in time series, and obtains the plurality of captured image data.
The relationship specifying unit identifies the change in the relative relationship and
The estimation unit estimates the behavior content of the person based on the combination of the type of the subject and the change in the relative relationship.
The behavior estimation device according to any one of claims 1 to 4. The estimation unit further determines the action content based on the result of determining whether or not the subject was included in any of the plurality of captured image data before the person detection unit detects the person. presume,
The behavior estimation device according to claim 5. The person detection unit further detects the attribute of the person and
The estimation unit estimates the behavior content based on the attributes of the person detected by the person detection unit.
The behavior estimation device according to any one of claims 1 to 6. The subject detection unit detects the first subject and the second subject included in the captured image, and detects the first subject and the second subject.
The relationship specifying unit is a first relative relationship, which is a relationship between the person and the first subject due to the movement of the person, and a second relative relationship, which is a relationship between the person and the second subject due to the movement of the person. The relationship and the third relative relationship, which is the relationship between the first subject and the second subject due to the movement of the person, are specified.
The estimation unit estimates the behavior content of the person based on the combination of the type of the subject and at least two relative relationships of the first relative relationship, the second relative relationship, or the third relative relationship. ,
The behavior estimation device according to any one of claims 1 to 7. It further has a setting reception unit that accepts the setting of the attribute of the subject included in the captured image.
The estimation unit estimates the action content based on the attributes of the subject.
The behavior estimation device according to any one of claims 1 to 8. The relationship specifying unit further specifies the positional relationship between the subject detected by the subject detecting unit and the subject for which the setting receiving unit has received the attribute setting.
The estimation unit estimates the action content based on the positional relationship specified by the relationship specifying unit and the attributes of the subject for which the setting receiving unit has accepted the setting.
The behavior estimation device according to claim 9. A place specifying part that specifies the place shown in the captured image, and
An output unit that outputs notification information when the action content estimated by the estimation unit and the location specified by the location identification unit satisfy a predetermined condition.
Have more,
The behavior estimation device according to any one of claims 1 to 10. The output unit outputs the notification information of the type corresponding to the combination of the location specified by the location specifying unit and the action content.
The behavior estimation device according to claim 11. Further having a color specifying unit for specifying at least one of the change in the color of the person or the change in the color of the subject.
The estimation unit estimates the action content based on at least one of the change in the color of the person specified by the color identification unit and the change in the color of the subject.
The behavior estimation device according to any one of claims 1 to 12. The image acquisition unit acquires a plurality of the captured image data, and obtains the plurality of captured image data.
The relationship specifying unit identifies a plurality of the relative relationships and
The estimation unit estimates the behavior content of the person based on the combination of each of the plurality of relative relationships and the type of the subject.
The behavior estimation device according to any one of claims 1 to 13. Computer runs,
Steps to acquire captured image data showing captured images,
The step of detecting the movement of a person included in the captured image and
The step of detecting the type of the subject included in the captured image and
A step of specifying a relative relationship between the person and the subject due to the movement of the person, and
A step of estimating the behavior content of the person based on the combination of the type of the subject and the relative relationship,
Behavior estimation method with. On the computer
Steps to acquire captured image data showing captured images,
The step of detecting the movement of a person included in the captured image and
The step of detecting the type of the subject included in the captured image and
A step of specifying a relative relationship between the person and the subject due to the movement of the person, and
A step of estimating the behavior content of the person based on the combination of the type of the subject and the relative relationship,
A program to execute. It is equipped with a behavior estimation device that estimates a person's behavior content based on an image captured by the image pickup device, and an information terminal that displays the behavior content of the person estimated by the behavior estimation device.
The behavior estimation device is
An image acquisition unit that acquires captured image data indicating the captured image, and an image acquisition unit.
A person detection unit that detects the movement of a person included in the captured image,
A subject detection unit that detects the type of subject included in the captured image, and
A relationship specifying unit that specifies the relative relationship between the person and the subject due to the movement of the person, and
An estimation unit that estimates the behavior of the person based on the combination of the type of the subject and the relative relationship.
A transmission unit that transmits behavior content information indicating the behavior content estimated by the estimation unit to the information terminal, and a transmission unit.
Have,
The information terminal is
An information receiving unit that receives the action content information, and
A display unit that displays the action content based on the action content information received by the information receiving unit, and a display unit that displays the action content.
Behavior estimation system with.

Images