JP7507524B1 - Attention-grabbing call-to-action system - Google Patents

Abstract

[Problem] To provide a system for calling out attention to an action that can make an appropriate call according to the attributes of an action that occurs when attention to an action, such as abnormal behavior, of an action that occurs in a video is detected.
[Solution] In the attention behavior calling system 1, feature points of a target behavior entity Z captured in a time-series image Y are detected, and when it is determined that an attention behavior has been performed based on the displacement of the stored feature points and the displacement of the detected feature points, a different call is made by voice according to the type of attention behavior. However, even when it is determined that an attention behavior has been performed, if the attribute or size of the target behavior entity Z satisfies a first predetermined condition, no call is made, or the content or manner of the call is changed.
[Selected figure] Figure 3

Description

The present invention relates to a system for calling attention to an action that can make an appropriate call according to the attributes of an action that is captured on video when abnormal or other attention-grabbing action of the action is detected.

There is a known technique for extracting from time-series images an entity that has behaved differently from a predetermined "normal behavior" when the entity's behavior in the time-series images differs from the "normal behavior" (see, for example, Patent Document 1).

Patent No. 6525179

By using the above technology, it is possible to call out to any abnormal behavior from a notification unit installed near the imaging means.

However, for example, children often behave in ways that are judged to be abnormal, and some of these behaviors would be considered abnormal if the behavior was an adult, but would not be considered abnormal if the behavior was a child. In such a situation, if calls were made to both adults and children in the same way, the frequent calls would not only become a distraction, but would also undermine the credibility of the calls.

Therefore, the present invention aims to provide a system for calling attention to an action that can make an appropriate call according to the attributes of an action that occurs when abnormal or other attention-grabbing action of an action that occurs in a video is detected.

The present invention provides an attention behavior calling system that includes an acquisition unit that acquires time-series images captured by a photographing means, a detection unit that detects feature points of a target behavior object captured in the time-series images, a storage unit that stores the displacement of the feature points of the behavior object when the behavior object performs an attention behavior, a determination unit that determines whether the attention behavior has been performed based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image, an estimation unit that estimates the attribute or size of the target behavior object based on the detected feature points or the appearance of the target behavior object, and a calling unit that is provided around the shooting range of the photographing means and that, when it is determined that the attention behavior has been performed, issues a different call by voice according to the type of the attention behavior, and the calling unit does not issue the call or changes the content or manner of the call even when it is determined that the attention behavior has been performed, if the attribute or size satisfies a first predetermined condition.

With this configuration, when abnormal or other noteworthy behavior of a target behavior entity is detected in a time-series image, it becomes possible to make an appropriate call according to the attributes of the target behavior entity. For example, it is possible to stop the call if the attribute is "child," and to increase the volume of the call if the attribute is "elderly."

In another aspect of the present invention, there is provided a program and method for calling attention to an action that correspond to the above-mentioned system for calling attention to an action.

The attention behavior calling system of the present invention makes it possible to make an appropriate call according to the attributes of an action subject when attention behavior such as abnormal behavior of an action subject captured in a video is detected.

FIG. 1 is an explanatory diagram of a time-series image according to a first embodiment of the present invention; 1 is a block diagram of a system for calling attention to actions according to a first embodiment of the present invention; 1 is a flowchart of a system for calling attention to behavior according to a first embodiment of the present invention; Block diagram of an attention behavior calling system according to a second embodiment of the present invention. 1 is a block diagram of a system for calling attention to behavior according to a modified example of the present invention;

The attention behavior calling system 1 according to the first embodiment of the present invention will be described below with reference to Figures 1 to 3.

As shown in FIG. 1, the attention behavior calling system 1 is for calling out to a target behavior entity Z captured in a time-series image Y (in FIG. 1, frames constituting an image) captured by a capture means X according to the type of behavior of the target behavior entity Z. In this embodiment, a human being is used as the target behavior entity Z, and for ease of understanding, the target behavior entity Z is simply displayed using only a skeleton.

As shown in FIG. 2, the attention behavior call system 1 includes an acquisition unit 2, a detection unit 3, a storage unit 4, a determination unit 5, an estimation unit 6, and a call unit 7. In this embodiment, the attention behavior call system 1 is provided integrally with the image capture means X.

The acquisition unit 2 acquires time-series images Y captured by the imaging means X.

The detection unit 3 detects feature points of the target action object Z captured in the time-series image Y.

There are various possible feature points, but in this embodiment, we will explain using an example of detecting joints as feature points.

When detecting joints as feature points, the following methods can be considered, for example:

First, the "joint identification criteria" and "acting object identification criteria" are stored in a memory unit (which may be memory unit 4 or another memory unit).

The "joint identification standard" is used to identify multiple human joints, and indicates the shape, orientation, size, etc. for each joint to identify them.

The "Behavior Identification Standards" indicate the "basic postures" of various human variations ("walking", "standing upright", etc.), the "range of motion of each joint", and the "distance between each joint" in a single human.

After detecting multiple joints that meet the above "joint identification criteria," it is possible to identify the joints contained in each of the target action objects Z by referring to the "action object identification criteria."

Note that feature points may be detected individually for each time-series image Y, or may be detected collectively from multiple time-series images Y if the chronological order can be identified.

The memory unit 4 stores the displacement of the feature points of the action object when the action object performs a target action. When joints are detected as feature points, the displacements of multiple joints are stored.

Examples of behaviors of interest include falling, punching, kicking, etc., and multiple behaviors may be stored without being limited to one. Furthermore, "premonitory behaviors" such as preparatory movements for shoplifting may also be stored as behaviors of interest. Note that in order to estimate behaviors other than the behaviors of interest (walking, stopping, etc.), the memory unit 4 may also store the displacement of feature point information when these behaviors occur.

The determination unit 5 determines whether or not attention behavior has occurred based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image Y.

For example, if the displacement of a detected feature point matches the displacement of a feature point of a focused behavior by a specified amount or more, it may be determined that the focused behavior has been performed.

The estimation unit 6 estimates the attributes or size of the target action object Z based on the detected feature points or the appearance of the target action object Z.

The attributes or size of the target action Z can be estimated using known methods.

For example, when identifying the target action object Z by joints as described above, it is possible to estimate the size of the target action object Z by determining the distance between the joint point of the head and the joint point of the ankle.

In addition to the size mentioned above, attributes can also be estimated from features such as posture and bone structure, as well as external appearance such as clothing, hair length, and hair color.

The calling unit 7 is provided around the shooting range of the image capturing means X, and when it is determined that an attention behavior has been performed, it issues a different call by voice according to the type of attention behavior.

Different appeals depending on the type of behavior in question could be, if the behavior in question is a criminal act (shoplifting, violent acts, etc.), then "Shoplifting (violent acts) is a crime," and if the behavior in question is an accident (falling, etc.), then "Are you OK?", "Can you get up?", "Are you hurt?", etc.

The above call is expected to stop any criminal activity in the event of such an act. In addition, in the event of an accident, it will be possible to determine whether or not rescue is required based on the actions and gestures of the target actor Z in response to the call.

For example, children often behave in ways that are judged to be attention-grabbing behaviors (abnormal behaviors), and among these behaviors, there are some that would be abnormal if the target behavior entity Z were an adult, but would not be considered abnormal if it were a child. In such a situation, if calls were made to both adults and children in the same way, the frequent calls would become a distraction and would likely undermine the credibility of the calls.

Therefore, in this embodiment, even if it is determined that attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call unit 7 does not make a call or changes the content or manner of the call.

For example, if the attribute is "child," it may be possible to not make a call. If the attribute is "elderly," it may be possible to turn up the volume of the call (changing the manner of the call) or change the call to a request for help from people nearby (changing the content of the call), etc.

Also, similar control may be performed simply when the size of the target action object Z satisfies a first predetermined condition (for example, when "the size (height) of the target action object Z is smaller than a predetermined value").

In this way, in the present embodiment, the attention behavior calling system 1 can make an appropriate call according to the attributes of the target behavior entity Z when an attention behavior such as an abnormal behavior of the target behavior entity Z is detected in the time-series image Y.

Next, we will explain the flow of calls in this embodiment using the flowchart in Figure 3.

First, when a time-series image Y is acquired by the acquisition unit 2 (S1), the detection unit 3 detects feature points of a target action object Z captured in the time-series image Y (S2).

In addition, the attributes or size of the target action object Z are estimated based on the feature points detected in S2 or the appearance of the target action object Z (S3).

Next, based on the displacement of the feature points stored in the memory unit 4 and the displacement of the feature points detected in S2, it is determined whether or not attention behavior has occurred (S4). S3 and S4 may be performed in the reverse order.

If it is determined that attention behavior has occurred (S4: YES), it is determined whether the attribute or size estimated in S3 satisfies a first predetermined condition (S5).

If the first predetermined condition is not met (S5: NO), a different voice call is made depending on the type of attention behavior (S6).

On the other hand, if the first predetermined condition is met (S5: YES), the call is not made or the content or manner of the call is changed (S7).

As described above, in the attention behavior call system 1 according to this embodiment, even if it is determined that attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call is not made, or the content or manner of the call is changed.

With this configuration, when abnormal or other noteworthy behavior of the target behavior entity Z is detected in the time-series image Y, it becomes possible to make an appropriate call according to the attributes of the target behavior entity Z. For example, it is possible to stop the call if the attribute is "child," and to increase the volume of the call if the attribute is "elderly."

Next, an attention behavior calling system 100 according to a second embodiment of the present invention will be described with reference to FIG. 4. Note that the same components as those in the first embodiment are given the same reference numerals and their description will be omitted.

In this embodiment, the attention behaviors stored in the memory unit 4 are determined through learning.

In detail, the attention behavior calling system 100 includes a learning side acquisition unit 11, a learning side detection unit 12, and a determination unit 13 in addition to the configuration of the first embodiment.

The learning side acquisition unit 11 acquires sample images (multiple sample time-series images) captured by an imaging means X installed to capture a specified range.

The learning side detection unit 12 detects the behavior of the sample behavioral entity captured in the sample video.

The behavior of the sample behavior body can be detected by storing the displacement of feature points (such as the movement of each joint) when the sample behavior body performs a specified behavior in a memory unit (which can be memory unit 4 or another memory unit), detecting the feature points in the same manner as detection unit 3, and detecting that "the specified behavior has been performed" if the displacement of the detected feature point information matches the displacement of the stored feature points by a specified amount or more.

The determination unit 13 determines one or more "normal behaviors" within a predetermined range based on the multiple behaviors detected by the learning side detection unit 12. In this embodiment, the determined "normal behaviors" are stored in the memory unit 4 as attention behaviors.

"Normal behavior" can be determined based on various criteria, but for example, behavior that has a predetermined (threshold) or higher ratio among all detected behaviors could be determined to be "normal behavior."

Then, the determination unit 5 determines that attention behavior has been performed when the displacement of the feature points detected from each time-series image Y does not correspond to "normal behavior."

In detail, if the displacement of the feature points detected from each time-series image Y does not correspond to "normal behavior," i.e., if it is determined that attention behavior has occurred (S3 in FIG. 3: YES), the operations of S4-S7 in FIG. 3 are performed, as in the first embodiment.

As described above, in the present embodiment, the attention behavior calling system 100 determines that attention behavior has been performed when the displacement of feature points detected from each time-series image Y does not correspond to the "normal behavior" determined by learning.

With this configuration, it is possible to call out to behavior that is inappropriate for the situation (predetermined range), even if the behavior is not clearly abnormal, such as violent behavior or falling. However, with this configuration, for example, the behavior of a child is more likely to be determined as attention-grabbing behavior (abnormal behavior). However, even in such a case, if the attribute or size meets the first predetermined condition, the call is not made, or the content or manner of the call is changed, thereby preventing frequent calls due to attention-grabbing behavior (abnormal behavior) of a "child" or the like.

The attention behavior calling system of the present invention is not limited to the above-described embodiment, and various modifications and improvements are possible within the scope of the claims.

For example, even if a place is not appropriate for an adult to "run," a child may run without any malicious intent. However, it is not appropriate to uniformly issue a call to the child when the action of "running" is unintentional and low risk. Therefore, depending on the malicious intent and the degree of danger of the attentional behavior, a call may not be issued, or the content or manner of the call may be changed.

In this case, as shown in FIG. 5, the attention behavior calling system 1 may further include a risk determination unit 8 that refers to a plurality of time-series images Y showing the attention behavior determined to have been performed and determines the risk level of the attention behavior determined to have been performed individually.

In this modified example, the "whether the attribute or size satisfies a first predetermined condition" that was also used in the embodiment is also used to determine malice. For example, if the attribute is "child" or "elderly", it can be considered that the malice is "low". It is also possible to simply consider that the malice is "low" if the size is below a predetermined level.

For example, when the behavior of interest is "running," it is possible to determine the degree of danger as "high danger" if the speed of the target behavior object Z is equal to or greater than a predetermined value, and "low danger" if it is less than the predetermined value.

For example, if the behavior of interest is "lying down," the risk is low to begin with, so it may be possible to determine the risk as "low risk" regardless of speed, etc. However, if the behavior of interest is "falling," it is preferable to determine the risk taking into account the speed of falling, etc.

Then, even if it is determined that attention behavior has occurred, if the attribute or size satisfies a first predetermined condition and the degree of danger satisfies a second predetermined condition set for each attribute or size, the call unit 7 does not make a call or changes the content or manner of the call.

For example, if the attribute is "child," it can be estimated that the vehicle is "low malicious intent," and if the second predetermined condition is "driving slowly," the danger is also low, so no call will be made.

On the other hand, if the attribute is "adult," it can be inferred that the person has "highly malicious intent," and the call may be made regardless of the level of danger.

With this configuration, when abnormal or other noteworthy behavior of the target behavior entity Z is detected in the time-series image Y, it becomes possible to issue a more appropriate appeal depending on the maliciousness and danger level of the noteworthy behavior.

The calls can also be further controlled depending on the location.

In this case, as shown in FIG. 5, the attention behavior calling system 1 may further include a background detection unit 9 that detects the background shown in the time-series image Y.

Then, the call unit 7 will either not make a call or will change the content or manner of the call depending on the detected background.

For example, if the background is stairs, the target action Z may be startled by the call and may risk falling, so "no call will be made." If the background is an elevator, the distance between the call unit 7 and the target action Z is close, so "lower the volume" may be considered.

With this configuration, when abnormal or other noteworthy behavior of the target behavior entity Z is detected in the time-series image Y, it becomes possible to make a more appropriate appeal depending on the background.

In this case, it will be even more effective if you also take into account the risk mentioned above.

For example, even if a child is running slowly, if there are stairs in the background, the danger is high, so a call may be made.

In addition, some users may want to be notified of any abnormal behavior, even that of children.

As shown in FIG. 5, the attention behavior call system 1 may further include a setting unit 10 that allows the user to set whether or not to make a call or to change the content or manner of the call when the age or size of the target behavior entity Z meets a predetermined condition.

In addition, in the above embodiment, the imaging means X is provided integrally with the interrogation unit 7, but it is sufficient if it is provided near the periphery of the imaging range of the imaging means X so that an interrogation can be made to the target action object Z.

In addition, in the above embodiment, all components are provided integrally, but this does not exclude, for example, some components (memory unit 4, determination unit 5, estimation unit 6, etc.) being provided in different locations (cloud, etc.).

The present invention can also be applied to a program and method corresponding to the processing performed by each component as a controller, and to a recording medium storing the program. In the case of a recording medium, the program is installed in a computer or the like. Here, the recording medium storing the program may be a non-transient recording medium. A non-transient recording medium may be, but is not limited to, a CD-ROM or the like.

Reference Signs List 1, 100 Attention behavior calling system 2 Acquisition unit 3 Detection unit 4 Storage unit 5 Determination unit 6 Estimation unit 7 Call unit 8 Danger level determination unit 9 Background detection unit 10 Setting unit 11 Learning side acquisition unit 12 Learning side detection unit 13 Decision unit X Shooting means Y Time series image Z Behavior body

Claims (7)

an acquisition unit for acquiring time-series images captured by an imaging means;
A detection unit that detects feature points of a target action object captured in the time-series images;
A storage unit that stores displacement of a feature point of a behavior object when the behavior object performs an abnormal behavior;
a determination unit that determines whether or not the abnormal behavior has occurred based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image;
an estimation unit that estimates an attribute or a size of the target action object based on the detected feature points or an appearance of the target action object;
a calling unit that is provided around the photographing range of the photographing means and that, when it is determined that the abnormal behavior has been performed, issues a different call by voice according to the type of the abnormal behavior;
Equipped with
The attention behavior calling system is characterized in that the call unit does not make the call or changes the content or manner of the call when the attribute or size satisfies a first predetermined condition, even if it is determined that the abnormal behavior has been performed. A learning side acquisition unit that acquires a sample video captured by the imaging means installed to capture a predetermined range;
A learning side detection unit that detects the behavior of a sample behavior object captured in the sample video;
A determination unit that determines one or more normal behaviors in the predetermined range based on the multiple behaviors detected by the learning side detection unit;
Further comprising:
The system for calling for attention to behavior as described in claim 1, characterized in that the determination unit determines that the abnormal behavior has been performed when the displacement of feature points detected from each time-series image does not correspond to the normal behavior. A risk determination unit that determines a risk of each of the abnormal behaviors determined to have been performed by referring to a plurality of time-series images showing the abnormal behaviors determined to have been performed,
The attention behavior calling system of claim 1, characterized in that the call unit does not make the call or changes the content or manner of the call when the attribute or size satisfies the first specified condition and the degree of danger satisfies a second specified condition set for each attribute or size, even if it is determined that the abnormal behavior has been performed. A background detection unit detects a background in the time-series images,
The attention behavior calling system according to claim 1 , wherein the calling unit does not make the call or changes the content or manner of the call depending on the detected background. The call system according to claim 1, further comprising a setting unit that allows a user to set whether or not to not make the call or to change the content or manner of the call when the age or size of the behavioral entity satisfies a predetermined condition. A program executed by a computer that stores a displacement of a feature point of a behavior object when the behavior object performs an abnormal behavior,
A step of acquiring time-series images captured by an imaging means;
detecting feature points of a target action object captured in the time-series images;
determining whether or not the abnormal behavior has occurred based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image;
estimating an attribute or a size of the target action based on the detected feature points or an appearance of the target action;
a step of providing a voice message corresponding to a type of abnormal behavior when the voice message is provided around the photographing range of the photographing means and when the abnormal behavior is determined to have occurred,
Equipped with
A program for calling out attention to behavior, characterized in that in the step of making a call, even if it is determined that abnormal behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call is not made or the content or manner of the call is changed. A method executed by a computer that stores displacements of feature points of an action object when the action object performs an abnormal action, comprising:
A step of acquiring time-series images captured by an imaging means;
detecting feature points of a target action object captured in the time-series images;
determining whether or not the abnormal behavior has occurred based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image;
estimating an attribute or a size of the target action based on the detected feature points or an appearance of the target action;
a step of providing a voice message corresponding to a type of abnormal behavior when the voice message is provided around the photographing range of the photographing means and when the abnormal behavior is determined to have occurred,
Equipped with
A method for calling out attention to behavior, characterized in that in the step of making a call, even if it is determined that abnormal behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call is not made or the content or manner of the call is changed.

Images