WO2018008702A1

WO2018008702A1 - Behavior detection system

Info

Publication number: WO2018008702A1
Application number: PCT/JP2017/024714
Authority: WO
Inventors: 洋登永吉; 大介勝又; 孝史野口; 健太郎大西
Original assignee: 株式会社日立システムズ
Priority date: 2016-07-07
Filing date: 2017-07-05
Publication date: 2018-01-11
Also published as: JP2018005752A; JP6841608B2

Abstract

The purpose of the present invention is to identify an action with high accuracy even when the types of actions to be recognized are increased. The behavior recognition system 10 has an action model storage part 21, a video photographing device 11, a position recognition section 15, and an action recognition section 16. The action model storage part 21 stores action models representing models of fundamental actions to be detected. The video photographing device 11 photographs a work activity of a subject to be recognized. The position recognition section 15 detects the position of the subject from photographic information photographed by the video photographing device 11. The action recognition section 16 recognizes the fundamental action of the subject. In addition, the action recognition section 16, upon acquiring a similar action model to the recognized fundamental action of the subject by searching through the action models stored in the action model storage part 21, acquires the meaning of the action corresponding to the acquired action model.

Description

Behavior detection system

The present invention relates to a behavior detection system, and more particularly to a technique effective for motion analysis for recognizing and analyzing human motion.

In recent years, there has been a growing need for recognition of human behavior from the standpoint of preventing employee fraud in stores and ensuring product safety through hygiene management technology using HACCP (Hazard Analysis and Critical Control Point).

As this type of behavior recognition technology, there is a technology that automatically recognizes the behavior and behavior of a moving object such as a human being, an animal, or a machine (for example, see Patent Document 1). This Patent Document 1 describes that, for example, a feature quantity extracted from a sensor attached to a person is compared with a feature quantity stored in a feature quantity database to recognize the action of the person.

JP-A-10-113343

However, in the technique of Patent Document 1 described above, the feature quantity acquired by the sensor attached to the person is only compared with the feature quantity stored in the database. There is a risk that the amount increases and it takes time to recognize the motion.

Also, even if the operation is the same, if the operation target, for example, the work place is different, it may mean a different operation. In such a case, it is difficult to recognize the motion, and as a result, there is a problem that the accuracy of motion recognition is lowered.

An object of the present invention is to provide a technique capable of identifying an operation with high accuracy even when the types of the recognized operation are increased.

The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

Of the inventions disclosed in this application, the outline of typical ones will be briefly described as follows.

That is, a typical behavior detection system recognizes an element action to be recognized and analyzes an action meaning indicating the meaning of the recognized element action. This behavior detection system includes an operation model storage unit, a photographing unit, a position recognition unit, and an operation recognition unit.

The behavior model storage unit stores a behavior model representing the recognized element behavior as numerical information. The photographing unit photographs a work operation to be recognized. The position recognition unit recognizes the position of the recognition target from the shooting information acquired by the shooting unit.

The behavior detection system has a work position operation table that associates the position of the recognition target, the element operation performed at the corresponding position, and the operation meaning of the element operation. The motion recognition unit refers to the work position motion table, acquires an element motion associated with the position recognized by the position recognition unit, and further extracts a motion model corresponding to the element motion from the motion model storage unit. . The motion recognition unit recognizes the element motion of the recognition target by comparing the extracted motion model with the motion of the recognition target. Further, the motion recognition unit extracts the motion meaning from the recognition target position and the recognized element motion with reference to the work position motion table.

Also, the motion recognition unit generates recognition information that associates the motion meaning corresponding to the acquired motion model, the position detected by the position recognition unit, and the time when the recognition target element motion is recognized.

Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

(1) The recognition accuracy of element motion can be improved.

(2) The recognition speed of element motion can be improved.

It is explanatory drawing which shows an example of a structure in the behavior detection system by one Embodiment. It is explanatory drawing which shows an example of a data structure in the place operation | movement table which the memory | storage part of FIG. 1 has. It is explanatory drawing which shows the other example of a data structure in the place operation | movement table of FIG. It is explanatory drawing which shows an example of the data structure in the behavior model accumulate | stored in the behavior model storage part which the memory | storage part of FIG. 1 has. It is a flowchart which shows an example of the operation | movement in the behavior detection system of FIG. It is explanatory drawing which shows the other structural example of the behavior detection system of FIG.

In the following embodiments, when it is necessary for the sake of convenience, the description will be divided into a plurality of sections or embodiments. However, unless otherwise specified, they are not irrelevant to each other. There are some or all of the modifications, details, supplementary explanations, and the like.

Further, in the following embodiments, when referring to the number of elements (including the number, numerical value, quantity, range, etc.), especially when clearly indicated and when clearly limited to a specific number in principle, etc. Except, it is not limited to the specific number, and may be more or less than the specific number.

Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily indispensable unless otherwise specified and apparently essential in principle. Needless to say.

Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., the shape of the component is substantially the case unless it is clearly specified and the case where it is clearly not apparent in principle. And the like are included. The same applies to the above numerical values and ranges.

In all the drawings for explaining the embodiments, the same members are, in principle, given the same reference numerals, and the repeated explanation thereof is omitted.

Hereinafter, embodiments will be described in detail.

<Configuration example of behavior detection system>
FIG. 1 is an explanatory diagram illustrating an example of a configuration in a behavior detection system 10 according to an embodiment.

The behavior detection system 10 is a system that recognizes the position and element motion of a worker who is a recognition target, and stores the meaning of the recognized element motion, time, and the like in association with each other. As shown in FIG. 1, the behavior detection system 10 includes a video photographing device 11, a recognition processing unit 12, and a display device 13.

The video photographing device 11 as a photographing unit is, for example, a web camera or a surveillance camera capable of photographing a color image, a depth sensor capable of photographing a distance to a photographing target, or the like. The recognition processing unit 12 extracts information related to a person's movement (hereinafter referred to as movement-related information) from the shooting information shot by the video shooting apparatus 11. The motion-related information is, for example, time series information of a human joint position or information obtained by abstracting it.

To obtain time series information of human joint positions from color information, Toshev et al. (A. Vision and Pattern Recognition, 2014, pp. 1653-1660.If a depth sensor is used, the method of Shotton et al. (J. Shotton, T. Sharp, A. Fitzgibbon, M. Finocchio, A. Blake, M. Cook, and R. Moore, “Real-time human pose recognition in parts from single depth images,” Communications of the ACM, vol. 56, no. 1, pp. 116-124, 2013 .) Can be used.

The above-described abstracted information is, for example, information representing the speed of joint movement, the distance between joints, the speed of joint movement expressed in frequency, or the combination thereof. Both are expressed as numerical information.

The display device 13 is composed of a display such as a liquid crystal monitor, for example, and displays recognition information described later under the control of the recognition processing unit 12.

The recognition processing unit 12 includes a position recognition unit 15, an operation recognition unit 16, and a storage unit 17. The position recognizing unit 15 recognizes the position of the worker, that is, where the worker is in the work place, from the photographing information photographed by the video photographing device 11.

This position recognition can be performed, for example, by giving in advance the angle of view, the distance to the floor, and the angle of the floor of the image capturing device 11. If the photographing apparatus is a normal color camera, the position in the workplace can be recognized from the size and position of the subject on the image. In the case of a depth camera, depth information can also be used, so that the position can be recognized with high accuracy. The motion recognition unit 16 recognizes the worker's elemental motion from the shooting information of the video shooting device 11.

The storage unit 17 includes a nonvolatile storage device such as a hard disk device (HDD) or a flash memory, and stores various types of information. The storage unit 17 includes a place operation table 20, an operation model storage unit 21, and a recognition information storage unit 22.

The place operation table 20 which is a work position operation table is a table in which a position is associated with an element operation of a worker performed at the position. The motion model storage unit 21 stores motion models of element motions to be detected. This motion model is a series of motion related information when, for example, a worker performs a standard work motion.

<Configuration example of location action table>
FIG. 2 is an explanatory diagram illustrating an example of a data configuration in the place operation table 20 included in the storage unit 17 of FIG.

The place action table 20 stores the area where the element action can be executed and the meaning of the action indicated by the element action in the area for the element action of the worker to be recognized. As shown in FIG. 2, the place operation table 20 is composed of data of “place”, “operation ID”, and “operation meaning”.

“Location” is data indicating which position or area of the workplace, and is represented by coordinates, for example. “Action ID” specifies the element action of the worker to be recognized in the “place”, and is indicated by a number in the example of FIG. The “motion meaning” is data indicating the meaning of the element motion of the worker to be recognized at the place.

<Other configuration examples of the location action table>
FIG. 3 is an explanatory diagram showing another example of the data structure in the place operation table 20 of FIG.

In the case of the location operation table 20 shown in FIG. 2, “operation ID” and “operation meaning” correspond to each “location”. However, in the location operation table 20 shown in FIG. In the table, “location” and “operation meaning” correspond to each other.

<Configuration example of motion model>
FIG. 4 is an explanatory diagram showing an example of a data configuration in the behavior model accumulated in the behavior model accumulation unit 21 included in the storage unit 17 of FIG.

The operation model stored in the operation model storage unit 21 includes “operation ID” and “operation model” as shown in FIG.

The “motion model” is a numerical value of the elemental motion of the worker to be recognized as described above. A series of motion related information when a worker performs standard work in advance. The motion related information captured once may be used as the motion model as it is, or the motion related information captured multiple times is used as the motion model. It may be used. In the latter case, motion-related information captured a plurality of times may be used as an operation model as it is, or an average may be used.

The recognition information storage unit 22 stores the recognition information generated by the motion recognition unit 16. The recognition information stored in the recognition information storage unit 22 includes, for example, “time”, “location”, “operation ID”, and “operation meaning”.

“Time” is, for example, the time when the worker started work. Alternatively, it may be from the time when the work is started to the time when the work is finished. This is the time when the motion recognition unit 16 recognizes the worker's element motion, for example.

“Location” indicates the position where the worker is working. “Operation ID” identifies the element operation of the worker to be recognized. The “motion meaning” indicates what meaning the “motion ID” has at the corresponding “location”.

<Operation example of behavior detection system>
Next, the operation of the behavior detection system 10 will be described.

FIG. 5 is a flowchart showing an example of the operation in the behavior detection system 10 of FIG. In FIG. 5, behavior recognition processing by the behavior detection system 10 will be described. The behavior recognition process is a process for recognizing a worker's work position and element motion, generating recognition information, and storing the recognition information.

Note that FIG. 5 illustrates the case where the behavior recognition process described below is executed by hardware such as the position recognition unit 15 and the action recognition unit 16, and the behavior recognition process is, for example, the recognition processing unit of FIG. 1. The program may be executed on the basis of software in a program format stored in a program storage memory (not shown) provided in FIG.

When executed based on software, the software is executed by, for example, a CPU (Central Processing Unit) (not shown) included in the recognition processing unit 12.

First, the position recognition unit 15 recognizes the position of the worker who started the work based on, for example, the above-described method (step S101).

Subsequently, the position recognition unit 15 outputs the recognized worker position to the operation recognition unit 16. The motion recognition unit 16 acquires the recognition target corresponding to the position recognized by the position recognition unit 15 with reference to the place motion table 20 of FIG. 2, that is, the element motion of the worker (step S102).

For example, in the place motion table 20 of FIG. 2, when the position recognized by the position recognition unit 15 is (1.0, 2.0), the “motion ID” of the recognition target element motion is “1” and “2”. 'Become.

Then, the motion recognition unit 16 reads all the motion models corresponding to the “motion ID” acquired in the process of step S102 from the motion model storage unit 21 of the storage unit 17 (step S103).

Subsequently, the motion recognition unit 16 calculates motion related information from the shooting information of the worker shot by the video shooting device 11 (step S104). Then, the motion recognition unit 16 operates the “motion ID” of the most similar motion model among the motion models read out in step S103 with respect to the motion-related information calculated in step S104. Obtained as a recognition result (step S105).

In the behavior model storage unit 21 of FIG. 4, for example, when the behavior model most similar to the work information is (0.53, 0.52, 0.33), the “motion ID” corresponding to the behavior model. Becomes '1'. Therefore, the motion recognition result is “1”. When a plurality of pieces of movement related information are accumulated in one movement model, for example, the movement related information calculated in the process of step S104 is compared with movement related information in each movement model, The most similar result is adopted as a comparison result with the corresponding behavior model.

Subsequently, the motion recognition unit 16 acquires the motion meaning corresponding to the recognition result by the process of step S105 from the place motion table 20 (step S106), generates recognition information, and the recognition information storage unit 22 of the storage unit 17. (Step S107).

For example, when the position recognized by the position recognition unit 15 in the process of step S101 is (1.0, 2.0) and the operation recognition result is “1” in the process of step S105, the location shown in FIG. Referring to the operation table 20, the operation meaning is “turn the lever”.

Also, the recognition information generated by the motion recognition unit 16 associates the time when the work is started, the position of the worker, the motion recognition result, and the motion meaning as described above.

This completes the behavior recognition process.

In this way, by recognizing not only the worker's elemental motion but also the worker's position, it is recognized that even the same elemental motion has different motion meanings depending on the worker's work position. be able to.

This can improve the recognition accuracy of work movements.

Also, as described above, since different motion meanings are recognized depending on the position of the worker, the load on the recognition processing of the motion recognition unit 16 can be reduced. As a result, the recognition processing time can be shortened, and the performance of the behavior detection system 10 can be improved. Further, by increasing the number of motion models stored in the motion model storage unit 21, it is possible to further improve the recognition accuracy of work motions.

<Display example of recognition information>
Here, the recognition information stored in the recognition information storage unit 22 may be displayed on the display device 13. For example, the operation recognition unit 16 reads the recognition information stored in the recognition information storage unit 22 and causes the display device 13 to display the display.

Alternatively, when a request for displaying recognition information is input by an input unit such as a mouse or a keyboard (not shown) included in the behavior detection system 10, the operation recognition unit 16 reads the recognition information in the recognition information storage unit 22 and displays the display device 13. You may make it display on.

Thereby, it is possible to efficiently confirm whether or not the worker is performing a predetermined work by browsing the recognition information displayed on the display device 13 by the supervisor.

Also, depending on the work, there is an arrangement that a predetermined work (hereinafter referred to as a prescribed work) is arranged to be performed in a predetermined time zone. The action recognition unit 16 may determine whether or not the prescribed work is being performed, and may output an alert to the display device 13 or the like when the prescribed work is not being performed by the worker.

In this case, the storage unit 17 includes a prescribed work information storage unit. The specified work information storage unit stores specified work information indicating the position, time zone, and operation meaning of the specified work.
Then, the motion recognition unit 16 searches the recognition information stored in the recognition information storage unit 22, and is the operation corresponding to the determined motion meaning being performed at the position defined in the prescribed work information? No, as an example, the order of work and the excess / shortage of work are determined.

At this time, the motion recognition unit 16 uses the time of the recognition information stored in the recognition information storage unit 22 to extract the corresponding specified work information, and is determined at a position determined in the extracted specified work information. It may be determined whether or not an operation corresponding to the operation meaning is performed.

When there is no recognition information similar to the prescribed work information, the action recognition unit 16 determines that the prescribed work has not been performed, and outputs an alert to the display device 13 or the like indicating that the prescribed work has not been performed.

This makes it possible for the supervisor to easily and quickly confirm whether or not the worker has performed the prescribed work.

As described above, the supervisor can efficiently check the operation of the worker.

<Other configuration examples of behavior detection system>
In the present embodiment, the operation model storage unit 21 is included in the storage unit 17. However, the operation model storage unit 21 may be connected via, for example, the Internet.

FIG. 6 is an explanatory diagram showing another configuration example of the behavior detection system 10 of FIG.

6 is different from the behavior detection system 10 in FIG. 1 in that the behavior model storage unit 21 includes, for example, storage on the cloud, so-called cloud storage. In this case, the behavior model storage unit 21 is connected to the recognition processing unit 12 through a communication line 30 such as the Internet.

For example, if the work contents and the workplace configuration are almost the same, the behavior model may be uploaded to the behavior model storage unit 21 through the communication line 30, and the operation model is individually input in advance at each workplace. Can be made unnecessary. Thereby, the man-hour at the time of using the behavior detection system 10 can be reduced.

As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

Note that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described.

Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. . In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

DESCRIPTION OF SYMBOLS 10 Behavior detection system 11 Image | video imaging device 12 Recognition processing part 13 Display apparatus 15 Position recognition part 16 Motion recognition part 17 Storage part 20 Location action table 21 Action model storage part 22 Recognition information storage part

Claims

A behavior detection system for recognizing an element action of a recognition target and analyzing an action meaning indicating the meaning of the recognized element action,
An action model accumulating unit for accumulating an action model serving as a model of an element action to be detected;
A photographing unit that photographs the work operation of the recognition target;
A position recognition unit for recognizing the position of the recognition target from shooting information shot by the shooting unit;
An action recognition unit for recognizing the element action of the recognition target;
Have
The motion recognition unit searches for the motion model stored in the motion model storage unit, acquires a motion model similar to the recognized element motion of the recognition target, and the motion corresponding to the acquired motion model A behavior detection system that captures meaning.
The behavior detection system according to claim 1,
The motion recognition unit generates recognition information that associates the motion meaning corresponding to the acquired motion model, the position detected by the position recognition unit, and the time when the recognition target element motion is recognized. Detection system.
The behavior detection system according to claim 2,
A behavior detection system including a recognition information storage unit that stores the recognition information generated by the motion recognition unit.
The behavior detection system according to claim 1,
A work position operation table that associates the position of the recognition target, the element operation performed at the position, and the operation meaning of the element operation;
The behavior recognition system, wherein the motion recognition unit extracts the motion meaning of the element motion corresponding to the position detected by the position recognition unit with reference to the work position motion table.
The behavior detection system according to claim 1,
The behavior detection system is a behavior detection system in which the behavior model accumulated in the behavior model accumulation unit is obtained by quantifying the motion element to be recognized.
The behavior detection system according to claim 3,
Having a prescribed work information storage unit for accumulating prescribed work information consisting of preset working conditions of the prescribed work;
The motion recognition unit searches the work information stored in the prescribed work information storage unit to determine whether there is the work information similar to the recognition information stored in the recognition information storage unit; A behavior detection system that outputs an alert when it is determined that there is no work information similar to the recognition information.
The behavior detection system according to claim 1,
The behavior model storage unit is a behavior detection system that is a cloud storage.