JP2021077177A - Operation recognition apparatus, operation recognition method, and operation recognition program - Google Patents

Abstract

To improve accuracy in recognition of a plurality of monitor targets.SOLUTION: A recognition unit recognizes a plurality of monitor objects based on a captured image. An identification number setting unit sets an identification number to each of the monitor objects. A tracking processing unit tracks each of monitor objects recognized in the captured image and set with the identification numbers, based on the region of each of the monitor objects recognized in the captured image. An operation recognition processing unit recognizes an operation of each of the monitor objects in a region of the captured image destination where each of the monitor objects is tracked. A recognition result output unit outputs a recognition results of an operation of each of the recognized monitor objects.SELECTED DRAWING: Figure 4

Description

The present invention relates to a motion recognition device, a motion recognition method, and a motion recognition program.

Today, for example, by capturing the behavior or movement of a monitored object such as a person, animal, or work machine with a camera device and analyzing this captured image, it is possible to visualize and recognize the behavior of the person, animal, work machine, or the like. The behavior recognition device is known.

For example, Patent Document 1 (Japanese Unexamined Patent Publication No. 2011-100175) discloses a human behavior determining device capable of determining the behavior of a person included in a crowded scene only by video processing. This person behavior determination device detects the center of gravity locus as a feature amount based on the position of the center of gravity and the locus of the center of gravity of the person determined to be the same by the image processing. Then, the behavior of the person is determined by collating the detected feature amount with the previously registered locus feature amount for each action.

However, in the case of the person behavior determination device disclosed in Patent Document 1, when there are a plurality of monitoring targets that are working or operating, it becomes difficult to continuously identify the same monitoring target, and the monitoring target There is a problem that the recognition accuracy is lowered. In particular, when a plurality of monitoring targets overlap each other on the captured image, or when they are hidden in the shadow and reappear, it becomes difficult to continuously identify the same monitoring target, and the monitoring target is recognized. The accuracy is significantly reduced.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a motion recognition device, a motion recognition method, and a motion recognition program for improving the recognition accuracy of a plurality of monitored objects.

In order to solve the above-mentioned problems and achieve the object, the present invention sets a recognition unit that recognizes a plurality of monitoring targets and an identification number for each monitoring target recognized by the recognition unit based on the captured image. Based on the identification number setting unit and the area of each monitoring target recognized in the captured image, the tracking processing unit that tracks each monitoring target with the identification number set recognized in the captured image, and each monitoring target In the result area of the tracked captured image, it has an operation recognition processing unit that recognizes the operation of each monitoring target and a recognition result output unit that outputs the recognition result of the recognized operation of each monitoring target.

According to the present invention, there is an effect that the recognition accuracy of a plurality of monitored objects can be improved.

FIG. 1 is a system configuration diagram of the behavior recognition system of the embodiment. FIG. 2 is a hardware configuration diagram of an action recognition device provided in the action recognition system of the embodiment. FIG. 3 is a functional block diagram of the action recognition device. FIG. 4 is a flowchart showing the flow of the behavior recognition process of the monitoring target in the behavior recognition device. FIG. 5 is a diagram for explaining the recognition operation of the worker by the recognition unit of the action recognition device. FIG. 6 is a diagram for explaining the calculation processing of the feature amount by the recognition unit of the action recognition device. FIG. 7 is a diagram showing a hierarchical structure of worker authentication processing by the recognition unit of the behavior recognition device. FIG. 8 is a diagram showing IDs set by the initial ID setting unit of the action recognition device for each authenticated worker. FIG. 9 is a diagram showing captured images of workers sequentially input to the tracking processing unit of the behavior recognition device. FIG. 10 is a diagram for explaining a tracking operation performed by the tracking processing unit of the action recognition device. FIG. 11 is another diagram for explaining the tracking operation performed by the tracking processing unit of the action recognition device. FIG. 12 is a diagram for explaining an operation in which the tracking processing unit of the action recognition device evaluates the feature amount of the prediction area of the worker and obtains the weighting coefficient. FIG. 13 is a diagram showing captured images of a plurality of frames of an operator input from the camera device to the action recognition device. FIG. 14 is a diagram for explaining spatiotemporal image data composed of captured images of a plurality of frames. FIG. 15 is a diagram showing an example of the behavior recognition result of the worker recognized by the behavior recognition processing unit of the behavior recognition device. FIG. 16 is a diagram showing an example of an action recognition result in which an operator has difficulty in detecting during walking in a series of actions from walking to shelving. FIG. 17 is a diagram showing an example of an action recognition result in which an operator has difficulty in detecting a series of actions from walking to shelving during shelving.

Hereinafter, the behavior recognition system of the embodiment as an application example of the motion recognition device, the motion recognition method, and the motion recognition program will be described.

(System configuration)
FIG. 1 is a system configuration diagram of the behavior recognition system of the embodiment. As shown in FIG. 1, the behavior recognition system includes a behavior recognition device 1 and a camera device 2. The camera device 2 images a monitored object such as a single person or a plurality of people, animals, and robots.

The action recognition device 1 has an input interface unit 3 and an action recognition processing unit 4. The interface 3 acquires a captured image to be monitored from the camera device 2. The behavior recognition processing unit 4 recognizes the behavior (movement) of the monitoring target based on the captured image acquired via the interface 3, and outputs the behavior recognition result to an external device such as a monitor device viewed by the monitor, for example. ..

(Hardware configuration of behavior recognition device)
FIG. 2 is a hardware configuration diagram of the action recognition device 1. As shown in FIG. 2, the action recognition device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a communication unit 14, and an HDD (Hard Disk Drive) 15. It has an input interface unit 3 and an output interface unit 17. Each of these parts 3, 11 to 15 and 17 is connected to each other via a bus line 18.

In addition to the camera device 2 described above, an operation unit 20 such as a keyboard device and a mouse device is connected to the input interface unit 3. The output interface unit 17 stores a monitor device (display unit) for displaying the action recognition result. The action recognition result may be output to an external storage device such as an HDD or a semiconductor memory via the output interface unit 17.

The server device 22 is connected to the communication unit 14 via, for example, a wide area network such as the Internet or a private network such as a LAN (Local Area Network). The communication unit 14 transmits and stores the action recognition result to the server device 22. As a result, the administrator or the like can access the server device 22 via a communication device such as a smart phone, a tablet terminal device, or a personal computer device to acquire the action recognition result, and can remotely monitor the monitoring target.

The HDD 15 stores an action recognition program that performs an action recognition process to be monitored. By executing this action recognition program, the CPU 11 realizes each function described below and executes the action recognition process to be monitored.

(Behavior recognition function)
FIG. 3 is a functional block diagram of each function realized by the CPU 11 executing the action recognition program. As shown in FIG. 3, the CPU 11 executes an action recognition program to execute an input unit 31, a recognition unit 32, an initial identification number setting unit (initial ID setting unit) 33, a tracking processing unit 34, and an action recognition processing unit 35. , The recognition result output unit 36, the monitoring target recognition dictionary input unit 37, and the action recognition dictionary input unit 38 are realized. The initial ID setting unit 33 is an example of the identification number setting unit. The action recognition processing unit 35 is an example of the action recognition processing unit.

Although the input units 31 to the action recognition dictionary input unit 38 are realized by software, some or all of them may be realized by hardware such as an IC (Integrated Circuit).

Further, the action recognition program may be provided by recording a file in an installable format or an executable format on a recording medium readable by a computer device such as a CD-ROM or a flexible disk (FD). Further, the action recognition program may be provided by recording on a recording medium readable by a computer device such as a CD-R, a DVD (Digital Versatile Disk), a Blu-ray disc (registered trademark), or a semiconductor memory. Further, the behavior recognition program may be provided in the form of being installed via a network such as the Internet, or may be provided by being incorporated in advance in a ROM or the like in the device.

(Behavior recognition processing)
The behavior recognition process of the monitoring target by the input unit 31 to the behavior recognition dictionary input unit 38 will be described with reference to the flowchart of FIG. By reading the action recognition program, the CPU 11 executes each process shown in the flowchart of FIG.

That is, first, the input unit 31 acquires the captured image from the camera device 2 (step S1). This captured image is a captured image obtained by capturing a work situation such as a work in which one or a plurality of workers, which is an example of a monitoring target, put a product in a workplace on a shelf.

Next, the monitoring target recognition dictionary input unit 37 inputs the monitoring target recognition dictionary for recognizing the worker into the recognition unit 32 (step S2). This monitoring target recognition dictionary is a dictionary (data group) showing a feature amount, a weighting coefficient, and an evaluation value threshold in each layer for calculating the evaluation value of each layer of the recognition unit 32. This monitoring target recognition dictionary provides rectangular vertex position coordinates, weighting coefficients, and evaluation thresholds in each layer for calculating the features described below for captured images of humans and non-human objects. It is formed by learning in advance.

Next, the recognition unit 32 refers to the monitoring target recognition dictionary and recognizes the worker in the captured image acquired by the input unit 31 (step S3). FIG. 5 is a diagram for explaining the recognition operation of the operator by the recognition unit 32. As shown in FIG. 5, the recognition unit 32 cuts out blocks 51, 52, ... Of a predetermined shape such as a rectangular shape within the range of the captured image 50. The upper left coordinates (Xs, Ys) of the block 51 and the lower right coordinates (Xe, Ye) of the block 51 are determined by the position of the block 51 and the size of the rectangle in the captured image 50.

The recognition unit 32 selects the block 51 in order from the largest size to the smallest size, and performs calculations such as the feature amount described below. That is, the processing time of the large block and the small block is the same. Further, as the blocks existing in the captured image 50, the number of large size blocks is small and the number of small size blocks is large. Therefore, the recognition unit 32 selects a block having a large size to a block having a small size in this order, and performs a calculation such as a feature amount. This enables quick detection of objects (monitored targets).

6 (a) to 6 (b) are diagrams for explaining the arithmetic processing of the feature amount. As shown in FIGS. 6 (a) to 6 (b), the recognition unit 32 adds the pixel values in the white area to the black-and-white rectangular area in the block with reference to the input monitoring target recognition dictionary. The processing is performed, and the difference from the total pixel value in the black pixel region is calculated as the feature amount h (x) in the block. Then, the recognition unit 32 calculates the evaluation value f (x) by performing a predetermined weighting process on the feature amount h (x).

The following equation (1) is an arithmetic expression of such an evaluation value f (x).

Here, as shown in FIG. 7, the recognition unit 32 calculates the evaluation value f (x) for each layer such as the first layer to the nth layer (n is a natural number). When the calculated evaluation value f (x) is smaller than the preset threshold value indicated by the monitoring target recognition dictionary, the recognition unit 32 determines that the object is a non-human object (non-human block) and evaluates the block. Cancel. On the other hand, when the calculated evaluation value f (x) is equal to or greater than a preset threshold value indicated by the monitoring target recognition dictionary, the recognition unit 32 determines that the block is a block in which a person is shown.

When the block in which a person is shown is recognized in this way, the initial ID setting unit 33 sets an identification number for each block (step S4). In the example of FIG. 8, three blocks are recognized as blocks in which a person is shown, and the initial ID setting unit 33 sets the identification numbers of ID1, ID2, and ID3 for each block (initial area). Is.

Next, the tracking processing unit 34 tracks the worker shown in each block (step S5). Specifically, as shown in FIG. 9, the tracking processing unit 34 is supplied with the first frame F0, the next frame F1 is supplied after Δt time, and so on. The captured images are supplied sequentially (in chronological order).

The tracking processing unit 34 defines the worker state "S (x, y, vx, by, Hx, Hy, M)" in the first frame. x and y are the coordinate values of the upper left point A of the block in which the worker (tracking target) shown in FIG. 9 surrounded by a rectangular frame is shown. Hx and Hy indicate the horizontal size and the vertical size of the block in which the worker is shown. vx and vy indicate the speed at which the worker in the block moves in the horizontal and vertical directions (the initial value is set to 0). M is a change in the variable magnification of the worker in the block (the rate of change in the size of the worker with respect to the previous frame: the initial value is set to 0).

As shown in FIGS. 10 and 11, the tracking processing unit 34 tracks the worker by repeatedly predicting, observing, and correcting the state of the worker. “St” shown in FIG. 11 indicates a worker (tracking target) state at time t, and “Yt” indicates an observation result at time t.

Tracking processing unit 34, the calculation of equation (2) below, to calculate the change of state to a state S _k from the state S _k-1 of the worker.

The tracking processing unit 34 predicts the state of the next worker of k from the state of the worker of k-1 by performing the following calculations of equations (3) to (9).

Next, tracking processing unit 34, by performing the calculation of the following equation (10) calculates the observed data Z _k of state S _k of the worker.

The observation data is a color histogram in the block (in the area where the tracking target is shown) shown in FIG. 9 of the operator. The tracking processing unit 34 calculates the color histogram by performing the calculation of the following equation (11). In equation (11), k () is the kernel that calculates the color histogram. Further, (11) In the equation, a is the magnification ratio, h _{(x i)} is a color pixel value. P is the frequency of the color histogram.

The kernel k in the equation (11) is calculated by the following equation (12).

By using the kernel k calculated by these 12 equations, a large value is calculated for the central portion of the block (subject area), and a smaller value is calculated as the value is closer to the periphery of the block. This makes it possible to reduce the influence of the periphery of the block (subject area).

_{Next, the tracking processing unit 34 observes the predicted state Sk} ⁽ⁱ⁾ by performing the calculation of the following equation (13), and calculates the weighted average of the observation result evaluation. At this time, the tracking processing unit 34 evaluates the feature quantities of N prediction regions as shown by a plurality of frames in FIG. 12, and obtains the weighting coefficient. The weighted average of the obtained weighting factors is the tracking result.

Specifically, the tracking processing unit 34 works by performing the operations of the following equations (14) to (20) by adding a random variable to the above equations (3) to (9). to predict the state _{S k} of the person. As r _{1 to r 7} , for example, a Gaussian random variable can be used.

Next, the tracking processing unit 34 ^{calculates the weighting coefficient π i} in the predicted state as follows. That is, first, the tracking processing unit 34 uses the histogram P calculated from the tracking area of the operator as a model, and performs the calculation of the following equation (21) as the color histogram q of the predicted N prediction areas. Then, calculate the Bhattacharyya coefficient.

このバタチャリア係数は、作業者の追尾領域のカラーヒストグラム（＝モデル）と、予測した領域のカラーヒストグラムの類似度を表す。このため、バタチャリア係数の値が大きいということは、両者の類似度が高いことを意味する。追尾処理部３４は、このバタチャリア係数を用いて、以下の（２２）式の演算を行うことで、予測した状態の重み係数π^ｉを算出する。

This batacharia coefficient represents the similarity between the color histogram (= model) of the tracking area of the operator and the color histogram of the predicted area. Therefore, a large value of the Batacharia coefficient means that the degree of similarity between the two is high. ^{The tracking processing unit 34 calculates the weighting coefficient π i} in the predicted state by performing the calculation of the following equation (22) using this batacharia coefficient.

In addition, "d" in the formula (22) is calculated by the following formula (23).

Tracking processing unit 34 is thus calculated weight coefficients [pi ^i, and, using the state S ⁱ _k expected, by carrying out calculation of the above equation (13), and calculates the tracking result of the operator.

Next, the tracking processing unit 34 uses the similarity ρ [p, q] of the tracking result represented by the batacharia coefficient calculated by performing the calculation of the above equation (21) as the certainty of worker tracking. The value of this certainty is in the range of 0 to 1.0. The higher the value of ρ [p, q], the higher the certainty.

If the certainty level is equal to or higher than a predetermined threshold value set in advance, the tracking processing unit 34 determines that the tracking is successful, and maintains the ID of each worker shown in FIG. 8 as it is. Further, the tracking processing unit 34 stores, for example, the position information (= rectangular information) of the worker determined to be successful in tracking and the image data of the rectangular area of the worker as the worker information in the storage unit. When the image information of the next frame is input, the tracking processing unit 34 performs the above-mentioned tracking processing, and when the tracking is successful, the worker ID is maintained and the worker information saved in the storage unit is updated. To do.

Next, during the tracking of such workers, the workers overlap each other and move to the shadow of the object, or move out of the imaging range of the camera device 2, so that the worker who was tracking can move from the captured image. It may disappear or become difficult to detect, making it difficult for the operator to track. In such a case, it becomes difficult to acquire the data of the worker whose ID is lost or difficult to detect.

In addition, "when tracking becomes difficult" is a synonym for "when tracking is not possible", "when tracking is discontinued", "when tracking becomes poor", and the like.

When the tracking of the worker is successful, as described above, the tracking processing unit 34 maintains the ID of the worker being tracked and updates the worker information in the storage unit. On the other hand, when tracking becomes difficult, the tracking processing unit 34 maintains the ID and worker information of the worker who has become difficult to track, which was detected in the last frame in which tracking was successful. For example, in the example of FIG. 8, when it becomes difficult to track the worker with ID3, the tracking processing unit 34 maintains the ID3 of the worker and the worker information detected in the frame in which the tracking was last successful.

When it becomes difficult to track the worker, the recognition unit 32 performs the recognition process of each worker again and resets the ID for each worker. When resetting this ID, among the workers who were re-recognized, the ID of the worker who had difficulty in tracking the worker who had the similarity closest to the similarity of the worker who had difficulty in tracking. To set. As a result, the same ID can be set for the same worker.

To specifically explain such an ID resetting operation, when it becomes difficult to track the operator, the recognition unit 32 again performs the operator based on the captured image acquired via the input unit 31. Re-recognition process is performed. It is assumed that, for example, three workers A, B, and C are recognized by this re-recognition process.

The initial ID setting unit 33 determines the similarity of the three workers A, B, and C that have been re-recognized, and the similarity (batacharia coefficient) of, for example, the worker of ID3 that was maintained when tracking became difficult. Each is calculated by performing the calculation of the above equation (21). Then, the initial ID setting unit 33 refers to the worker having the closest similarity to the ID3 worker maintained when the tracking becomes difficult among the three workers A, B, and C. And set "ID3". As a result, when the worker who has become difficult to track once becomes in a detectable state again, he / she sets the same ID as the ID set when the tracking becomes difficult and continues the tracking. Can be done.

Similarly, for example, when it becomes difficult to track two workers with ID1 and ID2, the initial ID setting unit 33 will perform when tracking becomes difficult among the three workers A, B, and C. “ID1” is set for the worker having the closest similarity to the maintained ID1 worker. Further, the initial ID setting unit 33 refers to the worker having the closest similarity to the ID2 worker maintained when tracking becomes difficult among the three workers A, B, and C. And set "ID2". Even after re-recognition, each worker can be tracked with the same ID.

Next, the action recognition processing unit 35 recognizes the worker's behavior based on the behavior recognition dictionary for recognizing the worker's behavior, which is input from the behavior recognition dictionary input unit 38 in step S6 of the flowchart of FIG. Perform the process (step S7).

Specifically, as shown in FIG. 13, the action recognition processing unit 35 extracts the spatiotemporal features of the rectangular region of the worker in a plurality of frames input via the input unit 31. The horizontal axis x and the vertical axis y of each frame shown in FIG. 13 are spatial coordinates. Further, FIG. 13 shows how the frames F1, F2, ... Are arranged in chronological order along the time axis t. That is, each frame is spatiotemporal (x, y, t) image data. Further, one pixel value I (x, y, t) in space-time is a function of spatial coordinates (x, y) and time t.

When the worker moves, a change point occurs in the spatiotemporal image data shown in FIG. The action recognition processing unit 35 recognizes the specific action of the worker based on this change point (= feature point in space-time). The action recognition processing unit 35 detects this change point (characteristic point in space-time) as follows.

That is, FIG. 14 shows spatiotemporal image data. The large cube shown in FIG. 14 is spatiotemporal image data. The horizontal axis represents the spatial coordinates x (pixels), and the vertical axis represents the spatial coordinates y (pixels). The time axis t is a time series axis of the captured image input at a predetermined frame rate such as 30 frames per second.

The action recognition processing unit 35 divides the spatiotemporal image data shown in FIG. 14 into blocks having M pixels in the x direction, N pixels in the y direction, and a T frame size (M × N × T) in the t direction. When the worker performs a specific action, the feature amount of the block corresponding to the action of the worker in the spatiotemporal image data becomes large (a large amount of change occurs in the spatiotemporal space). As described below, the action recognition processing unit 35 extracts a block having a large amount of change as a feature point.

When the action recognition processing unit 35 extracts the feature points from the spatiotemporal image data, the action recognition processing unit 35 first performs the calculation of the following equation (24) in order to remove the noise in the spatial direction (x, y) direction. Smooth the spatiotemporal image data.

I (x, y, t) shown in the equation (24) indicates the pixel value of the xy coordinate in the frame at time t. Further, g (x, y) shown in Eq. (24) is a kernel for smoothing processing. The "*" symbol means that the convolution process is performed. The smoothing process may be performed by averaging the pixels, or may be performed using a Gaussian smoothing filter.

Next, the action recognition processing unit 35 performs filtering processing on the time axis with respect to the spatiotemporal image data that has undergone the smoothing processing. As this filtering process, a Gabor filtering process represented by the following equation (25) is performed.

The (25) _{"g ev"} and _{"g od"} in the expression, respectively in the following (26) and (27), a kernel Gabor filter. The "*" symbol means that the convolution process is performed. “Τ” and “ω” are parameters of the Gabor filter kernel.

Next, the action recognition processing unit 35 performs the filtering process shown by the above equation (2) on all the pixels of the spatiotemporal image data shown in FIG. 13, and then divides the pixels as shown in FIG. The average value of the blocks is calculated by the following formula (28).

In the action recognition processing unit 35, the average value (M (x, y, t)) of the blocks calculated by the calculation of the formula (28) is equal to or higher than a predetermined threshold value (Thre) as shown in the following formula (29). If the value is, this block is extracted as a feature point.

Next, the action recognition processing unit 35 calculates the spatiotemporal edge information of the pixels of the block, which is the feature point extracted from the spatiotemporal image data, by performing the differential calculation of the following equation (30).

In the case of the example shown in FIG. 14, since one block has M × N × T pixels, it is possible to obtain M × N × T × 3 differential values. Therefore, each block can be described by a vector of three differential values of MxNxTx. That is, the feature points can be described by a vector of M × N × T × 3 dimensions.

The action recognition dictionary input from the action recognition dictionary input unit 38 to the action recognition processing unit 35 shown in FIG. 3 is an image of a specific action such as a worker carrying a baggage, walking, or placing the baggage on a shelf. It is M × N × T × 3D vector information calculated (learned) in advance by performing the calculation of the equation (30) based on the feature points detected from the captured image of the frame.

When creating an action recognition dictionary, the action recognition dictionary input unit 38 uses, for example, the K-means clustering method to generate feature points that are M × N × T × three-dimensional vectors, for example, K types. It is classified into the feature points of. By performing this classification process, feature points having similar features can be classified as the same type of feature points.

Next, the action recognition dictionary input unit 38 averages the M × N × T × 3D edge vectors of the same type of feature points for the classified K types of feature points, and calculates K average vectors Vk. To do. Each vector is a recognition vector that represents that type of feature point. The feature points obtained from the captured image of the specific behavior of the worker are distributed near the average vector Vk obtained from the learning data of the same specific behavior.

Utilizing this characteristic, the behavior recognition dictionary input unit 38 calculates the total number of blocks of each feature point group of K type, and calculates the recognition histogram H (k) which is the frequency of the feature point group. As described above, the distribution of the feature points to be recognized is close to the distribution of the feature points of the training data. Therefore, for example, the recognition histogram of the worker to be recognized is approximated to the learning histogram of the learning data of the same behavior (behavior) of the worker. Therefore, it is possible to create an action recognition dictionary for recognizing a specific action of a worker or the like with the histogram H (k) obtained from the learning data.

As an example, the behavior recognition dictionary can be created by using the machine learning method of SVM (Support Vector Machine). When creating a behavior recognition dictionary by this machine learning method, positive learning data learned from the captured image of the specific behavior of the worker to be recognized and negative learning from the captured image of the behavior of the worker different from the specific behavior. Create a behavior recognition dictionary with the learning data of.

The behavior recognition dictionary may be created by using other machine learning methods such as K Nearest Neighbor or Multilayer Perceptron, in addition to the SVM machine learning method.

Summarizing the behavior recognition operations of the behavior recognition processing unit 35 described above, the behavior recognition processing unit 35 is based on the above-mentioned time from the N-frame spatiotemporal image data input as the captured image (moving image) of the worker to be recognized. Extract spatial feature points. The action recognition processing unit 35 obtains an M × N × T × three-dimensional differential vector of each feature point block. The distances from the K learning average vectors Vk obtained from the differential vector and the input learning data are calculated, and the types of the feature point blocks are classified into the types of the learning average vectors Vk having the closest distance. By classifying the feature point blocks by this method, the feature point blocks can be classified into K types. The action recognition processing unit 35 creates a feature point histogram T (k) of the captured image (moving image) to be recognized based on the appearance frequency of each type of feature point block.

Then, the behavior recognition processing unit 35 uses the above-mentioned SVM machine learning method based on the behavior recognition dictionary input from the behavior recognition dictionary input unit 38 and the feature point histogram T (k) of the captured image to be recognized. , Performs recognition processing of the specific behavior of the worker. In the SVM recognition process using the SVM machine learning method, the worker's specific action and the recognition result other than the specific action are output.

The recognition result output unit 36 shown in FIG. 3 outputs the recognition result other than the specific action of the worker and the specific action to the monitoring device 23 via, for example, the output interface unit 17 (step S8). As a result, it is possible to monitor the monitoring target of the worker or the like via the monitoring device 23. In step S9 of the flowchart of FIG. 4, the CPU 11 determines whether or not such recognition processing is completed. When it is determined that the recognition process has not been completed (step S9: No), the process returns to step S1, and the above-mentioned recognition process for the specific action of the worker is repeatedly performed. When it is determined that the recognition process is completed (step S9: Yes), all the processes of the flowchart of FIG. 4 are completed.

The recognition result output unit 36 may transmit the recognition result to the server device 22 via the communication unit 14 and the network 21. In this case, the administrator or the like accesses the server device 22 via a communication device such as a smart phone, a tablet terminal device, or a personal computer device, and acquires the recognition result. This makes it possible to remotely monitor the monitoring target of the worker or the like.

Next, the recognition result output form of the recognition result output unit 36 will be described by using the behavior of the worker walking, moving to the position of the shelf, holding the product and putting it on the shelf as an example of the recognition target. In this case, as shown in FIG. 15, the recognition result output unit 36 outputs the start time and duration of the action based on the recognition result of the worker's action recognized by the action recognition processing unit 35. The difference (t2-t1) between the walking end time t2 and the walking start time t1 is the walking time.

Further, the difference (t3-t2) between the end time t3 of the shelving action and the start time t2 of the shelving action is the time of the shelving action. The shelving work time is the total time of the worker's walking time and the shelving work time, and is the time of the difference (t3-t1) between the shelving end time t3 and the walking start time t1. The recognition result output unit 36 outputs the walking time, the shelving action time, and the total time of the shelving work of each worker.

Next, for example, when the worker overlaps with another worker and the posture of the worker changes, which makes it difficult for the worker to recognize, the behavior recognition processing unit 35 interrupts the recognition process of the worker. FIG. 16 shows an example in which it becomes difficult for an operator to recognize an operator while walking while shelving products. The time between the time t3 and the time t2 shown in FIG. 16 indicates the time when the worker who is walking becomes difficult to recognize.

In such a case, the behavior recognition processing unit 35 determines whether or not the time difference between the time t3-time t2 at which the walking worker becomes difficult to recognize is equal to or less than the predetermined threshold value Thr_w. When a walking worker overlaps with another worker, it often becomes difficult to recognize temporarily, for example, for 2 seconds or 5 seconds. Therefore, the action recognition processing unit 35 sets the threshold value Thr_w to, for example, 2 seconds or 5 seconds, and the time difference of the time t3-time t2 at which the walking worker becomes difficult to recognize is 2 seconds or less or When it is 5 seconds or less, it is recognized that the worker is in a walking state during the time t3-time t2.

That is, if the time when the worker's recognition becomes difficult is less than a predetermined time, the action recognition processing unit 35 continuously performs the action (action) recognized before the recognition becomes difficult. Recognize that it was. As a result, during the period from the walking start time t1 to the shelving start time t4 shown in FIG. 16, even if the worker becomes difficult to recognize on the way, it is recognized that the worker is continuously in the walking state. To.

FIG. 17 shows an example in which the worker temporarily becomes difficult to recognize during shelving. Times t3 to time t4 shown in FIG. 17 are times during which the worker was temporarily difficult to recognize during shelving. In this case as well, as described above, if the time during which the worker has difficulty in recognizing is less than or equal to the time at which the threshold value Thr_w is reached, for example, 2 seconds or 5 seconds, the worker is difficult to recognize. It is recognized that the worker continued to carry out the shelving work even during the time that had been reached. As a result, during the period from the shelving start time t2 to the shelving end time t5 shown in FIG. 17, even if the worker becomes difficult to recognize on the way, the worker is continuously carrying out the shelving work. Is recognized.

In this way, if the time when the worker's recognition becomes difficult is less than a predetermined time, it is recognized that the action (movement) that was recognized before the recognition became difficult was continuously performed during that time. By doing so, even if the worker becomes difficult to recognize on the way, it is possible to measure the correct working time and the like.

The recognition result output unit 36 sets, for each worker, the work start time, the work end time, the work time (time from the start to the end of a series of work (operations) = required time), etc., which are the recognition results of the action. Output as the worker's behavior recognition result.

(Effect of embodiment)
As is clear from the above description, the behavior recognition system of the embodiment recognizes each monitoring target based on the captured images of the monitoring targets of a plurality of workers and the like, and sets an ID for each. Each monitoring target is tracked based on the set ID, but when the monitoring target becomes difficult to recognize, the ID and related information (position information and image data) of the monitoring target that has become difficult to recognize are maintained. In this state, the above-mentioned calculation of the equation (21) is performed to calculate the similarity of the monitored objects that have become difficult to recognize and the similarity of each monitored object that has been re-recognized. Then, among the re-recognized monitoring targets, the ID of the monitoring target that has become difficult to recognize is set for the monitoring target that has the similarity that most closely resembles the similarity of the monitoring target that has become difficult to recognize.

As a result, even if multiple monitoring targets are recognized at the same time and some or all of the monitoring targets are temporarily difficult to recognize, when the monitoring targets are re-recognized, before and after the recognition becomes difficult. , The same ID can be set as a monitoring target so that it can be tracked. Therefore, it is possible to accurately monitor a plurality of monitoring targets.

Further, the action recognition system of the embodiment extracts spatiotemporal feature points of the monitored area from the captured images of a plurality of frames, and responds to the actions (movements) of each monitored target based on the extracted spatiotemporal feature points. Detect feature quantity. Then, based on this feature amount, the action (behavior) of each monitoring target is recognized, and for example, the action start time, the action end time, the required time, etc. of each monitoring target are output as the authentication result. As a result, the behaviors of a plurality of monitored objects can be visualized respectively.

Finally, the embodiments described above are presented as an example and are not intended to limit the scope of the invention. Each of the novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention.

For example, in the description of the above-described embodiment, the monitoring target is described as a worker, but this may be an animal, a passerby on a road, a person gathered at a specific place, or another robot or the like. It may be a monitoring target. In this case as well, the same effect as described above can be obtained.

Further, the embodiment and the modification of the embodiment are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof.

1 Action recognition device 2 Camera device 3 Interface unit 11 CPU
15 HDD
31 Input unit 32 Recognition unit 33 Initial ID setting unit 34 Tracking processing unit 35 Action recognition processing unit 36 Recognition result output unit 37 Monitoring target recognition dictionary input unit 38 Behavior recognition dictionary input unit

Japanese Unexamined Patent Publication No. 2011-100175

Claims (8)

A recognition unit that recognizes multiple monitoring targets based on the captured image,
An identification number setting unit that sets an identification number for each of the monitoring targets recognized by the recognition unit,
Based on the region of each monitoring target recognized in the captured image, a tracking processing unit that tracks each monitoring target recognized in the captured image and for which an identification number is set, and a tracking processing unit.
An operation recognition processing unit that recognizes the operation of each of the monitored objects in the result area of the captured image that tracks each of the monitored objects.
An operation recognition device having a recognition result output unit that outputs a recognition result of each recognized operation of the monitored object. The motion recognition device according to claim 1, wherein the recognition result output unit outputs the start time of the operation of each monitoring target and the time from the start to the end of a series of operations as the recognition result. Claim 1 or claim, wherein the tracking processing unit maintains the identification number, position information, and image information of each monitoring target, and updates the position information and image information each time the tracking is successful. Item 2. The motion recognition device according to item 2. The tracking processing unit maintains the identification number, position information, and image information of the monitoring target whose tracking has become difficult.
The identification number setting unit calculates the similarity of the monitored object for which tracking has become difficult, and also calculates the similarity of each monitored object re-recognized by the recognition unit due to the difficulty in tracking. Of the similarities of each monitored object re-recognized by the recognition unit, it became difficult to track a monitored object having a similarity closest to the similarity of the monitored object for which tracking became difficult. The motion recognition device according to claim 3, wherein the same identification number as the identification number set for the monitoring target is set. From claim 1, the motion recognition processing unit extracts spatiotemporal feature points from the captured images of a plurality of frames, and recognizes the motion of each monitoring target based on the extracted spatiotemporal feature points. The motion recognition device according to any one of claims 4. When the time during which the tracking target is difficult to track by the tracking processing unit during a series of operations of the monitoring target is less than or equal to a predetermined time, the recognition result output unit continues the series even during the time during which the tracking becomes difficult. 1 to any one of claims 1 to 5, characterized in that the time information corresponding to the operation of the monitored object is output as the recognition result by recognizing that the operation of the above has been performed. The described motion recognition device. A recognition step in which the recognition unit recognizes a plurality of monitoring targets based on the captured image,
An identification number setting step in which the identification number setting unit sets an identification number for each of the recognized monitoring targets, and
A tracking processing step in which the tracking processing unit tracks each of the monitoring targets recognized in the captured image and having an identification number set based on the region of each monitoring target recognized in the captured image.
A motion recognition processing step in which the motion recognition processing unit recognizes the motion of each monitoring target in the result area of the captured image that tracks each monitoring target.
An operation recognition method in which a recognition result output unit has a recognition result output step for outputting a recognition result of each recognized operation of the monitored object. Computer,
A recognition unit that recognizes multiple monitoring targets based on the captured image,
An identification number setting unit that sets an identification number for each of the monitoring targets recognized by the recognition unit,
Based on the region of each monitoring target recognized in the captured image, a tracking processing unit that tracks each monitoring target recognized in the captured image and for which an identification number is set, and a tracking processing unit.
An operation recognition processing unit that recognizes the operation of each of the monitored objects in the result area of the captured image that tracks each of the monitored objects.
An operation recognition program characterized in that it functions as a recognition result output unit that outputs the recognition result of each recognized operation of the monitored object.

Images