JP2020154552A - Behavior recognition device, behavior recognition method, and program - Google Patents

Abstract

To provide a behavior recognition device, a behavior recognition method, and a program, capable of recognizing standard work of workers in a highly accurate and high-speed manner.SOLUTION: A video input section accepts input of a video captured by a fish-eye camera (capturing means), and an area splitting section splits an image included in the video into a plurality of areas with distinct distortions. A dictionary creation section creates a recognition dictionary for recognizing a specific behavior of a subject for each split area. A dictionary selection section selects a recognition dictionary to be used for recognizing the specific behavior of the subject detected from the video, from a plurality of the recognition dictionaries created by the dictionary creation section. Then, a behavior recognition section recognizes the specific behavior of the subject based on the recognition dictionary selected by the dictionary selection section.SELECTED DRAWING: Figure 12

Description

The present invention relates to a behavior recognition device, a behavior recognition method and a program.

In workplaces such as offices and factories, it is an important issue to improve the production efficiency of the workplace by visualizing the behavior of workers and analyzing the working hours. Therefore, it is effective to take a video of the workplace with a camera and analyze the obtained video to recognize and analyze the behavior of a specific standard work (hereinafter referred to as standard work) by the worker.

However, it takes a huge amount of analysis time and effort to visually analyze workplace videos taken with a camera, extract behaviors of standard work performed in a fixed procedure, measure the time of each movement, and visualize them. is necessary. Therefore, conventionally, in order to automatically recognize a human behavior, a method has been proposed in which a person is recognized from a captured video, the movement trajectory of the person is obtained from the center of gravity of the recognized person, and a specific behavior is recognized from the movement trajectory. There is.

When recognizing the behavior of an operator, it is desirable to capture the widest possible field of view with one camera in order to improve the efficiency of processing. Therefore, it is desirable to take a picture using a camera equipped with a wide-angle lens having a wide angle of view. However, images taken with a camera equipped with a wide-angle lens are distorted. When the image is distorted, the shape of the person in the image is distorted, and the recognition accuracy of the person deteriorates. Since it is necessary to trace the movement of the same person over time in order to recognize the standard work, the deterioration of the recognition accuracy of the person causes the deterioration of the recognition accuracy of the standard work. In order to prevent such deterioration of accuracy, it is desirable to recognize the standard operation after correcting the distortion of the image. However, since it takes time and effort to correct the distortion, there is a problem that it is difficult to recognize the standard work with high accuracy and high speed.

The present invention has been made in view of the above, and an object of the present invention is to provide a behavior recognition device, a behavior recognition method, and a program capable of recognizing a worker's standard work with high accuracy and high speed. ..

In order to solve the above-mentioned problems and achieve the object, the behavior recognition device of the present invention is a behavior recognition device that recognizes a specific behavior of a subject reflected in the moving image from a captured moving image, and includes a wide-angle lens. A moving image input unit for inputting the moving image shot by the shooting means, an area dividing section for dividing the image included in the moving image into a plurality of areas having different distortions, and specifying the subject for each of the divided areas. A dictionary creation unit that creates a recognition dictionary for recognizing an action, and a recognition dictionary used to recognize a specific action of a subject detected from the moving image from a plurality of recognition dictionaries created by the dictionary creation unit. It is characterized by including a dictionary selection unit for selection and an action recognition unit for recognizing a specific action of the subject based on the recognition dictionary selected by the dictionary selection unit.

According to the present invention, a worker's standard work can be recognized with high accuracy and high speed.

FIG. 1 is a hardware block diagram showing an example of the hardware configuration of the behavior recognition system according to the first embodiment. FIG. 2 is a diagram showing an example of a scene in which the behavior recognition system according to the first embodiment is used. FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the fisheye camera. FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the action recognition device. FIG. 5 is a diagram illustrating distortion of an image observed with a fisheye lens. FIG. 6 is a diagram for explaining the difference in distortion depending on the position of the image observed by the fisheye lens. FIG. 7 is a diagram showing an example of an image observed by the behavior recognition system of the first embodiment. FIG. 8 is a diagram for explaining the “walking” behavior among the specific behaviors recognized by the behavior recognition system. FIG. 9 is an enlarged view of a person in the image of FIG. 7. FIG. 10 is a diagram for explaining the “shelfing” behavior of putting a product on a shelf among the specific behaviors recognized by the behavior recognition system. FIG. 11 is a diagram showing an example of an enlarged view of a person performing a shelving action. FIG. 12 is a functional block diagram showing an example of the functional configuration of the action recognition processing unit. FIG. 13 is a functional block diagram showing an example of the functional configuration of the dictionary creation unit. FIG. 14 is a functional block diagram showing an example of the functional configuration of the action recognition unit. FIG. 15 is a diagram showing an example of a moving image input to the moving image input unit. FIG. 16 is a diagram illustrating a feature point detection method. FIG. 17A is a first diagram showing an example of the extracted feature points. FIG. 17B is a second diagram showing an example of the extracted feature points. FIG. 18 is a diagram illustrating the measurement of the duration of a specific action. FIG. 19 is a flowchart showing an example of the flow of creating a recognition dictionary. FIG. 20 is a flowchart showing an example of the flow of the recognition process of the specific action. FIG. 21 is a flowchart showing an example of a processing flow for recognizing a plurality of specific actions. FIG. 22 is a hardware block diagram showing an example of the hardware configuration of the behavior recognition system according to the second embodiment. FIG. 23 is a diagram showing an example of a scene in which the behavior recognition system according to the second embodiment is used. FIG. 24 is a functional block diagram showing an example of the functional configuration of the action recognition processing unit according to the second embodiment. FIG. 25 is a flowchart showing an example of the flow of the recognition process of the specific action in the second embodiment.

(First Embodiment)
The behavior recognition device, the behavior recognition method, and the first embodiment of the program will be described in detail with reference to the accompanying drawings.

(Explanation of hardware configuration of behavior recognition device)
FIG. 1 is a hardware block diagram showing an example of the hardware configuration of the behavior recognition system 100 according to the present embodiment. As shown in FIG. 1, the behavior recognition system 100 includes a fisheye camera 200 and a behavior recognition device 300.

The action recognition system 100 recognizes a specific action of a subject photographed by the fisheye camera 200. The specific action is, for example, a standard work such as “walking” or “shelving luggage” that is repeatedly performed in the work environment of the workplace.

The fisheye camera 200 is a video camera provided with a fisheye lens capable of observing a range of 360 ° in the entire circumference. It should be noted that the fisheye lens is provided as an example, and the fisheye camera 200 may be provided with a wide-angle lens. The fisheye camera 200 is an example of a photographing means.

The action recognition device 300 recognizes the specific action of the person (subject) in the moving image by analyzing the moving image taken by the fisheye camera 200. In order to recognize the specific behavior of the subject, a certain number of frames of images (continuous images, videos) are required. As the number of frames increases, the load of the processing for correcting the distortion of the fisheye camera 200 increases. The feature of this embodiment is that the moving image is analyzed without correcting the distortion.

The behavior recognition device 300 includes a behavior recognition processing unit 321 and an interface unit 322 that connects the behavior recognition processing unit 321 and the fisheye camera 200.

The action recognition processing unit 321 recognizes a specific action of a person (subject). The interface unit 322 converts the moving image taken by the fisheye camera 200 into a data format that can be recognized by the action recognition processing unit 321 and delivers it to the action recognition processing unit 321.

Next, a typical scene in which the behavior recognition system 100 is used will be described with reference to FIG. FIG. 2 is a diagram showing an example of a scene in which the behavior recognition system 100 according to the first embodiment is used.

As shown in FIG. 2, the behavior recognition system 100 is installed in a work environment in a workplace such as an office or a factory. Then, the fisheye camera 200 captures a moving image including a plurality of people H1 and H2 who are working in the work environment. Since it is efficient to capture the working environment with one camera, it is desirable that the fisheye camera 200 is provided with a wide-angle lens having a wide angle of view. In the present embodiment, the fisheye camera 200 includes a fisheye lens having a diagonal angle of view of 180 °. The people H1 and H2 are examples of subjects.

(Explanation of hardware configuration of fisheye camera)
First, the hardware configuration of the fisheye camera 200 will be described.

FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the fisheye camera 200. As shown in FIG. 3, the fisheye camera 200 includes a fisheye lens 217 having a diagonal angle of view of 180 degrees or more and a CCD (Charge Coupled Device) 203. The fisheye camera 200a is an example of a photographing means. The fisheye camera 200 incidents the subject light on the CCD 203 through the fisheye lens 217. Further, the fisheye camera 200 includes a mechanical shutter 202 between the fisheye lens 217 and the CCD 203. The mechanical shutter 202 blocks the incident light on the CCD 203. The opening and closing of the mechanical shutter 202 is controlled by the motor driver 206. Further, the lens position of the fisheye lens 217 is also controlled by the motor driver 206, and the autofocus function is realized.

The CCD 203 converts the optical image formed on the imaging surface into an electric signal and outputs analog image data. The image data output from the CCD 203 has noise components removed by the CDS (Correlated Double Sampling) circuit 204 and converted into digital image data (hereinafter, simply referred to as image data) by the A / D converter 205. After that, it is output to the image processing circuit 208.

The image processing circuit 208 uses an SDRAM (Synchronous DRAM) 212 that temporarily stores image data to perform various image processing such as YCrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, and color conversion processing. .. The white balance process is an image process that adjusts the color density of the image data, and the contrast correction process is an image process that adjusts the contrast of the image data. The edge enhancement process is an image process that adjusts the sharpness of the image data, and the color conversion process is an image process that adjusts the hue of the image data. Further, the image processing circuit 208 displays the image data subjected to signal processing and image processing on the LCD 216 (liquid crystal display).

The image data subjected to signal processing and image processing in the image processing circuit 208 is recorded in the memory card 214 via the compression / decompression circuit 213. The compression / decompression circuit 213 compresses the image data output from the image processing circuit 208 and outputs it to the memory card 214 according to the instruction acquired from the operation unit 215, and decompresses the image data read from the memory card 214 to obtain an image. Output to the processing circuit 208.

The fisheye camera 200a includes a CPU (Central Processing Unit) 209 that performs various arithmetic processes according to a program. The CPU 209 is a read-write memory RAM (Random Access) having a ROM (Read Only Memory) 211 which is a read-only memory for storing programs and the like, a work area used in various processing processes, and various data storage areas. It is interconnected with Memory) 210 by a bus line.

The timing of the CCD 203, the CDS circuit 204, and the A / D converter 205 is controlled by the CPU 209 via the timing signal generator 207 that generates the timing signal. Further, the image processing circuit 208, the compression / decompression circuit 213, and the memory card 214 are also controlled by the CPU 209.

The output of the fisheye camera 200 is input to the interface unit 322, which is the signal processing board of the behavior recognition device 300 shown in FIG.

(Explanation of hardware configuration of behavior recognition device)
Next, the hardware configuration of the action recognition device 300 will be described.

FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the action recognition device 300. As shown in FIG. 4, the action recognition device 300 includes a CPU (Central Processing Unit) 301 that controls the operation of the entire action recognition device 300, and a ROM (Read Only Memory) 302 and CPU 301 that store programs used to drive the CPU 301. It has a RAM (Random Access Memory) 303 used as a work area of the above. It also has an HD (Hard Disk) 304 that stores various data such as programs, and an HDD (Hard Disk Drive) 305 that controls reading or writing of various data to the HD 304 according to the control of the CPU 301.

The action recognition device 300 also has a media I / F 307, a display 308, and a network I / F 309. The media I / F 307 controls reading or writing (storage) of data to the media 306 such as a flash memory. The display 308 displays various information such as cursors, menus, windows, characters, or images. The network I / F 309 uses a communication network for data communication.

The action recognition device 300 also includes a keyboard 311, a mouse 312, a CD-ROM (Compact Disk Read Only Memory) drive 314, and a bus line 310. The keyboard 311 includes a plurality of keys for inputting characters, numerical values, various instructions, and the like. The mouse 312 selects and executes various instructions, selects a processing target, moves the cursor, and the like. The CD-ROM drive 314 controls reading or writing of various data to the CD-ROM 313 as an example of the removable recording medium. The bus line 310 is an address bus, a data bus, or the like for electrically connecting the above components.

The hardware of the illustrated behavior recognition device 300 does not need to be housed in one housing or as a group of devices. Further, in order to support cloud computing, the physical configuration of the behavior recognition device 300 of the present embodiment does not have to be fixed, and hardware resources are dynamically connected / disconnected according to the load. It may be composed of.

The program is distributed in an executable format or a compressed format in a state of being stored in a storage medium such as media 306 or CD-ROM 313, or is distributed from a server that distributes the program.

The program executed by the action recognition device 300 of the present embodiment has a modular configuration including each function shown below. The CPU 301 of the action recognition device 300 reads a program from a storage medium such as ROM 302 or HD 304 and executes the program, so that each module is loaded on the RAM 303 and exerts each function.

(Explanation of distortion generated by fisheye camera)
Next, the distortion generated in the image taken by the fisheye camera 200 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram illustrating distortion of an image observed with a fisheye lens. FIG. 6 is a diagram for explaining the difference in distortion depending on the position of the image observed by the fisheye lens.

Image I shown in FIG. 5 is an example of image I observed when a camera equipped with a standard lens or a telephoto lens captures a target in which squares composed of regular vertical and horizontal straight lines are drawn. .. As shown in FIG. 5, in the image I, vertical and horizontal linear grids are observed. Then, the ratio of the lengths of the vertical lines and the horizontal lines in each square is almost equal regardless of the position of the image I. That is, the distortion generated in the image I is very small.

On the other hand, the image J is an example of an image observed when the same target as described above is photographed by the fisheye camera 200 of the present embodiment. The fisheye camera 200 generates an image by a so-called equidistant projection method, which generates an image in which the distance from the center of the image is proportional to the direction (angle) of the observation object. Therefore, the magnitude of the distortion generated differs between the vicinity of the center of the image J and the peripheral portion.

Specifically, in the image J observed when the target is photographed, the vertical line is observed in an arc shape passing through the points C1 and C2 on the circumference of the circle circumscribing the image J. Further, the horizontal line of the target is observed in an arc shape passing through the points C3 and C4 on the circumference of the circle circumscribing the image J.

That is, in the vicinity of the center of the image J, the vertical and horizontal lines of the target are observed in a state close to a straight line. Then, the ratio of the lengths of the vertical lines and the horizontal lines in each square is almost the same. That is, the distortion that occurs is small. On the other hand, in the peripheral portion of the image J, both the vertical line and the horizontal line of the target are observed as curves. Furthermore, the ratio of vertical lines to horizontal lines in each square is different. As described above, in the image J, the larger the distance from the center of the image, the larger the distortion that occurs. The direction of the generated distortion is point-symmetrical with respect to the center of the image J.

Therefore, in the image J, the magnitude and direction of the movement that occurs when the person performs the same action differs depending on the position where the person is observed. That is, in order to recognize a person's behavior, a recognition dictionary may be prepared for each location of the image J, and the behavior may be recognized by using the recognition dictionary according to the position where the person is observed.

Specifically, as shown in FIG. 6, a plurality of regions R1, R2, R3, and R4 are set from the center of the image J to the periphery, and a recognition dictionary is created for each region R1, R2, R3, and R4. create. In this case, the distortion of the image is the smallest in the region R1 and the largest in the region R4. The areas R1, R2, R3, and R4 are examples of areas to be set, and the number of areas is not limited to four. As described above, the action recognition system 100 of the present embodiment sets similar areas at a plurality of different positions of the image J, and creates a recognition dictionary in each area. Then, the behavior recognition system 100 performs behavior recognition using a recognition dictionary created at a position closest to the position of the person detected from the captured moving image.

Even when a camera equipped with a wide-angle lens or an ultra-wide-angle lens is used instead of the fisheye camera 200, distortion larger than the center of the image occurs around the image as in the fisheye lens. .. Therefore, a method of performing behavior recognition using a recognition dictionary according to the position in the image is effective.

(Explanation of the actually observed image)
An example of an image observed by the behavior recognition system 100 will be described with reference to FIGS. 7 to 11. FIG. 7 is a diagram showing an example of an image observed by the behavior recognition system 100 of the first embodiment.

FIG. 7 is an example of a worker's specific behavior in the workplace. In particular, image J1 of FIG. 7 is a diagram showing an example of a specific action of “walking”. The "walking" action is an action that occurs when a worker moves from one specific action to another when performing a plurality of specific actions. And, in general, the longer the time required for the "walking" action, the lower the work efficiency. The action recognition system 100 recognizes the walking action as one of the specific actions.

FIG. 8 is a diagram for explaining the “walking” behavior among the specific behaviors recognized by the behavior recognition system 100. As shown in FIG. 8, the fisheye camera 200 photographs the person H1 who is walking in chronological order. In this case, a moving image (image sequence) of a walking person H1 is obtained.

FIG. 9 is an enlarged view j1 of a person in the image J1 of FIG. The action recognition system 100 recognizes a specific action by observing the time change of the region shown in FIG.

FIG. 10 is a diagram for explaining a “shelfing” action of putting a product on a shelf among the specific actions recognized by the action recognition system 100. As shown in FIG. 10, the fisheye camera 200 captures the person H1 who is shelving in a character series. In this case, a moving image (image sequence) of the person H1 who is shelving can be obtained.

FIG. 11 is a diagram showing an example of an enlarged view j2 of a person performing a shelving action. The action recognition system 100 recognizes a specific action by observing the time change of the region shown in FIG.

(Explanation of the functional configuration of the behavior recognition processing unit)
Next, the functional configuration of the action recognition processing unit 321 will be described with reference to FIG. FIG. 12 is a functional block diagram showing an example of the action recognition processing unit 321 according to the present embodiment. As shown in FIG. 12, the action recognition processing unit 321 includes a moving image input unit 331, an area division unit 332, a dictionary creation unit 333, a dictionary selection unit 334, an action recognition unit 335, and a duration measurement unit 336. To be equipped.

The moving image input unit 101 inputs a moving image taken by the fisheye camera 200 via the interface unit 322 (FIG. 1).

The area dividing unit 332 divides the image included in the moving image taken by the fisheye camera 200 into a plurality of areas having different distortions.

The dictionary creation unit 333 creates different recognition dictionaries for recognizing a person's specific behavior for each divided area.

The dictionary selection unit 334 selects a recognition dictionary to be used for recognizing a specific behavior of a person detected from a moving image from different recognition dictionaries created by the dictionary creation unit 333.

The action recognition unit 335 recognizes a person's specific action based on the recognition dictionary selected by the dictionary selection unit 334.

The duration measuring unit 336 measures the duration of the specific action based on the recognition result of the specific action.

(Explanation of the functional configuration of the dictionary creation unit)
Next, the functional configuration of the dictionary creation unit 333 will be described with reference to FIG. FIG. 13 is a functional block diagram showing an example of a schematic configuration of the dictionary creation unit 333 according to the present embodiment. As shown in FIG. 13, the dictionary creation unit 333 includes a feature point extraction unit 333a, a feature point classification unit 333b, a feature vector calculation unit 333c, a histogram creation unit 333d, and a recognition dictionary creation unit 333e.

Although the moving image taken by the fisheye camera 200 has distortion, the distortion is not corrected, and the dictionary creation unit 333 creates a recognition dictionary corresponding to a plurality of positions of the image included in the captured moving image. To do. That is, the dictionary creation unit 333 creates a recognition dictionary that recognizes a specific action (standard work) in a state where the distortion is large in a region where the distortion is large. Further, the dictionary creation unit 333 creates a recognition dictionary that recognizes a specific action in a state where the distortion is small in a region where the distortion is small. Therefore, when creating a recognition dictionary, the subject performs standard work at various positions in the image.

The feature point extraction unit 333a extracts feature points generated in association with a specific action (standard work) from a plurality of images included in the moving image taken by the fisheye camera 200. More specifically, the feature point extraction unit 333a cuts out T image frames from the input moving image, and with respect to the cut out T image frames, the feature points in spatiotemporal space (also referred to as spatiotemporal feature points). ) Is extracted. The feature point is that when the input moving image is divided into three-dimensional blocks of a predetermined size consisting of two axes in the spatial direction and one axis in the temporal direction, the average brightness of the image in the block is a predetermined value. It is a block that exceeds. The feature point extraction unit 333a extracts feature points from a plurality of moving images in order to generate highly accurate learning data.

The feature point extraction unit 333a detects a person from the images included in the moving image taken by the fisheye camera 200 by using a known person detection algorithm, and only for the detected human area. , The feature point extraction described above may be performed. According to this, since the area for extracting the feature points can be limited, the processing can be performed more efficiently.

The feature point classification unit 333b classifies (clusters) an M × N × T × three-dimensional vector representing the feature points extracted by the feature point extraction unit 333a by, for example, a known K-means method (K-means method). .. Assuming that the number of classes to be classified is K type, the feature point classification unit 333b classifies the feature points extracted from the moving image for learning into K types.

The feature vector calculation unit 333c averages the M × N × T × 3D vectors of the K types of feature points classified by the feature point classification unit 333b at the same type of feature points, and K average vectors Vk. Ask for. The average vector Vk calculated by the feature vector calculation unit 333c is a vector representing K types of feature points, respectively. The average vector Vk is an example of a learning vector.

The feature vectors obtained from the moving image of observing the specific behavior are distributed close to the average vector Vk obtained from the learning data of the same specific behavior. Using this characteristic, the action recognition unit 335 can perform highly accurate action recognition from a moving image having distortion taken by the fisheye camera 200 even in a state where the distortion is not corrected. That is, when the worker moves linearly, the movement of the person becomes curved in the region where the distortion of the captured moving image is large. However, since the recognition dictionary created by the dictionary creation unit 333 is learned as a curve of human movement, it is possible to recognize a specific action without correcting the distortion. Similarly, in the region where the distortion is small, the linear movement of the worker is learned as the linear movement, so that the specific action can be recognized without correcting the distortion.

The histogram creation unit 333d creates a learning histogram H (k) representing the appearance frequency of the average vector Vk. Specifically, for K types of feature points, the total number of blocks of each feature point group is calculated, and a learning histogram H (k) is created. The learning histogram H (k) shows the frequency of the feature point k group. The histogram creation unit 333d is an example of the learning histogram creation unit.

The recognition dictionary creation unit 333e creates a dictionary that recognizes a specific action in each area by the learning histogram H (k) obtained from the learning data of each area in N action recognition target areas. The recognition dictionary creation unit 333e creates a recognition dictionary by a machine learning method of SVM (Support Vector Machine). In addition, when the recognition dictionary creation unit 333e creates the recognition dictionary by the machine learning method of SVM, the positive learning data (plus learning data) including the specific action to be recognized and the specific action to be recognized are generated. A recognition dictionary may be created by preparing negative learning data (minus learning data) that is not included. That is, the recognition dictionary creation unit 333e creates a recognition dictionary that accepts positive learning data as correct data and excludes negative learning data as different data. As a result, the specific behavior and the behavior that is likely to be mistaken can be learned as negative learning data, so that the recognition rate of the specific behavior can be improved. When creating the recognition dictionary, other machine learning methods may be used in addition to the SVM machine learning method. For example, a machine learning method such as KNN (K Nearest Neighbor) or MLP (Multilayer perceptron) may be used.

(Explanation of the functional configuration of the behavior recognition unit)
Next, the functional configuration of the action recognition unit 335 will be described with reference to FIG. FIG. 14 is a functional block diagram showing an example of a schematic configuration of the action recognition unit 335 according to the present embodiment. As shown in FIG. 14, the action recognition unit 335 includes a feature point extraction unit 335a, a feature vector calculation unit 335b, a histogram creation unit 335c, and an action recognition unit 335d. The feature point extraction unit 335a has the same function as the feature point extraction unit 333a included in the dictionary creation unit 333.

The feature vector calculation unit 335b obtains spatiotemporal edge information (differential vector) at the feature points extracted by the feature point extraction unit 335a. The spatiotemporal edge information will be described in detail later.

The histogram creation unit 335c creates a specific behavior histogram T (k) representing the appearance frequency of the spatiotemporal edge information.

The action recognition unit 335d compares the specific action histogram T (k) created by the histogram creation unit 335c based on the differential vector obtained from the moving image with the learning histogram H (k) stored in the recognition dictionary. Recognize specific behavior by The distribution of feature points to be recognized is close to the distribution of feature points in the recognition dictionary. That is, since the specific behavior histogram T (k) obtained from the image of the recognition target performing the specific behavior and the learning histogram H (k) of the same specific behavior are similar, the distortion of the image is not corrected. , It is possible to recognize specific behavior.

(Explanation of images observed by the behavior recognition system)
Next, an example of an image observed by the behavior recognition system 100 will be described with reference to FIGS. 15 and 16. FIG. 15 is a diagram showing an example of a moving image (image string) input to the moving image input unit 331. Each image (frame) shown in FIG. 15 is an image taken by the fisheye camera 200 and is an image without distortion correction. The horizontal axis x and the vertical axis y of the captured image are spatial coordinates. The time axis of the image frames F1 and F2 is indicated by t. That is, the input image becomes spatiotemporal data in coordinates (x, y, t). The pixel value at one coordinate in space-time is a function of space coordinates (x, y) and time t. When a person moves when recognizing a specific behavior in the workplace described above, a change point occurs in the spatiotemporal data shown in FIG. The action recognition system 100 recognizes a specific action by finding this change point, that is, a feature point in space-time.

Next, a method for extracting feature points in the present embodiment will be described. As shown in FIG. 16, the spatiotemporal image data is divided into blocks. The large cube in FIG. 16 shows spatiotemporal image data. The horizontal axis x and the vertical axis y represent spatial coordinates. Each unit is a pixel. The time axis is indicated by t. For example, a moving image is input at a video rate of 30 frames / second, and a time-series image is input. By converting at this video rate, the actual time when the image was taken can be obtained. The spatio-temporal image data of FIG. 16 is divided into blocks of size (N, N, T). The size of one block is horizontal M pixel, vertical N pixel, and T frame. One square in FIG. 16 indicates one block. When a person performs a certain action, the feature amount of the block becomes large in the block in which the movement occurs in the spatiotemporal data. That is, a large amount of change occurs in space-time.

Next, a method of extracting a block having a large amount of change as a feature point will be described. In order to extract feature points from spatiotemporal image data, first, a smoothing process is performed to remove noise in the spatial direction, that is, in the (x, y) direction. The smoothing process is performed by the equation (1).

Here, I (x, y, t) is a pixel value of the (x, y) coordinate in the frame at time t. Further, g (x, y) is a kernel for smoothing processing. In addition, * is an operator indicating the convolution process. The smoothing process may be simply an averaging process of pixel values, or an existing Gaussian smoothing filter process may be performed.

Next, filtering processing is performed on the time axis. Here, the Gabor filtering process represented by the equation (2) is performed. The Gabor filter is a directional filter, and has the function of emphasizing parallel and evenly spaced lines existing in the region where the filter acts and removing noise existing between the lines. The _{g ev} and _{g od} in equation (2), respectively, is a kernel Gabor filter shown as Equation (3) Equation (4). In addition, * is an operator indicating the convolution process. In addition, τ and ω are kernel parameters in the Gabor filter.

After performing the filtering process shown in the above equation (2) on all the pixels of the spatiotemporal image shown in FIG. 15, the average value of R (x, y, t) in the divided block shown in FIG. 16 is obtained. In equation (5), the average value of the blocks of spatiotemporal coordinates (x, y, t) is obtained.

As shown in the equation (6), when the average value M (x, y, t) in the block is larger than the predetermined threshold value Thr_M, this block is used as a feature point.

(Explanation of how to describe feature points)
Next, a method of describing the feature points will be described with reference to FIGS. 17A and 17B. FIG. 17A is a first diagram showing an example of feature points extracted from moving images. FIG. 17B is a second diagram showing an example of feature points extracted from moving images. That is, FIG. 17A is an image k1 showing an example of feature points at time t, which is extracted from the image of the person performing the shelving shown in FIG. As shown in FIG. 17A, feature points are extracted in a moving portion. FIG. 17B is an image k2 showing an example of the feature points similarly extracted at time t + Δt.

After the feature points shown in FIG. 17A are extracted, the spatiotemporal edge information of the pixels in the block to which the feature points belong is obtained. That is, the edge information E (x, y, t) (differential vector) of the pixel is obtained by performing the differential operation shown in the equation (7).

Since one block contains M × N × T pixels, a differential value of M × N × T × 3 can be obtained by the equation (7). That is, the block including the feature points can be described by a vector of M × N × T × 3 differential values. That is, the feature points can be described by a vector of M × N × T × 3 dimensions. Then, the edge information E (x, y, t) is obtained in the same manner for the image k2 of FIG. 17B.

When creating a recognition dictionary that recognizes a specific action by learning, the dictionary creation unit 333 creates a recognition dictionary corresponding to a plurality of different positions in the image having different distortions.

Here, among the plurality of feature points extracted from the image k1, the feature points that are close to each other are considered to be the feature points that occur with the action of one person. That is, assuming that the area m1 shown in FIG. 17A is a human existence area, the recognition dictionary created by the dictionary creation unit 333 is stored in association with the representative point (for example, the position of the center of gravity) of the area m1.

The same applies to the region m2 formed by the feature points extracted from the image k2 of FIG. 17B. In this way, the dictionary creation unit 333 creates a recognition dictionary using an N-frame image including a specific action as one learning data.

(Explanation of the duration of a specific action)
Next, the duration of the specific action will be described with reference to FIG. FIG. 18 is a diagram illustrating the measurement of the duration of a specific action.

The duration measuring unit 336 measures the duration of the specific action based on the recognition result of the specific action. FIG. 18 shows an example in which it is recognized that the “walking” action is performed between the time t0 and the time t1, and the “shelfing” action is recognized between the time t2 and the time t3.

In FIG. 18, the duration measuring unit 336 determines that the duration of the “walking” behavior is (t1-t0) and the duration of the “shelfing” behavior is (t3-t2). When the number of specific actions to be recognized increases, the recognition process of each specific action is performed in the same manner, and the duration of the action is measured.

(Explanation of the flow of recognition dictionary creation process)
Next, the flow of the recognition dictionary creation process performed by the dictionary creation unit 333 will be described with reference to FIG. Note that FIG. 19 is a flowchart showing an example of the flow of creating a recognition dictionary.

The moving image input unit 331 inputs a moving image taken by the fisheye camera 200 (step S11).

The feature point extraction unit 333a extracts feature points from the input moving image (step S12).

The feature point classification unit 333b clusters the extracted feature points (step S13).

The feature vector calculation unit 333c calculates the average vector Vk (step S14).

The histogram creation unit 333d creates a learning histogram H (k) (step S15).

The recognition dictionary creation unit 333e creates a recognition dictionary (step S16). After that, the dictionary creation unit 333 ends the process of FIG. As described above, since it is necessary to create a plurality of recognition dictionaries at different positions (positions with different distortions) of the image, the process of FIG. 19 is repeatedly executed.

(Explanation of the flow of behavior recognition processing)
Next, the flow of the action recognition process performed by the action recognition processing unit 321 will be described with reference to FIG. Note that FIG. 20 is a flowchart showing an example of the flow of the recognition process of the specific action.

The moving image input unit 331 inputs a moving image taken by the fisheye camera 200 (step S21).

The feature point extraction unit 335a extracts feature points from the input moving image (step S22).

The feature vector calculation unit 335b calculates the average vector Vk (step S23).

The histogram creation unit 335c creates a specific behavior histogram T (k) (step S24).

The dictionary selection unit 334 selects a recognition dictionary (step S25). Specifically, the dictionary selection unit 334 selects a recognition dictionary created in the vicinity of the position of the feature point extracted by the feature point extraction unit 335a. That is, the dictionary selection unit 334 selects the recognition dictionary created at a position where the magnitude of distortion is close.

The action recognition unit 335 recognizes a specific action (step S26). The flow of the recognition process for the specific action will be described later (FIG. 21).

The duration measuring unit 336 measures the duration of the specific action (step S27).

Further, the duration measurement unit 336 outputs the type of the specific action and the measurement result of the specific action (step S28). After that, the action recognition unit 335 ends the process of FIG. 20.

(Explanation of the flow of recognition processing for specific actions)
Next, the flow of the recognition process of the specific action performed by the action recognition unit 335 will be described with reference to FIG. Note that FIG. 21 is a flowchart showing an example of a processing flow for recognizing a plurality of specific actions. In particular, FIG. 21 shows a flow of processing for recognizing that the “shelfing” action is performed after the “walking” action is performed among the specific actions.

The action recognition unit 335 recognizes the "walking" action (step S31).

Next, the action recognition unit 335 determines whether or not the "walking" action has been recognized (step S32). When it is determined that the "walking" action is recognized (step S32: Yes), the process proceeds to step S31. On the other hand, if it is not determined that the "walking" action is recognized (step S32: No), the process proceeds to step S33.

If No is determined in step S32, the action recognition unit 335 recognizes the "shelfing" action (step S33).

Next, the action recognition unit 335 determines whether or not the “shelfing” action has been recognized (step S34). When it is determined that the "shelfing" action has been recognized (step S34: Yes), the process of FIG. 21 is terminated, and the process proceeds to step S27 of FIG. On the other hand, if it is not determined that the "shelfing" action is recognized (step S34: No), the process returns to step S31.

The flowchart shown in FIG. 21 is an example, and the action recognition unit 335 performs processing according to the type and order of the specific actions to be recognized.

As described above, according to the behavior recognition device 300 of the first embodiment, the moving image input unit 331 inputs the moving image taken by the fisheye camera 200 (shooting means), and the area dividing unit 332 receives the moving image. The image included in the moving image is divided into a plurality of areas having different distortions. The dictionary creation unit 333 creates a recognition dictionary for recognizing a specific action of a person (subject) for each divided area. The dictionary selection unit 334 selects a recognition dictionary to be used for recognizing a specific behavior of a person detected from a moving image from a plurality of recognition dictionaries created by the dictionary creation unit 333. Then, the action recognition unit 335 recognizes a person's specific action based on the recognition dictionary selected by the dictionary selection unit 334. Therefore, since the recognition dictionary is created for each image area, it is possible to recognize a person's specific behavior (standard work) without correcting the distortion of the captured image.

Further, according to the behavior recognition device 300 of the first embodiment, the dictionary selection unit 334 corresponds to the position of the person (subject) detected from the image included in the moving image taken by the fisheye camera 200 (shooting means). Select a recognition dictionary. Therefore, it is possible to recognize a person's specific behavior (standard work) without correcting the distortion of the captured image.

Further, according to the action recognition device 300 of the present embodiment, the duration measurement unit 336 measures the duration of the specific action based on the recognition result of the specific action. Therefore, the duration of a specific action (standard work) can be easily and accurately measured.

Further, according to the behavior recognition device 300 of the first embodiment, the feature point extraction unit 333a extracts feature points from a plurality of images included in the moving image taken by the fisheye camera 200 (shooting means). .. The feature point classification unit 333b classifies the extracted feature points into K types. The feature vector calculation unit 333c obtains each K average vector Vk (learning vector) for each of the K types of feature point groups classified. Therefore, it is possible to easily learn the specific behavior of a person (subject).

Further, according to the action recognition device 300 of the first embodiment, the dictionary creation unit 333 performs the video (plus learning data) of the person performing the specific action and the specific action input by the video input unit 331. A learning histogram H (k) is created from a video of a person who has not performed (minus learning data) and an average vector Vk (learning vector) based on the feature amount of each feature point of each data, and plus learning data. A recognition dictionary is created based on the learning histogram H (k) generated from the above and the learning histogram H (k) generated from the minus learning data. Therefore, the accuracy of the recognition dictionary can be improved.

Further, according to the behavior recognition device 300 of the first embodiment, the feature point extraction unit 335a extracts feature points from a plurality of images included in the moving image taken by the fisheye camera 200 (shooting means). .. The feature vector calculation unit 335b calculates a feature vector indicating the size and direction of the spatiotemporal edge at the extracted feature points. The histogram creation unit 335c creates a specific behavior histogram T (k) based on the feature vector of the extracted feature points. Then, the behavior recognition unit 335d recognizes a person's specific behavior based on the specific behavior histogram T (k) and the learning histogram H (k) of the recognition dictionary. Therefore, the specific behavior of a person (subject) can be easily and accurately recognized.

Further, according to the behavior recognition device 300 of the first embodiment, the feature vector calculation unit 335b divides a plurality of input images into M × N × T size flocs, and differentiates each block. , M × N × T × 3D edge information E (x, y, t) (differential vector) is calculated. Then, the feature vector calculation unit 335b compares the calculated edge information E (x, y, t) with the K-type average vector Vk (learning vector) learned in advance, and based on the result of the comparison, the edge. The information E (x, y, t) is classified into the same kind of feature points as the closest average vector Vk. The histogram creation unit 335c creates a specific behavior histogram T (k) based on the classification result, and therefore easily creates a specific behavior histogram T (k) used for recognizing the specific behavior from the captured moving image. be able to.

Further, according to the action recognition device 300 of the first embodiment, the dictionary creation unit 333 and the action recognition unit 335 perform filtering processing on the time axis for the input moving image. Then, the feature point extraction units 333a and 335a extract the block as a feature point when the average value in the block of M × N × T is larger than a predetermined threshold value as a result of performing the filtering process. Therefore, the feature points can be easily extracted.

Further, according to the behavior recognition device 300 of the present embodiment, the filtering process is performed by the Gabor filtering process shown in the equations (2), (3), and (4). Therefore, by removing the noise of the captured moving image, it is possible to obtain an image that makes it easy to recognize a specific action.

Further, according to the behavior recognition device 300 of the first embodiment, the feature point extraction units 333a and 335a perform smoothing processing on each image before performing filtering processing on the time axis. Therefore, since the noise generated in the time axis direction is removed, the specific behavior of the person (subject) can be recognized with higher accuracy.

Further, according to the action recognition device 300 of the first embodiment, the action recognition unit 335 recognizes the specific actions in a predetermined order when recognizing the specific actions of a person, and when the specific actions are recognized, the specific actions are recognized. If the recognition result is output and the specific action is not recognized, the next specific action is recognized. Therefore, even when a plurality of specific actions occur consecutively, it can be reliably recognized.

Further, according to the behavior recognition device 300 of the first embodiment, the wide-angle lens is a fisheye lens. Therefore, it is possible to observe a wider range with one camera.

(Second Embodiment)
Next, the behavior recognition device, the behavior recognition method, and the second embodiment of the program will be described in detail with reference to the accompanying drawings.

(Explanation of hardware configuration of behavior recognition device)
FIG. 22 is a hardware block diagram showing an example of the hardware configuration of the behavior recognition system 100a according to the present embodiment. As shown in FIG. 22, the behavior recognition system 100a includes fisheye cameras 200 and 201 and a behavior recognition device 300a.

The behavior recognition system 100a has the same function as the behavior recognition system 100 described in the first embodiment, and recognizes a specific behavior of a person (subject) photographed by the fisheye cameras 200 and 201. The difference from the behavior recognition system 100 is that moving images taken by two fisheye cameras 200 and 201 can be input.

The behavior recognition device 300a includes a behavior recognition processing unit 321a and an interface unit 322a for connecting the behavior recognition processing unit 321a and the fisheye cameras 200 and 201.

The action recognition processing unit 321a recognizes a specific action of a person (subject). The interface unit 322a converts the moving image captured by the fisheye cameras 200 and 201 into a data format that can be recognized by the action recognition processing unit 321a, and delivers the moving image to the action recognition processing unit 321a.

Next, a typical scene in which the action recognition system 100a is used will be described with reference to FIG. FIG. 23 is a diagram showing an example of a scene in which the behavior recognition system 100a according to the second embodiment is used.

As shown in FIG. 23, the behavior recognition system 100a is installed in a work environment in a workplace such as an office or a factory. The fisheye cameras 200 and 201 capture moving images including a plurality of people H1 and H2 working in the work environment. In the present embodiment, the fisheye cameras 200 and 201 both include a fisheye lens having a diagonal angle of view of 180 °. Then, the two fisheye cameras 200 and 201 capture the same working environment from different directions. The people H1 and H2 are examples of subjects.

The behavior recognition system 100 described in the first embodiment can recognize predetermined behaviors of a plurality of people in an environment in which a plurality of workers are working, but enters the blind spot of another worker. The worker was unable to recognize the behavior because it could not be visualized. On the other hand, since the action recognition system 100a observes the work environment from different directions, the blind spots are reduced, and the predetermined actions of a plurality of people can be recognized more reliably. Further, since the behavior recognition system 100a can capture one worker with the two fisheye cameras 200 and 201, the behavior recognition can be performed using the images captured with smaller distortion.

Since the hardware configuration of the behavior recognition system 100a is the same as the hardware configuration of the behavior recognition system 100 except that the number of fisheye cameras increases, the description thereof will be omitted.

(Explanation of the functional configuration of the behavior recognition processing unit)
Next, the functional configuration of the action recognition processing unit 321a will be described with reference to FIG. 24. FIG. 24 is a functional block diagram showing an example of the functional configuration of the action recognition processing unit 321a according to the second embodiment. As shown in FIG. 24, the action recognition processing unit 321a includes the same person determination unit 337 in addition to the functional configuration (FIG. 12) of the behavior recognition processing unit 321 described in the first embodiment. Further, the action recognition processing unit 321a includes a dictionary selection unit 334a whose function has been changed, instead of the dictionary selection unit 334.

The dictionary selection unit 334a selects a recognition dictionary according to the image used for behavior recognition when the fisheye cameras 200 and 201 each photograph the same person. Specifically, the dictionary selection unit 334a compares the positions of the same person in the images taken by the fisheye cameras 200 and 201, and recognizes the specific behavior of the person in the position closer to the center of the image. The recognition dictionary of, that is, the recognition dictionary created at the position where the distortion is smaller is selected. Whether or not the same person appears in the images taken by the fisheye cameras 200 and 201 is determined by the same person determination unit 337, which will be described later.

The same person determination unit 337 determines whether or not the same person appears in the images captured by the fisheye cameras 200 and 201, respectively. Specifically, the same person determination unit 337 compares the feature vectors based on the feature points extracted from the images taken by the fisheye cameras 200 and 201, respectively, so that the type of the feature vector and the direction of the feature vector are similar. If so, it is determined that the same person is shown.

(Explanation of the flow of behavior recognition processing)
Next, the flow of the action recognition process performed by the action recognition processing unit 321a will be described with reference to FIG. 25. Note that FIG. 25 is a flowchart showing an example of the flow of the recognition process of the specific action in the second embodiment.

Steps S41 to S44 are the same processes as steps S21 to S24 (FIG. 20) described in the first embodiment.

Next, the same person determination unit 337 identifies an area representing the same person from the images taken by the fisheye cameras 200 and 201 (step S45).

Subsequently, the dictionary selection unit 334a identifies a fisheye camera that has captured the area closest to the center of the image among the areas representing the same person identified in step S45, and recognizes the area corresponding to the position. Select a dictionary (step S46). If the area representing the same person cannot be specified in step S45, the dictionary selection unit 334a selects a recognition dictionary corresponding to each of the detected areas.

The subsequent processes from step S47 to step S49 are the same as steps S26 to S29 (FIG. 20) described in the first embodiment.

As described above, in the behavior recognition device 300a of the second embodiment, the plurality of fisheye cameras 200 and 201 (photographing means) photograph the same area from different directions. Therefore, the blind spot in the observation range is reduced. Moreover, since the same person (subject) can be photographed from different directions, the recognition accuracy of behavior recognition can be improved.

Further, according to the behavior recognition device 300a of the second embodiment, the dictionary selection unit 334a detects the same person (subject) from the images included in the moving images taken by the plurality of fisheye cameras 200 and 201 (shooting means). ), The recognition dictionary with the smallest distortion is selected from the recognition dictionaries according to the position. Therefore, the recognition accuracy of the specific action can be improved.

Further, according to the behavior recognition device 300a of the second embodiment, the dictionary selection unit 334a detects the position of the same person from the images included in the moving images taken by the plurality of fisheye cameras 200 and 201 (shooting means). The recognition dictionary corresponding to the position near the center of the image is selected from the recognition dictionaries according to. Therefore, since the recognition dictionary created at the position where the distortion is small is selected, the recognition accuracy of the specific action can be improved.

Although the embodiment of the present invention has been described above, the above-described embodiment is presented as an example and is not intended to limit the scope of the present invention. This new embodiment can be implemented in a variety of other forms. In addition, various omissions, replacements, and changes can be made without departing from the gist of the invention. Further, this embodiment is included in the scope and gist of the invention, and is also included in the scope of the invention described in the claims and the equivalent scope thereof.

200,201 Fisheye camera (shooting means)
300,300a Behavior recognition device 321, 321a Behavior recognition processing unit 331 Video input unit 332 Area division unit 333 Dictionary creation unit 334 Dictionary selection unit 335 Behavior recognition unit 336 Duration measurement unit 333a, 335a Feature point extraction unit 333b Feature point classification unit 333c, 335b Feature vector calculation unit 333d Histogram creation unit (learning histogram creation unit)
335c Histogram creation unit 333e Recognition dictionary creation unit 335d Behavior recognition unit H1, H2 people (subjects)
H (k) Learning histogram T (k) Specific behavior histogram Vk average vector (learning vector)

Japanese Unexamined Patent Publication No. 2011-100175

Claims (17)

It is an action recognition device that recognizes the specific behavior of the subject in the video from the recorded video.
A video input unit for inputting the video shot by a shooting means equipped with a wide-angle lens,
A region dividing portion that divides the image included in the moving image into a plurality of regions having different distortions,
A dictionary creation unit that creates a recognition dictionary for recognizing a specific action of the subject for each of the divided areas.
A dictionary selection unit that selects a recognition dictionary to be used for recognizing a specific action of a subject detected from the moving image from a plurality of recognition dictionaries created by the dictionary creation unit.
An action recognition unit that recognizes a specific action of the subject based on the recognition dictionary selected by the dictionary selection unit.
Behavior recognition device equipped with. The dictionary selection unit selects the recognition dictionary according to the position of the subject detected from the image included in the moving image.
The behavior recognition device according to claim 1. Equipped with multiple shooting means to shoot the same area from different directions,
The behavior recognition device according to claim 1. The dictionary selection unit selects the recognition dictionary having the smallest distortion among the recognition dictionaries according to the positions of the same subjects detected from the images included in the moving images taken by the plurality of shooting means.
The behavior recognition device according to claim 3. The dictionary selection unit selects a recognition dictionary corresponding to a position close to the center of the image among the recognition dictionaries corresponding to the positions of the same subjects detected from images included in the moving images taken by the plurality of shooting means. ,
The behavior recognition device according to claim 3 or 4. A duration measuring unit for measuring the duration of the specific action based on the recognition result of the specific action is further provided.
The behavior recognition device according to any one of claims 1 to 4. The dictionary creation unit
A feature point extraction unit that extracts feature points from a plurality of images included in a moving image shot by the shooting means, and a feature point extraction unit.
A feature point classification unit that classifies the extracted feature points into K types,
A feature vector calculation unit that obtains K learning vectors for each of the K types of classified feature point groups,
The behavior recognition device according to any one of claims 1 to 6, further comprising a learning histogram creating unit that creates a learning histogram representing the frequency of appearance of the learning vector. The learning histogram creation unit
Each data is input from the moving image of the subject performing the specific action, which constitutes the positive learning data, and the moving image of the subject forming the negative learning data, which constitutes the positive learning data, and which is input by the moving image input unit. Using the learning vector based on the feature amount of the feature points of, each learning histogram is created.
The dictionary creation unit
The recognition dictionary is created based on the learning histogram generated from the positive learning data and the learning histogram generated from the negative learning data.
The behavior recognition device according to claim 7. The behavior recognition unit
A feature point extraction unit that extracts feature points from a plurality of images included in a moving image shot by the shooting means, and a feature point extraction unit.
A feature vector calculation unit that calculates a feature vector indicating the size and direction of the spatiotemporal edge at the extracted feature points, and a feature vector calculation unit.
A histogram creating unit for creating a histogram showing the appearance frequency of the feature vector at the feature point is provided.
Recognizing the specific behavior of the subject based on the histogram and the recognition dictionary.
The behavior recognition device according to claim 7 or 8. The feature vector calculation unit
By dividing a plurality of input images into M × N × T size flocs and differentiating each block, an M × N × T × 3D differential vector is calculated.
The calculated differential vector is compared with the pre-learned learning vector, and the differential vector is classified into the closest feature points of the same type as the learning vector based on the result of the comparison.
The histogram creation unit
Create the histogram based on the result of the classification.
The behavior recognition device according to claim 9. The dictionary creation unit and the action recognition unit
The input video is filtered on the time axis to perform filtering processing.
The feature point extraction unit
As a result of performing the filtering process, when the average value in the block of M × N × T is larger than a predetermined threshold value, the block is extracted as a feature point.
The behavior recognition device according to any one of claims 7 to 10. When g _ev and _good are the kernels of the Gabor filter represented by the following equations (1) and (2), * is a convolution process, and τ and ω are parameters of the kernel, the filtering process is performed. This is a Gabor filtering process using the following equation (3).

The behavior recognition device according to claim 11. The feature point extraction unit
Before performing the filtering process, a smoothing process is performed on each image.
The behavior recognition device according to claim 11 or 12. The behavior recognition unit
When recognizing the specific action of the subject, the specific action is recognized in a predetermined order, and when the specific action is recognized, the recognition result is output.
If a specific action is not recognized, recognize the next specific action,
The behavior recognition device according to any one of claims 1 to 13. The wide-angle lens is a fisheye lens.
The behavior recognition device according to any one of claims 1 to 14. When recognizing the specific behavior of the subject in the video from the recorded video,
A video input step for inputting the video shot by a shooting means equipped with a wide-angle lens, and
A region division step of dividing the image included in the moving image into a plurality of regions having different distortions,
A dictionary creation step of creating a different recognition dictionary for recognizing a specific action of the subject for each of the divided areas,
A dictionary selection step for selecting a recognition dictionary to be used for recognizing a specific action of a subject detected from the moving image from a plurality of recognition dictionaries created in the dictionary creation step.
An action recognition step that recognizes a specific action of the subject based on the recognition dictionary selected in the dictionary selection step,
Behavior recognition method to execute. A computer that controls an action recognition device that recognizes the specific behavior of the subject in the video from the recorded video.
A video input unit for inputting the video shot by a shooting means equipped with a wide-angle lens,
A region dividing portion that divides the image included in the moving image into a plurality of regions having different distortions,
A dictionary creation unit that creates a recognition dictionary for recognizing a specific action of the subject for each of the divided areas.
A dictionary selection unit that selects a recognition dictionary to be used for recognizing a specific action of a subject detected from the moving image from a plurality of recognition dictionaries created by the dictionary creation unit.
An action recognition unit that recognizes a specific action of the subject based on the recognition dictionary selected by the dictionary selection unit.
A program that works.

Images