JP7338182B2 - Action recognition device, action recognition method and program - Google Patents

Description

The present invention relates to an action recognition device, an action recognition method, and a program.

In a workplace such as an office or a factory, it is an important issue to improve the production efficiency of the workplace by visualizing the behavior of workers and analyzing the working hours and the like. Therefore, it is an effective means of recognizing and analyzing specific standard work (hereinafter referred to as "standard work") by workers by capturing videos of workplaces with cameras and analyzing the obtained videos.

However, visual analysis of workplace videos taken with a camera, extraction of standard work actions performed in a fixed procedure, measurement of the time for each action, and visualization of these actions requires a huge amount of analysis time and labor. is necessary. Conventionally, in order to automatically recognize human behavior, a method has been proposed in which a person is recognized from a captured video, the movement trajectory of the person is obtained from the weight of the recognized person, and specific behavior is recognized from the movement trajectory. there is

When recognizing a worker's behavior, it is desirable to photograph a field of view as wide as possible with one camera in order to improve processing efficiency. Therefore, it is desirable to use a camera equipped with a wide-angle lens with a wide angle of view. However, images captured by cameras with wide-angle lenses are distorted. When the image is distorted, the shape of the person in the image is distorted, and the accuracy of human recognition deteriorates. In order to recognize the standard work, it is necessary to trace the movement of the same person over time. In order to prevent such deterioration in accuracy, it is desirable to recognize the standard work after correcting the distortion of the image. However, since it takes time and effort to correct the distortion, there is a problem that it is difficult to recognize the standard work with high precision and high speed.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an action recognition device, an action recognition method, and a program capable of recognizing a worker's standard work with high precision and high speed. .

In order to solve the above-described problems and achieve the object, the action recognition device of the present invention is an action recognition device that recognizes a specific action of a subject captured in a captured moving image to be recognized, Input a video of a subject performing a specific action at multiple positions with different distortions within the observation range of the imaging means, taken by multiple imaging means equipped with a wide-angle lens and photographing the same area from different directions . a second moving image input unit for inputting a moving image to be recognized by the plurality of photographing means; and input by the first moving image input unit and the second moving image input unit an area dividing unit that divides an image included in a moving image into a plurality of areas each having a different distortion; a dictionary creating unit that creates a recognition dictionary for recognizing actions; The dictionary creation unit selects a recognition dictionary with the least distortion from among the plurality of recognition dictionaries created for each of the photographing means and for each of the regions , and based on the recognition dictionary selected by the dictionary selection unit. and an action recognition unit that recognizes the specific action of the subject.

ADVANTAGE OF THE INVENTION According to this invention, a worker's standard work can be recognized highly accurately and at high speed.

FIG. 1 is a hardware block diagram showing an example of the hardware configuration of an action recognition system according to the first embodiment. FIG. 2 is a diagram showing an example of a scene in which the action recognition system according to the first embodiment is used. FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the fisheye camera. FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the action recognition device. FIG. 5 is a diagram for explaining distortion of an image observed with a fisheye lens. FIG. 6 is a diagram for explaining the difference in distortion depending on the position of an image observed with a fisheye lens. FIG. 7 is a diagram illustrating an example of an image observed by the action recognition system of the first embodiment; FIG. 8 is a diagram explaining a "walking" action among the specific actions recognized by the action recognition system. FIG. 9 is an enlarged view of a person in the image of FIG. FIG. 10 is a diagram for explaining a "putting" action of placing a product on a shelf among the specific actions recognized by the action recognition system. FIG. 11 is a diagram showing an example of an enlarged view of a person who is taking the putaway action. 12 is a functional block diagram illustrating an example of a functional configuration of an action recognition processing unit; FIG. FIG. 13 is a functional block diagram showing an example of the functional configuration of a dictionary creation unit. 14 is a functional block diagram illustrating an example of a functional configuration of an action recognition unit; FIG. FIG. 15 is a diagram showing an example of a moving image input to the moving image input unit. FIG. 16 is a diagram for explaining the feature point detection method. FIG. 17A is a first diagram showing an example of extracted feature points. FIG. 17B is a second diagram showing an example of extracted feature points. FIG. 18 is a diagram illustrating measurement of the duration of specific actions. FIG. 19 is a flow chart showing an example of the flow of creating a recognition dictionary. FIG. 20 is a flowchart showing an example of the flow of specific action recognition processing. FIG. 21 is a flow chart showing an example of the flow of processing for recognizing a plurality of specific actions. FIG. 22 is a hardware block diagram showing an example of the hardware configuration of the action recognition system according to the second embodiment. FIG. 23 is a diagram illustrating an example of a scene in which the action recognition system according to the second embodiment is used; 24 is a functional block diagram illustrating an example of a functional configuration of an action recognition processing unit according to the second embodiment; FIG. FIG. 25 is a flowchart showing an example of the flow of specific action recognition processing according to the second embodiment.

(First embodiment)
A first embodiment of a behavior recognition device, a behavior recognition method, and a program will be described in detail below with reference to the accompanying drawings.

(Description of hardware configuration of action recognition device)
FIG. 1 is a hardware block diagram showing an example of the hardware configuration of an action recognition system 100 according to this embodiment. As shown in FIG. 1, the action recognition system 100 includes a fisheye camera 200 and an action recognition device 300. FIG.

The action recognition system 100 recognizes a specific action of a subject photographed by the fisheye camera 200. FIG. A specific action is, for example, standard work such as "walking" or "putting things in storage" that is repeatedly performed in the working environment of the workplace.

The fish-eye camera 200 is a video camera with a fish-eye lens capable of observing a full 360° range. Note that the provision of the fish-eye lens is merely an example, and the fish-eye camera 200 may be provided with a wide-angle lens. Note that the fisheye camera 200 is an example of a photographing unit.

The action recognition device 300 analyzes a moving image captured by the fisheye camera 200 to recognize a specific action of a person (subject) captured in the moving image. In order to recognize the specific behavior of the subject, a certain number of frames of images (continuous images, videos) are required. As the number of frames increases, the processing load for correcting distortion of the fisheye camera 200 increases. This embodiment is characterized in that a moving image is analyzed without correcting distortion.

The action recognition device 300 includes an action recognition processing unit 321 and an interface unit 322 that connects the action recognition processing unit 321 and the fisheye camera 200 .

The action recognition processing unit 321 recognizes a specific action of a person (subject). The interface unit 322 converts the moving image captured by the fisheye camera 200 into a data format recognizable by the action recognition processing unit 321 and transfers the converted data to the action recognition processing unit 321 .

Next, a typical scene in which the action recognition system 100 is used will be described with reference to FIG. FIG. 2 is a diagram showing an example of a scene in which the action recognition system 100 according to the first embodiment is used.

As shown in FIG. 2, the action recognition system 100 is installed in a work environment such as an office or a factory. Then, the fisheye camera 200 shoots a moving image including a plurality of people H1 and H2 working in the work environment. Since it is efficient to photograph the working environment with one camera, it is desirable that the fisheye camera 200 has a wide-angle lens with a wide angle of view. In this embodiment, the fisheye camera 200 is assumed to have a fisheye lens having a diagonal angle of view of 180°. Note that the people H1 and H2 are examples of subjects.

(Description of the hardware configuration of the fisheye camera)
First, the hardware configuration of the fisheye camera 200 will be described.

FIG. 3 is a hardware block diagram showing an example of the hardware configuration of the fisheye camera 200. As shown in FIG. As shown in FIG. 3, the fisheye camera 200 includes a fisheye lens 217 having a diagonal angle of view of 180 degrees or more and a CCD (Charge Coupled Device) 203 . It should be noted that the fisheye camera 200a is an example of a photographing means. The fisheye camera 200 causes subject light to enter the CCD 203 through the fisheye lens 217 . The fisheye camera 200 also has a mechanical shutter 202 between the fisheye lens 217 and the CCD 203 . A mechanical shutter 202 blocks incident light to the CCD 203 . Opening and closing of the mechanical shutter 202 is controlled by a motor driver 206 . The lens position of the fisheye lens 217 is also controlled by the motor driver 206 to realize an autofocus function.

The CCD 203 converts the optical image formed on the imaging surface into an electrical signal and outputs analog image data. Noise components are removed from the image data output from the CCD 203 by a CDS (Correlated Double Sampling) circuit 204, and converted into digital image data (hereinafter simply referred to as image data) by an A/D converter 205. After that, it is output to the image processing circuit 208 .

An image processing circuit 208 uses an SDRAM (Synchronous DRAM) 212 that temporarily stores image data to perform various image processing such as YCrCb conversion processing, white balance control processing, contrast correction processing, edge enhancement processing, and color conversion processing. . The white balance processing is image processing for adjusting the color density of image data, and the contrast correction processing is image processing for adjusting the contrast of image data. Edge enhancement processing is image processing that adjusts the sharpness of image data, and color conversion processing is image processing that adjusts the hue of image data. Further, the image processing circuit 208 displays image data that has undergone signal processing and image processing on the LCD 216 (liquid crystal display).

Image data subjected to signal processing and image processing in the image processing circuit 208 is recorded in the memory card 214 via the compression/decompression circuit 213 . The compression/decompression circuit 213 compresses the image data output from the image processing circuit 208 and outputs the image data to the memory card 214 according to an instruction obtained from the operation unit 215, and decompresses the image data read from the memory card 214 to generate an image. Output to the processing circuit 208 .

The fisheye camera 200a includes a CPU (Central Processing Unit) 209 that performs various arithmetic processing according to a program. The CPU 209 includes a ROM (Read Only Memory) 211 which is a read-only memory storing programs and the like, and a RAM (Random Access Memory) which is a readable and writable memory having a work area used in various processing processes and various data storage areas. Memory) 210 and are interconnected by bus lines.

CCD 203, CDS circuit 204 and A/D converter 205 are controlled in timing by CPU 209 via timing signal generator 207 which generates timing signals. Furthermore, the image processing circuit 208 , compression/decompression circuit 213 and memory card 214 are also controlled by the CPU 209 .

The output of the fisheye camera 200 is input to the interface section 322, which is the signal processing board of the action recognition device 300 shown in FIG.

(Description of hardware configuration of action recognition device)
Next, the hardware configuration of the action recognition device 300 will be described.

FIG. 4 is a hardware block diagram showing an example of the hardware configuration of the action recognition device 300. As shown in FIG. As shown in FIG. 4, the action recognition device 300 includes a CPU (Central Processing Unit) 301 that controls the overall operation of the action recognition device 300, a ROM (Read Only Memory) 302 that stores a program used to drive the CPU 301, a CPU 301 has a RAM (random access memory) 303 used as a work area for It also has an HD (Hard Disk) 304 that stores various data such as programs, and an HDD (Hard Disk Drive) 305 that controls reading and writing of various data to and from the HD 304 under the control of the CPU 301 .

Moreover, the action recognition apparatus 300 has media I/F307, the display 308, and network I/F309. A media I/F 307 controls reading or writing (storage) of data to a media 306 such as a flash memory. A display 308 displays various information such as a cursor, menus, windows, characters, or images. A network I/F 309 performs data communication using a communication network.

The action recognition device 300 also has a keyboard 311 , a mouse 312 , a CD-ROM (Compact Disk Read Only Memory) drive 314 and a bus line 310 . A keyboard 311 has a plurality of keys for inputting characters, numerical values, various instructions, and the like. A mouse 312 selects and executes various instructions, selects an object to be processed, moves a cursor, and the like. A CD-ROM drive 314 controls reading or writing of various data to a CD-ROM 313, which is an example of a removable recording medium. A bus line 310 is an address bus, a data bus, or the like for electrically connecting the components described above.

The illustrated hardware of the action recognition device 300 does not need to be housed in one housing or integrated into a device. In addition, in order to support cloud computing, the physical configuration of the action recognition device 300 of this embodiment does not have to be fixed, and hardware resources can be dynamically connected/disconnected according to the load. may consist of

The program is distributed while being stored in a storage medium such as the medium 306 or the CD-ROM 313 in an executable format or a compressed format, or distributed from a server that distributes the program.

The program executed by the action recognition device 300 of this embodiment has a module configuration including each function shown below. The CPU 301 of the action recognition device 300 reads a program from a storage medium such as the ROM 302 or the HD 304 and executes it, thereby loading each module onto the RAM 303 and exerting each function.

(Description of distortion that occurs with a fisheye camera)
Next, the distortion that occurs in the image captured by the fisheye camera 200 will be described with reference to FIGS. 5 and 6. FIG. FIG. 5 is a diagram for explaining distortion of an image observed with a fisheye lens. FIG. 6 is a diagram for explaining the difference in distortion depending on the position of an image observed with a fisheye lens.

The image I shown in FIG. 5 is an example of an image I observed when a target on which squares composed of regular vertical and horizontal straight lines is drawn is photographed with a camera equipped with a standard lens or a telephoto lens. . As shown in FIG. 5, vertical and horizontal linear grids are observed in the image I. FIG. The ratio of the length of the vertical line to the length of the horizontal line in each square is almost the same regardless of the position of the image I. That is, the distortion generated in image I is very small.

On the other hand, image J is an example of an image observed when the same target as described above is photographed by fisheye camera 200 of the present embodiment. The fisheye camera 200 generates an image by a so-called equidistant projection method, which generates an image in which the distance from the center of the image is proportional to the direction (angle) of the observed object. Therefore, the magnitude of distortion generated is different between the vicinity of the center of the image J and the peripheral portion.

Specifically, in the image J observed when the target is photographed, a vertical line is observed in the shape of an arc passing through the points C1 and C2 on the circumference of the circle circumscribing the image J. A horizontal line of the target is observed as an arc passing through points C3 and C4 on the circumference of a circle circumscribing image J. FIG.

That is, near the center of the image J, the vertical and horizontal lines of the target are observed to be nearly straight. The ratio of the length of the vertical line to the horizontal line in each square is almost equal. That is, the generated strain is small. On the other hand, in the peripheral portion of image J, both the vertical and horizontal lines of the target are observed as curved lines. Furthermore, the ratio of vertical lines to horizontal lines in each square is different. Thus, in the image J, the greater the distance from the center of the image, the greater the distortion that occurs. The direction of the generated distortion is point-symmetrical with respect to the center of the image J. FIG.

Therefore, in the image J, the magnitude and direction of movement that occurs when the person performs the same action differs depending on the position where the person is observed. That is, in order to recognize a person's action, a recognition dictionary may be prepared for each location of the image J, and action recognition may be performed using the recognition dictionary corresponding to the position where the person is observed.

Specifically, as shown in FIG. 6, a plurality of regions R1, R2, R3, and R4 are set from the center to the periphery of the image J, and a recognition dictionary is created for each of the regions R1, R2, R3, and R4. create. In this case, the image distortion is the smallest in the region R1 and the largest in the region R4. Note that the regions R1, R2, R3, and R4 are examples of regions to be set, and the number of regions is not limited to four. In this manner, the action recognition system 100 of this embodiment sets similar regions at a plurality of different positions on the image J, and creates a recognition dictionary for each region. Then, the action recognition system 100 performs action recognition using a recognition dictionary created at a position closest to the position of the person detected from the captured video.

Note that even if a camera equipped with a wide-angle lens or an ultra-wide-angle lens is used instead of the fisheye camera 200, distortion larger than that in the center of the image occurs in the periphery of the image, similar to the case of the fisheye lens. . Therefore, a method of performing action recognition using a recognition dictionary corresponding to a position within an image is effective.

(Description of the image actually observed)
Examples of images observed by the action recognition system 100 will be described with reference to FIGS. 7 to 11. FIG. FIG. 7 is a diagram showing an example of an image observed by the action recognition system 100 of the first embodiment.

FIG. 7 is an example of specific actions of workers in the workplace. In particular, the image J1 in FIG. 7 is a diagram showing an example of the specific action "walking". A "walking" action is an action that occurs when a worker performs a plurality of specific actions and moves from one specific action to another specific action. And generally, the longer the time required for the "walking" action, the lower the working efficiency. The action recognition system 100 recognizes a walking action as one of specific actions.

FIG. 8 is a diagram for explaining the "walking" action among the specific actions recognized by the action recognition system 100. As shown in FIG. As shown in FIG. 8, the fisheye camera 200 photographs a person H1 who is walking in chronological order. In this case, a moving image (image sequence) of the walking person H1 is obtained.

FIG. 9 is an enlarged view j1 of a person in image J1 of FIG. The action recognition system 100 recognizes a specific action by observing the time change of the area shown in FIG.

10A and 10B are diagrams for explaining a "putting" action of placing a product on a shelf, among the specific actions recognized by the action recognition system 100. FIG. As shown in FIG. 10, the fish-eye camera 200 photographs a character sequence of a person H1 who is doing shelving. In this case, a moving image (image sequence) of the person H1 who is doing the shelving is obtained.

FIG. 11 is a diagram showing an example of an enlarged view j2 of a person who is taking the putaway action. The action recognition system 100 recognizes a specific action by observing the time change of the area shown in FIG.

(Description of the functional configuration of the action recognition processing unit)
Next, the functional configuration of the action recognition processing unit 321 will be described with reference to FIG. 12 . FIG. 12 is a functional block diagram showing an example of the action recognition processing section 321 according to this embodiment. As shown in FIG. 12, the action recognition processing unit 321 includes a video input unit 331, a region dividing unit 332, a dictionary creating unit 333, a dictionary selecting unit 334, an action recognizing unit 335, and a duration measuring unit 336. Prepare.

The moving image input unit 331 inputs a moving image captured by the fisheye camera 200 via the interface unit 322 (FIG. 1).

The region dividing unit 332 divides the image included in the moving image captured by the fisheye camera 200 into a plurality of regions with different distortions.

The dictionary creation unit 333 creates a different recognition dictionary for recognizing specific human behavior for each divided area.

A dictionary selection unit 334 selects a recognition dictionary to be used for recognizing a specific behavior of a person detected from a moving image from among the different recognition dictionaries created by the dictionary creation unit 333 .

The action recognition unit 335 recognizes specific human actions based on the recognition dictionary selected by the dictionary selection unit 334 .

The duration measurement unit 336 measures the duration of the specific action based on the recognition result of the specific action.

(Description of the functional configuration of the dictionary creation part)
Next, with reference to FIG. 13, the functional configuration of the dictionary creation unit 333 will be described. FIG. 13 is a functional block diagram showing an example of a schematic configuration of the dictionary creation unit 333 according to this embodiment. As shown in FIG. 13, the dictionary creation unit 333 includes a feature point extraction unit 333a, a feature point classification unit 333b, a feature vector calculation unit 333c, a histogram creation unit 333d, and a recognition dictionary creation unit 333e.

Although the moving image captured by the fisheye camera 200 has distortion, the distortion is not corrected. do. That is, the dictionary creating unit 333 creates a recognition dictionary for recognizing specific actions (standard work) in a highly distorted region in a highly distorted region. In addition, the dictionary creating unit 333 creates a recognition dictionary for recognizing the specific action in a state where the distortion is small in the area where the distortion is small. Therefore, when creating a recognition dictionary, subjects perform a standard task at various positions in the image.

The feature point extraction unit 333a extracts feature points that occur along with a specific action (standard work) from among a plurality of images included in the moving image captured by the fisheye camera 200. FIG. More specifically, the feature point extraction unit 333a extracts T image frames from the input moving image, and extracts feature points in spatiotemporal space (also referred to as spatiotemporal feature points) for the extracted T image frames. ). A feature point is an input video that is divided into three-dimensional blocks of a predetermined size consisting of two axes in the spatial direction and one axis in the time direction. is a block that exceeds . Note that the feature point extraction unit 333a extracts feature points from a plurality of moving images in order to generate highly accurate learning data.

Note that the feature point extracting unit 333a uses a known human detection algorithm to detect a person from images included in the moving image captured by the fisheye camera 200, and detects only the area of the detected person. , the feature point extraction described above may be performed. According to this, it is possible to limit the area from which the feature points are extracted, so that the processing can be performed more efficiently.

The feature point classification unit 333b classifies (clusters) the M×N×T×3-dimensional vector representing the feature points extracted by the feature point extraction unit 333a, for example, by a known K-means method. . Assuming that the number of classes to be classified is K types, the feature point classification unit 333b classifies the feature points extracted from the learning video into K types.

The feature vector calculation unit 333c averages the M×N×T×3-dimensional vectors of the same type of feature points among the K types of feature points classified by the feature point classification unit 333b to obtain K average vectors Vk Ask for Each of the average vectors Vk calculated by the feature vector calculator 333c is a vector representing K types of feature points. Note that the average vector Vk is an example of a learning vector.

Feature vectors obtained from videos in which a specific action is observed are distributed near the average vector Vk obtained from learning data of the same specific action. Using this characteristic, the action recognition unit 335 can perform highly accurate action recognition from a distorted moving image captured by the fisheye camera 200 even without correcting the distortion. That is, when the worker moves linearly, the movement of the person becomes curvilinear in a region where the distortion of the captured moving image is large. However, since the recognition dictionary created by the dictionary creating unit 333 is learned assuming that human movements are curvilinear, it is possible to recognize the specific action without correcting the distortion. Similarly, in areas where distortion is small, the linear movements of the worker are learned as linear movements, so specific actions can be recognized without correcting the distortion.

The histogram creation unit 333d creates a learning histogram H(k) representing the appearance frequency of the average vector Vk. Specifically, for K types of feature points, the total number of blocks in each feature point group is calculated to create a learning histogram H(k). A learning histogram H(k) indicates the frequency of k groups of feature points. Note that the histogram creation unit 333d is an example of a learning histogram creation unit.

The recognition dictionary creation unit 333e creates a dictionary for recognizing specific actions in each of the N action recognition target regions, based on the learning histogram H(k) obtained from the learning data of each region. The recognition dictionary creating unit 333e creates a recognition dictionary by a machine learning method of SVM (Support Vector Machine). Note that when creating the recognition dictionary by the SVM machine learning method, the recognition dictionary creation unit 333e combines positive learning data (plus learning data) including specific actions to be recognized and specific actions to be recognized. A recognition dictionary may be created by preparing negative learning data (negative learning data) that does not include such data. That is, the recognition dictionary creating unit 333e creates a recognition dictionary that accepts positive learning data as correct data and excludes negative learning data as different data. As a result, actions that are likely to be mistaken for specific actions can be learned as negative learning data, so that the recognition rate of specific actions can be improved. It should be noted that other machine learning methods than the SVM machine learning method may be used when creating the recognition dictionary. For example, machine learning methods such as KNN (K Nearest Neighbor) and MLP (Multilayer Perceptron) may be used.

(Description of the functional configuration of the action recognition unit)
Next, the functional configuration of the action recognition unit 335 will be described with reference to FIG. 14 . FIG. 14 is a functional block diagram showing an example of a schematic configuration of the action recognition unit 335 according to this embodiment. As shown in FIG. 14, the action recognition section 335 includes a feature point extraction section 335a, a feature vector calculation section 335b, a histogram creation section 335c, and an action recognition section 335d. The feature point extraction unit 335a has the same function as the feature point extraction unit 333a included in the dictionary creation unit 333. FIG.

The feature vector calculation unit 335b obtains spatio-temporal edge information (differential vector) at the feature points extracted by the feature point extraction unit 335a. Details of the spatio-temporal edge information will be described later.

The histogram creation unit 335c creates a specific action histogram T(k) representing the appearance frequency of spatio-temporal edge information.

The action recognition unit 335d compares the specific action histogram T(k)) created by the histogram creation unit 335c based on the differential vector obtained from the moving image and the learning histogram H(k) stored in the recognition dictionary. to recognize specific behaviors. The distribution of feature points to be recognized is close to the distribution of feature points in the recognition dictionary. That is, since the specific action histogram T(k) obtained from the recognition target image performing the specific action and the learning histogram H(k) of the same specific action are similar, the image distortion correction is not performed. , it is possible to recognize specific actions.

(Description of images observed by the action recognition system)
Next, examples of images observed by the action recognition system 100 will be described with reference to FIGS. 15 and 16. FIG. FIG. 15 is a diagram showing an example of a moving image (image sequence) input to the moving image input unit 331. As shown in FIG. Each image (frame) shown in FIG. 15 is an image captured by the fisheye camera 200 and is an image without distortion correction. The horizontal axis x and vertical axis y of the photographed image are spatial coordinates. The time axis of the image frames F1 and F2 is indicated by t. That is, the input image becomes spatio-temporal data at coordinates (x, y, t). A pixel value at one spatio-temporal coordinate is a function of spatial coordinates (x, y) and time t. When a person moves when recognizing the above-mentioned specific behavior in the workplace, a change point occurs in the spatio-temporal data shown in FIG. 15 . The action recognition system 100 recognizes the specific action by finding this change point, that is, the spatio-temporal feature point.

Next, a method for extracting feature points in this embodiment will be described. As shown in FIG. 16, the spatio-temporal image data is divided into blocks. Large cubes in FIG. 16 indicate spatio-temporal image data. The horizontal axis x and the vertical axis y represent spatial coordinates. Each unit is a pixel. Also, the time axis is indicated by t. For example, a moving image is input at a video rate of 30 frames/second, and time-series images are input. By converting at this video rate, the actual time the image was taken can be determined. The spatio-temporal image data of FIG. 16 is divided into blocks of size (N, N, T). The size of one block is horizontal M pixels, vertical N pixels, and T frames. One square in FIG. 16 indicates one block. When a person performs a certain action, the feature amount of the block in which movement occurs in the spatio-temporal data increases. That is, a large amount of change occurs in time and space.

Next, a method of extracting a block with a large amount of change as a feature point will be described. In order to extract feature points from spatio-temporal image data, first, smoothing processing is performed to remove noise in the spatial direction, that is, the (x, y) direction. The smoothing process is performed by Equation (1).

Here, I(x, y, t) is the pixel value of the (x, y) coordinates in the frame at time t. Also, g(x, y) is a kernel for smoothing processing. Also, * is an operator indicating convolution processing. The smoothing process may simply be an averaging process of pixel values, or an existing Gaussian smoothing filter process may be performed.

Next, filtering processing is performed on the time axis. Here, the Gabor filtering process shown in Equation (2) is performed. The Gabor filter is a directional filter, and has the effect of emphasizing parallel and equally spaced lines existing in the region where the filter is applied and removing noise existing between the lines. g _ev and g _od in equation (2) are kernels of Gabor filters indicated by equations (3) and (4), respectively. Also, * is an operator indicating convolution processing. Furthermore, τ and ω are parameters of the kernel in the Gabor filter.

After performing the filtering process shown in the above formula (2) on all pixels of the spatio-temporal image shown in FIG. 15, the average value of R(x, y, t) in the divided blocks shown in FIG. 16 is obtained. The average value of the block of the spatio-temporal coordinates (x, y, t) is obtained by Equation (5).

As shown in Equation (6), if the average value M(x, y, t) in the block is greater than a predetermined threshold Thre_M, this block is taken as a feature point.

(Description of how to describe feature points)
Next, a description method of feature points will be described with reference to FIGS. 17A and 17B. FIG. 17A is a first diagram showing an example of feature points extracted from a moving image. FIG. 17B is a second diagram showing an example of feature points extracted from a moving image. That is, FIG. 17A is an image k1 showing an example of feature points at time t, extracted from the image of the person who is doing the shelving shown in FIG. As shown in FIG. 17A, feature points are extracted from moving parts. FIG. 17B is an image k2 showing an example of feature points similarly extracted at time t+Δt.

After the feature points shown in FIG. 17A are extracted, the spatio-temporal edge information of the pixels in the block to which the feature points belong is obtained. That is, the edge information E(x, y, t) (differential vector) of the pixel is obtained by performing the differential operation shown in Equation (7).

Since one block includes M.times.N.times.T pixels, equation (7) yields M.times.N.times.T.times.3 differential values. That is, a block containing feature points can be described by a vector of M×N×T×3 differential values. That is, feature points can be described by M×N×T×3-dimensional vectors. Edge information E(x, y, t) is similarly obtained for image k2 in FIG. 17B.

When creating a recognition dictionary for recognizing a specific action through learning, the dictionary creation unit 333 creates recognition dictionaries corresponding to a plurality of different positions with different distortions in an image.

Here, among the plurality of feature points extracted from the image k1, the closest feature points are considered to be feature points that occur as a result of the action of one person. That is, the recognition dictionary created by the dictionary creation unit 333 is stored in association with the representative point (for example, the position of the center of gravity) of the region m1, assuming that the region m1 shown in FIG. 17A is the region where a person exists.

The same applies to the area m2 formed by the feature points extracted from the image k2 in FIG. 17B. In this way, the dictionary creation unit 333 creates a recognition dictionary using N frames of images including a specific action as one set of learning data.

(Explanation of duration of specific action)
Next, with reference to FIG. 18, the duration of specific actions will be described. FIG. 18 is a diagram illustrating measurement of the duration of specific actions.

The duration measurement unit 336 measures the duration of the specific action based on the recognition result of the specific action. FIG. 18 shows an example in which an action of "walking" is recognized from time t0 to time t1, and an action of "putting" is recognized from time t2 to time t3.

The duration measuring unit 336 determines that the duration of the “walking” behavior is (t1-t0) and the duration of the “putting away” behavior is (t3-t2) in FIG. Also when the number of specific behaviors to be recognized increases, the recognition processing for each specific behavior is similarly performed, and the duration of the behavior is measured.

(Description of the flow of recognition dictionary creation processing)
Next, with reference to FIG. 19, the flow of recognition dictionary creation processing performed by the dictionary creation unit 333 will be described. Note that FIG. 19 is a flowchart showing an example of the flow of creating a recognition dictionary.

The moving image input unit 331 inputs a moving image captured by the fisheye camera 200 (step S11).

The feature point extraction unit 333a extracts feature points from the input moving image (step S12).

The feature point classification unit 333b clusters the extracted feature points (step S13).

The feature vector calculator 333c calculates the average vector Vk (step S14).

The histogram creation unit 333d creates a learning histogram H(k) (step S15).

The recognition dictionary creation unit 333e creates a recognition dictionary (step S16). After that, the dictionary creating unit 333 terminates the processing of FIG. 19 . As described above, it is necessary to create a plurality of recognition dictionaries at different positions of the image (positions with different distortions), so the process of FIG. 19 is repeatedly executed.

(Explanation of flow of action recognition processing)
Next, the flow of action recognition processing performed by the action recognition processing unit 321 will be described with reference to FIG. Note that FIG. 20 is a flowchart showing an example of the flow of recognition processing for a specific action.

The moving image input unit 331 inputs a moving image captured by the fisheye camera 200 (step S21).

The feature point extraction unit 335a extracts feature points from the input moving image (step S22).

The feature vector calculator 335b calculates the average vector Vk (step S23).

The histogram creation unit 335c creates a specific action histogram T(k) (step S24).

The dictionary selection unit 334 selects a recognition dictionary (step S25). Specifically, the dictionary selection unit 334 selects a recognition dictionary created near the position of the feature point extracted by the feature point extraction unit 335a. That is, the dictionary selection unit 334 selects recognition dictionaries created at positions where the magnitudes of distortion are close to each other.

The action recognition unit 335 recognizes the specific action (step S26). Note that the flow of recognition processing for specific actions will be described later (FIG. 21).

The duration measurement unit 336 measures the duration of the specific action (step S27).

Further, the duration measurement unit 336 outputs the type of specific action and the measurement result of the specific action (step S28). After that, the action recognition unit 335 ends the processing of FIG. 20 .

(Description of the flow of specific action recognition processing)
Next, with reference to FIG. 21, the flow of specific action recognition processing performed by the action recognition unit 335 will be described. Note that FIG. 21 is a flowchart showing an example of the flow of processing for recognizing a plurality of specific actions. In particular, FIG. 21 shows the flow of processing for recognizing that the "putting away" action was performed after the "walking" action among the specific actions.

The action recognition unit 335 recognizes a "walking" action (step S31).

Next, the action recognition unit 335 determines whether or not a "walking" action has been recognized (step S32). If it is determined that the "walking" action has been recognized (step S32: Yes), the process proceeds to step S31. On the other hand, if it is not determined that the "walking" action has been recognized (step S32: No), the process proceeds to step S33.

If determined as No in step S32, the action recognition unit 335 recognizes the action of "shelving" (step S33).

Next, the action recognizing unit 335 determines whether or not the action of "shelving" has been recognized (step S34). If it is determined that the "put away" action has been recognized (step S34: Yes), the process of FIG. 21 is terminated and the process proceeds to step S27 of FIG. On the other hand, if it is not determined that the "putting away" action has been recognized (step S34: No), the process returns to step S31.

Note that the flowchart shown in FIG. 21 is an example, and the action recognition unit 335 performs processing according to the type and order of specific actions to be recognized.

As described above, according to the action recognition device 300 of the first embodiment, the moving image input unit 331 inputs a moving image captured by the fisheye camera 200 (capturing means), and the area dividing unit 332 An image included in a moving image is divided into a plurality of regions with different distortions. The dictionary creating unit 333 creates a recognition dictionary for recognizing a specific action of a person (subject) for each divided area. A dictionary selection unit 334 selects a recognition dictionary to be used for recognizing a specific behavior of a person detected from a moving image from among the plurality of recognition dictionaries created by the dictionary creation unit 333 . Based on the recognition dictionary selected by the dictionary selection unit 334, the action recognition unit 335 recognizes the specific human action. Therefore, since a recognition dictionary is created for each area of an image, it is possible to recognize a specific human action (standard work) without correcting the distortion of the photographed image.

Further, according to the action recognition device 300 of the first embodiment, the dictionary selection unit 334 detects the position of the person (subject) from the image included in the moving image captured by the fisheye camera 200 (capturing means). Select a recognition dictionary. Therefore, it is possible to recognize specific human behavior (standard work) without correcting the distortion of the photographed image.

Also, according to the action recognition device 300 of the present embodiment, the duration measurement unit 336 measures the duration of the specific action based on the recognition result of the specific action. Therefore, the duration of specific behavior (standard work) can be measured easily and accurately.

Further, according to the action recognition device 300 of the first embodiment, the feature point extraction unit 333a extracts feature points from a plurality of images included in a moving image captured by the fisheye camera 200 (capturing means). . The feature point classification unit 333b classifies the extracted feature points into K types. The feature vector calculator 333c obtains K average vectors Vk (learning vectors) for each of the K kinds of classified feature point groups. Therefore, it is possible to easily learn the specific behavior of a person (subject).

Further, according to the action recognition device 300 of the first embodiment, the dictionary creation unit 333 combines the video (plus learning data) of the person performing the specific action, input by the video input unit 331, with the specific action. Using the average vector Vk (learning vector) by the feature amount of the feature points of each data, the learning histogram H(k) is created from the video of the person who has not performed (negative learning data), and the plus learning data. and the learning histogram H(k) generated from the negative learning data, a recognition dictionary is created. Therefore, it is possible to improve the accuracy of the recognition dictionary.

Further, according to the action recognition device 300 of the first embodiment, the feature point extraction unit 335a extracts feature points from a plurality of images included in a moving image captured by the fisheye camera 200 (capturing means). . The feature vector calculator 335b calculates a feature vector indicating the magnitude and direction of the spatio-temporal edge at the extracted feature point. The histogram creating unit 335c creates a specific action histogram T(k) based on the feature vectors of the extracted feature points. Then, the action recognition unit 335d recognizes a person's specific action based on the specific action histogram T(k) and the learning histogram H(k) of the recognition dictionary. Therefore, the specific behavior of a person (subject) can be easily and accurately recognized.

Further, according to the action recognition device 300 of the first embodiment, the feature vector calculation unit 335b divides a plurality of input images into M×N×T size blocks, and performs differentiation processing on each block. , M×N×T×3-dimensional edge information E(x, y, t) (differential vector). Then, the feature vector calculation unit 335b compares the calculated edge information E (x, y, t) with the pre-learned K kinds of average vectors Vk (learning vectors), and based on the comparison result, calculates the edge Information E(x, y, t) is classified into the same kind of feature points as the nearest mean vector Vk. The histogram creation unit 335c creates the specific action histogram T(k) based on the classification result, and thus easily creates the specific action histogram T(k) used for recognizing the specific action from the captured moving image. be able to.

Further, according to the action recognition device 300 of the first embodiment, the dictionary creation unit 333 and the action recognition unit 335 perform filtering processing on the input moving image on the time axis. Then, when the average value in the M×N×T block is larger than a predetermined threshold as a result of the filtering processing, the feature point extraction units 333a and 335a extract the block as a feature point. Therefore, feature points can be easily extracted.

Further, according to the action recognition device 300 of the present embodiment, the filtering process is performed by the Gabor filtering process shown in Equations (2), (3), and (4). Therefore, by removing noise from the captured moving image, it is possible to obtain an image that facilitates recognition of the specific action.

Further, according to the action recognition device 300 of the first embodiment, the feature point extraction units 333a and 335a perform smoothing processing on each image before performing filtering processing on the time axis. Therefore, since noise occurring in the direction of the time axis is removed, the specific action of a person (subject) can be recognized with even higher accuracy.

Further, according to the action recognition device 300 of the first embodiment, when recognizing a specific action of a person, the action recognition unit 335 recognizes the specific action in a predetermined order, and when the specific action is recognized, When the recognition result is output and the specific action is not recognized, the next specific action is recognized. Therefore, even if a plurality of specific behaviors occur consecutively, they can be reliably recognized.

Further, according to the action recognition device 300 of the first embodiment, the wide-angle lens is a fisheye lens. Therefore, a wider range can be observed with a single camera.

(Second embodiment)
Next, a second embodiment of the action recognition device, action recognition method, and program will be described in detail with reference to the accompanying drawings.

(Description of hardware configuration of action recognition device)
FIG. 22 is a hardware block diagram showing an example of the hardware configuration of the action recognition system 100a according to this embodiment. As shown in FIG. 22, the action recognition system 100a includes fisheye cameras 200 and 201 and an action recognition device 300a.

The action recognition system 100a has the same function as the action recognition system 100 described in the first embodiment, and recognizes a specific action of a person (subject) photographed by the fisheye cameras 200,201. The difference from the action recognition system 100 is that moving images captured by two fisheye cameras 200 and 201 can be input.

The action recognition device 300a includes an action recognition processing section 321a and an interface section 322a that connects the action recognition processing section 321a and the fisheye cameras 200 and 201 to each other.

The action recognition processing unit 321a recognizes a specific action of a person (subject). The interface unit 322a converts the moving images captured by the fisheye cameras 200 and 201 into a data format recognizable by the action recognition processing unit 321a, and transfers the converted data to the action recognition processing unit 321a.

Next, a typical scene in which the action recognition system 100a is used will be described with reference to FIG. FIG. 23 is a diagram showing an example of a scene in which the action recognition system 100a according to the second embodiment is used.

As shown in FIG. 23, the action recognition system 100a is installed in a work environment such as an office or a factory. The fisheye cameras 200 and 201 capture moving images including a plurality of people H1 and H2 working in the work environment. In this embodiment, the fisheye cameras 200 and 201 are both equipped with a fisheye lens having a diagonal angle of view of 180°. The two fisheye cameras 200 and 201 photograph the same work environment from different directions. Note that the people H1 and H2 are examples of subjects.

The action recognition system 100 described in the first embodiment can recognize predetermined actions of a plurality of workers in an environment where a plurality of workers are working. Since it is not possible to visualize the worker who is doing this, it was not possible to perform action recognition. On the other hand, since the action recognition system 100a observes the work environment from different directions, the blind spots are reduced and the predetermined actions of a plurality of people can be recognized more reliably. Furthermore, since the action recognition system 100a can photograph one worker with the two fisheye cameras 200 and 201, action recognition can be performed using images photographed with less distortion.

Note that the hardware configuration of the action recognition system 100a is the same as the hardware configuration of the action recognition system 100 except that the number of fisheye cameras is increased, so the description is omitted.

(Description of the functional configuration of the action recognition processing unit)
Next, the functional configuration of the action recognition processing section 321a will be described with reference to FIG. FIG. 24 is a functional block diagram showing an example of the functional configuration of the action recognition processing section 321a in the second embodiment. As shown in FIG. 24, the action recognition processing unit 321a includes a same person determination unit 337 in addition to the functional configuration (FIG. 12) of the action recognition processing unit 321 described in the first embodiment. Also, the action recognition processing unit 321a includes a dictionary selection unit 334a whose function is changed instead of the dictionary selection unit 334. FIG.

The dictionary selection unit 334a selects a recognition dictionary corresponding to an image used for action recognition when the fisheye cameras 200 and 201 respectively photograph the same person. Specifically, the dictionary selection unit 334a compares the positions of the same person in the images captured by the fisheye cameras 200 and 201, and recognizes the specific behavior of the person who appears in a position closer to the center of the image. , that is, the recognition dictionary created at a position with less distortion. The same person determination unit 337, which will be described later, determines whether or not the same person is captured in the images captured by the fisheye cameras 200 and 201. FIG.

The same person determination unit 337 determines whether the images captured by the fisheye cameras 200 and 201 include the same person. Specifically, the same person determination unit 337 compares feature vectors based on feature points extracted from images captured by the fisheye cameras 200 and 201, respectively, to determine whether the types of feature vectors and the orientations of the feature vectors are similar. If so, it is determined that the same person is photographed.

(Explanation of flow of action recognition processing)
Next, the flow of action recognition processing performed by the action recognition processing unit 321a will be described with reference to FIG. Note that FIG. 25 is a flow chart showing an example of the flow of specific action recognition processing in the second embodiment.

Steps S41 to S44 are the same processes as steps S21 to S24 (FIG. 20) described in the first embodiment.

Next, the same person determination unit 337 identifies areas representing the same person from the images captured by the fisheye cameras 200 and 201 (step S45).

Next, the dictionary selection unit 334a identifies the fish-eye camera that captured the area closest to the center of the image from among the areas representing the same person identified in step S45, and recognizes the corresponding position. A dictionary is selected (step S46). In step S45, if the regions representing the same person cannot be identified, the dictionary selection unit 334a selects recognition dictionaries corresponding to the respective detected regions.

The processing from step S47 to step S49 that follows is the same as the processing from step S26 to step S29 (FIG. 20) described in the first embodiment.

As described above, in the action recognition device 300a of the second embodiment, the plurality of fisheye cameras 200 and 201 (photographing means) photograph the same area from different directions. Therefore, blind spots in the observation range are reduced. In addition, since the same person (subject) can be photographed from different directions, the recognition accuracy of action recognition can be improved.

Further, according to the action recognition device 300a of the second embodiment, the dictionary selection unit 334a detects the same person (subject ), the recognition dictionary with the least distortion is selected from among the recognition dictionaries corresponding to the positions of the . Therefore, it is possible to improve the recognition accuracy of the specific action.

Further, according to the action recognition device 300a of the second embodiment, the dictionary selection unit 334a detects the positions of the same person detected from the images included in the moving images captured by the plurality of fisheye cameras 200 and 201 (capturing means). A recognition dictionary corresponding to a position near the center of the image is selected from among the recognition dictionaries corresponding to . Therefore, since a recognition dictionary created at a position with little distortion is selected, it is possible to improve the recognition accuracy of the specific action.

Although the embodiments of the present invention have been described above, the above-described embodiments are presented as examples and are not intended to limit the scope of the present invention. This novel embodiment can be implemented in various other forms. Also, various omissions, replacements, and changes can be made without departing from the scope of the invention. Moreover, this embodiment is included in the scope and gist of the invention, and is included in the scope of the invention described in the claims and its equivalents.

200, 201 fisheye camera (photographing means)
300, 300a action recognition device 321, 321a action recognition processing unit 331 moving image input unit 332 region division unit 333 dictionary creation unit 334 dictionary selection unit 335 action recognition unit 336 duration measurement unit 333a, 335a feature point extraction unit 333b feature point classification unit 333c, 335b feature vector calculation unit 333d histogram creation unit (learning histogram creation unit)
335c Histogram creation unit 333e Recognition dictionary creation unit 335d Action recognition unit H1, H2 Person (subject)
H(k) learning histogram T(k) specific action histogram Vk average vector (learning vector)

JP 2011-100175 A

Claims (14)

A behavior recognition device for recognizing a specific behavior of a subject captured in a captured moving image to be recognized,
Input a video of a subject performing a specific action at multiple positions with different distortions within the observation range of the imaging means, taken by multiple imaging means equipped with a wide-angle lens and photographing the same area from different directions . a first moving image input unit for
a second moving image input unit for inputting a moving image to be recognized by the plurality of photographing means;
an area dividing unit that divides an image included in the moving images input by the first moving image input unit and the second moving image input unit into a plurality of areas with different distortions;
a dictionary creation unit for creating a recognition dictionary for recognizing the specific behavior of the subject for each of the photographing means and each of the regions from the moving image input by the first moving image input unit ;
The dictionary creating unit creates a dictionary for each photographing means and for each area according to the position of the same subject detected from each of the areas of the image included in the moving image to be recognized input from the different photographing means. a dictionary selection unit that selects a recognition dictionary with the least distortion from among the plurality of recognition dictionaries obtained;
an action recognition unit that recognizes the specific action of the subject based on the recognition dictionary selected by the dictionary selection unit;
An action recognition device comprising: The dictionary selection unit selects a position near the center of the image from among the recognition dictionaries corresponding to the positions of the same subject detected from the images included in the moving image to be recognized input by the second moving image input unit. select the corresponding recognition dictionary,
The action recognition device according to claim 1 . Further comprising a duration measurement unit that measures the duration of the specific action based on the recognition result of the specific action,
The action recognition device according to claim 1 or 2 . The dictionary creation unit
a feature point extraction unit for extracting feature points from a plurality of images included in the moving image input by the first moving image input unit ;
a feature point classification unit that classifies the extracted feature points into K types;
a feature vector calculation unit that obtains K learning vectors for each of the K types of classified feature point groups;
The action recognition device according to any one of claims 1 to 3 , further comprising a learning histogram creation unit that creates a learning histogram representing the appearance frequency of the learning vector. The learning histogram creation unit
from the moving image of the subject performing the specific action, which constitutes the positive learning data, and the moving image of the subject not performing the specific action, which constitutes the negative learning data, which are input by the first moving image input unit; , using the learning vectors based on the feature amounts of the feature points of each data, create a learning histogram,
The dictionary creation unit
creating the recognition dictionary based on the learning histogram generated from the positive learning data and the learning histogram generated from the negative learning data;
The action recognition device according to claim 4 . The action recognition unit is
a feature point extracting unit for extracting feature points from among a plurality of images included in the moving image to be recognized input by the second moving image input unit ;
a feature vector calculation unit that calculates a feature vector indicating the magnitude and direction of the spatiotemporal edge at the extracted feature point;
a histogram creation unit that creates a histogram representing the frequency of appearance of the feature vectors at the feature points,
Recognizing the specific behavior of the subject based on the histogram and the recognition dictionary;
The action recognition device according to claim 4 or 5 . The feature vector calculator,
A plurality of input images are divided into blocks of size M×N×T, and each block is differentiated to calculate an M×N×T×3-dimensional differential vector,
Comparing the calculated differential vector with the learning vector learned in advance, classifying the differential vector into the same type of feature points as the closest learning vector based on the result of the comparison,
The histogram creating unit
creating the histogram based on the results of the classification;
The action recognition device according to claim 6 . The dictionary creation unit and the action recognition unit are
Filtering the moving image input by the first moving image input unit and the moving image to be recognized input by the second moving image input unit on the time axis,
The feature point extraction unit is
As a result of performing the filtering process, if the average value in the M × N × T block is larger than a predetermined threshold, extracting the block as a feature point;
The action recognition device according to any one of claims 4 to 7 . Let g _ev and g _od be the kernel of the Gabor filter shown in the following equations (1) and (2), * be the convolution process, and τ and ω be the parameters of the kernel. A Gabor filtering process using the following equation (3),

The action recognition device according to claim 8 . The feature point extraction unit is
Performing a smoothing process on each image before performing the filtering process,
The action recognition device according to claim 8 or 9 . The action recognition unit is
when recognizing the specific action of the subject, recognizing the specific action in a predetermined order, and outputting the recognition result when the specific action is recognized,
If the specific action is not recognized, recognize the next specific action,
The action recognition device according to any one of claims 1 to 10 . The wide-angle lens is a fisheye lens,
The action recognition device according to any one of claims 1 to 11 . When recognizing the specific behavior of the subject in the video from the captured video to be recognized,
Input a video of a subject performing a specific action at multiple positions with different distortions within the observation range of the imaging means, taken by multiple imaging means equipped with a wide-angle lens and photographing the same area from different directions . a first video input step of
a second moving image input step of inputting a moving image to be recognized by the plurality of photographing means;
an area dividing step of dividing an image included in the moving images input in the first moving image input step and the second moving image input step into a plurality of areas each having a different distortion;
a dictionary creation step of creating a recognition dictionary for recognizing the specific behavior of the subject for each of the photographing means and each of the regions from the moving image input in the first moving image input step ;
The dictionary creation step creates the dictionary for each imaging means and for each area according to the position of the same subject detected from each of the areas of the image included in the moving image to be recognized input from the different imaging means . a dictionary selection step of selecting a recognition dictionary with the least distortion from among the plurality of recognition dictionaries obtained;
an action recognition step of recognizing the specific action of the subject based on the recognition dictionary selected in the dictionary selection step;
An action recognition method that performs A computer that controls an action recognition device that recognizes a specific action of a subject captured in a captured video to be recognized,
Input a video of a subject performing a specific action at multiple positions with different distortions within the observation range of the imaging means, taken by multiple imaging means equipped with a wide-angle lens and photographing the same area from different directions . a first moving image input unit for
a second moving image input unit for inputting a moving image to be recognized by the plurality of photographing means;
an area dividing unit that divides an image included in the moving images input by the first moving image input unit and the second moving image input unit into a plurality of areas with different distortions;
a dictionary creation unit for creating a recognition dictionary for recognizing the specific behavior of the subject for each of the photographing means and each of the regions from the moving image input by the first moving image input unit ;
The dictionary creating unit creates a dictionary for each photographing means and for each area according to the position of the same subject detected from each of the areas of the image included in the moving image to be recognized input from the different photographing means. a dictionary selection unit that selects a recognition dictionary with the least distortion from among the plurality of recognition dictionaries obtained;
an action recognition unit that recognizes the specific action of the subject based on the recognition dictionary selected by the dictionary selection unit;
A program that works as

Images