KR20220072499A - Method, apparatus and system for recognizing behavior based on multi-view video - Google Patents

Abstract

The present invention relates to a multi-view image-based behavior recognition method, apparatus and system. The method for recognizing an action according to an embodiment of the present invention is performed by an electronic device, and at the same time n (where n is a natural number greater than or equal to 3) uses a video captured in different views of an object or event. As a method for recognizing an action, n classifiers according to a machine learning technique using learning data including input data for images of different views and result data for behavior types of objects or events in the image, respectively learning; inputting images of n different views of the target region to each learned classifier to obtain n behavior types output by each classifier; and deriving at least one behavior type from among n behavior types obtained from each classifier as a result by using an ensemble of classifiers.

Description

Multi-view image-based behavior recognition method, apparatus and system

The present invention relates to a multi-view image-based behavior recognition method, apparatus and system, and more particularly, to a multi-view video in which a target area is simultaneously photographed in various directions or angles, within the target area. It relates to a method, apparatus and system for recognizing various actions of an object or event.

There is a technology for recognizing various actions of an object or event in the video by analyzing a video captured by a camera, that is, an image-based action recognition technology. Conventional image-based behavior recognition technology (hereinafter, referred to as “conventional technology”) analyzes a single image captured by one camera to determine the behavior of an object.

Meanwhile, in recent years, in order to respond to various monitoring purposes and targets, a system (eg, an intelligent CCTV system, etc.) that provides a multi-view shot from various directions or angles simultaneously with respect to a target area is required. In particular, in such a system, at least three or more cameras need to be installed. For example, such an intelligent CCTV system may be a living crime prevention system, an automatic object tracking system, an illegal parking/stop enforcement system, a road crime prevention system, a signal/speeding control system, an illegal garbage discharge control system, and the like.

However, in the prior art, there is a problem in that it is difficult to analyze the behavior recognition in the multi-view image captured by these three or more cameras. This is because the prior art is merely an action recognition technology for a single image. That is, since the prior art is a behavior recognition technology through a complex image processing algorithm, if it is applied to a multi-view image as it is, the amount of computation is very large and the computation speed is very slow, making it impossible to cope with a real-time environment.

In addition, the prior art has a problem in that when the object is covered or the action of the object is a complicated action, the accuracy of recognition on the object is lowered.

KR10- 2019-0050551 A

In order to solve the problems of the prior art as described above, the present invention examines the action recognition of an object or an event in the image for a video shot in three or more multi-views of a target area at the same time. An object of the present invention is to provide a method, an apparatus and a system for performing it quickly and accurately.

However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

Behavior recognition method according to an embodiment of the present invention for solving the above problems is performed by an electronic device, and at the same time, n (where n is a natural number equal to or greater than 3) is a video shot with different views. ) as a method of recognizing the behavior of an object or event using a machine learning technique ( training n classifiers according to machine learning); inputting images of n different views of the target region to each learned classifier to obtain n behavior types output by each classifier; and deriving at least one behavior type from among n behavior types obtained from each classifier as a result by using an ensemble of classifiers.

In the learning step, the input data for each of the n classifiers may include images of different views,

In the acquiring, images of different views of the target region may be input to each classifier.

The learning may include: performing calibration between each camera so that an object or event is located in a specific area in the image in n images of different views captured by each camera; and learning the n classifiers using n images of different views captured by each camera on which the calibration has been performed.

The specific region may correspond to a central region in the image.

In the learning step, the input data for each of the n classifiers includes images of different views, information on frames in which objects or events appear in the images, and objects or events that appear in the image disappear. Each frame may include information.

The device for recognizing an action according to an embodiment of the present invention is a device for recognizing the action of an object or event by using a video shot with n (where n is a natural number of 3 or more) different views at the same time, A memory for storing n classifiers pre-learned according to machine learning using learning data including input data for images of different views and result data for behavior types of objects or events in the image, respectively ; and a control unit that controls recognition of an action for an object or event in images of n different views of the target region by using each classifier stored in the memory.

Behavior recognition device according to another embodiment of the present invention is a device for recognizing the behavior of an object or event using a video shot with n (where n is a natural number of 3 or more) different views at the same time. , using training data including input data for images of different views and result data on the types of actions of objects or events in the image, respectively, to receive n classifiers pre-learned according to machine learning communication department; and a control unit that controls recognition of an action for an object or event in images of n different views of the target region by using each classifier received from the communication unit.

The controller may input images of n different views of the target region to each classifier to obtain n types of behaviors output by each classifier, and use an ensemble of classifiers from each classifier. At least one behavior type among the acquired n behavior types may be derived as a result.

A behavior recognition system according to an embodiment of the present invention includes a plurality of cameras that simultaneously take n (where n is a natural number of 3 or more) different views of video; and an electronic device for recognizing an action of an object or event by using an image of a different view captured by each camera.

The electronic device uses n pre-learned n pieces of pre-learned data according to machine learning using input data for images of different views and learning data including result data for behavior types of objects or events in the image, respectively. memory to store the classifier; and a control unit that controls recognition of an action for an object or event in images of n different views of the target region by using each classifier received from the communication unit.

The controller may input images of n different views of the target region to each classifier to obtain n types of behaviors output by each classifier, and use an ensemble of classifiers from each classifier. At least one behavior type among the acquired n behavior types may be derived as a result.

The camera may be a camera for CCTV. That is, the behavior recognition system according to an embodiment of the present invention may be a CCTV system.

The present invention configured as described above has an advantage in that it is possible to process behavior recognition for a plurality of images rather than a single image.

That is, the present invention analyzes a video taken in three or more multi-views of a target area at the same time using a learning model of a machine learning technique to act on an object or an event in the image. Since recognition can be performed more quickly and accurately, there is an advantage in that it is possible to respond to a real-time environment.

In addition, the present invention has the advantage of being able to further improve the accuracy of behavior recognition by processing a plurality of images at the same time, and finally analyzing the behavior recognition through the ensemble method of the analysis result of the learning model of the machine learning technique.

In addition, when the present invention is applied to a CCTV system such as a video security system in use in public institutions or in the private sector, there is an advantage in that it is possible to prevent crime, improve the arrest rate of criminals, and guarantee personal safety.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

1 shows a block diagram of a system 10 according to an embodiment of the present invention.
2 is a block diagram showing a behavior recognition apparatus 200 according to an embodiment of the present invention.
3 is a block diagram of the controller 250 in the behavior recognition apparatus 200 according to an embodiment of the present invention.
4 is a flowchart of a behavior recognition method according to an embodiment of the present invention.
5 is a detailed flowchart of steps S100 and S200 of a behavior recognition method according to an embodiment of the present invention.
6 shows an example of an image captured by each camera 100 of the system 10 according to an embodiment of the present invention.
FIG. 7 shows an example of an image according to a result of performing calibration between each camera 100 in FIG. 6 .
8 illustrates a behavior recognition process performed according to S200 of the behavior recognition method according to an embodiment of the present invention.

The above object and means of the present invention and its effects will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form as the case may be, unless otherwise specified in the phrase. In this specification, terms such as “include”, “provide”, “provide” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may indicate one of the words listed together, or a combination of two or more. For example, “or B” and “at least one of B” may include only one of A or B, or both A and B.

In the present specification, descriptions according to “for example” and the like may not exactly match the information presented, such as recited properties, variables, or values, tolerances, measurement errors, limits of measurement accuracy, and other commonly known factors The embodiments of the present invention according to various embodiments of the present invention should not be limited by effects such as modifications including .

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in between. It should be understood that there is On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In this specification, when it is described that a certain element is 'on' or 'in contact with' another element, it may be directly in contact with or connected to the other element, but another element may exist in the middle. It should be understood that On the other hand, when it is described that a certain element is 'directly on' or 'directly' of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between the elements, for example, 'between' and 'directly between', etc. may be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. In addition, the above terms should not be construed as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first component' may be referred to as a 'second component', and similarly, a 'second component' may also be referred to as a 'first component'.

Unless otherwise defined, all terms used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows a block diagram of a system 10 according to an embodiment of the present invention.

The system 10 according to an embodiment of the present invention is a system that collects images for a plurality of location areas and analyzes the images, and can recognize an action of an object or an event in the image. . That is, the system 10 according to an embodiment of the present invention may be a system for protecting lives and property, such as crime prevention and disaster monitoring, based on display or analysis of images.

For example, the system 10 according to an embodiment of the present invention may be a Closed Circuit Television (CCTV) system. In other words, a CCTV system is a system that transmits images using wired or wireless facilities to a specific recipient in a specific building or facility. Its uses are diverse.

Referring to FIG. 1 , a system 10 according to an embodiment of the present invention may include a plurality of cameras 100 and a behavior recognition device 200 .

The camera 100 is provided with n (however, n is a natural number equal to or greater than 3) (100-1, 100-2, ... 100-n). Each of these cameras 100 may provide images of n multiple views having different views by simultaneously photographing the area to be monitored from various directions or angles. For example, the camera 100 may be a CCTV camera, but is not limited thereto.

For example, each camera 100 may be implemented as a rotating camera. In this case, the rotation camera may be a Pan-Tilt-Zoom (PTZ) camera capable of pan, tilt, and zoom functions. In this case, the pan function refers to a function in which the camera 100 rotates along a first direction (eg, a horizontal direction), and the tilt function refers to a function in which the camera 100 moves in a second direction (eg, a different direction from the first direction). It refers to the function of rotational movement along the vertical direction). In addition, the zoom function refers to a function in which the camera 100 enlarges and captures an image based on the central region of the currently captured image with respect to the monitoring target region. That is, the n rotation cameras can provide multiple views by photographing the area to be monitored in various directions or angles.

Each camera 100 transmits the captured image information to the behavior recognition device 200 through a wired or wireless transmission path. Also, the camera 100 may transmit the captured image information to the behavior recognition apparatus 200 through a network such as the Internet. For example, the camera 100 may be a camera to which an IP address is not assigned on a network or an IP camera to which an IP address is assigned, but is not limited thereto.

On the other hand, since each camera 100 captures a monitoring target area in different directions or angles, images captured by each camera 100 may have different views, that is, a multi-view. . According to the number of cameras 100, n images of these different views may be simultaneously captured. That is, the first image from the first camera 100 , the second image from the second camera 200 , and the nth image from the n-th camera 100 may be photographed, respectively, and these images are time-series continuous image frames. It may be a video having information about .

In each image, a 'point' may refer to any one pixel in an image frame, and a 'region' may refer to a plurality of contiguous points (ie, a plurality of pixels) within an image frame. For example, in an image, a 'center point' of a certain area may refer to a pixel located at the center of the corresponding area. Also, in an image, a 'center coordinate point' may refer to a pixel located at the center of a frame of the image, and the 'center region' may be an area including the central coordinate point.

The behavior recognition apparatus 200 is a device that receives and processes image information captured by the camera 100 , and controls recognition of various actions of objects or events in the image. That is, the behavior recognition apparatus 200 may display the received image information for monitoring or the like through the display. Also, the behavior recognition apparatus 200 may perform learning and behavior recognition control for n images captured by each camera 100 according to a behavior recognition method to be described later.

In addition, the behavior recognition apparatus 200 may analyze the received image information. For example, the behavior recognition device 200 recognizes an object or event to be monitored, such as a person or a vehicle, from the received image information, and the recognized behavior is an abnormal behavior (eg, parking violation, signal violation, intrusion, fire, violence, theft, speculation, etc.), information on this may be delivered to the user through various warning devices.

That is, the behavior recognition device 200 is a device that performs control processing on the camera 100 or control processing on image information captured by the camera 100, and may be an electronic device or a computing network capable of computing. have.

For example, the electronic device includes a desktop personal computer (PC), a laptop personal computer (PC), a tablet personal computer (PC), a netbook computer, a workstation, and a personal digital assistant (PDA). , a smartphone (smartphone), a smart pad (smartpad), or a mobile phone (mobile phone), etc., but is not limited thereto.

2 is a block diagram showing a behavior recognition apparatus 200 according to an embodiment of the present invention.

As shown in FIG. 2 , the behavior recognition apparatus 200 may include an input unit 210 , a communication unit 220 , a display 230 , a memory 240 , and a control unit 250 .

The input unit 210 generates input data in response to input of various users (monitor, etc.), and may include various input means. For example, the input unit 210 may include a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), a menu button (menu button) and the like may be included, but is not limited thereto. For example, the input unit 210 may receive commands for various settings.

The communication unit 220 is configured to communicate with other devices. For example, the communication unit 220 is 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energe (BLE), near field communication (NFC), Wireless communication such as Wi-Fi communication may be performed or wired communication such as cable communication may be performed, but is not limited thereto. For example, the communication unit 220 may receive image information captured by the camera 100 from each camera 100 , and receive information on pre-learned learning models (each classifier and ensemble model) from other devices. can do. In addition, the communication unit 220 may transmit image information, information on the learned learning model, an action recognition result, and the like to another device.

The display 230 displays various image data on a screen, and may be configured as a non-emission panel or a light emitting panel. For example, the display 230 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a micro electromechanical system (MEMS). mechanical systems) display, or an electronic paper display, etc., but is not limited thereto. In addition, the display 230 may be implemented as a touch screen or the like in combination with the input unit 210 . For example, the display 230 may display image information captured by the camera 100 , setting information, an action recognition result, and the like on a screen.

The memory 240 stores various types of information necessary for the operation of the behavior recognition apparatus 200 . The stored information may include image information captured by the camera 100, setting information, information about a learning model, information about an ensemble technique, a behavior recognition result, and program information related to a behavior recognition method to be described later, but is limited thereto. it is not going to be For example, the memory 240 may be a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type depending on the type. ), a Sagneto-optical media type, a multimedia card micro type, a flash memory type, a read only memory type, or a random access memory type), but is not limited thereto. In addition, the memory 240 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system according to its purpose/location, but is not limited thereto.

The controller 250 may perform various control operations of the behavior recognition apparatus 200 . That is, the controller 250 may control the execution of a behavior recognition method to be described later, and the remaining components of the behavior recognition apparatus 200 , that is, the input unit 210 , the communication unit 220 , the display 230 , and the memory 240 . You can control actions such as For example, the control unit 250 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in a corresponding processor.

3 is a block diagram showing the control unit 250 in the behavior recognition apparatus 200 according to an embodiment of the present invention.

The control unit 250 controls the execution of the behavior recognition method according to an embodiment of the present invention, and as shown in FIG. 3 , the learning unit 251 , the calibration unit 252 , the classifier processing unit 253 , and the ensemble processing unit (254). For example, the learning unit 251 , the calibration unit 252 , the classifier processing unit 253 , and the ensemble processing unit 254 may be a hardware configuration of the control unit 250 or a software process performed by the control unit 250 . , but is not limited thereto.

Hereinafter, the behavior recognition method according to the present invention will be described in more detail.

4 is a flowchart of a behavior recognition method according to an embodiment of the present invention, and FIG. 5 is a detailed flowchart of S100 and S200 of the behavior recognition method according to an embodiment of the present invention.

A method for recognizing an action according to an embodiment of the present invention is a method for recognizing an action of an object or an event in an image by using a video shot simultaneously from n cameras 100 in n different views. As shown in FIG. 4 , S100 and S200 are included.

S100 is a step of learning the learning model (each classifier, ensemble model) using the training data. In this case, the classifier is a learning model that is learned according to machine learning and performs a classification function, and may be provided with n equal to the number of images having different views. This is because learning is performed using images of different views for each classifier, and classification for behavior recognition is performed using images of different views for each learned classifier. That is, the first classifier may be trained by using the first image, the second classifier using the second image, and the nth classifier using the nth image, respectively. In addition, the learned first classifier may perform classification for behavior recognition by using the first image, the learned second classifier using the second image, and the learned n-th classifier using the n-th image, respectively.

However, each image used in S100 is an image prepared for learning (ie, an image for learning data), and is included in input data of the learning data. In addition, the output data of the training data includes a label for an action type of an object or event in the image. A detailed description thereof will be provided later.

Specifically, as shown in FIG. 5 , S100 may include S101 to S105.

First, in S101, n cameras 100 for photographing a monitoring target area in various directions or angles are installed at a plurality of points. For example, n rotation cameras may be installed at a plurality of points.

6 shows an example of an image captured by each camera 100 of the system 10 according to an embodiment of the present invention, and FIG. 7 is a result of performing calibration between each camera 100 in FIG. An example of the following image is shown.

Thereafter, in S102 , the calibration unit 252 performs calibration between each camera 100 . In this case, the calibration refers to an operation of adjusting the shooting direction of each camera 100 so that an object or an event in each image is located within the same specific area. That is, the calibrator 252 performs a pan or tilt function for each camera 100 so that in n images of different views captured by each camera 100 , an object or event is located in a specific area within the image. , calibration may be performed between each camera 100 .

For example, the specific region may be a central region within the image frame, but is not limited thereto. Through such calibration, it is possible to maintain consistency with respect to the location of an object or event of each image included in the training data, thereby improving learning efficiency and thus increasing the accuracy of classification.

For example, as shown in FIG. 6 , the monitoring target area including the object areas OR1 , OR2 , OR3 including the object O has three rotating cameras 100-1, 100-2, 100-3. may be implemented to photograph. In this case, each center point of OR1, OR2, and OR3 may be referred to as CP1, CP2, and CP3. In particular, in the first image, OR1 ( CP1 ) is located in the central region of the image, but in the second and third images, OR2 and OR3 ( CP2 and CP3 ) are located in regions deviating from the central region in the image, respectively.

Accordingly, as shown in FIG. 7 , a calibration in which the shooting direction of the first rotating camera 100-1 is left as it is and the shooting directions of the second and third rotating cameras 100-2 and 100-3 are adjusted. can be performed. As a result of this calibration, OR2 and OR3 (CP2 and CP3) may be displaced to OR2' and OR3' (CP2' and CP3'), respectively. That is, OR2' and OR3' (CP2' and CP3') may be respectively located in the central region in the image, and in particular, the moved positions CP2' and CP3' may correspond to the central coordinate points in the image, respectively.

Thereafter, in S103 , the learning unit 251 acquires images of n different views. That is, instead of an image of a single view, images of n different views captured by each camera unit 100 may be acquired.

Thereafter, in S104 , the learning unit 251 generates a GT (Ground-Truth) of training data for machine learning learning by using the acquired image. In this case, the training data includes a pair of input data and output data. The input data includes an image of another view acquired, and the result data is a label (ie, action name) for an action type of an object or event in the image. ) is included. Hereinafter, the training data for the k-th classifier (where k is a natural number of 3 or more and n or less) is referred to as “k-th training data”, and the input data and output data of the k-th training data are referred to as “k-th input data” and It is referred to as “k-th output data”.

For example, the first to nth result data may include, but are not limited to, labels of types of actions for parking violations, signal violations, trespassing, fire, violence, theft, or speculation.

In particular, in each training data, it is preferable that the first to nth input data include images of different views. That is, the first input data may include a first image, the second input data may include a second image, and the nth input data may include an nth image.

In addition, in each learning data, input data includes, in addition to images of different views, information on frames in which objects or events appear in the images (hereinafter referred to as “first frames”), and objects that appeared in the images. Alternatively, information on a frame in which the event disappears (hereinafter, referred to as a “second frame”) may be further included. Since each image is a moving picture having information on continuous image frames in a time series, time information about the action of the object or event in the video (that is, information about the start time and end time of the action) is obtained from the learning data. This is to provide additional input data. In this case, the first input data may include the first image, information on the first frame in the first image, and information on the second frame in the first image, respectively. In addition, the second input data may include the second image, information on the first frame in the second image, and information on the second frame in the second image, respectively. Also, the n-th input data may include an n-th image, information on a first frame in the n-th image, and information on a second frame in the n-th image, respectively.

Thereafter, in S105 , the learning unit 251 learns n classifiers using the prepared training data. That is, the learning unit 251 may learn n classifiers according to a machine learning technique using the training data generated based on n images of different views captured by each camera 100 on which calibration has been performed. . In this case, the classifier may be learned according to a machine learning technique of supervised learning.

For example, the machine learning technique may include, but is not limited to, an artificial neural network, a decision tree, a Gaussian process regression, a support vector machine, random forests, or deep learning. In addition, deep learning techniques can include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, etc. However, the present invention is not limited thereto.

The classifier may include a plurality of layers and have a function on the relationship between input data and output data through learning. That is, the classifier expresses the relationship between input data and output data in multiple layers (ie, layers), and these multiple representation layers are also referred to as “neural network”. When input data is input to the learned classifier, output data according to a corresponding expression layer may be output. Each layer in the neural network consists of at least one filter, and each filter may have a matrix of weights.

Also, in S105 , after learning the classifier, the learning unit 251 may additionally train an ensemble model, which is a learning model according to an ensemble of classifiers. In this case, the ensemble technique is a technique for deriving at least one behavior type from among the n predicted behavior types as a final result using the result data output (predicted) by each classifier.

For example, the ensemble technique includes a first method based on majority vote/voting, such as voting, bagging, pasting, random forest, etc., and Ada Boost, gradient boost, etc. It may be a second method based on boosting such as (Gradient Boost), and the first method may be preferable, but is not limited thereto.

That is, in the ensemble model, the output data of each classifier may be included in the input data, and the label of the action type that the user wants to classify as the final result for the object or event in the corresponding image may be included in the output data.

As a result of performing S100, a learning model capable of performing behavior recognition according to an ensemble technique may be trained based on the results of a plurality of classifiers for each image captured from three or more different views.

However, the above-described S100 is not performed in the behavior recognition apparatus 200, but may be performed in another apparatus. In this case, the above-described S100 is performed by the control unit of the other device, and the learning model (each classifier, ensemble model) learned according to the execution result may be transmitted to the behavior recognition device 200 through the communication unit 220 . .

8 illustrates a behavior recognition process performed according to S200 of the behavior recognition method according to an embodiment of the present invention.

Next, S200 is a step of performing behavior recognition using the learned learning model, that is, each classifier and ensemble model.

Specifically, as shown in FIGS. 5 and 8 , S200 may include S201 to S203.

First, in S201 , the classifier processing unit 253 obtains an output (prediction) for n behavior types by using each of the learned n classifiers. That is, by inputting images of n different views of the monitoring target region to each learned classifier, n behavior types output by each classifier can be obtained. In this case, each image is an actual surveillance image acquired in real time by each camera 100 , and a label is not added unlike the image of the learning data.

That is, by inputting the first image to the learned first classifier, the first output for the action type may be obtained. Also, by inputting the second image to the learned second classifier, a second output for the action type may be obtained. In addition, an n-th output for the action type may be obtained by inputting an n-th image to the learned n-th classifier.

However, the images of n different views input to each classifier may be images after calibration of each camera 100 is performed, but is not limited thereto. In addition, the calibration of each camera 100 may be performed through the control unit 250 , ie, the calibration unit 252 , as in S102 described above.

Meanwhile, the classifier processing unit 253 may additionally input time information about an action of an object or event (ie, information about a start time and an end time of the corresponding action) to each classifier in addition to the image information. In this case, the classifier processing unit 253 may derive time information about a frame in which the object or event appears and disappears by utilizing various object/event extraction algorithms for extracting an object or event from an image.

Thereafter, in S202 , the ensemble processing unit 254 derives at least one behavior type among n behavior types obtained from each classifier as a result by using an ensemble of classifiers technique. That is, by inputting the output of each classifier to the learned ensemble model, at least one action type may be finally determined as an action on an object or event in n images.

Thereafter, in S203 , the controller 250 may control to display the result of the action recognition according to S201 or S202 on the display 230 . Also, the control unit 250 may transmit the result of recognition of the action to another device through the communication unit 220 .

In addition, after performing S200, when the recognized behavior is an abnormal behavior (eg, parking violation, signal violation, intrusion, fire, violence, theft, speculation, etc.), the control unit 250 provides information about this to various warning devices. It can also be delivered to the user through

The present invention configured as described above has an advantage in that it is possible to process behavior recognition for a plurality of images rather than a single image. That is, according to the present invention, an action on an object or event in the image is analyzed using a learning model of a machine learning technique for a video shot simultaneously in three or more multi-views of a target region. Since recognition can be performed more quickly and accurately, there is an advantage in that it is possible to respond to a real-time environment. In addition, the present invention has the advantage of being able to further improve the accuracy of behavior recognition by processing a plurality of images at the same time, and finally analyzing the behavior recognition through the ensemble technique of the analysis result of the learning model of the machine learning technique. In addition, when the present invention is applied to a CCTV system such as a video security system used in public institutions or private companies, there is an advantage in that it is possible to prevent crime, improve the arrest rate of criminals, and guarantee personal safety.

In the detailed description of the present invention, although specific embodiments have been described, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and their equivalents.

10: CCTV system 100: camera unit
110: fixed camera 120: rotating camera
200: CCTV operating device 210: input unit
220: communication unit 230: display
240: memory 250: control unit
251: learning unit 252: calibration unit
253: classifier processing unit 254: ensemble processing unit

Claims (9)

As a method of recognizing an action of an object or event using a video captured by an electronic device and simultaneously n (where n is a natural number of 3 or more) different views,
learning n classifiers according to a machine learning technique using input data for images of different views and learning data including result data for behavior types of objects or events in the images, respectively;
inputting images of n different views of the target region to each learned classifier to obtain n behavior types output by each classifier; and
deriving at least one behavior type among n behavior types obtained from each classifier as a result by using an ensemble of classifiers;
How to include.
According to claim 1,
In the learning step, the input data for each of the n classifiers includes images of different views,
In the acquiring, images of different views of the target region are input to each classifier.
3. The method of claim 2,
The learning step is
performing calibration between each camera so that an object or event is located in a specific area in the image in n images of different views captured by each camera; and
learning the n classifiers using n images of different views captured by each camera on which calibration has been performed;
How to include.
4. The method of claim 3,
The specific region corresponds to a central region in the image.
3. The method of claim 2,
In the learning step, the input data for each of the n classifiers includes images of different views, information on frames in which objects or events appear in the images, and objects or events that appear in the image disappear. How to include information in each frame.
Simultaneously, as a device for recognizing an action of an object or event using a video shot with n different views (where n is a natural number of 3 or more),
A memory for storing n classifiers pre-learned according to machine learning using learning data including input data for images of different views and result data for behavior types of objects or events in the image, respectively ; and
A control unit for controlling the recognition of an object or an event in the image of n different views of the target region by using each classifier stored in the memory;
The control unit is
Images of n different views of the target region are input to each classifier to obtain n behavior types output by each classifier, and n behavior types obtained from each classifier using an ensemble of classifiers A device for deriving at least one kind of behavior as a result.
Simultaneously, as a device for recognizing an action of an object or event using a video shot with n different views (where n is a natural number of 3 or more),
A communication unit for receiving n classifiers pre-learned according to machine learning by using input data for images of different views and learning data including result data for action types of objects or events in the image, respectively ; and
A control unit for controlling the recognition of an action for an object or an event in the image of n different views of the target area by using each classifier received from the communication unit;
The control unit is
Images of n different views of the target region are input to each classifier to obtain n behavior types output by each classifier, and n behavior types obtained from each classifier using an ensemble of classifiers A device for deriving at least one kind of behavior as a result.
a plurality of cameras simultaneously capturing n (provided that n is a natural number of 3 or more) different views; and
An electronic device for recognizing an action of an object or event by using an image of a different view captured by each camera;
The electronic device is
A memory for storing n classifiers pre-learned according to machine learning using learning data including input data for images of different views and result data for behavior types of objects or events in the image, respectively ; and
A control unit for controlling the recognition of an action for an object or an event in the image of n different views of the target area by using each classifier received from the communication unit;
The control unit is
Images of n different views of the target region are input to each classifier to obtain n behavior types output by each classifier, and n behavior types obtained from each classifier using an ensemble of classifiers A system that derives at least one kind of behavior as a result.
9. The method of claim 8,
The camera is a CCTV camera system.

Images