KR101876433B1 - Activity recognition-based automatic resolution adjustment camera system, activity recognition-based automatic resolution adjustment method and automatic activity recognition method of camera system - Google Patents

Abstract

The present invention relates to a technology for recognizing a human activity in a low resolution image and a camera system for automatically adjusting a photographing resolution of a camera based on the technology. An activity recognition-based automatic resolution adjustment camera system comprises: a data server for generating learning data which is basic to recognize human activity in a low resolution image based on a high resolution learning image, and generating a human activity recognition model from the learning data; and an activity recognition-based automatic resolution adjustment camera for photographing a low resolution image and photographing an image by changing a conventional resolution mode to a higher resolution mode when it is determined to be an emergency situation through a first activity recognition for recognizing a human activity in the low resolution image by using the human activity recognition model transmitted from the data server. Therefore, according to the present invention, it is possible to protect privacy of a subject, reduce battery consumption of the camera, and to efficiently utilize a storage space.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and apparatus for automatically recognizing a motion in a video camera, and a method for automatically recognizing a motion in a video camera, OF CAMERA SYSTEM}

The present invention relates to a technique for recognizing human behavior in a low-resolution image, and a camera and a system for automatically adjusting a shooting resolution of the camera based on the technology. It uses computer vision technology to recognize human behavior in very low resolution images and adjusts the camera's shooting and storage resolution according to its contents.

Infringement of the privacy of the subject by consecutive shooting of high resolution images without consent has emerged as an important social issue. High-resolution cameras, including robotic cameras and wearable cameras, are increasingly used both in public and private places. As a result, people are worried that important personal information will be recorded without consent. For example, if a home (home security or smart home service) camera is hacked, there is a risk that your personal life will be watched or recorded by someone else throughout the 24 hours. In addition, high-resolution cameras such as a robot camera and a wearable camera have a problem that the battery consumption is large due to continuous high-resolution image capturing and the storage space is required, which makes it difficult to use for a long time.

 Therefore, it is necessary to develop a secure camera device from the privacy invasion problem, and to design a privacy vision computer vision algorithm which makes it possible. In other words, there is a need to recognize important events and human behaviors in the image, and to prevent random high-resolution image capturing based on the recognition.

In response to these societal demands, there have been recent attempts to recognize human behavior from low-resolution images. However, most of the previous researches have limited in that they do not consider the intrinsic properties of low-resolution sensors. Referring to FIG. 6, when converting a high-resolution image to a low-resolution image, images originating from exactly the same scene often have completely different pixel (i.e., RGB) values due to inherent limitations that a single pixel can capture from a scene . With low-resolution transforms, even a plurality of low-resolution images originating from exactly the same scene can be completely different visual data. In other words, most of the previous studies did not take into account the characteristics of these low resolution transformations.

In addition, there is a problem that the technique of recognizing the human motion based on the conventional convolution neural network (CNN) is not suitable for the low resolution image because it does not consider the behavior recognition from the low resolution image.

Japanese Patent Application Laid-Open No. 10-2017-0028591

The present invention relates to a behavior recognition based resolution automatic adjustment camera system which shoots a low resolution image in a normal state in order to protect the privacy of a subject and then takes a high resolution image when it is determined through a behavior recognition technology for a low resolution image, An automatic resolution control method, and a camera system automatic in-image behavior recognition method.

The present invention relates to a behavior recognition-based resolution automatic camera system that can reduce the camera battery consumption by automatically adjusting the shooting resolution of the camera according to the importance of the situation through the low-resolution image behavior recognition technology, And an automatic adjustment method.

In the present invention, a plurality of low-resolution images transformed from a high-resolution image are mapped to an embedding space in consideration of the characteristics of a low-resolution image conversion, thereby learning a convolutional neural network, And a method for automatically recognizing a behavior in a video camera system capable of increasing the recognition rate.

The technical problem to be solved by the present invention is not limited to the above-mentioned technical problems, and various technical problems can be included within the scope of what is well known to a person skilled in the art from the following description.

According to an aspect of the present invention, there is provided an automatic recognition-based resolution automatic camera system, which generates learning data based on recognition of human behavior in a low-resolution image based on a high-resolution learning image, A data server for generating a behavior recognition model; Resolution image, and when a crisis is detected through the first behavior recognition that recognizes human behavior in a low-resolution image using the human behavior recognition model transmitted from the data server, the image is changed to a high- And a behavior recognition based resolution automatic adjustment camera for photographing.

In one embodiment, the behavior recognition-based resolution automatic adjustment camera may further include a second behavior recognition unit that recognizes human behavior in the image using the human behavior recognition model for an image photographed in the high- In case of a crisis, it is possible to store a high resolution image, and in case of a crisis, a low resolution image can be stored.

In one embodiment, the behavior recognition-based resolution automatic adjustment camera includes a photographing unit for photographing a low-resolution image; A communication unit for receiving a human behavior recognition model used for recognizing human behavior in the low resolution image from the data server and transmitting the received human behavior recognition model to the image analysis unit; An image analyzing unit for determining whether the user is in a crisis state through a first behavior recognition process for recognizing a human behavior in a low resolution image using the human behavior recognition model received from the communication unit; A control unit for controlling the photographing unit to change to a higher resolution mode than the existing resolution and to photograph the photographing unit if a crisis state is determined by the image analysis unit; And a storage unit for storing a low-resolution image or a high-resolution image photographed by the photographing unit.

In one embodiment, the image analyzing unit may determine whether the image is a crisis state through a second behavior recognition that recognizes human behavior in the image using the human behavior recognition model for the image photographed in the high resolution mode .

In one embodiment, when the situation determined through the second behavior recognition of the image analysis unit corresponds to a crisis situation, the control unit stores an image photographed in high resolution in the storage unit, and if the situation is not a crisis, So that the photographed image is stored in the storage unit.

In one embodiment, the data server includes: a resolution converter for converting a high-resolution human behavior image into a plurality of low-resolution images; A plurality of low resolution images generated by the resolution conversion unit are received and divided into a spatial stream and a time stream, and convolution and pooling are performed for each stream, and a fully connected layer is added A convolution neural network learning unit for learning a convolutional neural network; And a human behavior recognition model generation unit for generating a human behavior recognition model based on data learned by the convolution neural network learning unit.

In one embodiment, the resolution conversion unit converts a first batch composed of n low-resolution images converted from the same high-resolution image and a plurality of different high-resolution images into n Resolution images can be generated as learning data and transmitted to the convolution neural network learning unit.

In one embodiment, the convolutional neural network learning unit receives the first and second arrangements and divides each of the images into a spatial stream and a temporal stream, and performs convolution and pooling for each stream, , And applying a fully connected layer to the embedding space and then adjusting the embedding distance by mapping the embedding space to the embedding space.

According to an embodiment of the present invention, there is provided a method for automatically adjusting a resolution based on a behavior recognition, the method comprising: (a) (b) receiving a human behavior recognition model used by the communication unit of the camera for recognizing human behavior in a low-resolution image from an external data server and transmitting the human behavior recognition model to an image analysis unit of the camera; (c) determining whether the image analyzing unit is in a crisis state through a first behavior recognition that recognizes human behavior in a low-resolution image using the human behavior recognition model received from the camera communication unit; (d) when the image analyzing unit determines that the image capturing unit is in a crisis state, the control unit of the camera controls the image capturing unit to change the image capturing unit to a higher resolution mode than the existing resolution image capturing unit; And (e) storing a low-resolution image or a high-resolution image captured by the photographing unit in a storage unit of the camera.

In one embodiment, a step (b) is performed between steps (d) and (e), wherein the image analyzing unit changes the mode to the high resolution mode, 2, a step of judging whether or not the situation is a crisis through behavior recognition;

In one embodiment, in the step (e), when the situation determined through the second behavior recognition of the image analysis unit corresponds to a crisis situation, the image photographed at the high resolution is stored in the storage unit, The image photographed at the low resolution may be stored in the storage unit.

A method for automatically recognizing in-vivo behavior of a camera system according to an embodiment of the present invention includes the steps of: (a) converting a high-resolution human behavior image into a plurality of low-resolution images by a resolution conversion unit of a data server; (b) a convolutional neural network learning unit of the data server receives the plurality of low-resolution images and divides them into a spatial stream and a time stream, performs convolution and pooling for each stream, layer is further applied to learn a Convolutional Neural Network; (c) generating a human behavior recognition model based on data learned by the convolutional neural network learning unit; (d) photographing a low-resolution image of a photographing part of the camera; And (e) recognizing human behavior in the low-resolution image photographed by the photographing unit using the human behavior recognition model received from the data server by the image analysis unit of the camera.

In one embodiment, the step (a) may include: converting a resolution conversion unit of the data server from the same high-resolution image to generate a first batch consisting of n low-resolution images having different resolutions and a plurality of different high- And generating a second batch composed of n low-resolution images having different resolutions, as training data, and transmitting the generated training data to the convolution neural network learning unit.

In one embodiment, in the step (b), the convolutional neural network learning unit of the camera receives the first and second arrangements and divides each of the images into a spatial stream and a temporal stream, Conversation and pooling are performed, and a fully connected layer is further applied. Then, the embedding distance is adjusted by mapping to an embedding space, thereby learning for motion recognition in an image And < / RTI >

The motion recognition-based resolution automatic adjustment camera of the present invention, its system, and resolution adjusting method shoot a low-resolution image at normal times, and when a dangerous situation is determined through a behavior recognition technology for a low-resolution image, Privacy protection can be achieved.

The automatic recognition-based resolution automatic camera and its system and resolution adjustment method of the present invention can reduce the camera battery consumption by automatically adjusting the shooting resolution of the camera according to the importance of the situation through the low resolution image behavior recognition technology, .

The method for automatically recognizing behavior in a low-resolution image of the present invention learns a convolutional neural network by mapping a plurality of low-resolution images transformed from a high-resolution image into an embedding space in consideration of characteristics of a low-resolution image transformation , The human motion recognition rate in the photographed low-resolution image can be increased.

FIG. 1 is a block diagram illustrating a configuration of a behavior recognition-based automatic resolution camera system according to an exemplary embodiment of the present invention. Referring to FIG.
FIG. 2 is a view for explaining an operation concept of an automatic recognition-based resolution automatic camera system according to an embodiment of the present invention.
FIG. 3 is a block diagram illustrating a configuration of a camera for automatic recognition of a resolution-based resolution according to an exemplary embodiment of the present invention.
FIG. 4 is a flowchart illustrating a flow of operation of a camera for automatically recognizing a resolution-based resolution according to an exemplary embodiment of the present invention.
FIG. 5 is a diagram illustrating a detailed configuration of a data server in a configuration of an automatic recognition-based resolution camera system according to an exemplary embodiment of the present invention.
FIG. 6 is a diagram for explaining that even when a high-resolution image is converted to a low-resolution image, a low-resolution image derived from the same high-resolution image may have a different value.
FIG. 7 is a view showing a structured manner in which a convolutional neural network learning unit according to an embodiment of the present invention divides an input image into a spatial stream and a temporal stream.
8 is a diagram illustrating a structure of a two-stream convolutional neural network applied to convolutional neural network learning according to an embodiment of the present invention.
9 is a diagram for explaining how the convolutional neural network learning unit according to an embodiment of the present invention performs learning for intra-image behavior recognition using an embedding space.
10 is a diagram illustrating a structure of a multi-siamese convolutional neural network applied to convolutional neural network learning according to an embodiment of the present invention.
FIG. 11 is a flowchart illustrating a step of learning a convolutional neural network among methods of automatically recognizing behaviors in an image of a camera based on a resolution-based automatic resolution control according to an exemplary embodiment of the present invention.
FIG. 12 is a view comparing the accuracy of the behavior recognition between the case where the method of automatically detecting the behavior in the camera of the camera system according to the embodiment of the present invention and the accuracy of the behavior recognition between the prior art.

DETAILED DESCRIPTION OF THE EMBODIMENTS Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

The embodiments disclosed herein should not be construed or interpreted as limiting the scope of the present invention. It will be apparent to those of ordinary skill in the art that the description including the embodiments of the present specification has various applications. Accordingly, any embodiment described in the Detailed Description of the Invention is illustrative for a better understanding of the invention and is not intended to limit the scope of the invention to embodiments.

The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the following detailed description. Also, although one or more functional blocks of the present invention are represented as discrete blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

In addition, the expression "including any element" is merely an expression of an open-ended expression, and is not to be construed as excluding the additional elements.

Further, when a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it should be understood that there may be other components in between.

Also, the expressions such as 'first, second', etc. are used only to distinguish a plurality of configurations, and do not limit the order or other features between configurations.

FIG. 1 is a diagram illustrating a configuration of an automatic recognition-based resolution camera system according to an exemplary embodiment of the present invention. FIG. 2 illustrates an operation concept of an automatic recognition-based resolution automatic camera system according to an exemplary embodiment of the present invention FIG.

Referring to FIG. 1, the behavior recognition-based automatic resolution camera system 1 of the present invention includes a behavior recognition-based resolution automatic adjustment camera 10 and a data server 20. The behavior recognition-based resolution automatic adjustment camera 10 photographs a subject, the data server 20 receives a high-resolution image, performs learning for human behavior recognition, and recognizes human behavior in a low-resolution image based on the learned data To generate a human behavior recognition model. The motion recognition based resolution control camera 10 of the present invention may include various camera devices. For example, it can be a smart home camera, a robot camera, a security CCTV, a wearable camera, a car black box camera, and the like.

The data server 20 can perform learning for recognizing human behavior in a low-resolution image from a high-resolution image stored on-line or a high-resolution image on-line. The data server 20 can generate learning data by learning a convolutional neural network, and can generate a human behavior recognition model based on the generated learning data. Detailed description of the detailed configuration of the data server 20 and a technique for recognizing human behavior in an image will be described below with reference to FIGS. 5 to 10. FIG.

Referring to FIG. 2, the automatic recognition-based resolution automatic resolution camera 10 (hereinafter referred to as 'camera') shoots a low-resolution moving image to prevent privacy invasion of a subject. The camera 10 of the present invention recognizes human behavior in a low-resolution image (for example, 16x12 pixels) by applying a behavior recognition technique to the photographed image. The camera 10 can pick up an image of a high resolution (for example, 4K pixels) by selecting an optimal camera resolution according to a result of behavior recognition, and can store the captured image in a memory or transmit it to the data server 20. Therefore, the behavior recognition-based resolution automatic adjustment camera 10 of the present invention can solve the problem that the privacy of the subject can be violated by continuously photographing a high resolution image even in a conventional camera or the like. In addition, the present invention reduces battery consumption of the camera 10 and enables efficient use of the storage space. The concrete operation of the behavior recognition based resolution automatic adjustment camera 10 of the present invention will be described with reference to FIG. 3 to FIG.

FIG. 3 is a block diagram illustrating a configuration of a camera for automatic recognition of a resolution-based resolution according to an exemplary embodiment of the present invention.

3, the behavior recognition-based automatic resolution camera 10 may include a photographing unit 100, a communication unit 200, an image analysis unit 300, a storage unit 400, and a control unit 500 .

When the control unit 500 receives a control signal for changing from the control unit 500 to the high-resolution image shooting mode while shooting a subject at a low resolution, the shooting unit 100 can shoot a subject at a high resolution.

The communication unit 200 receives the human behavior recognition model used for recognizing human behavior in the low resolution image from the data server 20 and transmits the received human behavior recognition model to the image analysis unit 300. The data server 20 can perform learning to recognize human behavior in a low-resolution image from a high-resolution image stored on-line or a high-resolution image on-line. The high-resolution image is used to learn a Convolutional Neural Network to perform human behavior recognition in a low-resolution image photographed by the photographing unit 100. [ A high-resolution image as a learning image may be a publicly available source. A publicly available source can be, for example, an open source such as YouTube. In addition, high-resolution images can be obtained from sources that publicly provide video online. The communication unit 200 may transmit the image stored in the storage unit 400 to the data server 20.

The communication unit 200 may include at least one of a wireless communication module and a wired communication module. The wireless communication module may include a wireless network communication module, a wireless local area network (WLAN), a wireless communication module such as a Wi-Fi, a wireless fidelity or WiMAX, and a wireless personal area network (WPAN) Or the like.

The image analysis unit 300 can receive the human behavior recognition model from the communication unit 200 and recognize the human behavior in the low resolution image photographed by the photographing unit 100. [ The image analyzing unit 300 can recognize the human behavior in the low-resolution image photographed by the photographing unit 100 using the human behavior recognition model, and judge whether or not the recognized human behavior corresponds to the crisis situation recognition).

The storage unit 400 may store a low-resolution image or a high-resolution image photographed by the photographing unit 100. The storage unit 400 stores images (weapons, naked images, drugs, blood, etc.) to be recognized or analyzed by the image or image analysis unit 300 taken by the image sensing unit 100 as a dangerous situation, And may be in the form of a flash memory for providing images in a real-time streaming manner.

To this end, the storage unit 400 includes a main storage unit and an auxiliary storage unit, and stores an application program necessary for the functional operation of the action recognition-based resolution automatic adjustment camera 10. The storage unit 400 may include a program area and a data area. The program area is an area for storing an operating system or the like for booting the automatic recognition-based resolution automatic camera 10, and the data area is an area for storing an image photographed by the photographing part 100.

The control unit 500 may control the photographing unit 100, the communication unit 200, the image analysis unit 300, and the storage unit 400. The control unit 500 can control the photographing unit 100 to change the mode to a higher resolution mode than the existing resolution and take a picture if the crisis is determined by the image analysis unit 300. [ This is to maximize the privacy of the photographed subject and to minimize the battery consumption of the camera 10 and the utilization of the storage space. That is, the normal photographing unit 100 photographs with a low resolution image (ex. 16x12 pixels), and when receiving the control signal, it can photograph with a high resolution image (ex. 720x480, 1280x720, 1920x1080, 4K, etc.). The control unit 500 may control the photographing unit 100 to operate at a predetermined cycle to minimize battery consumption.

FIG. 4 is a flowchart illustrating a flow of operation of a camera for automatically recognizing a resolution-based resolution according to an exemplary embodiment of the present invention.

Referring to FIG. 4, the photographing unit 100 photographs a normal low-resolution image and transmits the photographed image to the image analysis unit 300 (S210). The image analyzing unit 300 determines whether the user is in a crisis state by recognizing a first action in the low resolution image using the human behavior recognition model received from the communication unit 200 at step S220. The control unit 500 transmits a control signal so that the low resolution image is stored in the storage unit 400 when the image analysis unit 300 determines that the human action in the low resolution image does not correspond to the crisis situation.

If the human behavior in the low-resolution image is determined to be a crisis by the image analysis unit 300, the control unit 500 transmits a control signal to the imaging unit 100 to capture the subject as a high-resolution image (S240). The image analyzing unit 300 determines whether a high-resolution image photographed by the photographing unit 100 is in a crisis state through a second behavior recognition that recognizes human behavior in the image using the human behavior recognition model at step S250. When the image analysis unit 300 determines that the human action in the high-resolution image is a crisis state, the control unit 500 transmits a control signal to store the high-resolution image in the storage unit 400, A control signal is transmitted so that an image photographed at a low resolution is stored in the storage unit 400 (S260).

As described above, the behavior recognition-based resolution automatic adjustment camera 10 of the present invention performs two human behavior recognition on the photographed image to confirm whether the human behavior recognition is correct, and can finally store the photographed image. Therefore, it is possible to reduce the error of the behavior recognition, to protect the privacy of the object and to appropriately use the storage space.

FIG. 5 is a diagram illustrating a detailed configuration of a data server in a configuration of an automatic recognition-based resolution camera system according to an exemplary embodiment of the present invention.

5, the data server 20 may include a resolution conversion unit 21, a convolutional neural network learning unit 22, and a human behavior recognition model generation unit 23.

The resolution conversion unit 21 can convert the high-resolution human behavior image received from the communication unit 200 into a plurality of low-resolution images. Multiple low-resolution images can be obtained using a technique called Inverse Super Resolution (ISR). The main motivation of Inverse Super Resolution (ISR) is that a high resolution image contains a quantity of information corresponding to a low resolution image set, and a behavior recognition method can be performed by applying another low resolution image transformation to a single high resolution image. (M. S. Ryoo, B. Rothrock, C. Fleming, and H. J. Yang. Privacy-preserving human activity recognition from extreme low resolution. In AAAI, 2017.)

Resolution conversion section 21 generates n low-resolution images (i.e., V _ik ) for each high-resolution training image X _i by applying transform sets F _k and D _k as shown in the following equation (1) .

Where _Fk is the camera motion transformation and _Dk is the downsampling operator. Although F _k can be any affine transformation, the present invention regards the combination of transformation, scaling, and rotation as motion transformation. Also, standard mean downsampling is used for D _k . A sufficient number of low resolution transformed images V _ik generated from the learning samples X _{i are} provided to the convolutional neural network CNN and the classifier 330 so that efficient learning can be performed.

The convolutional neural network learning unit 22 receives a plurality of low resolution images generated by the resolution conversion unit 21 and divides them into a spatial stream and a time stream (optical flow), and performs convolution and pooling ) And apply a fully connected layer to learn the Convolutional Neural Network.

Referring to FIG. 7, the convolutional neural network learning unit 22 can know that spatial streams and time streams derived from low-resolution images are used as parameters. At this time, the spatial resolution of the low-resolution image is 16 × 12 pixels. More specifically, the spatial stream is configured to input the RGB pixel values of each frame (i.e., the input dimension is 16x12x3), and the temporal stream is the 10-frame connection of the X and Y optical flow images (i.e., 16x12x20) Lt; / RTI > The X and Y optical flow images are constructed by calculating the "x (and y) optical flow magnitude" per pixel.

The human behavior recognition model generation unit 23 can generate a human behavior recognition model for recognizing human behavior in an image photographed by the photographing unit 100 based on data learned by the convolution neural network learning unit 22 have. The human behavior recognition model may correspond to a classifier. The classifier can shorten the image information and motion information included in the image based on the learned data when a new image is input, and can recognize the behavior in the image using the reduced information. The image analysis technique of the present invention can be applied not only to a convolutional neural network (CNN) but also to a support vector machine (SVM).

8 is a diagram illustrating a structure of a two-stream convolutional neural network applied to convolutional neural network learning according to an embodiment of the present invention.

Referring to FIG. 8, the convolutional neural network learning unit 22 can perform learning for behavior recognition using a two-stream convolutional neural network. A two-stream convolution neural network is applied to each frame of the image. They are summarized using time pyramids and generate a single video representation. Let a 2-stream network applied to each frame V ^t of video V at time t be h (V ^t ). Then, the expression f (V;?) Is calculated according to the following equation (2).

Here, "," denotes the vector concatenation operator, T is the number of frames in the image V, and fc represents the set of fully connected layers to be applied at the top of the connection. The size of h (V ^t ) is 512-D: 256 x 2. θ is a set of parameters in CNN that require learning from learning data. max is a temporary maximum pooling operator that computes the maximum value of each element. In the example, four levels of time pyramids were used (i.e., a total of 15 maximum poolings).

A fully connected layer and a softmax layer can be applied to f (V) to learn the classifier. Let g be such a layer, y = g (f (V;?)). Where y is an activity class label. The learning g (f (V; [theta]) having classification loss using the low resolution image generated by the resolution converting unit 21 can provide a basic image classification model. Figure 8 shows the overall structure of a two-stream CNN model using this time pyramid.

9 is a diagram for explaining how the convolutional neural network learning unit 22 according to an embodiment of the present invention performs learning for motion recognition in an image using an embedding space.

The two-stream network design of FIG. 8 can classify behavioral images by learning model parameters optimized for classification, but does not consider the characteristics of ultra-low resolution images in which different low resolution data are generated due to different transformations applied to the same scene . In order for the classifier to better reflect the characteristics of the ultralow-resolution image, the embedding space, which maps to the same embedding location, regardless of the conversion to different low-resolution images with the same semantic content, . Embedding learning is commonly optimized for both embedded learning and classification using the learned embedding in an end-to-end manner, and more generalized (i.e., overfilled) Enables learning of classifiers.

Referring to FIG. 9, embedding learning is performed by minimizing the embedding distance between negative pairs while maximizing the embedding distance between positive pairs.

10 is a diagram illustrating a structure of a multi-siamese convolutional neural network applied to convolutional neural network learning according to an embodiment of the present invention.

Referring to FIG. 10, the resolution converting unit 21 converts a first batch composed of n low-resolution images having different resolutions and a plurality of different high-resolution images converted from the same high-resolution image, A second batch composed of n low-resolution images having the same size as the training data, and transmits the training data to the convolutional neural network learning unit 22. [

The convolutional neural network learning unit 22 receives the first and second arrangements, divides each of the images into a spatial stream and a temporal stream, performs convolution and pooling for each stream, After the layer (Fully connected layer) is further applied, the embedding distance is mapped by mapping to the embedding space, so that learning for motion recognition in an image can be performed.

The Siamese neural network is a concept often used to learn similarity measures between two inputs, with two networks sharing the same parameters. The goal of the Siamese network is to learn the embedding space in which similar items (low-resolution images in the case of the present invention) are placed nearby. More specifically, a sample corresponding to a positive pair that should be located close in the embedding space and a sample corresponding to a negative pair that should be located remotely are used for learning.

(V;?) is applied to the arbitrary low-resolution images V _i and V _j twice during the learning, x _i = f (V;?) ;?) and x _j = f (V _j ;?) are obtained. Where (x _i, x _j) may ssangil the positive or negative of the pair. The contrast loss for learning the network parameter [theta] is expressed by the following equation (3).

Where m is a predetermined margin, B is the batch of low resolution learning examples used, i and j are the index of the learning pair in the batch. y ' _{( i, j )} is a binary number, 1 for positive pairs and 0 for negative pairs.

In embedding learning for low-resolution intra-image behavior recognition, the positive pair is composed of two low-resolution images derived from the same high-resolution image, and the negative pair is composed of two low-resolution images derived from different high-resolution images. Figure 9 shows embedding learning with contrast loss. In addition, since the goal is to finally classify the low-resolution images by learning y = g (f (V;?)), The network must be learned by using the combining loss function as shown in the following equation (4).

Where L _class (θ) is the standard classification loss of the network y = g (f (V; θ)), and λ ₁ , λ ₂ are the weights.

The multi-siamese convolutional neural network of the present invention has 2n network copies sharing the same parameter θ for f (V; θ), unlike the standard cyamises network with only two network sharing parameters. The multi-siamese convolutional neural network maps each copy to each of the n different low resolution transforms (i.e., F _k ). Therefore, the embedding distance can be kept small by using the contrast loss. And the multi-siamuse convolutional neural network has n more network copies using images that do not match the scene of the first n branch to form a negative learning pair.

Let X _ik = f (V _ik;? ), where V _ik is obtained by applying the transform F _k to X _i . Two types of placement are randomly generated based on placement B of the original high resolution learning image. B ₁ is the arrangement of the low-resolution images generated in the single high-resolution image Xi, and B ₂ is the arrangement of the randomly selected low-resolution images. B ₁ produces a positive pair, and B ₂ produces a negative pair. The size of B ₁ and B ₂ should be n. B ₁ is obtained by applying n different low-resolution transforms to each example Xi of B, and each result V _ik = D _k F _k X _i is provided to the first n branches of the multi-siamese network. The low-resolution example Vj of B ₂ is provided directly to the remaining n branches of the siamese network. Therefore, the new loss function is expressed by the following equation (5).

That is, the multi-siamese convolution neural network model of the present invention simultaneously considers a plurality of low-resolution transforms for embedding learning. The new loss function essentially takes all pairs of n low resolution transforms as positive pairs and considers the same number of negative pairs using a separate arrangement.

The final loss function is calculated by combining the multi-cimase contrast loss with the standard classification loss performed in equation (4). FIG. 10 shows the overall process of learning multi-cyamics embedding and classifier. This can be seen as Siamese CNN, which combines multiple contrast losses of different low resolution pairs.

The multi-siamese convolution neural network model of the present invention uses three fully connected layers for embedding learning and classification. After applying the time pyramid, we obtain an intermediate representation of 7680-D (ie, 15 × 256 × 2) per image. Next we get a fully connected layer of size 8192. Embedding learning occurs after the second full-connect layer, where x has a dimension of 8192-D. Classification is performed by adding another fully connected layer and a soft max layer at the top.

FIG. 11 is a flowchart illustrating a step of learning a convolutional neural network among methods of automatically recognizing behaviors in an image of a camera based on a resolution-based automatic resolution control according to an exemplary embodiment of the present invention.

Referring to FIG. 11, the data server 20 receives a high-resolution learning image (S1101). The resolution conversion unit 21 converts the received high-resolution human behavior image into a plurality of low-resolution images (S1102). At this time, the resolution converting unit 21 converts a first batch composed of n low resolution images having different resolutions and a plurality of different high resolution images converted from the same high resolution image, (S1103), and transmits the second batch to the convolutional neural network learning unit 22. The convolutional neural network learning unit 22 generates a second batch of low-resolution images as learning data (S1103). The convolutional neural network learning unit 22 receives a plurality of low-resolution images and divides them into a spatial stream and a temporal stream, performs a convolution and a pooling for each stream, and performs au, a fully connected layer, And the convolutional neural network is learned (S1104). At this time, the convolutional neural network learning unit 22 receives the first and second arrangements, divides each of the images into a spatial stream and a time stream, and performs convolution and pooling for each stream After the Fully connected layer is further applied, the embedding distance is mapped to the embedding space to perform learning for intra-image motion recognition (S1105).

FIG. 12 is a view comparing the accuracy of the behavior recognition between the case where the method of automatically detecting the behavior in the camera of the camera system according to the embodiment of the present invention and the accuracy of the behavior recognition between the prior art.

FIG. 12 shows the results of comparing the accuracy of classification using a 16x12 HMDB dataset or a DogCentric activity dataset in the case of (1) applying basic 1-stream CNN, (2) applying 2-stream CNN, Table. For the above two CNNs, we divided into (i) learning without using multiple low-resolution transforms, (ii) learning with multiple low-resolution transformations applied but without embedding learning, and (iii) learning with Siamese embedding learning. Respectively.

The number of low-resolution transformations used in the experiment is 15. These transformations were randomly selected from a uniform pool. A total of 25 motion transforms F _k were provided in the X and Y directions of {-5, -2.5, 0, +2.5, + 5}%. In addition, a total of 75 transforms were provided with three different rotations of the angle {-5, 0, 5} degrees. 27 transforms out of 75 were randomly selected.

Referring to FIG. 12A, it can be seen that the case of applying the siamese learning to the 2-stream CNN has the highest accuracy. In other words, it can be seen that learning the embedding space using the cyamuse network structure has a great effect on the behavior classification. The use of contrast loss based on multiple low resolution transformations is more robust in classifier learning and reduces overfitting of training data.

12B is a table showing the result of comparing the present invention with the conventional technique. As shown in the table, it can be seen that the classification accuracy according to the present invention is higher than 8%. That is, the method of applying the embedding learning to the 2-stream multi-Siamese CNN, which is one embodiment of the present invention, yields the best result in the 16x12 resolution motion recognition.

12C and 12D are tables showing the results of experiments using DogCentric activity dataset. As a result, it can be seen that the automatic detection method of the behavior in the camera system of the camera according to the embodiment of the present invention shows the best accuracy in the behavior recognition, and thus is the most effective in privacy protection.

The automatic recognition method based on behavior recognition and the automatic in-video behavior recognition method according to an embodiment of the present invention may be implemented in a form of a program command that can be executed through various computer means and recorded in a computer readable medium . The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

1: Behavior recognition based resolution automatic camera system
10: Behavior recognition based resolution automatic adjustment camera
20: Data server
100:
200:
300: Image analysis section
310: resolution conversion unit
320: convolutional neural network learning unit
330:
400:
500:

Claims (14)

A data server for generating learning data based on recognition of human behavior in the low resolution image based on the high resolution learning image and generating a human behavior recognition model from the learning data; And
A low-resolution image is taken,
Resolution image based on a human behavior recognition model received from the data server, and a first action recognition unit for recognizing a human action in the low-resolution image, An adjustment camera,
The data server comprising:
A resolution converter for converting the high resolution human behavior image into a plurality of low resolution images;
A convolutional neural network learning unit for learning a convolutional neural network; And
And a human behavior recognition model generation unit that generates a human behavior recognition model based on data learned by the convolution neural network learning unit,
Wherein the convolutional neural network learning unit comprises:
A plurality of low resolution images generated by the resolution conversion unit are received and divided into a spatial stream and a time stream, and convolution and pooling are performed for each stream, and a fully connected layer is added After application,
Characterized in that learning for behavior recognition in an image is performed by being mapped to an embedding space and controlling an embedding distrance.
The method according to claim 1,
Wherein the behavior recognition-based resolution automatic adjustment camera comprises:
Determining whether the captured image is a crisis state through a second behavior recognition that recognizes human behavior in the image using the human behavior recognition model,
If the situation is a crisis, you can save the images shot at high resolution,
And stores the image photographed at a low resolution in case of a non-crisis situation.
The method according to claim 1,
Wherein the behavior recognition-based resolution automatic adjustment camera comprises:
A photographing unit for photographing a low-resolution image;
A communication unit for receiving a human behavior recognition model used for recognizing human behavior in the low resolution image from the data server and transmitting the received human behavior recognition model to the image analysis unit;
An image analyzing unit for determining whether the user is in a crisis state through a first behavior recognition process for recognizing a human behavior in a low resolution image using the human behavior recognition model received from the communication unit;
A control unit for controlling the photographing unit to change to a higher resolution mode than the existing resolution and to photograph the photographing unit if a crisis state is determined by the image analysis unit; And
And a storage unit for storing a low-resolution image or a high-resolution image photographed by the photographing unit.
The method of claim 3,
Wherein the image analyzing unit comprises:
And determining whether the image is a crisis state through a second behavior recognition that recognizes human behavior in the image using the human behavior recognition model for the image photographed by changing to the high resolution mode.
5. The method of claim 4,
Wherein,
When the situation determined through the second behavior recognition of the image analysis unit corresponds to a crisis situation, an image photographed in high resolution is stored in the storage unit,
Wherein the control unit controls the camera to store an image photographed at a low resolution in a non-crisis state in the storage unit.
delete The method according to claim 1,
Wherein the resolution conversion unit comprises:
A first batch consisting of n low resolution images having different resolutions converted from the same high resolution image and a second batch composed of n low resolution images having different resolutions converted from a plurality of different high resolution images, Is generated as learning data,
To the convolutional neural network learning unit.
delete A method for automatically adjusting a resolution based on a behavior recognition by a camera,
(a) photographing an image of a camera at a low resolution;
(b) receiving a human behavior recognition model used by the communication unit of the camera for recognizing human behavior in a low-resolution image from an external data server and transmitting the human behavior recognition model to an image analysis unit of the camera;
(c) determining whether the image analyzing unit is in a crisis state through a first behavior recognition that recognizes human behavior in a low-resolution image using the human behavior recognition model received from the camera communication unit;
(d) when the image analyzing unit determines that the image capturing unit is in a crisis state, the control unit of the camera controls the image capturing unit to change the image capturing unit to a higher resolution mode than the existing resolution image capturing unit; And
(e) storing a low-resolution image or a high-resolution image captured by the photographing unit in a storage unit of the camera,
Wherein the human behavior recognition model is generated based on data learned by the convolutional neural network learning unit of the data server,
The convolutional neural network learning unit divides a plurality of low resolution images input from the data conversion unit into a spatial stream and a time stream, performs convolution and pooling for each stream, , And then the convergence neural network is learned by mapping the embedding distance to the embedding space, thereby automatically learning the resolution based on the behavior recognition.
10. The method of claim 9,
Between step (d) and step (e)
Determining whether the image is a crisis state through a second behavior recognition that recognizes human behavior in the image using the human behavior recognition model with respect to the image photographed by changing the image analysis unit to the high resolution mode, How to adjust the resolution automatically.
11. The method of claim 10,
The step (e)
When the situation determined through the second behavior recognition of the image analyzer corresponds to a crisis situation, the image photographed at the high resolution is stored in the storage unit,
And if it is not a crisis, the image photographed at the low resolution is stored in the storage unit.
A method for automatically recognizing a behavior in an image, the method comprising:
(a) converting a high-resolution human behavior image into a plurality of low-resolution images by resolution conversion of a data server;
(b) a convolutional neural network learning unit of the data server receives the plurality of low-resolution images and divides them into a spatial stream and a time stream, performs convolution and pooling for each stream, layer is applied to the embedding space and then the embedding distance is adjusted by mapping the embedding space to the convolutional neural network;
(c) generating a human behavior recognition model based on data learned by the convolutional neural network learning unit;
(d) photographing a low-resolution image of a photographing part of the camera; And
(e) recognizing a human behavior in a low-resolution image photographed by the photographing unit using a human behavior recognition model received from the data server by the image analysis unit of the camera, .
13. The method of claim 12,
The step (a)
Resolution conversion unit of the data server is converted from the same high-resolution image and converted into a first batch composed of n low-resolution images having different resolutions and n low-resolution images having different resolutions converted from a plurality of different high- Generates a second batch composed of learning data,
To the convolutional neural network learning unit, the automatic recognition of intra-video motion in a camera system.
delete

Images