KR20190054702A - Method and apparatus for detecting action of object in viedio stream - Google Patents

Abstract

The present invention provides a method for identifying an action of an object from an image, which is capable of correctly detecting the action, and a device thereof. More specifically, the method for identifying an action of an object from an image comprises: a step of acquiring an image segment; a step of determining whether an action of an object exists in the image segment through a first neural network; a step of generating one or more future images successive to the image segment through a second neural network to form an integrated image segment when the action of the object exists in the image segment; and a step of detecting an action type and time point of the object in the integrated image segment through a third neural network, wherein the first neural network is independently learned and the second and third neural networks are dependently learned.

Description

METHOD AND APPARATUS FOR DETECTING ACTION OF OBJECT IN VIEDIO STREAM FIELD OF THE INVENTION [0001]

The present invention relates to a method and apparatus for recognizing an action of an object included in an image from an image.

Behavior recognition technology in the field of computer vision classifies the behavior of objects in the image depending on the image information obtained by inputting color (RGB) or depth image from the image sensor. Since the image sensor can be easily installed and operated in various places, recently, an image sensor is used to monitor a dangerous situation and to detect a specific event. However, in order to use behavior recognition technology in many applications, it is important to decide which features of the image to use, depending on the context in which the technology is to be used. The categories of features that can be used in the image can be roughly divided into two categories: a hand-crafted feature designed by the researcher and a deep feature extracted through the deep learning. The handcraft feature frequently used in the image includes a Dense Trajectories descriptor representing the human trajectory information and a Histogram of Oriented Gradients (HOG) descriptor representing the appearance information. The Deep Feature includes a Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) which learns appearance information are mainly used. In recent years, due to the breakthrough of deep-running technology, a deep-run-based engineer has been used as a representative.

As such, the behavior recognition technology shows high performance due to the development of deep learning technology. However, since the conventional behavior recognition technology uses an image in which a specific behavior is generated as learning data, there are still limitations in performing behavior recognition from real life images including a series of various actions or frames in which no action occurs. In other words, in order to recognize such a continuous action, a preprocessing process is required in which continuous action is cut into clip units so that only one action is included before the recognition technology is applied. Therefore, in order to recognize continuous action without such preprocessing, it is necessary to detect a part where the action occurs in the image and to recognize a certain action.

U.S. Patent No. 9,648,035 (entitled "User behavioral risk assessment"

Disclosure of Invention Technical Problem [6] The present invention provides a method and system for more accurately detecting a behavior by learning a start point and an end point of a section in which a behavior occurs together with a behavior type for real- I want to.

According to an aspect of the present invention, there is provided a method of recognizing an object in an image, the method comprising: acquiring an image segment; Determining whether an action of an object exists in an image segment through a first neural network; Constructing an integrated image segment by generating one or more future image frames contiguous to the image segment through a second neural network if the behavior of the object exists in the image segment; And detecting a behavior type and an action point of an object in the integrated image segment through the third neural network. At this time, the first neural network is learned independently, and the second neural network and the third neural network are learned dependently.

According to a second aspect of the present invention, there is provided a behavior recognition apparatus comprising: a memory for storing a program for recognizing an action of an object included in an image; And a processor for executing the program. At this time, the processor acquires an image segment as the program is executed, determines whether the behavior of the object exists in the image segment through the first neural network, and if there is an action of the object in the image segment, One or more future image frames contiguous to the image segment are generated through the neural network to form an integrated image segment and a behavior type and an action point of the object in the integrated image segment are detected through the third neural network. At this time, the first neural network is learned independently, and the second neural network and the third neural network are learned dependently.

A third aspect of the present invention provides a computer-readable recording medium having recorded thereon a program for performing the method of the first aspect on a computer.

According to the above-mentioned problem solving means, an embodiment of the present invention can detect a dangerous situation or recognize a specific situation in an environment in which an image is input in real time by generating future image frames. According to an embodiment of the present invention, a segment in which an action occurs in an image and a segment in which an action occurs in an image are detected to detect a segment in which the action occurs, and furthermore, an action point (i.e., It is possible to efficiently apply it to robot control requiring continuous observation by detecting behavior repeatedly depending on only visual information without learning prior to time.

1 shows a behavior recognition apparatus according to an embodiment of the present invention.
2 is an example of a neural network according to an embodiment of the present invention.
FIG. 3 shows a configuration of a behavior recognition apparatus according to an embodiment of the present invention.
4 shows an example of a detection network according to an embodiment of the present invention.
FIG. 5 is a flowchart illustrating a method of recognizing an action of an object in an image of the processor of FIG. 3 according to an embodiment of the present invention.
6 is a diagram for explaining a processor according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as " including " an element, it does not exclude other elements unless specifically stated otherwise.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings.

1 shows a behavior recognition apparatus 10 according to an embodiment of the present invention.

As shown in FIG. 1, the behavior recognition apparatus 10 recognizes the behavior of an object from a real-time image 12 using a neural network 11. The neural network 11 may be a set of algorithms for extracting various attribute information from the real-time image using the result of statistical machine learning and identifying the behavior of the object in the real-time image based on the extracted attribute information. In addition, the neural network 11 may be implemented with software or an engine for executing the above-described algorithm set. Software or a neural network implemented by an engine or the like may be executed by a processor in the behavior recognition apparatus 10. [

Meanwhile, the behavior recognition device 10 may be a video device such as a camera, a closed circuit television (CCTV), a black-box, and the like, but is not limited thereto. Such as a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, an IOT hub, an IOT server, Navigation, kiosks, home appliances, and the like.

2 is an example showing a neural network 11 according to an embodiment of the present invention.

2, the neural network 11 according to an embodiment of the present invention includes a proposal network 210, a future frame generation network 220, and a detection network detection network 230.

First, the poster network 210 receives an image segment 13 and abstracts various attributes included in the image segment to determine whether the image segment 13 includes an action of the object (i.e., an action / presence part) can do. Here, the image segment 13 may be obtained in real time as a set of a plurality of consecutive image frames. The abstracting of the attributes in the image segment can be performed by detecting attribute information such as object appearance information, object motion information, and the like in the image segment and determining a core attribute that can represent the image segment from among the detected attribute information.

At this time, the poster network 210 may include a plurality of abstraction layers 211 and a binary classifier 212, and the behavior recognition apparatus 10 may include a plurality of abstraction layers 211, The attribute information can be extracted from the attribute information storage unit 13. Here, the attribute information may include polygon, edge, depth, sharpness, saturation, brightness, depth, and temporal / spatial variation values thereof, as non-limiting examples. The behavior recognition apparatus 10 acquires the feature map based on the attribute information extracted from the last abstraction layer. Here, the feature map may include at least one attribute vector representing the attribute of the image segment as a combination of extracted attribute information.

The feature map may be applied as the input data of the binary classifier 212. The binary classifier 212 can determine whether there is an object action in the image segment based on the feature map. The behavior recognition apparatus 10 may acquire an object behavior part (i. E., Presence or non-existence of behavior) as a result of inputting the feature map to the binary classifier 212. [

The future image creator network 220 operates when the result of the above-mentioned transport network 210 indicates that the behavior of the object exists in the image segment 13. The future image generator network 220 generates one or more image frames contiguous to the image segment.

Specifically, the future image generator network 220 includes a plurality of abstraction layers 221 for extracting various attribute information from the image segment 13, and one or more future image frames contiguous to the image segment based on the extracted attribute information And a plurality of creator layers 222 that are arranged in a matrix. At this time, the attribute information of the image segment may include polygon, edge, depth, sharpness, saturation, brightness, depth, and temporal / spatial variation values thereof as a non-limiting example.

The behavior recognition apparatus 10 may acquire a feature map combining attribute information extracted from the last abstraction layer among a plurality of abstraction layers 221 and apply the feature map as input data of the first constructor layer. The plurality of creator layers 222 modify the attribute information based on the learned information or synthesize feature information of other previously stored images to generate a future image frame. In this case, the learned information is a space-time change value of each attribute information based on the behavior of the object, and is expressed by parameters of each creator layer. Further, the feature information of the other images may include polygon, edge, depth, sharpness, saturation, brightness, depth, and temporal / spatial variation values thereof as non-limiting examples.

Also, according to an implementation, the future image generator network 220 may determine a future image frame to be substantially contiguous to the image segment, and send it to the abstraction layer 221 and the constructor layer 222, And a discriminator layer (not shown). Such a future image generator network may be, by way of example, but not exclusively, a Laplacian Generative Adversarial Network.

Next, the behavior recognition apparatus 10 constructs an extended image segment by adding one or more future image frames obtained through the future image generator network 220 to the image segment 13. [

Next, the detection network 230 receives the extended image segment and abstracts various attributes included in the extended image segment, thereby detecting a behavior type and an action time included in the extended image segment. Here, the behavior type may be determined by the learning data of the detection network 230 and may include, by way of non-limiting example, human motion, attack, accident, injury, etc., can do. In addition, the action point includes a point in time at which the action of the object included in the extended image segment indicates the start and end points of the action.

The detection network 230 includes a plurality of abstraction layers 231 and one or more classifiers 232 for discriminating behavior types and behavior points. The behavior recognition apparatus 10 extracts attribute information from the extended image segment based on the plurality of abstraction layers 231 through the plurality of abstraction layers 231, The feature map obtained from the information can be applied to the input data of the classifier 232. [ Illustratively, the detection network 230 applies the feature map as input data to a first classifier (not shown) to identify the behavior type of the object in the aggregated image segment and to determine the resultant value and / The map may be applied as input data of a second classifier (not shown) to detect whether the time at which the behavior of the object in the integrated image segment is manifested corresponds to the end or start time of the behavior type. In this case, the first classifier may be a multi-class classifier, and the second classifier may be a binary classifier, but the present invention is not limited thereto, and the first classifier and the second classifier may be implemented with a plurality of binary classifiers.

FIG. 3 shows a configuration of a behavior recognition apparatus 10 according to an embodiment of the present invention. The behavior recognition apparatus 10 includes a memory 310 and a processor 320.

The memory 310 may store programs (one or more instructions) for processing and control of the processor 320. Programs stored in the memory 310 may be divided into a plurality of modules according to functions. According to one embodiment, the memory 310 may store a program that recognizes the behavior of an object included in an image from the image. The program may include a neural network module.

The neural network module may include a plurality of layers and classifiers included in a poster network, a future image generator network, and a detection network. In this case, each of the abstraction layers included in the networks includes a 3D convolutional layer including one or more instructions for extracting attribute information of an image from an input image to generate a feature map, and / And a pooling layer including one or more instructions for determining a representative value from the attribute information. Each creator layer included in the future image generator network may include a deconvolutional layer including one or more instructions for modifying extracted attribute information based on the feature map or synthesizing feature information of other previously stored images, . ≪ / RTI >

4 shows an example of a detection network according to an embodiment of the present invention.

The processor 320 includes a connection path (e.g., a bus) that transmits and receives signals with one or more cores (not shown) and a graphics processing unit (not shown) and / or other components .

According to one embodiment, the processor 320 may process one or more instructions contained in each network in a neural network module in parallel.

Hereinafter, with reference to FIG. 5, a method in which the processor 320 recognizes an action of an object in an image will be described.

First, the processor 320 obtains an image segment from the image (S510). At this time, the image may be obtained in real time from the image sensor provided in the behavior recognition apparatus 10, or may be received from the external imaging apparatus. A video segment is a set of consecutive video frames.

Thereafter, the processor 320 determines whether the action of the object exists in the image segment through the poster network (S520).

As discussed above, the poster network comprises a plurality of abstraction layers and a binary classifier. The processor 320 extracts attribute information of an image segment through a plurality of abstraction networks. For example, the processor 320 may extract linear information from an image frame using the first layer of the plurality of layers. Further, the device can extract the linear change value from the second layer by applying the extracted linear information to the second layer connected to the first layer as input data. As described above, the device can extract various attribute information by inputting image segments to each of a plurality of layers or applying attribute information extracted from a previous layer as input data. Subsequently, the processor 320 extracts the feature map by combining the attribute information extracted from the last layer.

The processor 320 may apply the extracted feature map to the input data of the binary classifier to determine whether the action of the object exists in the image segment.

If it is determined that there is no action of the object in the image segment, the processor 320 repeats steps S510 and S520 for the next image segment. This allows the processor 320 to minimize unnecessary computational load. Meanwhile, the next image segment may include some image frames of the current image segment (for example, the last image frame of the current image segment, etc.) in a duplicate manner.

However, if there is an action of the object in the image segment, the processor 320 generates one or more future image frames continuous to the image segment through the future image creator network to configure an integrated image segment (S530).

The future image generator network includes a plurality of abstraction layers and a plurality of producer layers. The processor 320 obtains the feature map through a plurality of abstraction layers included in the future image creator network as in the case of the present invention, and the attribute information of the image segment is transformed based on the feature map through a plurality of creator layers , And generates one or more future image frames in which feature information of other previously stored images is synthesized. Processor 320 constructs an integrated image segment by adding one or more future image frames to the image segment.

In addition, the future image generator network may further include a plurality of discriminator layers for determining the probability that one or more future image frames generated are substantially contiguous to the image segment, and for feeding back the result to the abstraction layer and the constructor layer . The processor 320 may repeatedly generate a future image frame based on the probability value calculated through the plurality of discriminator layers to improve the probability value of the discriminator layer to generate a more accurate future image frame.

Then, the processor 320 detects the action type and the action point of the object in the integrated image segment through the detection network (S540).

As described above, the detection network includes a plurality of abstraction layers and one or more classifiers. The processor 320 extracts the attribute information of the integrated image segment through the plurality of abstraction layers included in the detection network and obtains the feature map by combining the attribute information extracted from the last abstraction layer. The processor 320 then applies the feature map to the input data of one or more classifiers. Illustratively, the processor 320 may apply the feature map to the first classifier as input data to identify the behavior type of the object in the aggregated image segment, and to compare the result of the first classifier and / It is possible to detect whether the time at which the behavior of the object in the integrated image segment is expressed corresponds to the end or start time of the action type.

Thereafter, the processor 320 repeats S510 to S540 again to detect the behavior type and the action point of the object from the image received in real time. When the detected behavior type corresponds to a predetermined abnormal behavior, the processor 320 informs the behavior type and the action timing information through the notification device provided in the behavior recognition apparatus 10, The above-described information can be transmitted.

Each neural network in steps S520 to S540 may perform the attribute information extraction operation, classification operation, future image frame generation, and the like through the previously performed learning. For example, the processor 320 classifies the image segment performed before S510 into a behavior or a non-behavior, and as a result of generating a future image frame contiguous to the image segment, It is possible to learn each layer and classifier included in each network in the direction of increasing the accuracy of the behavior type and the behavior timing discrimination result. This will be described later in more detail with reference to FIG.

6 is a diagram for explaining a processor 320 according to an embodiment of the present invention. Referring to FIG. 6, the processor 320 includes a data learning unit 610 and a data recognizing unit 620.

The data learning unit 610 performs independent learning on the proposal network, and the future image generator network and the detection network perform interdependent learning. At this time, the learning may be to adjust parameters (e.g., weights, etc.) of each layer and / or classifier class, or to omit and / or add at least one layer in the operation.

Specifically, the data learning unit 610 independently learns the first criterion for detecting whether the behavior of the object exists from the input data (that is, the image segment) of the picture network. For example, the data learning unit 610 can input learning information segments and information about the behavior of the objects in the learning image segment (i.e., "behavior" or "non-behavior") to the proposal network to learn.

In addition, the data learning unit 610 inputs a learning image segment and a frame contiguous to the learning image segment to the future image generator network to learn. In addition, the data learning unit 610 inputs the result values (i.e., the integrated image segments) of the future image generator network and the behavior types and behavior points of the objects in the integrated image segment to the detection network. At this time, the data learning unit 610 further inputs the value of the loss function of the detection network to the future image generator network so that the future image generator network can learn based on the value of the loss function. Here, the loss function is illustratively a negative log-likelihood function, but is not limited thereto. In this way, the data learning unit 610 can perform the interdependent learning on the future image generator network and the detection network, thereby more accurately detecting the behavior type and the behavior time of the object.

The data recognition unit 620 can determine the behavior type and the action point of the object from the image segment based on the learned criterion through the data learning unit 610. [ This has been described above with reference to FIGS. 1 to 5, and thus a detailed description thereof will be omitted.

At least one of the data learning unit 610 and the data recognizing unit 620 may be manufactured in the form of at least one hardware chip and mounted on the behavior recognition apparatus 10. [ Alternatively, at least one of the data learning unit 610 and the data recognition unit 620 may be implemented as a software module. When at least one of the data learning unit 610 and the data recognition unit 620 is implemented as a software module (or a program module including an instruction), the software module may be a computer-readable, And may be stored in non-transitory computer readable media. In this case, at least one software module may be provided by an operating system (OS) or by a predetermined application.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

10: Behavior recognition device
11: Neural network
12: Real time image 13: Image segment
210: proposal network
220: Future image generator network
230: detection network
310: memory 320: processor
610: Data learning unit 620: Data recognition unit

Claims (13)

A method for a behavior recognition apparatus to recognize an action of an object in an image,
Obtaining an image segment;
Determining whether there is an action of an object in the image segment through a first neural network;
Constructing an integrated image segment by generating one or more future image frames contiguous to the image segment through a second neural network if there is an action of the object in the image segment; And
Detecting a behavior type and an action point of the object in the integrated image segment through a third neural network,
Wherein the first neural network is learned independently and the second neural network and the third neural network are learned dependently. The method according to claim 1,
Wherein the second neural network is learned based on a value of a loss function of the third neural network and the third neural network is learned based on a result of the second neural network. The method according to claim 1,
Wherein the step of determining whether or not the behavior of the object exists
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of layers included in the first neural network; And
And applying the feature map as input data to a binary classifier included in the first neural network to determine whether the behavior of the object exists in the image segment. The method of claim 3,
Wherein the attribute information includes at least one of polygon, edge, depth, sharpness, saturation, brightness, depth, and their temporal / spatial variation values included in the image segment Recognition method. The method according to claim 1,
The step of constructing the integrated video segment
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of first layers included in the second neural network;
Generating one or more future image frames in which attribute information of the image segment is modified based on the feature map or feature information of other previously stored images is synthesized through a plurality of second layers included in the second neural network step; And
And constructing the integrated image segment by adding the one or more future image frames to the image segment. 6. The method of claim 5,
The step of constructing the integrated video segment
Determining, via a plurality of third layers included in the second neural network, a probability that the generated one or more future image frames are substantially continuous to the image segment; And
And feedbacking the determined probability value to the first layer and the second layer to regenerate the one or more future image frames. The method according to claim 1,
The step of determining the action type and the action point
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of layers included in the third neural network; And
Applying the feature map to the first classifier as input data to identify the behavior type of the object in the integrated image segment and applying the result of the first classifier to the input data of the second classifier, Detecting whether a point at which a behavior is manifested corresponds to an end or a starting point of the behavior type. A memory for storing a program for recognizing an action of an object included in the image; And
And a processor for executing the program,
The processor, as the program is executed,
Acquiring an image segment, determining whether an action of the object exists in the image segment through the first neural network,
If there is an action of an object in the image segment, constructing an integrated image segment by generating one or more future image frames continuing to the image segment through a second neural network, Detecting an action type and an action point of the object,
Wherein the first neural network is learned independently and the second neural network and the third neural network are learned dependently. 9. The method of claim 8,
Wherein the second neural network is learned based on a value of a loss function of the third neural network and the third neural network is learned based on a result value of the second neural network. 9. The method of claim 8,
The processor
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of layers included in the first neural network, applying the feature map to input data to a binary classifier included in the first neural network, And determines whether or not there is an action of the object in the segment. 9. The method of claim 8,
The processor
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of first layers included in the second neural network, and acquiring the feature map through a plurality of second layers included in the second neural network One or more future image frames in which attribute information of the image segment is modified or feature information of other images previously stored is synthesized,
And the one or more future image frames are added to the image segment to form the integrated image segment. 9. The method of claim 8,
The processor
Acquiring a feature map by combining attribute information extracted from a last layer among a plurality of layers included in the third neural network, and applying the feature map to the first classifier as input data to determine a behavior type of the object in the integrated image segment And detecting whether the time at which the behavior of the object in the integrated image segment is manifested corresponds to the end or the start time of the behavior type by applying the result value of the first classifier to the input data of the second classifier In behavior recognition device. A computer-readable recording medium recording a program for performing the method according to any one of claims 1 to 7 on a computer.

Images