KR102376479B1 - Method, device and system for controlling for automatic recognition of object based on artificial intelligence - Google Patents

Abstract

Provided is a CCTV control method for artificial intelligence (AI)-based automatic object recognition to enable a manager to perform video monitoring effectively. According to one embodiment of the present invention, a method of controlling a CCTV for AI-based automatic object recognition is executed by a device. The method comprises the following steps: acquiring first video information generated by capturing a first area from a first CCTV when the first area is being photographed through the first CCTV; extracting an image of the video most recently acquired from the first video information as a first image; generating a first input signal by encoding the first image; inputting the first input signal to an artificial neural network and acquiring a first output signal on the basis of a result of the input of the artificial neural network; detecting whether an object exists in the first area on the basis of the first output signal; detecting a type of object located in the first area on the basis of the first output signal when the object exists in the first area; recognizing a state in which a first object is located in the first area when the object located in the first area is detected as the first object; and transmitting the first video information acquired after extraction of the first image to a manager terminal when it is recognized that the first object is located in the first area, so that the first video information is displayed on a screen of the manager terminal.

Description

CCTV control method, device and system for automatic object recognition based on artificial intelligence {METHOD, DEVICE AND SYSTEM FOR CONTROLLING FOR AUTOMATIC RECOGNITION OF OBJECT BASED ON ARTIFICIAL INTELLIGENCE}

The following embodiments relate to a technology for controlling a CCTV for automatic object recognition based on artificial intelligence.

CCTV (Closed-Circuit Television) is increasingly installed for theft prevention and intrusion prevention. Complaints about CCTV installation are flooding in parks and alleys, and these CCTVs play a big role in crime prevention and detection of natural disasters.

However, the biggest drawback of the existing CCTV is that it is mainly used as a post-mortem measure when a crime, theft, or intrusion occurs. That is, at the moment of theft/intrusion, etc., it is insufficient to immediately deal with it only by shooting through CCTV.

Of course, a 24-hour monitor agent can monitor the video recorded through CCTV, but it is almost impossible to monitor the video recorded through numerous CCTVs 24 hours a day. In addition, CCTV installed in houses or apartment complexes is mostly only a post-mortem measure, and there is no monitor agent.

Therefore, when an object is not recognized in a specific area, the video captured through CCTV is not provided to the manager, but only when the object is recognized in a specific area Therefore, there is a need for research on a method to efficiently perform image monitoring.

Korean Patent No. 10-1470314 Korean Patent No. 10-1470316 Korean Patent Registration No. 10-1765722 Korean Patent No. 10-1942808

According to an embodiment, when shooting for the first area is being performed through the first CCTV, the first image information generated by the shooting of the first area is obtained from the first CCTV, and from the first image information, the most recent extracts the image of the obtained image as a first image, encodes the first image to generate a first input signal, inputs the first input signal to the artificial neural network, and based on the result of the input of the artificial neural network, Obtain a first output signal, and detect whether an object exists in the first region based on the first output signal, and if it is detected that the object exists in the first region, based on the first output signal, The type of the object located in the first area is detected, and when the object located in the first area is detected as the first object, it is recognized as a state that the first object is located in the first area, and the first object is located in the first area CCTV for automatic recognition of objects based on artificial intelligence, which controls so that the first image information is displayed on the screen of the manager terminal by transmitting the first image information obtained after extraction of the first image to the manager terminal when recognized as being located It aims to provide a control method, apparatus and system.

The object of the present invention is not limited to the object mentioned above, and other objects not mentioned will be clearly understood from the description below.

According to an embodiment, in a method of controlling a CCTV for automatic object recognition based on artificial intelligence, performed by a device, when shooting for a first area is performed through a first CCTV, the first obtaining first image information generated by shooting of the first area from CCTV; extracting an image of the most recently acquired image from the first image information as a first image; generating a first input signal by encoding the first image; inputting the first input signal to an artificial neural network, and obtaining a first output signal based on a result of the input of the artificial neural network; detecting whether an object exists in the first area based on the first output signal; detecting a type of an object located in the first area based on the first output signal when it is detected that the object is present in the first area; recognizing that the first object is located in the first area when the object located in the first area is detected as the first object; and when it is recognized that the first object is located in the first area, the first image information obtained after extraction of the first image is transmitted to the manager terminal, and the first image information is displayed on the screen of the manager terminal There is provided a CCTV control method for automatic recognition of objects based on artificial intelligence, comprising the step of controlling to be displayed on the .

In the CCTV control method for automatic object recognition based on artificial intelligence, when the first object is recognized as being located in the first area, based on the first image information obtained after the extraction of the first image, the tracking and analyzing the movement of the first object; As a result of tracking and analyzing the movement of the first object, if it is confirmed that the first object has moved out of the first area in the direction of the second area, a second CCTV that is filming the second area checking; acquiring second image information generated by shooting of the second area from the second CCTV when the second area is being photographed through the second CCTV; and transmitting the second image information to the manager terminal, and controlling the second image information to be displayed on the screen of the manager terminal subsequent to the first image information.

In the obtaining of the first image information, a first camera is installed in a first direction of the first zone among the first CCTVs, and a second camera is installed in a second direction of the first zone, when a third camera is installed in a third direction of the first zone, controlling only the first camera to operate to perform photographing; When the first area is photographed in the first direction through the first camera, 1-1 image information generated by photographing the first area in the first direction from the first camera obtaining; extracting an image of the most recently acquired image from the 1-1 image information as a 1-1 image; generating a 1-1 input signal by encoding the 1-1 image; inputting the 1-1 input signal to an artificial neural network, and obtaining a 1-1 output signal based on a result of the input of the artificial neural network; detecting whether an object exists in the first area based on the 1-1 output signal; when it is detected that no object exists in the first area, controlling only the first camera to operate to perform photographing; detecting the number of objects located in the first area based on the 1-1 output signal when it is detected that an object exists in the first area; when it is determined that the number of objects located in the first area is smaller than a preset reference value, controlling the first camera and the second camera to operate to perform photographing; and controlling the first camera, the second camera, and the third camera to operate to perform photographing when it is determined that the number of objects located in the first area is greater than the reference value.

According to one embodiment, when an object is not recognized in a specific area, the image captured through CCTV is not provided to the manager, and only when the object is recognized in a specific area, from the point in time when the object is recognized, By providing an image to a manager, there is an effect that image monitoring can be performed efficiently.

On the other hand, the effects according to the embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is a diagram schematically showing the configuration of a system according to an embodiment.
2 is a flowchart illustrating a process of controlling a CCTV for automatic object recognition based on artificial intelligence according to an embodiment.
3 is a flowchart illustrating a process of tracking an object moving through a plurality of CCTVs according to an embodiment.
4 is a view showing a first CCTV according to an embodiment.
5 is a flowchart for explaining a process of controlling the operation of a CCTV camera according to the number of objects according to an embodiment.
6 is a flowchart illustrating a process of obtaining result image information corresponding to request information according to an exemplary embodiment.
7 is a flowchart illustrating a process of extracting and generating second result image information and third result image information according to an exemplary embodiment.
8 is a diagram for explaining an artificial neural network according to an embodiment.
9 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

The embodiments may be implemented in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like.

1 is a diagram schematically showing the configuration of a system according to an embodiment.

Referring to FIG. 1 , a system according to an embodiment may include a plurality of CCTVs 100 , a manager terminal 200 , and a device 300 .

First, a plurality of CCTVs 100 are installed in a specific place for each zone, and are devices that generate image information by photographing the installed zones, and are installed in the first CCTV 110 installed in the first zone and the second zone. It may include a second CCTV 120 and the like.

Each of the plurality of CCTV 100 may be composed of a plurality of cameras, and a motion detector camera having a function of detecting a moving object within the monitoring area, an infrared camera having a heat detection, a high-definition RGB camera taking a license plate image, a more reliable It may consist of a combination of a camera with a radar attached, a pan-tilt camera for performing automatic tracking, and a network camera capable of performing IP communication.

A plurality of CCTVs 100 may be connected to the device 300 through a network, and may operate according to a control signal received from the device 300 .

The manager terminal 200 is a terminal used by a manager who checks images captured by a plurality of CCTVs 100 , and may be configured to communicate with the device 300 by wire or wireless.

The device 300 may be a server owned by a person or organization that provides services using the device 300, a cloud server, or a peer-to-peer (p2p) set of distributed nodes. may be The device 300 may be configured to perform all or part of an arithmetic function, a storage/referencing function, an input/output function, and a control function of a typical computer.

The device 300 may be configured to communicate with a plurality of CCTVs 100 by wire or wireless, and control the operation of each of the plurality of CCTVs 100 to control whether to shoot, a shooting angle, a shooting direction, whether to store an image, etc. can

The device 300 may be configured to communicate with the manager terminal 200 by wire or wireless, and may control the operation of the manager terminal 200 to control which information to display on the screen of the manager terminal 200 . .

2 is a flowchart illustrating a process of controlling a CCTV for automatic object recognition based on artificial intelligence according to an embodiment.

Referring to FIG. 2 , first, in step S201 , when the device 300 is shooting for the first area through the first CCTV 110 , the first area is captured from the first CCTV 110 . The generated first image information may be acquired. In this case, the first CCTV 110 is capturing the first area, and the device 300 may acquire the first image information generated by the photographing of the first area in real time.

In step S202 , the device 300 may extract an image of an image most recently acquired from the first image information as the first image. In this case, the device 300 may extract an image of the most recently acquired image from the first image information acquired in real time for each preset period as the first image.

For example, in a state in which the first image information is being acquired in real time, the device 300 may extract an image of an image captured at the current time from the first image information as the first image every preset period.

In step S203 , the device 300 may generate a first input signal by encoding the first image.

Specifically, the device 300 may generate the first input signal by encoding the pixels of the first image with color information. The color information may include, but is not limited to, RGB color information, brightness information, and saturation information. The device 300 may convert color information into a numerical value, and may encode an image in the form of a data sheet including this value.

In step S204 , the device 300 may input the first input signal to the artificial neural network.

According to an embodiment, the artificial neural network is implemented as a convolutional neural network, and the convolutional neural network is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network stacks an input signal with a convolutional layer and a pooling layer sequentially. The convolution layer includes a convolution operation, a convolution filter, and an activation function. The calculation of the convolution filter is adjusted according to the matrix size of the target input, but a 9X9 matrix is generally used. The activation function generally uses, but is not limited to, a ReLU function, a sigmoid function, and a tanh function. The pooling layer is a layer that reduces the size of the input matrix, and uses a method of extracting representative values by tying pixels in a specific area. In general, the average value or the maximum value is often used for the calculation of the pooling layer, but is not limited thereto. The operation is performed using a square matrix, which is usually a 9x9 matrix. The convolutional layer and the pooling layer are repeated alternately until the corresponding input becomes small enough while maintaining the difference.

According to one embodiment, the classification neural network has a hidden layer and an output layer. In a convolutional neural network for an image processing method of CCTV, there are generally three or more hidden layers, and 100 nodes for each hidden layer are specified, but more can be specified in some cases. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. A total of 50 output layer nodes of a convolutional neural network can be made. A detailed description of the convolutional neural network will be described later with reference to FIG. 8 .

In step S205 , the device 300 may obtain a first output signal that is an output value of the artificial neural network based on the result of the input of the artificial neural network, which is the convolutional neural network.

According to an embodiment, 50 output layer nodes of the convolutional neural network may include upper 25 output layer nodes and lower 25 output layer nodes. The top 25 output layer nodes among 50 output layer nodes of the convolutional neural network may indicate the object recognition probability. The lower 25 nodes may indicate object types corresponding to the upper 25 nodes. A detailed description of the output of the convolutional neural network will be described later with reference to FIG. 8 .

In step S206 , the device 300 may detect whether an object exists in the first region based on the first output signal. At this time, the device 300 checks the output value for the object recognition probability based on the first output signal, analyzes the object recognition probability in the first area, and determines that the object recognition probability at a specific point is greater than a preset reference value. If confirmed, it may be detected that the object is present in the first area.

In step S207 , the device 300 may determine whether the object exists in the first area through the detection result of whether the object exists.

If it is detected that the object does not exist in the first area in step S207, the process returns to step S202, and the device 300 extracts the image of the most recently acquired image from the first image information as the first image again. . At this time, the device 300 may continue to acquire the image information generated by the shooting of the first area from the first CCTV (110).

If it is detected that the object exists in the first area in step S207 , in step S208 , the device 300 may detect the type of the object located in the first area based on the first output signal. Here, the type of object may include various types of object types such as people, animals, and vehicles.

In this case, the device 300 may detect the type of the object located in the first area by checking the output value of the object type based on the first output signal. For example, when the output value of the object type is “1” based on the first output signal, the device 300 detects the type of the object located in the first area as “person” and determines the type of the object. If the output value for the '2' is confirmed, the type of the object located in the first area may be detected as "vehicle".

In step S209 , when the object located in the first area is detected as the first object, the device 300 may recognize that the first object is located in the first area. For example, when an object located in the first region is detected as a “person”, the device 300 may recognize that a “person” is located in the first region.

In step S210, when the device 300 is recognized as a state in which the first object is located in the first area, the first image information obtained after extraction of the first image is transmitted to the manager terminal 200, and the first image information can be controlled to be displayed on the screen of the manager terminal 200 . At this time, the device 300 further transmits a notification message indicating that the first object is recognized in the first zone to the manager terminal 200, and controls the notification message to be displayed on the screen of the manager terminal 200 as a pop-up first. can do.

That is, when it is recognized that the first object is located in the first area, the device 300 may control so that the first image information obtained from the point in time when the first object is recognized is displayed on the screen of the manager terminal 200 . there is.

3 is a flowchart illustrating a process of tracking an object moving through a plurality of CCTVs according to an embodiment.

Referring to FIG. 3 , first, in step S301 , when an object located in the first area is detected as the first object, the device 300 may recognize that the first object is located in the first area.

In step S302 , the device 300 may track and analyze the movement of the first object based on the first image information obtained after the extraction of the first image.

Specifically, in a state in which the device 300 continues to acquire the first image information from the first CCTV 110 in real time, when it is recognized that there is a first object in the first area at the first time point, at the first time point Based on the obtained first image information, the position of the first object in the first area may be identified and analyzed, and the movement of the first object is tracked based on the first image information obtained after the first time point. can be analyzed.

In step S303 , as a result of tracking and analyzing the movement of the first object, the device 300 may determine whether the first object leaves the first area.

If it is confirmed in step S303 that the first object does not deviate from the first area, the process returns to step S302, and the device 300 may continue to track and analyze the movement of the first object based on the first image information.

If it is determined in step S303 that the first object has moved out of the first zone, in step S304 , the device 300 may determine that the first object has moved out of the first zone in the direction of the second zone.

For example, when the right area of the first area is set as the second area, the device 300 tracks the movement of the first object located in the first area, and as a result, the first object moves in the right direction When it is confirmed that the first area is out of the first area, it may be confirmed that the first object has moved from the first area to the second area.

In step S305 , the device 300 may check the second CCTV 120 that is shooting the second area.

In step S306 , the device 300 acquires the second image information generated by the shooting of the second area from the second CCTV 120 when the second area is being photographed through the second CCTV 120 . can do. In this case, the second CCTV 120 is capturing the second area, and the device 300 may acquire the second image information generated by the photographing of the second area in real time.

In operation S307 , the device 300 may transmit the second image information to the manager terminal 200 , and control so that the second image information following the first image information is displayed on the screen of the manager terminal 200 .

The device 300 may control to display the second image information subsequent to the first image information on the screen of the manager terminal 200 based on the positions of the first and second regions.

For example, when the first zone is located in the left direction of the second zone, the device 300 processes the first image information to be pushed to the left and the second image information to come in from the right at the same time, so that the first image Second image information may be controlled to be displayed on the screen of the manager terminal 200 following the information.

4 is a view showing a first CCTV according to an embodiment.

Referring to FIG. 4 , the first CCTV 110 may include a first camera 111 , a second camera 112 , a third camera 113 , and a fourth camera 114 .

The first camera 111 is installed in a first direction of the first area, and thus may perform photographing of the first area in the first direction. For example, since the first camera 111 is installed in the south direction of the first area, it is possible to photograph the first area in the south direction.

The second camera 112 is installed in the second direction of the first area, so that the first area may be photographed in the second direction. For example, the second camera 112 is installed in the west direction of the first area, so that the first area may be photographed in the west direction.

The third camera 113 is installed in the third direction of the first area, and thus may perform photographing of the first area in the third direction. For example, the third camera 113 is installed in the east direction of the first area, so that the first area may be photographed in the east direction.

The fourth camera 114 is installed in the fourth direction of the first area, so that the first area may be photographed in the fourth direction. For example, since the fourth camera 114 is installed in the north direction of the first area, it is possible to photograph the first area in the north direction.

According to one embodiment, the first CCTV 110 may be implemented with a different number of cameras depending on the embodiment. For example, the first CCTV 110 may be implemented as one first camera 111, may be implemented only with the first camera 111 and the second camera 112, the first camera 111, It may be implemented with the second camera 112 and the third camera 113 , including all of the first camera 111 , the second camera 112 , the third camera 113 and the fourth camera 114 . can be implemented.

In addition, the first CCTV 110 may be implemented by further including a camera installed in a different direction in addition to the first camera 111 , the second camera 112 , the third camera 113 and the fourth camera 114 . there is.

5 is a flowchart for explaining a process of controlling the operation of a CCTV camera according to the number of objects according to an embodiment.

Referring to FIG. 5 , first, in step S501 , the device 300 may control only the first camera 111 of the first CCTVs 110 to operate and take pictures. For example, if the first CCTV 110 includes the first camera 111 , the second camera 112 , the third camera 113 and the fourth camera 114 , the device 300 is the first It is possible to control that only one camera 111 is operated to perform photographing, and control not to operate with respect to the second camera 112 , the third camera 113 , and the fourth camera 114 .

In step S502 , the device 300 captures the first area in the first direction from the first camera 111 when the first area is being photographed in the first direction through the first camera 111 . The generated 1-1 image information may be acquired. In this case, the first camera 111 is capturing the first area in the first direction, and the device 300 records the 1-1 image information generated by capturing the first area in the first direction in real time. can be obtained

Thereafter, the device 300 extracts the image of the most recently acquired image from the 1-1 image information as the 1-1 image, encodes the 1-1 image to generate a 1-1 input signal, The 1-1 input signal may be input to the artificial neural network, and the 1-1 output signal may be obtained based on a result of the input of the artificial neural network.

In step S503 , the device 300 may detect whether an object exists in the first region based on the 1-1 output signal.

If it is detected that the object does not exist in the first area in step S503, the process returns to step S501, and the device 300 may control only the first camera 111 to operate to perform photographing.

If it is detected that the object exists in the first area in step S503 , in step S504 , the device 300 may detect the number of objects located in the first area.

For example, if the device 300 detects that the first object exists in the first region based on the 1-1 output signal, based on the 1-1 output signal, the first object is located in the first region It is possible to detect the number of first objects by checking how many are present. To this end, the artificial neural network may output, as a result of the input, information including not only the object recognition probability and object type, but also the number of objects.

In step S505 , the device 300 may determine whether the number of objects located in the first area is smaller than a reference value. Here, the reference value may be set differently depending on the embodiment.

If it is confirmed in step S505 that the number of objects located in the first zone is smaller than the reference value, in step S506, the device 300 operates the first camera 111 and the second camera 112 of the first CCTV 110 to control the shooting.

For example, when the first CCTV 110 includes the first camera 111 , the second camera 112 , the third camera 113 and the fourth camera 114 , the first camera 111 . When it is confirmed that the number of objects located in the first area is smaller than the reference value, the device 300 operates not only the first camera 111 but also the second camera 112 to perform a photographing operation. and control not to operate with respect to the third camera 113 and the fourth camera 114 .

If it is confirmed in step S505 that the number of objects located in the first zone is greater than the reference value, in step S507, the device 300 includes the first camera 111, the second camera 112 and the second of the first CCTV 110. 3 It is possible to control the camera 113 to operate to perform photographing.

For example, when the first CCTV 110 includes the first camera 111 , the second camera 112 , the third camera 113 and the fourth camera 114 , the first camera 111 . When it is confirmed that the number of objects located in the first area is greater than the reference value, the device 300 not only the first camera 111 but also the second camera 112 and the third camera 113 It is possible to control the operation to perform photographing, and control not to operate with respect to the third camera 113 .

According to an embodiment, the device 300 captures the first area in the second direction from the second camera 112 when the first area is photographed in the second direction through the second camera 112 . The first-second image information generated by photographing may be acquired. At this time, the second camera 112 is capturing the first area in the second direction, and the device 300 records 1-2 image information generated by capturing the first area in the second direction in real time. can be obtained

In addition, when the device 300 captures the first area in the third direction through the third camera 113 , the device 300 captures the first area in the third direction from the third camera 113 The 1-3 image information may be acquired. In this case, the third camera 113 is capturing the first area in the third direction, and the device 300 records the 1-3 image information generated by capturing the first area in the third direction in real time. can be obtained

On the other hand, if it is detected that the object does not exist in the first area in step S503, since only the first camera 111 of the first CCTV 110 is operating, the device 300 receives the first from the first camera 111. -1 Image information can be obtained. When only the 1-1 image information is obtained, the device 300 may set the 1-1 image information as the first image information.

After step S506, in step S508, the device 300 has the first camera 111 and the second camera 112 of the first CCTV 110 in operation, so the 1-1 image from the first camera 111 is Information may be acquired, and 1-2 th image information may be acquired from the second camera 112 . When the 1-1 image information and the 1-2 image information are obtained, the device 300 may generate the first image information by combining the 1-1 image information and the 1-2 image information. In this case, in the device 300 , the 1-1 image information is image information generated by photographing the first area in the first direction, and the 1-2 image information is an image generated by photographing the first area in the second direction. Since it is information, the 1-1 image information and the 1-2 th image information may be combined in consideration of the first direction and the second direction.

After step S507, in step S509, the device 300 has the first camera 111, the second camera 112, and the third camera 113 of the first CCTV 110 are operating, so the first camera 111 ) may obtain 1-1 image information, obtain 1-2 image information from the second camera 112 , and obtain 1-3 image information from the third camera 113 . 1-1 image information When the 1-1 image information and the 1-3 image information are obtained, the device 300 stores the 1-1 image information, the 1-2 image information, and the 1-3 image information In combination, the first image information may be generated. In this case, in the device 300 , the 1-1 image information is image information generated by photographing the first area in the first direction, and the 1-2 image information is an image generated by photographing the first area in the second direction. information, and since the 1-3 image information is image information generated by photographing the first area in the third direction, considering the first direction, the second direction, and the third direction, the 1-1 image information, the first -2 image information and 1-3 image information can be combined.

After steps S508 and S509, returning to step S503, the device 300 may detect whether an object exists in the first area.

Specifically, the first image information is generated by combining the 1-1 image information and the 1-2 image information, or the 1-1 image information, the 1-2 image information, and the 1-3 image information are combined Thus, after generating the first image information, the device 300 extracts an image of the most recently acquired image from the first image information as a first image, encodes the first image to generate a first input signal, , after inputting the first input signal to the artificial neural network, obtaining the first output signal based on the result of the input of the artificial neural network, and determining whether an object exists in the first region based on the first output signal can be detected.

6 is a flowchart illustrating a process of obtaining result image information corresponding to request information according to an exemplary embodiment.

Referring to FIG. 6 , first, in step S601 , the device 300 may collect image information from a plurality of CCTVs 100 . At this time, the image information may include an image captured by each of the plurality of CCTVs 100, the time taken, location information of each of the plurality of CCTVs 100, and unique identification information for each of the plurality of CCTVs 100. .

In step S602, the device 300 may generate tag information corresponding to the image information through the artificial intelligence module. At this time, the tag information includes object information (for example, two men, one car, two buildings, etc.) that appears in the image information, information on the state of the object (for example, moving, stopping, unexpected situation, etc.), It may include environmental information (weather information, etc.).

The artificial intelligence module can perform learning so that information about an object can be derived from image information using a deep learning technique, which is a field of machine learning.

In addition, the artificial intelligence module may calculate the weight of a plurality of inputs in the function through learning through deep learning.

In addition, various models such as RNN (Recurrent Neural Network), DNN (Deep Neural Network), and DRNN (Dynamic Recurrent Neural Network) may be used as an AI network model used for such learning. Here, RNN is a deep learning technique that considers current data and past data simultaneously. Recurrent neural network (RNN) refers to a neural network in which connections between units constituting an artificial neural network constitute a directed cycle. Furthermore, various methods may be used for a structure capable of constructing a recurrent neural network (RNN), for example, a fully recurrent network, a hopfield network, an Elman network, an ESN (Echo). state network), long short term memory network (LSTM), bi-directional RNN, continuous-time RNN (CTRNN), hierarchical RNN, and secondary RNN are representative examples. In addition, as a method for learning a recurrent neural network (RNN), methods such as gradient descent, Hessian Free Optimization, and Global Optimization Method may be used.

In addition, a machine learning model can be created by using a Supported Vector Machine (SVM) neural network algorithm that is well used for pattern recognition as an artificial intelligence network model. Here, SVM, that is, support vector machine, is a supervised learning model for pattern recognition and data analysis as one of the fields of machine learning, and can be mainly used for classification and regression analysis. When lost, the SVM algorithm can create a non-stochastic binary linear classification model that determines which category the new data belongs to based on the given data set.

Through this, the artificial intelligence module can more accurately generate tag information from image information.

In step S603 , the device 300 may store image information and tag information in a database. In this case, the database is a component included in the device 300 , and various pieces of information may be classified and stored.

In step S604 , the device 300 may receive first request information for searching for an image from the user terminal. Here, the first request information may include first object information, first time information, and first place information. That is, the device 300 may receive the first request information from the user terminal used by the user in order to search for an image desired by the user from among the images captured by the plurality of CCTVs 100 . To this end, the device 300 may be configured to communicate with the user terminal by wire or wireless, receives information from the user terminal, controls the operation of the user terminal according to the received information, and displays which information on the screen of the user terminal. You can control what is displayed.

The first request information may include an object desired by the user, first object information about the state of the object, first time information for selecting a specific time, and first place information for selecting a specific place. For example, when a user wants to check a man passing by a place A, the first object information, the man, the moving information, and the first place information, the information A, are transmitted to the device 300 through the user terminal. can In this case, the first time information may not be specified, and in this case, it may be requested to search all times.

In step S605 , the device 300 may extract first result image information corresponding to the first request information from among the image information stored in the database. In this case, the tag information and the first request information may not completely match. Accordingly, the device 300 calculates the similarity between the words included in the tag information and the words included in the first request information based on word-embedding, and extracts the first result image information based on the similarity. You may. At this time, word-embedding is a natural language processing modeling technology that maps words to low-dimensional real vectors to determine the similarity and importance between words and places semantically similar words closely. can

The first result image information may be all image information corresponding to the first request information. It can be cumbersome to do.

Accordingly, the device 300 may sort all the image information corresponding to the first requested information in the order of the highest matching score between the first requested information and the tag information, and extract only the top 10 as the first result image information. there is.

In step S606 , the device 300 may transmit the first result image information to the user terminal.

7 is a flowchart illustrating a process of extracting and generating second result image information and third result image information according to an exemplary embodiment.

Referring to FIG. 7 , first, in step S701 , the device 300 may generate second request information based on tag information included in the first result image information. In this case, the second request information may include second object information, second time information, and second place information.

According to one embodiment, the first result image information is extracted based on the user's first request information, and there is a concern that the desired image may not be searched because a keyword for searching for the image desired by the user is not included in the first request information there is

Accordingly, in order to more accurately and closely search for an image desired by the user, the device 300 performs the second object information, the second time based on the tag information of the first result image information extracted based on the user's first request information. The second request information including the information and the second place information may be generated.

For example, if you want to find a woman wearing blue clothes passing by place A, the user may mistake it for a man in black clothes, and as the first request information, A, black clothes, men, and moving can be transmitted to the device 300 . there is. At this time, although there may be an image that 100% matches the first requested information, an image that only partially matches may be extracted as the first result image information. In this case, in the first result image information to which only a portion is matched, as tag information, A, blue clothes, women, and moving information may be stored. In this case, the second request information may be generated as A, blue clothes, woman, and moving. Through this, although the user has delivered the first request information by mistake, it is possible to obtain a substantially desired image.

In step S702, the device 300 may select the second CCTV 120 located within a preset threshold distance range based on the location information of the first CCTV 110 that has captured the first result image information. This is to extract image information containing context and circumstances according to the movement of the moving object as will be described later, and the critical distance range may be set based on the moving speed of the object. For example, the critical distance range when the object is a person may be narrower than the critical distance range when the object is a car.

In step S703 , the device 300 may extract second resultant image information corresponding to the second request information from among the image information obtained from the second CCTV 120 .

It may be very inefficient to extract the second result image information from all image information stored in the database based on the second request information. This is because it can lead to the expansion of meaningless data. Therefore, it is preferable to first filter by the distance from the first CCTV 110 and then extract the second result image information based on the second request information.

In step S704 , the device 300 may calculate a movement direction of the object corresponding to the first object information based on the first result image information.

Thereafter, the device 300 may generate the third result image information based on the moving direction of the object and the location information of the second CCTV 120 .

Specifically, in step S705, the device 300 compares the positions of the first CCTV 110 and the second CCTV 120 based on the movement direction of the object, and the second CCTV 120 determines the movement direction of the object. It can be confirmed whether the center is located before the first CCTV (110). This is to intensively extract only the image in which the object desired by the user is captured.

If it is confirmed in step S705 that the location of the second CCTV 120 is located before the first CCTV 110 with respect to the moving direction of the object, in step S706, the device 300 performs the second result of the image information. Third result image information may be generated by extracting only an image shifted to the past by a preset shift time range from the 1-time information. In this case, the shift time range may be set based on the moving speed of the object. For example, the shift time range when the object is a person may be shorter than the shift time range when the object is a car.

If it is confirmed in step S705 that the position of the second CCTV 120 is located after the first CCTV 110 with respect to the moving direction of the object, in step S707, the device 300 performs the second result of the image information. Third result image information may be generated by extracting only the image shifted forward by the shift time range from the 1-time information.

When the third result image information is generated, the device 300 may transmit the first result image information, the second result image information, and the third result image information to the user terminal.

8 is a diagram for explaining an artificial neural network according to an embodiment.

According to an embodiment, the device 300 may generate an input signal by encoding an image of a video generated by shooting through CCTV, and the artificial neural network 801 receives the input signal as an input, and the object recognition probability 802 ) and the object type 803 can be output.

Encoding according to an embodiment may be performed by storing color information for each pixel of an image in the form of a digitized data sheet, and the color information may include RGB color, brightness information, and saturation information of one pixel. , but not limited thereto.

According to an embodiment, the artificial neural network 801 is implemented as a convolutional neural network, and the artificial neural network 801 may be composed of a feature extraction neural network 810 and a classification neural network 820, and the feature extraction neural network 810 is An operation of separating an object and a background from an image extracted from image information may be performed, and the classification neural network 820 may recognize an object therefrom and perform an operation of identifying an object recognition probability and an object type. In the method for the feature extraction neural network 810 to distinguish an object and a background, the change in each value of color information from the data sheet of the first input signal encoding the image is 30 in 6 or more of 8 pixels including one pixel. A group of pixels detected as having a change of % or more may be used as the boundary between the object and the background, but is not limited thereto.

According to an embodiment, the feature extraction neural network 810 stacks the input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes a convolution operation, a convolution filter, and an activation function. The calculation of the convolution filter is adjusted according to the matrix size of the target input, but a 9X9 matrix is generally used. The activation function generally uses, but is not limited to, a ReLU function, a sigmoid function, and a tanh function. The pooling layer is a layer that reduces the size of the input matrix, and uses a method of extracting representative values by tying pixels in a specific area. In general, the average value or the maximum value is often used for the calculation of the pooling layer, but is not limited thereto. The operation is performed using a square matrix, which is usually a 9x9 matrix. The convolutional layer and the pooling layer are repeated alternately until the corresponding input becomes small enough while maintaining the difference.

The classification neural network 820 according to an embodiment classifies the object separated from the background through the feature extraction neural network 810 according to its shape and continuity, and determines the object recognition probability 802 and the object type 803 . can Information stored in the database can be used for object comparison. The classification neural network 820 prioritizes the task of identifying the object recognition probability 802 , and may facilitate the recognition of the object type 803 according to the shape and size of the identified object.

According to an embodiment, the classification neural network 820 has a hidden layer and an output layer. In the artificial neural network 801 in the device 300 , there are generally 5 or more hidden layers, and 80 nodes of each hidden layer are designated, but more may be specified in some cases. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the artificial neural network 801 may be 50 in total.

In the output of the artificial neural network 801 according to an embodiment, the upper 25 nodes among 50 nodes of the output layer may indicate the object recognition probability 802, and the lower 25 nodes may indicate object types ( 803) can be indicated. The method of matching the top 25 nodes with the bottom 25 nodes is a method of matching the top nth node and the bottom nth node, and the nth node in the whole corresponds to the 25+nth node in the whole can proceed with For example, the 1st node corresponds to the 26th node, the 2nd node corresponds to the 27th node, the 10th node corresponds to the 35th node, and the 25th node corresponds to the 50th node. can proceed in this way. The object recognition probability 802 may be output as code information corresponding to the probability, but is not limited thereto. An output layer node having no output value among 50 output layer nodes of the artificial neural network 801 may output the number '0' as its output value. Among the top 25 nodes, nodes including this number '0' may be regarded as having no corresponding object recognition probability. If there are more than 25 classified object types, the remaining object types may be automatically processed subsequently after all output values generated in advance are processed.

According to an embodiment, the artificial neural network 801 may check the object recognition probability 802 and the object type 803 to determine the object recognition probability for each object type, and the object recognition probability is greater than a preset reference value. If confirmed, we can generate an output that the object exists.

For example, the artificial neural network 801 receives an image extracted from the first image information generated by photographing of the first area, and then analyzes the object recognition probability for each type of object. As a result, the object recognition probability of the first object If it is confirmed that the value is greater than this reference value, an output value may be generated as the presence of the first object in the first area. At this time, when it is confirmed that the first object exists in the first area, the artificial neural network 801 analyzes how many objects exist in the first area based on the image extracted from the first image information, and the first area It is also possible to generate the number of first objects existing in the output value.

According to an embodiment, the artificial neural network 801 may learn by receiving the first learning signal generated by the corrected correct answer input by the administrator when the administrator discovers a problem in object identification according to the artificial neural network 801 . The problem of object recognition according to the artificial neural network 801 may mean that there is a problem in the object recognition probability 802 and the object type 803 .

The first learning signal according to an embodiment is created based on the error between the correct answer and the output value, and in some cases, SGD using delta, a batch method, or a method following a backpropagation algorithm may be used. According to the first learning signal, the artificial neural network 801 performs learning by modifying the existing weights, and in some cases, momentum may be used. A cost function can be used to calculate the error, and a cross entropy function can be used as the cost function.

9 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

The device 300 according to an embodiment includes a processor 310 and a memory 320 . The processor 310 may include at least one of the devices described above with reference to FIGS. 1 to 8 , or perform at least one method described above with reference to FIGS. 1 to 8 . A person or organization using the apparatus 300 may provide a service related to some or all of the methods described above with reference to FIGS. 1 to 8 .

The memory 320 may store information related to the above-described methods or a program for implementing the above-described methods. The memory 320 may be a volatile memory or a non-volatile memory.

The processor 310 may execute a program and control the device 300 . The code of the program executed by the processor 310 may be stored in the memory 320 . The device 300 may be connected to an external device (eg, a personal computer or a network) through an input/output device (not shown), and may exchange data through wired/wireless communication.

The device 300 may be used to train an artificial neural network or to use a learned artificial neural network. The memory 320 may include a learning or learned artificial neural network. The processor 310 may learn or execute an artificial neural network algorithm stored in the memory 320 . The apparatus 300 for learning an artificial neural network and the apparatus 300 for using the learned artificial neural network may be the same or may be separate.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (3)

In a method of controlling a CCTV for automatic object recognition based on artificial intelligence, performed by a device,
acquiring first image information generated by shooting of the first area from the first CCTV when the first area is being photographed through the first CCTV;
extracting an image of the most recently acquired image from the first image information as a first image;
generating a first input signal by encoding the first image;
inputting the first input signal to an artificial neural network, and obtaining a first output signal based on a result of the input of the artificial neural network;
detecting whether an object exists in the first area based on the first output signal;
detecting a type of an object located in the first area based on the first output signal when it is detected that the object is present in the first area;
recognizing that the first object is located in the first area when the object located in the first area is detected as the first object; and
When it is recognized that the first object is located in the first area, the first image information obtained after extraction of the first image is transmitted to the manager terminal, and the first image information is displayed on the screen of the manager terminal control to be displayed;
The step of obtaining the first image information includes:
Among the first CCTVs, a first camera is installed in a first direction of the first zone, a second camera is installed in a second direction of the first zone, and a third camera is installed in a third direction of the first zone. when a camera is installed, controlling only the first camera to operate to perform photographing;
When the first area is photographed in the first direction through the first camera, 1-1 image information generated by photographing the first area in the first direction from the first camera obtaining;
extracting an image of the most recently acquired image from the 1-1 image information as a 1-1 image;
generating a 1-1 input signal by encoding the 1-1 image;
inputting the 1-1 input signal to an artificial neural network, and obtaining a 1-1 output signal based on a result of the input of the artificial neural network;
detecting whether an object exists in the first area based on the 1-1 output signal;
when it is detected that no object exists in the first area, controlling only the first camera to operate to perform photographing;
detecting the number of objects located in the first area based on the 1-1 output signal when it is detected that an object exists in the first area;
when it is determined that the number of objects located in the first area is smaller than a preset reference value, controlling the first camera and the second camera to operate to perform photographing;
controlling the first camera, the second camera, and the third camera to operate to perform photographing when it is confirmed that the number of objects located in the first area is greater than the reference value;
When the first area is photographed in the second direction through the second camera, 1-2 th image information generated by photographing the first area in the second direction from the second camera obtaining;
When the first area is photographed in the third direction through the third camera, 1-3 image information generated by photographing the first area in the third direction from the third camera obtaining;
setting the 1-1 image information as the first image information when only the 1-1 image information is obtained;
generating the first image information by combining the 1-1 image information and the 1-2 th image information when the 1-1 image information and the 1-2 image information are obtained; and
When the 1-1 image information, the 1-2 image information, and the 1-3 image information are obtained, the 1-1 image information, the 1-2 image information, and the 1-3 image Combining the information, comprising the step of generating the first image information,
CCTV control method for automatic object recognition based on artificial intelligence. The method of claim 1,
when it is recognized that the first object is located in the first area, tracking and analyzing the movement of the first object based on the first image information obtained after extraction of the first image;
As a result of tracking and analyzing the movement of the first object, if it is confirmed that the first object has moved out of the first area and moved in the direction of the second area, a second CCTV that is filming the second area checking;
acquiring second image information generated by shooting of the second area from the second CCTV when the second area is being photographed through the second CCTV; and
Transmitting the second image information to the manager terminal, further comprising the step of controlling the second image information to be displayed on the screen of the manager terminal subsequent to the first image information,
CCTV control method for automatic object recognition based on artificial intelligence. delete

Images