KR102550117B1 - Method and System for Video Encoding Based on Object Detection Tracking - Google Patents

Abstract

The present invention relates to a method and a system for video encoding based on object detection tracking. The system for video encoding and transmission based on object detection tracking according to one embodiment of the present invention may comprise: an image analysis device detecting and tracking an area that requires identification of objects within a captured image when compressing the captured image from a camera and transmitting the same to another device and compressing tracked objects differently depending on whether they are objects of interest to allow the other device to restore the objects of interest at a higher resolution than the standard resolution; and an external device operating as the other device, receiving object data of tracked objects provided by the image analysis device to restore the object of interest at a high resolution, and using the high-resolution object of interest to identify and recognize the tracked object. Accordingly, the quality of the video can be enhanced.

Description

Video encoding method based on object detection tracking, and system thereof {Method and System for Video Encoding Based on Object Detection Tracking}

The present invention relates to a video encoding and transmission method based on object detection and tracking, and a system thereof, and more particularly, to extract an object using a deep learning-based object detection and tracking system mounted on an artificial intelligence (AI) camera and After encoding in high resolution through a separate video encoder in units of detected objects, the system and monitoring system that transmits after convergence with video analysis metadata receive it and output it in high resolution in units of objects, and the secondary analysis server utilizes it It relates to a video encoding and transmission method and system based on object detection tracking capable of object identification and recognition.

A function of detecting and tracking an object by analyzing an image within a camera and generating an event based on the detected function is being used. In the past, motion-based object detection technology, that is, motion-based detection technology was used, but recently, deep learning-based object detection technology is commonly used. Deep learning-based object detection technology starts from simply identifying the type of object such as a person or vehicle, and has recently become generalized to analyze object attributes or identify objects. Attribute analysis of an object refers to a function of classifying an object, such as recognizing a clearly distinguishable clothing of a pedestrian or classifying a gender. In addition, object identification refers to a function of verifying or identifying whether a person is the same person through facial features or whole body features.

In the case of recent CCTV cameras, IP cameras are common, and in this case, compression technology through an encoder is used due to limitations in network resources and storage capacity. The problem is that in order to improve analysis performance, high-resolution and high-quality images must be received, but in real environments, compression rates are often used with a very high compression rate.

Of course, face image acquisition method and device technology using an IP camera have been disclosed in the past, but in the case of this technology, face detection and feature extraction are transmitted together with metadata, but the metadata contents are coordinates for reconstructing the image on the screen. It does not include a function to select video quality for each type of information or object.

Korea Patent Registration No. 10-1051389 (2011.07.18) Korea Patent Registration No. 10-2091088 (2020.03.13) Korea Patent Registration No. 10-1399785 (2014.05.20)

An embodiment of the present invention extracts an object using a deep learning-based object detection and tracking system mounted on an artificial intelligence (AI) camera, encodes the detected object in high resolution through a separate video encoder, and analyzes the image. A video encoding and transmission method based on object detection and tracking in which systems and monitoring systems that transmit and converge with meta data receive it and output it in high resolution in object units, and the secondary analysis server utilizes this to identify and recognize objects, And the purpose is to provide the system.

A video encoding and transmission system based on object detection and tracking according to an embodiment of the present invention detects and tracks an area in which an object needs to be identified in a captured image when compressing a captured image of a camera and transmitting the same to another device, and An image analysis device that compresses an object differently depending on whether it is an object of interest so that the object of interest is reconstructed at a high resolution higher than the reference resolution in the other device, and a tracking object that operates as the other device and is provided by the image analysis device and an external device for receiving object data, reconstructing the object of interest in the high resolution, and identifying and recognizing the tracked object using the reconstructed high resolution object of interest.

The image analysis device may be provided in the camera, and the external device may include a device provided at a remote location via a communication network to perform at least one operation of monitoring, data storage, search, and image analysis.

The video analysis apparatus may include an encoding unit for each multi-object that differently compresses the object of interest and the object of interest.

The image analysis device may compress a human object and a vehicle object in the photographed image as the object of interest to be reconstructed at the high resolution.

The video analysis device transmits metadata of the tracked object as the object data, and generates map coordinates for the tracked object on pre-stored map data in relation to the shooting location of the camera to determine the location of the tracked object. It can be included in the metadata and transmitted.

A video encoding and transmission method based on object detection and tracking according to an embodiment of the present invention, when an image analysis device compresses a captured image of a camera and transmits it to another device, detects and tracks a region in the captured image where object identification is required. and compressing the tracked object differently depending on whether it is an object of interest so that the object of interest is reconstructed in a high resolution higher than a reference resolution in the other device, and an external device operating as the other device, the image analysis device Receiving object data of a tracking object provided by , reconstructing the object of interest in the high resolution, and identifying and recognizing the tracking object using the reconstructed high resolution object of interest.

The image analysis device may be provided in the camera, and the external device may include a device provided at a remote location via a communication network to perform at least one operation of monitoring, data storage, search, and image analysis.

In the multi-object encoding unit included in the video analysis device, compression of the object of interest and the object of interest may be performed differently.

In the differently compressing, the human object and the vehicle object in the photographed image as the object of interest may be compressed to be reconstructed at the high resolution.

The video analysis device transmits the metadata of the tracked object as the object data, and generates map coordinates for the tracked object on pre-stored map data in relation to the shooting location of the camera to determine the location of the tracked object. The method may further include transmitting the metadata by including it in the metadata.

According to an embodiment of the present invention, the quality of video is degraded due to the low bandwidth transmitted in a conventional bandwidth environment, and in particular, severely moving objects, such as pedestrians and vehicles, are severely blocked, making it difficult to identify objects. problems that arise can be resolved.

In addition, according to an embodiment of the present invention, the object identification ability can be improved by detecting and tracking an area requiring object identification in the edge equipment, compressing only this part in high quality and transmitting the video as a video.

In the embodiment of the present invention, video quality can be obtained by encoding the object of interest by generating video and still images by cutting each object of interest through an object detection and tracking system mounted in an AI camera.

Furthermore, the embodiment of the present invention has the advantage that a person can also identify or recognize a vehicle when a high-quality image is acquired, and the image or video clip obtained in this high resolution is used as input data that can be used for the purpose of secondary video analysis. It can be.

1 is a diagram of a video encoding system based on edge-based object detection and tracking according to an embodiment of the present invention and a monitoring, storage, search, and analysis device associated therewith.
2 is a diagram showing detailed operations of a multi-object detection and tracking unit.
3 and 4 are diagrams showing detailed operations of the encoding unit for each multi-object.
5 is a diagram showing detailed operations of a multi-object video stream unit.
FIG. 6 is a diagram illustrating the detailed operation of FIG. 5 schematically.
7 is a diagram illustrating a video encoding system based on object detection tracking according to another embodiment of the present invention.
8 is a block diagram illustrating a detailed structure of the edge type video analysis device of FIG. 1 or the edge device of FIG. 7 .
9 is a flowchart illustrating a video encoding and transmission method based on object detection tracking according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram of a video encoding system based on edge-based object detection and tracking according to an embodiment of the present invention and a monitoring, storage, search, and analysis device associated therewith.

As shown in FIG. 1, a video encoding system based on edge-based object detection and tracking according to an embodiment of the present invention and a monitoring, storage, search, and analysis device associated therewith (hereinafter, a video encoding system based on object detection and tracking) ( 90) includes some or all of the edge-type image analysis device 100, a communication network (not shown), and the image processing system 110.

Here, "including some or all" means that some components constituting the image processing system 110, i.e., devices, are omitted so that the video encoding system 90 based on object detection and tracking is configured or the image processing system 110 is configured. This means that some of the devices or components constituting the communication network can be integrated into a network device (eg, a wireless switching device, etc.), and will be described as including all of them to help a sufficient understanding of the present invention.

The edge-type video analysis device 100 may be installed in an arbitrary place, such as a public place or a crime-prone area, and may be installed in a camera that captures the place. Alternatively, it may be installed and operated adjacent to the camera. When installed adjacently, it is preferable not to perform separate compression. The camera may be a CCTV camera or an IP camera, but may include an artificial intelligence (AI) camera in which a deep learning-based object detection and tracking system is implemented. The edge-type image analysis device 100 can operate in conjunction with various types of cameras, and substantially, the edge-type image analysis device 100 captures a high-resolution image taken from a camera, that is, a resolution corresponding to the resolution of the camera. You can see that the video is provided. In this regard, it is preferable that the edge-type image analysis device 100 is mounted in a camera and operated while being connected to the camera. Through this, since processing (eg, encoding/decoding, etc.) of the captured image does not occur, it is possible to receive an image corresponding to the resolution of the camera. Of course, since the resolution of a camera depends on the performance of a lens, a high-resolution image can be obtained when a color filter or the like provided in the lens has high resolution. However, in an embodiment of the present invention, a resolution equal to or higher than the reference resolution is defined as a high resolution. Therefore, a lower resolution may be a lower resolution, and the reference resolution may be regarded as a minimum resolution for implementing a technique according to an embodiment of the present invention.

The edge-type image analysis device 100 may perform an image analysis operation on a photographed image received from a camera. For example, when a camera generates 60 video frames per second, the edge-type image analysis device 100 may analyze an image of a video frame and track an object in the image according to a time change. Largely, the image may include a background image and an object within the background image. Here, the object may be a person or a vehicle, and various types of object objects may be included. Of course, animals may also be included, and trees may also be included. Objects may include various types of objects in the background image depending on the location where the camera is installed and operated. Meta data of the tracking object can be created in , and location information of the tracking object can be included in the meta data. Of course, the edge-type video analysis device 100 can increase the accuracy of object detection or classification by mounting an artificial intelligence deep learning program therein. For example, when objects overlap, it may be difficult to recognize a specific object. In this case, the type of the held object may be determined based on information in the next video frame while object detection is suspended. This process can detect and classify objects through learning through deep learning.

The edge-type image analysis device 100 according to an embodiment of the present invention may divide an object of interest and an object of interest when analyzing an image. Here, the object of interest may be a person or a vehicle. In addition, the uninteresting object may include objects other than the interest object. Accordingly, the edge-type image analysis apparatus 100 may provide compression of the object of interest and non-interest objects differently when compressing and transmitting a captured image or, more precisely, when transmitting an analysis result obtained by analyzing the captured image. For example, the non-interest object may be compressed and provided according to a compression rate or compression method preset in the edge-type video analysis device 100, or more accurately, according to the performance of an encoder provided, but the object of interest may be provided by a remote device In order to reconstruct an object of interest in high resolution, compressed data of video data may be transmitted with a compression rate lower than that of an object of non-interest. If the compression, that is, the encoding, is strong, a lot of video data is omitted, and thus, a lot of compression loss may occur in the process. Therefore, in an embodiment of the present invention, compression of a non-interest object (or background image) and an object of interest are performed differently, and furthermore, it is possible to transmit data without performing compression in the case of an object of interest. Since this can be made in various ways according to the intention of the system designer, the embodiment of the present invention will not be particularly limited to any one form.

In addition, when the edge-type image analysis device 100 generates and transmits meta data for a background image of a captured image or a tracked object in a background image, for example, compression information (eg, provided for decoding) of an object of interest that is compressed differently ), and location information of the object of interest in the background image may be transmitted together, and based on this, restoration of the object of interest may be performed. Of course, the location information here is location information within a photographed image, and may be map coordinates on map data using map data of a photographed location.

As a result of the above, the edge-type image analysis device 100 generates video and still images by cutting each object of interest through the object detection and tracking system mounted on the AI camera, and encodes the object of interest. When restoring, high-resolution video quality can be obtained. Acquiring a high-quality image can make identification or recognition of a vehicle much easier for humans. In addition, the image or video clip obtained in this high resolution can be used as input data that can be used for the purpose of secondary video analysis, thereby increasing accuracy.

In addition, in the embodiment of the present invention, the quality of video is degraded due to the low bandwidth transmitted in an environment where the bandwidth is considerably low, and in particular, severely moving objects, such as pedestrians and vehicles, are severely blocked, making it difficult to identify objects. Many of these problems can be solved. The edge equipment detects and tracks the area that needs object identification, compresses only this part to high quality, and transmits it as a video to increase the object identification ability.

An image processing system 110 according to an embodiment of the present invention includes a system capable of live monitoring a multi-object video stream provided by an edge-type video analysis device 100 and a system for storing/playing/searching such a multi-object video stream. system, and finally an analysis system that performs secondary analysis based on these multi-object video streams.

A monitoring system such as the monitoring device 111 receives a multi-object video stream provided by the edge type video analysis device 100 . The monitoring system includes a method of outputting a background image and outputting video for each of multiple objects thereon. Also, the monitoring system includes a method of outputting video for each of multiple objects on top of a real-time video stream. The monitoring system may include a method of selecting and outputting only an object of interest when outputting multiple objects. For example, it may be output based on the type of object of interest, property information, event information of an incident or accident, and the like.

In addition, the storage system constituting the image processing system 110 receives and stores the multi-object video stream provided by the edge-type image analysis device 100 . A storage system such as the storage/playback/search device 112 includes a system for storing both a background image and a video stream for each object along with a real-time image. The playback and retrieval system such as the storage/playback/search device 112 includes a function of outputting the same as a real-time video stream by moving and outputting video encoded for each multi-object to its original position. and a system for selecting and outputting only the video of the object of interest. When playing in the playback system, a function of selecting a real-time video stream as a background or a learned background image may be provided.

A secondary analysis system such as the secondary analysis device 113 performs secondary analysis using object detection and tracking information received from the edge-type video analysis device 100, which is a primary analysis server. In addition, it includes a system that performs higher performance by referring to an image of an object unit encoded with high quality. For example, an analysis operation for detecting an event related to a human or vehicle incident or accident may be performed.

Of course, the image processing system 110 operates in conjunction with the video encoding system of FIG. 1, that is, the edge-type image analysis device 100, and may include various types of devices. Representatively, as shown in FIG. 1, a server device for providing various video services such as a monitoring device 111, a storage/playback/search device 112, and a secondary analysis device 113 may be included. The monitoring device 111 may be installed and operated in a control center such as a government agency, and the storage / playback / search device 112 may include devices such as various servers that provide video playback or search services. The difference analysis device 113 may include a device that performs a new analysis operation based on the image analysis result provided by the edge-type image analysis device 100 . For example, when the monitoring device 111 of the control center needs to know an incident or accident occurring at a location through a photographed image of a camera installed at an arbitrary location, the secondary analysis device 113 is an edge-type image analysis device Based on the video analysis result provided by (100), an event such as an incident or accident may be determined and notified to the monitoring device 111.

In summary, the monitoring device 111 is installed in a control center, etc. so that control personnel can detect events such as incidents and accidents that occur in a place where a camera is installed. Therefore, the monitoring device 111 may include a computer owned by a control agent or the like. In addition, the storage/reproduction/search device 112 may store video frame data of a moving image provided by the edge-type image analysis device 100, for example, and may also store audio data together. Accordingly, when the monitoring device 111 wants to view a video on a specific date and in a specific place again, it can be provided through search or the like so that it can be reproduced. For example, when a crime or a traffic accident occurs in a specific place and a police station or the like requests an image of the place, a service related thereto may be provided.

Furthermore, the secondary analysis device 113 may mean that a video analysis is primarily performed on an image captured by a camera, and a new image analysis operation is performed based on the analysis result. Of course, in this process, it is possible to perform a secondary analysis operation by using the video stored in the storage / playback / search device 112 together, and representatively, it is possible to perform an operation related to event determination such as an incident or an accident. have been explained Of course, even in the case of the secondary analysis device 113, a deep learning program can be stored to determine events of accidents and accidents. The secondary analysis device 113 can determine incidents and accidents through learning of the big data after forming big data, and by applying a deep learning program, it will be possible to increase the accuracy of the judgment.

In addition to the above, the edge-type image analysis device 100 and the image processing system 110 of FIG. 1 may perform various operations, and other details will be continuously reviewed with reference to FIGS. 2 to 5.

2 is a diagram showing detailed operations of a multi-object detection and tracking unit, FIGS. 3 and 4 are diagrams showing detailed operations of an encoding unit for each multi-object, FIG. 5 is a diagram showing detailed operations of a multi-object video stream unit, FIG. 6 is FIG. 5 It is a diagram showing the detailed operation of.

Referring to FIGS. 2 to 6 together with FIG. 1 for convenience of explanation, the edge-type image analysis device 100 according to an embodiment of the present invention, as shown in FIG. 1, includes an image input unit 101 and a background image generator. (102), a multi-object detection and tracking unit 103, a multi-object encoding unit 104, and a multi-object video stream unit 105. It is not very different from the meaning, so I would like to replace it with the contents.

The image input unit 101 receives an image within the edge image analysis device 100 to perform a video encoding and transmission method based on object detection and tracking in the edge image analysis device 100 . That is, the edge-type video analysis device 100 can be considered to include a device that receives an image from an external camera or an image input system capable of capturing an image inside an edge device such as an IP camera.

The background image generator 102 performs a function of generating a background image from a captured image provided by the image input unit 101 . That is, the edge-type video analysis device 100 includes a background image generation unit 102 that generates a background image that becomes a background of each object stream.

The multi-object detection and tracking unit 103 analyzes the input image input to the image input unit 101 to simultaneously detect and track multiple objects. The edge-type image analysis device 100 includes a multi-object detection and tracking unit 103 that detects objects from an input image and tracks objects detected in a series of frames. In the process of performing such tracking, tracking information may be generated in the form of direction vector information or scalar information. Of course, this may be generated in the form of map coordinates on map data.

2 shows detailed operations of the multi-object detection and tracking unit. The multi-object detection and tracking unit 103 receives real-time images from the video input unit 101 and detects multi-objects (S200 and S210). That is, an object of interest, such as a person or a vehicle, in the background image and other non-interest objects may be detected. Of course, object detection may be performed through an object detector, and a DCNN-based object detector may be used as the object detector to increase detection accuracy. Alternatively, after learning various types of images of people and vehicles, it is possible to detect and classify objects based on them. In addition, the multi-object detection and tracking unit 103 performs a tracking operation of the detected multi-objects, and can be regarded as motion tracking according to time changes. For example, if the object detected in the (n-1)th video frame and the object in the nth video frame have the same properties (eg shape, color, etc.), the two objects are the same object and their respective positions (information) Based on this, movement can be tracked. Then, the multi-object detection and tracking unit 103 may generate metadata for the multi-object. Metadata can be seen as having various information related to objects of interest or non-interest, which are tracking objects. For example, various information such as the shape, type, and color of an object may be included. Also, metadata may include object location information. Alternatively, compression information related to compression of the object of interest may be included.

The multi-object encoding unit 104 corresponds to a part that generates a video in units of multi-tracking object information. The edge-type video analysis apparatus 100 may include a multi-object video generation unit that generates a video stream for each multi-object based on an input image and multi-object tracking information, that is, an encoding unit for each multi-object.

The multi-object encoding unit 104 may compress video data for various types of objects provided by the multi-object detection and tracking unit 103, that is, perform an encoding operation. In this process, the multi-object encoding unit 104 may output video data, more precisely, object data, by dividing an object of interest, such as a person or a vehicle, from an object of interest and other non-interest objects, and compressing them using different compression methods. Of course, the multi-object encoding unit 104 according to an embodiment of the present invention performs different compressions depending on the type of object, more precisely, whether or not it is an object of interest, but may also generate object metadata. Although the above multi-object detection tracking unit 103 can generate metadata, the multi-object detection tracking unit 103 can provide the position of an object in a captured image, that is, coordinate information, and in the embodiment of the present invention, Image processing may be performed in a form of generating and providing map coordinates for map data of an arbitrary region. Accordingly, the encoding unit 104 for each multi-object may perform an operation for generating metadata in addition to performing different compression operations depending on whether an object of interest (or a non-interest object, a background image, etc.) exists. In this respect, the multi-object encoding unit 104 may be referred to as a multi-object video generator, and the multi-object encoding unit 104 may be included in the multi-object video generator.

The multi-object encoding unit 104 may include a first encoder for compressing, or encoding, a non-interest object, and a second encoder for compressing an object of interest. However, in an embodiment of the present invention, it may be possible to perform compression only on objects of interest according to a preset compression method, and to transmit objects of interest without performing compression. Of course, a currently used standardization technique may be applied to the preset compression method. It may be configured by intra, inter, or a combination thereof. On the other hand, in the case of the second encoder, the object of interest may be encoded in a method different from that of the first encoder, and encoding is preferably performed in a direction of increasing resolution when reconstructing the object of interest. Accordingly, since the compression rate may mean the degree of compression, the second encoder may have a lower compression rate than the first encoder. Above all, in the embodiment of the present invention, the compression of the object of interest and the object of interest are different, and various compression methods may be used. In addition, when an object of interest such as a person or a vehicle object is classified based on preset information, the detection accuracy of the object of interest and the non-interest object may be increased through learning by configuring the object detector as a deep learning-based detector.

3 and 4 show detailed operations of the multi-object encoding unit 104 of FIG. 1 . When an image is input through the image input unit 101, the multi-object encoding unit 104 may generate a background image and perform an update operation of the generated background image (S300, S310). is changing. In addition, the multi-object encoding unit 104 may encode a background image video (S320). At this time, encoding of the background image may be performed using a preset encoding method, that is, a standardized encoder. Also, the multi-object encoding unit 104 may perform encoding for each object. Real-time video and multi-object tracking information are received and images are generated for each object (S400, S410, S420). A corresponding object image may be classified into an object of interest and an object of non-interest. Video encoding is performed for each object ID (S430), and in this process, the object of interest and the object of interest may be compressed differently. Of course, in the embodiment of the present invention, only the background image is compressed with the existing standardized encoding technology to transmit data, and in the case of objects, it is possible to transmit data with other compression methods. The shape will not be particularly limited.

The multi-object video stream unit 105 of FIG. 1 may correspond to a kind of communication unit. The multi-object video stream unit 105 collects background image, multi-object tracking information, and video information for each multi-object and transmits the information through streaming. Of course, the multi-object video stream unit 105 may perform a data synthesis operation, include the corresponding data in a designated data packet, and provide the data to the image processing system 110. In this way, the edge-type video analysis device 100 includes a real-time video stream unit that encodes the entire input video and provides it as a video stream.

FIG. 5 is a diagram showing detailed operations of the multi-object video stream unit of FIG. 1, and FIG. 6 is a diagram illustrating detailed operations of FIG. As shown in FIG. 5, the multi-object video stream unit 105 of FIG. 1 according to an embodiment of the present invention receives metadata information of multiple objects, encoded video information for each object, and encoded background image information, and transmits them Synchronize with multiple information streams (S500 to S530). For example, as shown in (a) of FIG. 6, when a person and a vehicle object in one unit video frame are detected and transmitted as a stream, they can be simultaneously transmitted by synchronizing time as shown in (b) of FIG. (S540).

7 is a diagram illustrating a video encoding system based on object detection tracking according to another embodiment of the present invention.

As shown in FIG. 7 , a video encoding system 190 based on object detection and tracking according to another embodiment of the present invention includes some or all of an edge device 700, a communication network 710, and an image processing device 720. can include

Here, "including some or all" means that some components such as the communication network 710 are omitted so that the video encoding system 190 based on object detection and tracking is configured, or some components such as the image processing device 720 are included. It means something that can be integrated and configured in a network device (eg, wireless switching device, etc.) constituting the communication network 710, and will be described as including all of them to help a sufficient understanding of the present invention.

The edge device 700 or image processing device 720 in FIG. 7 is not significantly different from the edge type image analysis device 100 or image processing system 110 in FIG. 1 . Therefore, the details are to be replaced with those in FIG. 1 .

However, in the case of the image processing device 720 of FIG. 7, the monitoring device 111, the storage/playback/search device 112, and the secondary analysis device 113 of FIG. 1 are integrated into one device. Show what you can do. Accordingly, the image processing device 720 of FIG. 7 may perform the same operation as in FIG. 1 including functional blocks such as a monitoring unit, a storage/playback/search unit, and a secondary analysis unit. However, in the case of FIG. 1, although the data operation processing speed may be faster than that of FIG. 7, it may be expensive due to the use of a large number of equipment. Therefore, since this configuration may be variously made according to the intention of the system designer, the embodiment of the present invention will not be particularly limited to any one form.

The communication network 710 includes both wired and wireless communication networks. For example, as the communication network 710, a wired or wireless Internet network may be used or interlocked. Here, the wired network includes an Internet network such as a cable network or a public switched telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), and Wibro networks. meaning to include Of course, the communication network 710 according to an embodiment of the present invention is not limited thereto, and may be used as an access network of a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 710 is a wired communication network, an access point within the communication network can access an exchange of a telephone company, but in the case of a wireless communication network, access to an SGSN or GGSN (Gateway GPRS Support Node) operated by a communication company to process data, Data can be processed by connecting to various repeaters such as BTS (Base Transceiver Station), NodeB, and e-NodeB.

Communications network 710 may also include an access point. Access points include small base stations such as femto or pico base stations that are often installed in buildings. Here, the femto or pico base station is classified according to the maximum number of edge devices 700 that can be connected to the small base station. Of course, the access point may include a short-range communication module for performing short-range communication such as ZigBee and Wi-Fi with the edge device 700 and the like. The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, ZigBee, infrared (IrDA), RF (Radio Frequency) such as UHF (Ultra High Frequency) and VHF (Very High Frequency), and ultra-wideband communication (UWB) in addition to Wi-Fi. can Accordingly, the access point may extract the location of the data packet, designate the best communication path for the extracted location, and forward the data packet to the next device, for example, the image processing device 720 along the designated communication path. An access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

8 is a block diagram illustrating a detailed structure of the edge type video analysis device of FIG. 1 or the edge device of FIG. 7 .

As shown in FIG. 8 , the edge device 700 or the edge type image analysis device 100 (hereinafter referred to as an edge device) according to an embodiment of the present invention includes a communication interface unit 800 and a control unit 810 , includes some or all of the edge-based image analysis unit 820 and the storage unit 830.

Here, "including part or all" means that the edge device 700 is configured by omitting some components such as the storage unit 830, or some components such as the edge-based image analysis unit 820 are the controller ( 810), which can be integrated into other components such as, etc., and will be described as including all of them to help a sufficient understanding of the invention.

The communication interface unit 800 may be installed in an arbitrary place and receive a photographed image of a camera photographing the corresponding place. In fact, since the edge device 700 of FIG. 7 can be mounted in a camera, it can operate in conjunction with the lens unit of the camera. Also, the communication interface unit 800 may perform an operation for transmitting an image captured by a camera to the image processing device 720 via the communication network 710 of FIG. 7 .

The communication interface unit 800 may transmit the captured image of the edge device 700 according to the embodiment of the present invention as a stream under the control of the controller 810, and in the process of doing so, the object of interest and the object of interest according to the embodiment of the present invention may be transmitted. Video data according to different compression methods of the object of interest may be transmitted, or data according to different compression methods of the background image and the object in the image may be transmitted. This is so that, for example, the image processing device 720 of FIG. 7 can restore high-resolution images, especially in objects such as people or vehicles, when video data that is compressed and transmitted is restored. Of course, when the metadata is transmitted, since information related to compression of the tracking object and information such as location information in the background image are provided together, the image processing device 720 uses a predetermined communication protocol based on this (e.g. encoding/decoding of the tracking object). etc.), the image can be restored.

The control unit 810 is in charge of overall control operations of the communication interface unit 800 of FIG. 8 , the edge-based image analysis unit 820 and the storage unit 830 . When an image captured by a camera is received through the communication interface unit 800, the controller 810 may temporarily store it in the storage unit 830. In addition, the control unit 810 provides the corresponding captured image to the edge-based image analysis unit 820 so that the image analysis is performed, and object data made of different compression methods generated in this process are combined with the image data of the background image. The communication interface unit 800 may be controlled to transmit.

The edge-based image analysis unit 820 separates a background image and an object from an image captured by a camera, and then performs data compression on the background image according to the performance of an encoder loaded at the time of shipment. Compression may be performed in various ways by intra, inter, or a combination thereof. On the other hand, the edge-based image analysis unit 820 may not perform compression on the separated object, or reduce the compression rate compared to the background image, that is, operate to restore a high-resolution object having a higher resolution than the background image during restoration. Of course, it is possible to construct a separate structure including pixel values of video data related to an object in a data packet for transmission of such data. In other words, it is desirable to implement a technique according to an embodiment of the present invention without changing the standardized specifications of data packets.

Also, in the case of objects, the edge-based image analysis unit 820 according to an embodiment of the present invention may perform compression differently by separating an object of interest from an object of interest. For example, in the traffic control center, since people or vehicles may be a main object of interest when monitoring an image, the object of interest is not compressed or the compression rate is reduced so that the object of interest is restored in high resolution. Therefore, in the embodiment of the present invention, heterogeneous encoders may be included for the above operation.

Furthermore, the edge-based video analysis unit 820 may synthesize the encoded background video, encoded video information for each object, and furthermore, metadata information of multiple objects, perform multi-information stream synchronization, and transmit the synthesized corresponding data Data packets to be transmitted as a stream may be generated and output. Accordingly, the corresponding data packet may be transmitted to the image processing device 720 of FIG. 7 through the communication interface unit 800 under the control of the controller 810 . Of course, according to an embodiment of the present invention, the image processing device 720 may include a separate decoder for restoring or decoding an object of interest, through which a restoration operation may be performed. Since encoding and decoding are a kind of agreement and communication protocol, when an encoding module according to an embodiment of the present invention is installed in the edge device 700, a corresponding decoding module may be installed in the image processing device 720.

The storage unit 830 may temporarily store various types of data processed under the control of the controller 810 . The control unit 810 may temporarily store an image taken by the camera in the storage unit 830 and then retrieve and provide the image to the edge-based image analysis unit 820 .

In addition to the above, the communication interface unit 800, the control unit 810, the edge-based image analysis unit 820, and the storage unit 830 of FIG. 8 can perform various operations, and other details have been sufficiently described above. , I want to replace it with those contents.

The communication interface unit 800, the control unit 810, the edge-based image analysis unit 820, and the storage unit 830 of FIG. 8 according to an embodiment of the present invention are composed of hardware modules physically separated from each other, but each module will be able to store and execute software for performing the above operation therein. However, since the corresponding software is a set of software modules, and each module can be formed of hardware, it will not be particularly limited to the configuration of software and hardware. For example, the storage unit 830 may be hardware storage or memory. However, since it is possible to store information in a software manner (repository), the above content will not be particularly limited.

Meanwhile, as another embodiment of the present invention, the control unit 810 may include a CPU and a memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), a command interpreter, and a registry, and the memory may include RAM. The control circuit performs control operations, the operation unit performs operation of binary bit information, and the instruction interpretation unit performs an operation of converting high-level language into machine language and machine language into high-level language, including an interpreter or compiler. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of operation of the edge-type image analysis device 100 or the edge device 700, the program stored in the edge-based image analysis unit 720 is copied and loaded into memory, that is, RAM. Then, by executing it, the data operation processing speed can be rapidly increased. In the case of a deep learning model, it can be loaded into GPU memory rather than RAM, and can be executed by accelerating the execution speed using the GPU.

9 is a flowchart illustrating a video encoding and transmission method based on object detection tracking according to an embodiment of the present invention.

Referring to FIG. 9 together with FIG. 1 for convenience of explanation, the edge-type video analysis device 100 according to an embodiment of the present invention compresses a captured image of a camera and transmits it to another device such as a server, an object in the captured image Detects and tracks the region that needs to be identified, and compresses the tracked object differently depending on whether it is an object of interest, that is, whether it is an object of interest or not, so that the object of interest is reconstructed with a high resolution higher than the reference resolution in another device (S900). Of course, in the embodiment of the present invention, a separate encoder may be further mounted and used in addition to the encoder previously installed in the current device. The background image can be compressed through an encoder that is basically installed in the device. In addition, compression with a lower compression rate or degree of compression than compression of the background image may be performed on objects in the background image. This is to ensure that high-resolution reconstruction is performed on the object.

Of course, in the embodiment of the present invention, the background image and the non-interest object may be compressed through an encoder pre-loaded in the device, but high-resolution reconstruction may be performed by varying the compression rate only for the object of interest. Furthermore, in the case of objects of interest or non-interest objects, it is possible to process data without performing compression or encoding itself, so the embodiment of the present invention will not be particularly limited to any one form. For example, when there is a lot of traffic in the communication network 710 as shown in FIG. 7 , compression may be performed on objects as well, but compression may not be performed on objects in a low-traffic communication environment. In this way, since the system may be properly selected and operated according to the communication environment, the embodiment of the present invention will not be particularly limited to any one form.

In addition, an external device such as the image processing system 110 of FIG. 1 receives the object data (metadata form) of the tracking object provided by the image analysis device 100, restores the object of interest in high resolution, and restores the Identification and recognition of the tracked object is performed using the high-resolution object of interest (S910). For example, in this process, a clear identification of a person or a vehicle can be made, and as a result, the efficiency of control in a control center or the like can be increased.

In addition to the above, the edge-type image analysis device 100 and the image processing system 110 of FIG. 1 can perform various operations, and since other details have been sufficiently described above, they will be replaced with those contents.

On the other hand, even if all the components constituting the embodiment of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the components may be selectively combined with one or more to operate. In addition, although all of the components may be implemented as a single independent piece of hardware, some or all of the components are selectively combined to perform some or all of the combined functions in one or a plurality of pieces of hardware. It may be implemented as a computer program having. Codes and code segments constituting the computer program may be easily inferred by a person skilled in the art. Such a computer program may implement an embodiment of the present invention by being stored in a computer-readable non-transitory computer readable media, read and executed by a computer.

Here, the non-transitory readable recording medium means a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is common in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications and implementations are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: edge type video analysis device 101: video input unit
102: background image generation unit 103: multi-object detection and tracking unit
104: multi-object encoding unit 105: multi-object video stream unit
110: image processing system 111: monitoring device
112: storage / playback / search device 113: secondary analysis device

Claims (10)

When a captured image captured by a camera is compressed and transmitted to another device, an area in the captured image where an object needs to be identified is detected and tracked, and the tracked object is compressed differently depending on whether it is an object of interest, and the other device detects the object of interest. An image analysis device for restoring to a high resolution higher than the reference resolution; and
An external device operating as the other device, receiving object data of a tracking object provided by the video analysis device, reconstructing the object of interest in the high resolution, and identifying and recognizing the tracking object using the reconstructed high resolution object of interest. Including; device;
The video analysis device is provided in the camera, and the external device includes a device provided at a remote location via a communication network to perform at least one operation of monitoring, data storage, search, and image analysis,
After the image analysis device separates the background image and the object from the captured image, the background image performs data compression according to the performance of the pre-mounted encoder, and the separated object does not perform compression or is better than the background image. Compress by reducing the compression rate,
The video analysis device includes an encoding unit for each multi-object that performs compression of the object of interest and the object of interest differently when compressing the separated object,
The video analysis device transmits metadata of the tracked object as the object data, and generates map coordinates for the tracked object on pre-stored map data in relation to the shooting location of the camera to determine the location of the tracked object. A video encoding system based on object detection and tracking that transmits a solution by including it in the metadata. delete delete According to claim 1,
The video encoding system based on object detection and tracking, wherein the image analysis device compresses a human object and a vehicle object in the photographed image as the object of interest to be reconstructed at the high resolution. delete When an image analysis device compresses an image captured by a camera and transmits it to another device, it detects and tracks a region in the captured image where an object needs to be identified, compresses the tracked object differently depending on whether it is an object of interest, and then compresses the other device. reconstructing the object of interest in a high resolution higher than a reference resolution in a device; and
An external device operating as the other device receives the object data of the tracking object provided by the image analysis device, restores the object of interest in the high resolution, identifies the tracking object using the restored high resolution object of interest, and recognizing; including,
The video analysis device includes an edge device provided in the camera, and the external device includes a device provided at a remote location via a communication network to perform at least one operation of monitoring, data storage, search, and image analysis,
After the image analysis device separates the background image and the object from the captured image, the background image performs data compression according to the performance of the pre-mounted encoder, and the separated object does not perform compression or is better than the background image. Compress by reducing the compression rate,
An encoding unit for each multi-object constituting the video analysis device performs compression of the object of interest and the object of interest differently,
The video analysis device transmits the metadata of the tracked object as the object data, and generates map coordinates for the tracked object on the map data stored in relation to the shooting location of the camera to determine the location of the tracked object. A video encoding method based on object detection and tracking further comprising: transmitting including the metadata in the metadata. delete delete According to claim 6,
The step of compressing differently from each other,
A video encoding method based on object detection and tracking, which compresses a human object and a vehicle object in the photographed image as the object of interest to be reconstructed at the high resolution. delete

Images