KR20240070798A - Apparatus and method for surveillance - Google Patents

Abstract

The present invention relates to a monitoring device and method, which performs monitoring of an object by having artificial intelligence hardware capable of AI calculation processing.
A surveillance device according to an embodiment of the present invention includes an image sensor that captures a surveillance area and generates an image, detects an object in the image, and cuts out an object area containing the detected object from the area of the image to produce a cropped image. A preprocessor that generates, an AI source that performs AI calculation processing, and an AI application that performs AI analysis on objects included in the image by having the AI source perform AI calculation processing on the cropped image, An NPU controller that causes the AI source to process an AI operation requested by the AI application, and transmits the AI operation processing request of the AI application and the AI operation processing result of the AI source between the AI application and the NPU controller. and an NPU driver that converts instructions between the AI application and the NPU controller to be compatible.

Description

Surveillance device and method {Apparatus and method for surveillance}

The present invention relates to a monitoring device and method, and more specifically, to a monitoring device and method that performs monitoring of an object by having artificial intelligence hardware capable of AI calculation processing.

Cameras can be used for surveillance of a target location. Users can perform surveillance on the location by referring to the video captured by the camera.

Artificial intelligence algorithms can be used to detect and track objects included in images. Computation processing for artificial intelligence algorithms can be performed by separate artificial intelligence hardware.

In general, artificial intelligence hardware can perform computational processing on images of preset resolution. Even if the original image is a high-resolution image, artificial intelligence hardware can perform calculation processing on a low-resolution image that reduces the resolution of the original image. Because artificial intelligence hardware performs computational processing on low-resolution images, the reliability of computational processing results by artificial intelligence hardware may be reduced.

Therefore, the emergence of inventions that improve the reliability of computational processing results by artificial intelligence hardware is required.

Public Patent Publication No. 10-2022-0094416 (2022.07.06)

The problem to be solved by the present invention is to provide a monitoring device and method that performs monitoring of an object equipped with artificial intelligence hardware capable of AI calculation processing.

The objects of the present invention are not limited to the problems mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

In order to achieve the above task, a surveillance device according to an embodiment of the present invention includes an image sensor that captures a surveillance area and generates an image, detects an object in the image, and includes an image sensor in the area of the image that contains the detected object. A preprocessor that cuts out the object area and generates a cropped image, an AI source that performs AI calculation processing, and AI analysis of the object included in the image by having the AI source perform AI calculation processing on the cropped image. An AI application that performs, an NPU controller that causes the AI source to process an AI operation requested by the AI application, and an AI operation processing request of the AI application and the AI source between the AI application and the NPU controller. It includes an NPU driver that delivers AI operation processing results and converts instructions between the AI application and the NPU controller to be compatible.

The resolution of the cropped image includes the same resolution as the image.

The preprocessor generates the cropped image in a size that is smaller than or equal to the size of the image that can be processed by the AI source.

The AI source is provided in a plurality, and the NPU driver monitors the resources of the plurality of AI sources, selects an AI source to perform AI operation processing among the plurality of AI sources with reference to the monitoring result, and the preprocessing. The unit generates the cropped image in a size that is smaller than or equal to the size of the image that can be processed by the selected AI source.

The preprocessor transmits crop information of the cropped image to the AI application, and the crop information includes crop position information indicating coordinates at which the object area is located among the areas of the image, and the object area among the frames of the image. Contains time information indicating the time of inclusion.

A surveillance method according to an embodiment of the present invention is a surveillance method that performs AI analysis on an object included in an image using an AI source that performs AI calculation processing, comprising: capturing a surveillance area to generate an image; Detecting an object in the image, generating a cropped image by cutting out an object area containing the detected object from an area of the image, and having the AI source perform AI operation processing on the cropped image. It includes performing AI analysis on objects included in the image.

Specific details of other embodiments are included in the detailed description and drawings.

According to the monitoring device and method of the present invention as described above, the artificial intelligence hardware processes only the area of the original image with a resolution that can be processed by the artificial intelligence hardware, so the reliability of the computational processing results by the artificial intelligence hardware is improved. there is.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram showing a monitoring system.
Figure 2 is a block diagram of a monitoring device according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining the function of the NPU controller shown in FIG. 2.
FIG. 4 is a diagram for explaining the function of the AI data processing unit shown in FIG. 3.
Figure 5 is a diagram showing a data packet transmitted by the AI data processing unit.
FIG. 6 is a block diagram of the preprocessor shown in FIG. 3.
Figure 7 is a diagram showing how a cropped image is generated.
Figure 8 is a diagram to explain the function of an AI application.
Figure 9 is a diagram showing an object detected in an image.
Figure 10 is a diagram showing the creation of a cropped image including a plurality of objects.
Figure 11 is a diagram showing what is generated on a cropped image with reduced resolution.
FIG. 12 is an exemplary diagram of a computing device implementing the monitoring device of FIG. 2.
Figure 13 is a flowchart showing a monitoring method according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings. The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely intended to ensure that the disclosure of the present invention is complete and to provide common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

1 is a diagram showing a monitoring system.

Referring to FIG. 1, the surveillance system 10 includes a surveillance device 100, a network video recorder (NVR) 150, a video management system (VMS) 160, and a user terminal 170. ) and consists of

The surveillance device 100, a network video recorder (NVR) 150, a video management system (VMS) 160, and the user terminal 170 can be connected to each other through the network 30. there is.

The monitoring device 100 has the function of capturing images and processing AI calculations. In the present invention, image is used to encompass still picture and moving picture. Moving pictures are also commonly called videos.

The monitoring device 100 may perform video analysis (VA) based on the captured image and AI operation processing results. Metadata obtained through VA may be stored in the network video recorder 150.

The video management system 160 can search for an image in which an event occurs among stored images using metadata according to a user's search request and take actions (alarm, object tracking, etc.) corresponding to the event.

The user terminal device 170 may be implemented as a personal computer, mobile terminal, etc., and may be connected to the surveillance device 100, the network video recorder 150, the video management system 160, etc. through the network 30.

In this way, the monitoring device 100 can perform VA through AI calculation, and for this purpose, it has AI processing functions internally based on various hardware (SoC: system on chip) and software. However, since each manufacturer of the monitoring device 100 uses its own SoC, it is not directly compatible with various software-based AI applications, which may cause the inconvenience of having to install a dedicated AI application for each SoC.

Therefore, in the present invention, the monitoring device 100 converts a command function for AI processing, for example, an application programming interface (API), to be compatible in the software layer in an open platform environment, and in hardware, it is an NPU. By adding a controller, the same AI application can be used interchangeably in various SoC environments.

Figure 2 is a block diagram showing the configuration of a monitoring device 100 according to an embodiment of the present invention.

As shown, the monitoring device 100 is divided into a top application layer (Layer2), a SW layer (Layer1) located below the application layer (Layer2), and a HW layer (Layer0) located below the SW layer (Layer1). It can be configured.

Here, the AI application 110 belongs to the application layer (Layer2), the NPU driver 120 belongs to the SW layer (Layer1), and a plurality of AI sources 51, 54, 55, embedded in the monitoring device 100 56) and the NPU controller 130 belong to the HW layer (Layer0).

The control unit or CPU 56 serves as a controller that controls the operations of other components of the monitoring device 100. In addition, memory (not shown) is a storage medium that stores results performed by the CPU 56 or data necessary for the operation of the CPU 150, and may be implemented as volatile memory or non-volatile memory.

First, looking at the HW layer (Layer0), the image sensor 60 captures an image from an external subject. For example, the image sensor 60 can generate an image by photographing a surveillance area. The image sensor 60 may be implemented with a known imaging device such as CCD or CMOS, and outputs pixels included in the captured image as digital values.

The Image Signal Processor (ISP) 65 may perform pre-processing on the captured image before storing it. Preprocessing may include image processing such as white balance, gamma adjustment, and up/down sampling. The image processed by the ISP 65 is provided to the NPU controller 130 as input video.

Multiple AI sources (51 to 56) can be embedded in the HW layer (Layer0). Each AI source (51 to 56) is communicatively connected to the NPU controller (130). The embedded AI sources 51 to 56 are SoCs that are built into the monitoring device 100 itself and have the ability to process AI operations, including a neural processing unit (NPU) 51 and a graphic processing unit (GPU) 54. ), a digital signal processor (DSP) 55, and a central processing unit (CPU) 56.

The original purpose of the GPU 54 is computer graphics processing, but the reason GPU is used for AI calculation processing is because graphics processing and AI calculation are performed in similar ways. For example, NVIDIA's GPU can process AI calculations using CUDA, and GPUs from other manufacturers can also process AI calculations by providing their own APIs.

In comparison, the NPU 51 is less versatile than the GPU because it is intended only for AI calculations, but since it is a chip for processing dedicated to AI calculations, the processing speed of AI calculations is considerably higher. However, due to the low versatility of the NPU (51), it is difficult to learn other AI algorithms. This is because the NPU (51) implements the neural network that makes up the deep learning network of the AI algorithm in hardware. Therefore, when implementing an algorithm with a different structure, it may take more time and cost. However, in its original purpose, the NPU (51) shows higher efficiency compared to the GPU (54). Manufacturers of such NPU 51 include ARM, Samsung Electronics, Apple, Qualcomm, and Huawei.

Meanwhile, by expanding from the embedded plurality of AI sources 51 to 56, AI sources 52 and 53 outside the monitoring device 100 may also be connected to the NPU controller 130. Hereinafter, the available AI source 50 includes the AI sources 51, 54, 55, and 56 embedded in the monitoring device 100 and the external AI sources 52 and 53.

The separate external SoC 52 is in the form of an AI source that can be directly wired/wireless connected to the monitoring device 100 through the connection socket 75, for example, a USB terminal. Since this separate external SoC 52 does not require communication through a separate network, it can be useful in an environment where the NPU 51 is not installed inside the monitoring device 100.

The cloud server 53 is a type of AI source that is installed in a large data center, edge server, gateway, etc. and can be connected to the monitoring device 100 through network communication. Therefore, for connection to the cloud server 53, the monitoring device 100 may include a network interface 70 such as Ethernet or WLAN.

Above the HW layer (Layer0), the SW layer (Layer1) exists. In the SW layer (Layer 1), software consisting of an NPU driver (120), an operating system (OS) such as Windows, Android, and IOS (121), and a software development kit (SDK) (123) are installed.

The SDK (123) is a development tool that is generally distributed to use extended functions of a hardware platform such as the NPU (51), an operating system (121), and a programming language. In the present invention, the SDK (123) is used for the expansion function of the NPU (51). Generally, not only the operating system 121 but also the SDK 123 includes an API (Application Programming Interface). Generally, an API is referred to as a contract between an information provider and an information user and constitutes the content needed by the consumer (call) and the content needed by the producer (response). These APIs make it easier for users to develop complex applications, debug them, and add new features as needed.

The NPU driver 120 may transmit the AI operation processing request of the AI application 110 and the AI operation processing result of the AI source 50 between the AI application 110 and the NPU controller 130. Additionally, the NPU driver 120 can convert instructions between the AI application 110 and the NPU controller 130 to be compatible.

To this end, the NPU driver 120 can perform conversion between an API (Application Programming Interface) used to control the available AI source 50 and an extended API. That is, the NPU driver 120 can convert different APIs for each AI source manufacturer into a standardized extended API (EAPI) usable by the AI application 110 and transmit it to the AI application 110. Likewise, the NPU driver 120 may convert the extended API into an API defined for each manufacturer of the AI source at the request of the AI application 110 and provide it to the NPU controller 130.

According to the request of the AI application 110 through the NPU driver 120, the NPU controller 130 can cause the AI source to process the AI operation requested by the AI application 110. In the present invention, a plurality of AI sources 50 are provided, and the NPU controller 130 may select at least one of the plurality of AI sources 50 and command to perform AI operation processing.

The NPU driver 120 provides information on available AI sources 50, including AI sources 51, 54, 55, 56 embedded in the monitoring device 100 and AI sources 52, 53 external to the monitoring device 100. Current resources can be monitored. In addition, the NPU driver 120 may refer to the monitoring result and select an AI source to perform AI calculation processing among the plurality of AI sources 51, 54, 55, 56, 52, and 53. Specifically, the NPU driver 120 selects the AI source with the highest priority among AI sources whose current monitored resources are greater than or equal to a threshold, and processes AI calculations by the selected AI source.

In addition, even after selecting the AI source, the NPU driver 120 monitors the current resources of the selected AI source and, if the current resources fall below the threshold, selects the AI source with the next priority of the selected AI source. You can. Alternatively, when the current resource falls below the threshold, the NPU driver 120 may select the selected AI source and the AI source with the next priority together to distribute and process the AI operation to a plurality of AI sources.

Among these available AI sources 50, the NPU 51 capable of dedicated AI calculation processing and embedded in the monitoring device 100 has the highest priority (first group). Next, although external to the monitoring device 100, the separate external SoC 52 and the cloud server 53, which have a dedicated NPU built-in, have the next priority (second group). Finally, the remaining SoCs embedded in the supervisor 100, GPU 54, DSP 55, and CPU 56, have the lowest priority (third group).

Due to the above priorities, the NPU controller 130 performs AI calculations in the order of high AI calculation processing ability among AI sources whose current resources are guaranteed to be above the threshold, and later, when the resource situation changes, the next-priority AI source is selected. AI calculations can be performed.

Like this, you can group priorities into three groups, but you can also prioritize by individual AI source overall. For example, the individual priorities are preferably in the following order: NPU (51), separate external SoC (52), cloud server (53), GPU (54), DSP (55), and CPU (56).

Additionally, the application layer (Layer2) is located above the SW layer (Layer1) as the highest layer. The AI application 110 in the highest layer (Layer2) requests the NPU driver 120 to perform an AI operation on the image captured by the image sensor 60. As a result, the AI application 110 receives the processing results of the AI operation through the NPU controller 130 and the NPU driver 120. The AI application can perform video analytics on captured images using AI processing results according to the request. This video analysis is a process of extracting objects and events such as people, animals, and cars from the input video and generating metadata that matches the input video. When a pair of input video and metadata is created in this way, object search and/or event search by the video management system 170 described above is possible.

Meanwhile, for AI computation, a learning model needs to be provided for repeated AI learning. It is desirable that such a learning model (neural network model) be created in advance by the AI application 110 and provided to the selected AI source through the NPU controller 130. Otherwise, a process of securing a separate learning model using the input video is necessary. The process of securing a learning model for AI learning in advance is as follows.

The learning model generator 111 (see FIG. 3) uses raw data or training data to be input as learning data to a neural network model. The learning model generator 111 uses the acquired training data to learn the learning model to have a standard for determining how to classify certain data. In this case, the learning model generator 111 may learn a neural network model through supervised learning using at least part of the learning data as a judgment standard. Alternatively, the learning model generator 111 may learn a neural network model through unsupervised learning and self-learn using learning data without supervision to discover the standard. When the neural network model is learned, the learning model generator 111 finally stores the learning model (neural network model) in memory.

The AI application 110 delivers the generated AI learning model to the NPU controller 130, and the NPU controller 130 provides the delivered learning model 131 to the selected AI source.

FIG. 3 is a diagram for explaining the function of the NPU controller shown in FIG. 2.

Referring to FIG. 3, the NPU controller 130 may be configured to include a learning model transmission unit 131, a preprocessor 132, an AI data processing unit 133, and an image processing unit 135.

As described above, the learning model previously created by the learning model generator 111 is transmitted to the learning model transmission unit 131 of the NPU controller 130. At this time, the learning model transmission unit 131 may provide the delivered learning model to the AI source as is, or may convert it into an AI parameter (neural network parameter) that can be used in the AI source and provide it to the AI source.

The image processing unit 135 processes the properties of the captured image into properties supported by the AI source and then provides the processed image to the AI source. Such image processing is the process of changing various image properties into image properties that can be supported by the selected AI source. Image properties include 3D color space (e.g. RGB, YCbCr, etc.) and resolution. , frame-rate, bitrate, and color depth (e.g., 256 colors, true color, etc.).

The AI data processing unit 133 serves to transmit commands downward or upward between the NPU driver 120 and the available AI source 50. The AI input/output processing unit 113 of the AI application 110 transmits the extended API (EAPI) to the NPU driver 120 when requesting an AI operation, and the NPU driver 120 transmits EAPI from the available AI source 50. It is converted into a supported API and delivered to the AI data processing unit 133. Afterwards, the AI data processing unit 133 provides the converted API to an AI source selected from among the available AI sources 50.

The processing results obtained by performing AI calculations on the selected AI source are provided to the AI data processing unit 133. At this time, the AI data processing unit 133 generates the processing result as a standardized data packet and provides it to the AI input/output processing unit 113 of the AI application 110 through the NPU driver 120. Specific examples of such data packets will be described later with reference to FIG. 5.

The NPU controller 130 provides a learning model, AI operation request (API format), and processed image to the available AI source 50, and receives the processing results of the AI operation from the available AI source 50.

Meanwhile, the AI source 50 can only perform calculation processing on images of limited size. For example, the AI source 50 can perform calculation processing only on images having a size of 600Х600 or less, and the size of the image that can be processed may be different for each AI source 50.

If the size of the image captured by the image sensor 60 exceeds the size of the image that can be processed by the AI source 50, the AI application 110 must reduce the captured image and transmit it to the AI source 50. Images that have been reduced in size have reduced resolution, and if AI processing is performed based on this, the reliability of image analysis may be reduced.

In order to prevent a decrease in reliability of image analysis, the monitoring device 100 according to an embodiment of the present invention may be provided with a preprocessor 132. The preprocessor 132 may detect an object in an image and generate a cropped image by cutting out an object area containing the detected object from the area of the image. The preprocessor 132 may generate a cropped image with a size smaller than or equal to the size of the image that can be processed by the AI source 50. Accordingly, the resolution of the cropped image may be the same as the resolution of the image generated by the image sensor 60. As described above, the NPU driver 120 may select a portion of the plurality of AI sources 50 to perform AI calculation processing. The preprocessor 132 may generate a cropped image with a size smaller than or equal to the AI image size of the AI source 50 selected by the NPU driver 120. The detailed configuration of the preprocessing unit 132 will be described later with reference to FIG. 6.

The AI application 110 can perform AI analysis on objects included in the image by having the AI source 50 perform AI calculation processing on the cropped image. As the AI source 50 performs AI calculation processing on cropped images whose resolution has not been reduced, reliability of image analysis can be improved.

FIG. 4 is a block diagram showing in more detail the configuration of the AI data processing unit 133 included in the NPU controller 130 of FIG. 3.

The AI data processing unit 133 receives the processing results of AI operations from available AI sources 50 and provides them to the NPU driver 120 in the form of standardized data packets.

The compatibility determination unit 136 receives the AI processing result from the selected AI source and then determines the compatibility of the data format representing the AI processing result. If the data format is not compatible according to the determination result, the format processing unit 137 processes the processing result of the AI operation, and the packetization unit 139 generates a processed data packet. On the other hand, if there is compatibility in the data format according to the determination result, the packetization unit 139 can transmit the data packet obtained by packetizing it as is without processing to the NPU driver 120.

Here, compatibility refers to whether the processing result of the AI operation is data in a format that the AI application 110 can recognize. In general, the processing results of AI operations are dependent on the AI source and are not standardized. For example, the number of data fields in the processing results provided by one AI source may be different from that of other AI sources, and the classification criteria or number of classifications for objects or events may be different. Additionally, there are cases where the same object or event is defined with a different name.

Considering these compatibility issues, the format processing unit 137 processes the processing results of incompatible AI operations into compatible data and transmits it to the packetization unit 139, and the packetization unit 139 always produces standardized data packets. It is provided to the NPU driver 120 in the form of.

Figure 5 is a diagram showing a data packet transmitted by the AI data processing unit.

The data packet 90 basically consists of a header 91 and a payload 92. The header 91 includes an AI identifier (e.g. 0x0919) indicating that it is a data packet with AI processing results. The AI identifier indicates that the data packet 90 contains AI processing results and is used to distinguish it from various other types of data packets.

Payload 92 may consist of a number of standardized data fields. The payload 92 specifically includes a field 93 indicating the order of video frames, a field 94 indicating the time when the video frame was created, a field 95 indicating the type of object identified within the video frame, and an identification field 95. It may include a field 96 indicating the probability that the identified object is correct and fields 97 and 98 indicating the location of the identified object within the video frame. Additionally, the payload 92 may include a field 99 for recording additional data according to user definition.

FIG. 6 is a block diagram of the preprocessor shown in FIG. 3.

Referring to FIG. 6, the preprocessing unit 132 includes an information collection unit 132a and an information generation unit 132b.

The information collection unit 132a may collect image size information (hereinafter referred to as AI image size) that can be processed by the AI source 50. AI image size may be received from AI source 50. To this end, the AI source 50 can store the AI image size and transmit the corresponding AI image size to the information collection unit 132a at the request of the information collection unit 132a or in a separate event.

In the present invention, a plurality of AI sources 50 are provided, and the AI image size for each AI source 50 may be different. The information collection unit 132a may receive the size of each AI image from a plurality of AI sources 50.

The information generator 132b may generate information that can be used by the AI application 110 for AI analysis. The information generator 132b includes an object detection unit 132c, a crop information generator 132d, and a crop image generator 132e.

The object detection unit 132c may detect an object in the image generated by the image sensor 110. For example, the object detection unit 132c may detect an object in an image using an algorithm such as Features from Accelerated Segment Test (FAST), Scale Invariant Feature Transform (SIFT), or Speed Up Robust Feature (SURF). The object detection unit 132c first detects the object before the AI application 110 detects the object in the image.

The crop information generator 132d may set an object area for the object detected by the object detection unit 132c and generate crop information for the object area. The object area may have a preset size or its size may be set in real time to include the entire selected object. In any case, the size of the object area may be set to be smaller than or equal to the AI image size that can be processed by the AI source 50. As described above, the NPU driver 120 selects a portion of the plurality of AI sources 50 to perform AI calculation processing, and the crop information generator 132d selects the AI source selected by the NPU driver 120 ( The size of the object area can be set to be smaller or the same size as the AI image size in 50).

Crop information represents information used to extract a partial image corresponding to an object area from an image. Crop information may include crop location information and time information. The crop position information indicates the coordinates at which the object area is placed in the area of the video, and the time information indicates the point in time at which the object area is included in the frame of the video image. For example, the time information may be a timestamp when the object area is set.

The cropped image generator 132e may generate a cropped image by referring to the cropped information. The cropped image represents the image of the portion of the image area corresponding to the object area. The crop image generator 132e may check the object area with reference to the crop position information, extract image data included in the corresponding object area, and generate a crop image. Additionally, the crop image generator 132e may generate a crop image by applying the object area to a specific frame among the frames of the image with reference to time information.

The preprocessor 132 may transmit the crop information generated by the crop information generator 132d and the crop image generated by the crop image generator 132e to the AI application 110. Crop information and cropped images can be transmitted to the AI application 110 through the NPU driver 120. The AI application 110 can perform AI analysis on objects included in the image using crop information and cropped images.

The above has explained that the cropped image is generated by the preprocessor 132, but according to some embodiments of the present invention, the cropped image may also be generated by the AI application 110. In this case, the crop image generator 132e may be provided in the AI application 110.

The preprocessor 132 can detect an object in an image and identify an object area that includes the detected object among the areas of the image. The AI application 110 creates a cropped image by cutting out the object area in the image, and allows the AI source 50 to perform AI operation processing on the cropped image to perform AI analysis on the object included in the image. . Hereinafter, the description will focus on the generation of the cropped image in the pre-processing unit 132.

Figure 7 is a diagram showing how a cropped image is generated.

Referring to FIG. 7, the image 500 may include at least one object 510.

The object detection unit 132c may detect the object 510 by scanning the image 500. Thus, when the image 500 includes the object 510, the crop information generator 132d can set the object area 520 for each object 510. To detect the object 510, the object detection unit 132c may use a separate detection algorithm. The object detection unit 132c can detect a preset object as the object 510 by using a detection algorithm. Therefore, if a preset object is not included in the image 500, the object detection unit 132c does not detect the object 510, and the crop information generation unit 132d does not set the object area 520. You can.

The crop image generator 132e may generate the crop image 600 by referring to the image data included in the object area 520. FIG. 7 shows that the image includes three objects 510. In this case, the crop image generator 132e can generate three crop images 600.

Figure 8 is a diagram for explaining the function of the AI application, and Figure 9 is a diagram showing an object detected in an image.

Referring to FIG. 8, the AI application 110 may perform analysis on the image by referring to the original image, cropped image, and crop information.

The AI application 110 may receive the original image generated by the image sensor 110, the cropped image generated by the crop image generator 132e, and the crop information generated by the crop information generator 132d. .

The AI application 110 can perform AI analysis on objects included in the image using an AI model. For example, the AI application 110 can detect and track an object in an image using an object detection model. The AI application 110 can detect objects in an image using R-CNN or YOLO (You Only Look Once) algorithm as an object detection model. The YOLO algorithm is an AI algorithm suitable for surveillance cameras that process real-time video because the speed of detecting objects in video is relatively fast.

Analyzing images using AI models requires a large amount of computational processing. The AI application 110 may allow the AI source 50 to perform calculation processing on image data. To this end, the AI application 110 may transmit image data of the cropped image to the AI source 50. The AI source 50 may perform AI operation processing on the delivered image data and deliver the results to the AI application 110. The AI application 110 may perform AI analysis by referring to the processing results of the AI source 50. For example, the AI application 110 may determine whether the object in question is a person, a car, or a motorcycle.

With reference to the processing results of the AI source 50, the AI application 110 may generate an image 700 in which object detection has been completed (hereinafter referred to as an object detection image). When described with reference to FIG. 9 , the object detection image 700 may include an object 710 and a detection area 720. The detection area 720 may be provided in the form of a square box surrounding the edge of the object 710. The user can check the size and location of the object 710 by referring to the detection area 720.

When the object 710 moves, the detection area 720 may move along with the object. Accordingly, the user can easily track the moving object 710.

Figure 10 is a diagram showing the creation of a cropped image including a plurality of objects.

Referring to FIG. 10 , the crop information generator 132d may set an object area 530 for a plurality of objects 510 .

The size of the object area 530 may be the same as or smaller than the AI image size. When a plurality of objects 510 are detected by the object detection unit 132c, the crop information generator 132d may set the object area 530 to include all objects 510.

The crop image generator 132e may generate the crop image 620 by referring to the image data included in the object area 530. The cropped image 620 is transmitted to the AI application 110, and the AI application 110 can transmit the image data of the cropped image 620 to the AI source 50. Since the size of the cropped image 620 is the same as or smaller than the size of the AI image that can be processed by the AI source 50, the AI source 50 can perform AI calculation processing on the image whose resolution has not been reduced.

Figure 11 is a diagram showing what is generated on a cropped image with reduced resolution.

Referring to FIG. 11 , the crop information generator 132d may set an object area 510 for a plurality of objects 510 .

The size of the object area 510 cannot be set larger than the AI image size. Accordingly, depending on the distribution situation of the plurality of objects 510, not all of the plurality of objects 510 may be included in the object area 510. In this case, the cropped image generator 132e sets an extended object area 540 that includes all of the plurality of objects 510, and reduces the size of the extended object area 540 by the AI image size to create a cropped image 630. can be created. The cropped image 630 may include image data of an image with reduced resolution compared to the original image 500.

Although the resolution is partially reduced, the reliability of image analysis can be improved because the reduction in resolution is smaller than when the entire original image 500 is reduced.

Figure 13 is a flowchart showing a monitoring method according to an embodiment of the present invention.

Referring to FIG. 13, the image sensor 110 of the monitoring device 100 according to an embodiment of the present invention can generate an image by photographing the surveillance area (S810).

The preprocessor 132 may detect the object 510 in the image generated by the image sensor 110 and generate a cropped image by cutting out the object area containing the detected object 510 from the area of the image ( S820). Here, the size of the object area is smaller than or equal to the AI image size, and the resolution of the cropped image may be the same as the resolution of the image.

The cropped image generated by the preprocessor 132 is transmitted to the AI application 110, and the AI application 110 causes the AI source 50 to perform AI operation processing on the cropped image to determine the objects included in the image. AI analysis can be performed on (510) (S830). Because the resolution of the cropped image is the same as the resolution of the original image, the reliability of image analysis can be improved.

Although embodiments of the present invention have been described with reference to the above and the attached drawings, those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will understand that it exists. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: surveillance system 30: network
50: AI source 60: Image sensor
100: Monitoring device 110: AI application
111: Learning model creation unit 113: AI input/output processing unit
120: NPU driver 130: NPU controller
131: Learning model transmission unit 132: Preprocessing unit
132a: Information collection unit 132b: Information generation unit
132c: object detection unit 132d: crop information generation unit
132e: Crop image generation unit 133: AI data processing unit
135: Image processing unit 136: Compatibility determination unit
137: Format processing unit 139: Packetization unit
150: network video recorder 160: video management system
170: User terminal

Claims (6)

An image sensor that captures the surveillance area and generates an image;
a preprocessor that detects an object in the image and generates a cropped image by cutting out an object area containing the detected object from an area of the image;
AI source that performs AI computational processing;
An AI application that performs AI analysis on objects included in the image by causing the AI source to perform AI calculation processing on the cropped image;
an NPU controller that causes the AI source to process AI operations requested by the AI application; and
An NPU driver that transmits the AI operation processing request of the AI application and the AI operation processing result of the AI source between the AI application and the NPU controller, and converts instructions between the AI application and the NPU controller to be compatible. Monitoring devices, including. According to claim 1,
A surveillance device wherein the resolution of the cropped image is the same as the resolution of the image. According to claim 1,
The preprocessor is a surveillance device that generates the cropped image in a size that is smaller than or equal to the size of the image that can be processed by the AI source. According to claim 1,
The AI source is provided in plural numbers,
The NPU driver monitors the resources of the plurality of AI sources and selects an AI source to perform AI operation processing among the plurality of AI sources with reference to the monitoring results,
The preprocessor is a monitoring device that generates the cropped image in a size that is smaller than or equal to the size of the image that can be processed by the selected AI source. According to claim 1,
The preprocessor transmits crop information of the cropped image to the AI application,
The crop information is,
Crop position information indicating coordinates at which the object area is located among the areas of the image; and
A monitoring device including time information indicating a point in time at which the object area is included in a frame of the image. In a surveillance method of performing AI analysis on objects included in an image using an AI source that performs AI calculation processing,
Generating an image by photographing the surveillance area;
Detecting an object in the image, and generating a cropped image by cutting out an object area containing the detected object from an area of the image;
A surveillance method comprising performing AI analysis on an object included in the image by having the AI source perform AI calculation processing on the cropped image.

Images