KR20210040709A - Device and Method for adding target class thereon - Google Patents

Abstract

As a preferred embodiment of the present invention, a terminal comprises: an MSE buffer for storing some sections of a received image; a first interface for selecting a target class from an image displayed on a display; and a second interface for selecting at least one specific point in the image displayed on the display, wherein the display can estimate and display the target class on a captured image received in real time from the at least one image capturing device after the target class is selected.

Description

Device and Method for adding target class thereon

The present invention relates to a method of analyzing an image by applying deep learning to a network surveillance camera, and in more detail, a method of performing transfer learning in order for a user to additionally add a new class in a network surveillance camera. About.

Conventional network surveillance cameras are implemented as embedded systems, so after learning using a given data set, the learned model parameters and labels are fixed, and objects are inferencing within the embedded system. ) Only the task was performed.

However, deep learning through the conventional method of classifying and analyzing each object in an image acquired from a network surveillance camera into only fixed classes classified through pre-learning has limited results.

KR 10-2019-0062030 A

In a preferred embodiment of the present invention, while monitoring an image received from a network surveillance camera by a user, a new target class is selected through the image, and the target class added by the user by performing transfer learning in the network surveillance camera. We intend to perform deep learning inference by reflecting the update of.

In a preferred embodiment of the present invention, a terminal includes: a receiver configured to receive an image captured by at least one image capturing device in real time; A Media Source Extensions (MSE) buffer for storing a partial section of the received image; A display displaying the received image; A first interface for selecting a target class from an image displayed on the display; And a second interface for selecting at least one specific time point from the image displayed on the display, wherein the display is configured to display the captured image received in real time from the at least one image capturing device after the target class is selected. It is characterized in that the target class can be estimated and displayed.

As a preferred embodiment of the present invention, a deep learning unit learns the target class based on frames within a predetermined period of time based on the selected specific time point, and the target class learned through the transmission unit and a deep learning variable related thereto are learned. It is characterized in that the transmission to at least one image capturing apparatus. In this case, the MSE buffer is characterized in that a partial section of the image received in the live monitoring process is decoded, rendered, displayed on the display, and stored in the MSE buffer for a predetermined period of time. The deep learning unit is characterized in that it performs transfer learning on the target class using a data augumentation method using all frames within a predetermined time based on the selected specific time point.

In another embodiment of the present invention, a terminal includes a display; A memory for storing one or more instructions; A buffer for storing a partial section of the received live image; And a processor that executes the one or more instructions stored in the memory. A first interface for selecting a target class in the live image; and a second interface for selecting at least one specific point in the live image, wherein the processor executes the one or more instructions, The display is controlled to output a live image received in real time, and the target class is estimated and displayed on the live image received in real time after the target class is selected, and the memory performs deep learning. And storing a learning result obtained by learning the target class from frames before and after the at least one specific time point.

In another preferred embodiment of the present invention, a method for displaying by adding a target class in a terminal comprises: receiving a live image captured by at least one image capturing device; Storing a partial section of the live video in a Media Source Extensions (MSE) buffer; Displaying the received live image on a display; Designating a target class in the live image displayed on the display; Selecting at least one specific time point from the live image displayed on the display using an instant playback interface; And estimating and displaying the target class on a live image received from the at least one image capturing device after the target class is selected, and including, in a time series order within a predetermined time based on the selected at least one specific time point. It is characterized in that transfer learning is performed on the target class using a plurality of frames arranged as.

In a preferred embodiment of the present invention, image analysis is not performed only for a fixed class, but an image analysis for a target desired by the user may be performed for a class additionally selected by the user through real-time transfer learning.

1 is a preferred embodiment of the present invention, showing a video surveillance system for performing real-time transfer learning in CCTV.
2 is a block diagram of a terminal according to a preferred embodiment of the present invention.
3 illustrates an example of storing a partial section of an image in an MSE buffer using an instant playback interface as a preferred embodiment of the present invention.
4 shows an example of performing real-time transfer learning as a preferred embodiment of the present invention.
FIG. 5 is a flowchart illustrating a display by adding a target class in a terminal in the video surveillance system as a preferred embodiment of the present invention.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present disclosure.

In the present specification, in particular, the “above” and similar designations used in the claims may indicate both the singular and the plural. Further, unless there is a description that clearly specifies the order of the steps describing the method according to the present disclosure, the described steps may be performed in a suitable order. The present disclosure is not limited according to the order of description of the described steps.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented with various numbers of hardware and/or software components that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for a predetermined function. In addition, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as an algorithm executed on one or more processors. In addition, the present disclosure may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as “mechanism”, “element”, “means” and “composition” can be used widely, and are not limited to mechanical and physical configurations.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a preferred embodiment of the present invention, showing a video surveillance system for performing real-time transfer learning in CCTV.

The image monitoring system 100 includes a user terminal 120 and an image capturing device 110. The image capturing device 110 is capable of wired or wireless communication with the terminal 120, and examples of the image capturing device 110 include a CCTV, a camera, a terminal capable of capturing an image, and the like.

Terminal 120 is a mobile phone, tablet PC, digital camera, camcorder, notebook, computer, tablet PC, desktop, e-book terminal, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, wearable It may be implemented in various electronic devices such as a wearable device. In addition, the terminal 120 also includes all of the physical devices including the components of FIG. 2.

As a preferred embodiment of the present invention, the image capturing device 110 includes a second deep learning unit 112, and the second deep learning unit 112 uses a pre-trained data set. Therefore, it is assumed that deep learning has been performed. The second deep learning unit 112 may analyze a pre-learned class among images captured by the image capturing apparatus 110 and provide a result in the form of metadata.

In a preferred embodiment of the present invention, the second deep learning unit 112 is also capable of transfer learning for a new target class. The process of performing transfer learning is as follows.

While the user monitors the image transmitted from the CCTV 110, the first interface S121 selects an image at point t that includes the new target class. An example of a target class is a man wearing a hat.

After that, the user implements to collect all images of the time point t+x to t-y including the selected target class using the second interface S122. In a preferred embodiment of the present invention, when a new target class is added in a live image, the same target class may be learned in a data augmentation format by using a feature having the same target class before and after a specific time t reference.

The terminal 120 performs data augmentation using all of the series of time series images collected through the second interface S122 through the first deep learning unit 122 (S123), and transfer learning for a new target class. (transfer learning) and update (S124) are performed. The terminal 120 transmits the updated deep learning parameter and new target class information to the CCTV 110.

The CCTV 110 updates the received deep learning variable and new target class information and then performs deep learning inferencing on the image.

Referring to FIG. 2, the terminal 200 includes a processor 210, a memory 220, a buffer 230, and a display 240, and the display 240 includes a first interface 250 and a second interface 260. ) Support.

The memory 220 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM, ROM, SRAM, EEPROM, PROM, magnetic memory, magnetic disk, optical disk, etc., and a processor ( 210) can be stored in a program for processing and control. The processor 210 is

The memory 220 may store one or more instructions, and the processor 210 may control the display 240 to output a live image received in real time by executing one or more instructions. The memory 220 may store a learning result obtained by learning a target class from frames before and after at least one specific time point through deep learning.

The processor 210 may control to estimate and display the selected target class in a live image received in real time after the target class is selected by the user as in the exemplary embodiment of FIGS. 3 to 4.

The buffer 230 may store a partial section of the received live image. Referring to FIG. 3, the buffer 230 may use the MSE buffer 330 and may store some sections 320 of the live image. In addition, the buffer 230 may store a plurality of frames 331 to 334 within a predetermined time interval based on a specific time point selected by the user through the second interface 260 in time series order.

The first interface 250 is implemented to select a target class from a live image displayed on the display 240. The second interface 260 is implemented to select at least one specific time point from the live image displayed on the display 240. In a preferred embodiment of the present invention, the second interface 260 may be an instant playback interface.

3 to 4 illustrate an example of storing an image using an MSE buffer while monitoring a live image in the terminals 310a and 310b as a preferred embodiment of the present invention.

In a preferred embodiment of the present invention, the live video is decoded by the decoder 332 and then rendered through the rendering unit 334 and displayed on the display 310b of the user terminal. The MSE buffer 330 used in the user terminal 310b does not immediately remove the image shown to the user, but holds some of the frames 331 to 334 for a predetermined period of time.

As a preferred embodiment of the present invention, when a user selects a specific object as a new target class in the terminal 310a, the user uses an interface such as instant playback, t-1 second, t second, t+1 second, A time series image corresponding to t+2 seconds is played back and used as training data of the deep learning unit 350. The data augmentation unit 340 increases the training data through scaling, translation, rotation, flipping, and add noise, and then the deep learning unit 350 Provide the collected data in ).

FIG. 5 shows a method of adding and displaying a target class in a terminal as a preferred embodiment of the present invention.

In a video surveillance system including a CCTV and a user terminal, the user terminal receives a live image photographed by at least one CCTV (S510). The terminal decodes and renders the received live image and displays it on the display, and does not remove the displayed image immediately, but retains the displayed image in the MSE buffer (S520, S530). It should be noted that steps S520 and S530 may be sequentially performed, step S520 may be performed after step S530 is performed, or steps S520 and S530 may be performed simultaneously.

The user monitors at least one live image displayed on the display of the terminal, and if there is a new target class to be added, the user may select the target class using a finger, a stylus, or various input interfaces (S540). Thereafter, the user selects at least one specific time point t from the live image displayed on the display using the instant playback interface (S550).

The first deep learning unit in the terminal collects a series of time-series images held in the MSE buffer, transfers and updates the deep learning variables and classes of the new target class through data augmentation, and transmits information about this to at least one CCTV. .

At least one CCTV performs deep learning inference based on the received deep learning variable and class of the new target class. In the terminal, after the new target class is selected, the selected target class is estimated and displayed in the live image received from the at least one image capturing apparatus (S560).

The device according to the present invention includes a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a user interface such as a touch panel, keys, buttons, etc. It may include a device and the like. Methods implemented as software modules or algorithms may be stored on a computer-readable recording medium as computer-readable codes or program instructions executable on the processor. Here, computer-readable recording media include magnetic storage media (eg, hard disks, etc.) and optical reading media (eg, CD-ROMs, DVDs: Digital Versatile Discs). The computer-readable recording medium is distributed over networked computer systems, so that computer-readable codes can be stored and executed in a distributed manner. The medium is readable by a computer, stored in the memory, and executed by the processor.

In the specification of the present invention (especially in the claims), the use of the term “above” and a similar designating term may correspond to both the singular and the plural. In addition, when a range is described in the present invention, the invention to which individual values falling within the range are applied (unless otherwise stated), and each individual value constituting the range is described in the detailed description of the invention. Same as. Finally, unless explicitly stated or contradicted to the order of the steps constituting the method according to the present invention, the steps may be performed in a suitable order. The present invention is not necessarily limited according to the order of description of the above steps. The use of all examples or illustrative terms (for example, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited by the above examples or illustrative terms unless limited by the claims. It does not become. In addition, those skilled in the art can recognize that various modifications, combinations, and changes may be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

So far, the present invention has been looked at around its preferred embodiments. Those of ordinary skill in the art to which the present invention pertains will be able to understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the above description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Claims (11)

A receiver configured to receive an image captured by at least one image capturing device in real time;
A Media Source Extensions (MSE) buffer for storing a partial section of the received image;
A display displaying the received image;
A first interface for selecting a target class from an image displayed on the display;
A second interface for selecting at least one specific point in the image displayed on the display; Including,
And the display is configured to estimate and display the target class on a captured image received in real time from the at least one image capturing device after the target class is selected. The method of claim 1,
And learning the target class based on frames within a predetermined period of time based on the selected specific time point, and transmitting the learned target class and a deep learning variable related thereto to the at least one image capturing apparatus. The method of claim 1, wherein the display
And displaying the captured image received in real time to support live monitoring, and selecting the target class in the live monitoring process using the first interface. The method of claim 3,
And the MSE buffer decodes a partial section of the image received in the live monitoring process, renders it, displays it on the display, and stores it in the MSE buffer for a predetermined period of time. The terminal according to claim 1, wherein when the at least one specific time point is selected through the second interface, frames within a predetermined time based on the selected specific time point are selected among frames stored in the MSE buffer. . The method of claim 5,
A terminal, characterized in that transfer learning is performed on the target class using a data augumentation method using all frames within a predetermined time based on the selected specific time point. The method of claim 1,
And performing transfer learning on the target class by using a plurality of frames arranged in chronological order within a predetermined time based on the selected specific time point. The method of claim 1, wherein the second interface
Terminal, characterized in that the instant playback (instant playback) interface. display;
A memory for storing one or more instructions;
A buffer for storing a partial section of the received live image; And
A processor that executes the one or more instructions stored in the memory;
A first interface for selecting a target class in the live image; And
Including; a second interface for selecting at least one specific point in the live image,
The processor controls the display to output a live image received in real time by executing the one or more instructions, and estimates and displays the target class on the live image received in real time after the target class is selected. Control to be able to do it,
Wherein the memory stores a learning result obtained by learning the target class from frames before and after the at least one specific time point through deep learning. 10. The terminal of claim 9, wherein the target class is learned by performing learning on one object using all frames before and after the at least one specific time point. As a method of displaying by adding a target class in a terminal,
Receiving a live image captured by at least one image capturing device;
Storing a partial section of the live video in a Media Source Extensions (MSE) buffer;
Displaying the received live image on a display;
Designating a target class in the live image displayed on the display;
Selecting at least one specific time point from the live image displayed on the display using an instant playback interface; And
Estimating and displaying the target class in a live image received from the at least one image capturing device after the target class is selected, in a time series order within a predetermined time based on the selected at least one specific time point And performing transfer learning on the target class using a plurality of arranged frames.

Images