KR102351433B1 - Apparatus for Reularization of Video Overfitting and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a video overfitting normalization device and a driving method thereof. According to an embodiment of the present invention, the video overfitting normalization device comprises: a communication interface unit for receiving video frames of a photographed image; and a control unit configured to determine an overfitting degree for each specified number of video frames among the received video frames and perform learning by dropping a pixel value of an arbitrary video frame to a meaningless value during data operation processing based on a probability value (p) adjusted according to the determined overfitting degree.

Description

Apparatus for Regularization of Video Overfitting and Driving Method Thereof

The present invention relates to an apparatus for normalizing video overfitting and a driving method of the apparatus, and more particularly, when it is necessary to determine the class from continuous operation in an environment where overfitting is easy to occur or the nature of video In order to prevent summation, a video overfit normalization device and its device that drop all pixel values of arbitrary video frames to meaningless values (eg 0, gray, etc.) It relates to a method of driving a device.

Various computer vision technologies using CCTV (Closed Circuit Television) and mobile phone cameras are applied to photos (images) and videos (videos) to detect and recognize desired objects such as people, cars, and animals from photos or videos. It is trying to improve the accuracy of object detection and recognition by grafting it with machine learning methods such as deep learning, but it has not reached a satisfactory level.

For example, there is an overfitting problem when learning an existing deep learning network, and various regularization techniques have been developed to solve this problem. As such, one of the most easily and widely used techniques in the image classification field is a dropout technique. Dropout refers to learning while dropping the neural network inside the network to 0 with probability p. Since the neural network is cut with a probability of p every time for the same input value (image, whatever), overfitting is avoided by learning the network with different features. Similar technologies include dropconnect and dropblock.

The field that classifies several consecutive images is video classification, which is similar to image classification, but is a field in which more severe overfitting occurs. Of course, many regularization techniques have been studied to prevent overfitting in video classification as well.

As described above, although regularization technology is being studied to prevent overfitting in video classification, there is a problem that it does not show satisfactory results for grafting it to deep learning technology of artificial intelligence.

Korean Patent Publication No. 10-2020-0041154 (2020.04.21) Korea Patent Publication No. 10-2021-0024935 (2021.03.08) Korean Patent Publication No. 10-2018-0075368 (2018.07.04)

In an embodiment of the present invention, for example, in an environment where overfitting is easy to occur or when it is necessary to identify a class from a continuous operation due to the nature of the video, in order to prevent overfitting in a moving picture, the entire pixel value of an arbitrary video frame is set to a meaningless value in the learning stage. An object of the present invention is to provide an apparatus for overfitting normalization of a moving image that drops to (eg, 0, gray, etc.) to allow learning of a video frame according to the drop and a method of driving the apparatus.

A video overfit normalization apparatus according to an embodiment of the present invention includes a communication interface unit for receiving a video frame of a captured image, and a degree of overfitting for each designated number of video frames among the received video frames, and determining the overfitting The control unit includes a control unit for learning by dropping pixel values of arbitrary video frames to meaningless values during data operation processing based on the probability value p adjusted according to the degree.

The controller may determine the number of video frames to be maintained for the learning and the video frames for the drop from the specified number of video frames, respectively, based on the probability value.

The control unit may determine the degree of overfitting (val_loss(t)/train_loss(t)) as a difference in a ratio between a training loss value and a validation loss value for the specified number of video frames. have.

The controller may increase the number of video frames dropped from the specified number of video frames as the degree of overfitting increases.

The controller may reduce the number of dropped video frames when a class of the specified number of video frames is to be determined through continuous operations due to the characteristics of the video.

In order to change the pixel value of the dropped video frame to a meaningless value, the controller may make all the pixel values have the same value.

The controller may perform an operation by masking a previously generated masking video frame to the dropped video frame in order to change a pixel value of the dropped video frame.

In addition, in the video overfitting normalization apparatus according to an embodiment of the present invention, the communication interface unit receives a video frame of a captured image, and determines the degree of overfitting for each specified number of video frames among the received video frames, and a control unit for learning by dropping pixel values of arbitrary video frames to meaningless values during data operation processing based on the probability value p adjusted according to the determined degree of overfitting.

According to an embodiment of the present invention, it will be possible to solve the overfitting problem that occurs during deep learning network training.

In addition, according to an embodiment of the present invention, it will be possible to approach normalization for video overfit prevention during network learning.

1 is a diagram showing a deep learning network system according to an embodiment of the present invention;
2 is a diagram for explaining a technique for preventing overfitting that occurs in video classification;
3 is a block diagram illustrating the detailed structure of the video overfitting normalization apparatus of FIG. 1, and
4 is a flowchart illustrating a driving process of the video overfitting normalization apparatus of FIG. 1 .

Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating a deep learning network system according to an embodiment of the present invention, and FIG. 2 is a diagram for explaining a technique for preventing overfitting that occurs in video classification.

As shown in FIG. 1 , the deep learning network system 90 according to an embodiment of the present invention includes some or all of the user device 100 , the communication network 110 , and the video overfitting normalization device 120 .

Here, "including some or all" means that the communication network 110 or the video overfitting normalization device 120 is omitted, so that the user device 100 operates in a standalone form to overfit the video according to the embodiment of the present invention. A normalization operation may be performed, and some or all of the components constituting the video overfit normalization device 120 may be integrated into a network device (eg, a wireless switching device, etc.) constituting the communication network 110 It is described as including all things to help a sufficient understanding of the invention as meaning.

The user device 100 may include a photographing device, such as a camera, which is installed in various places in an industrial site and performs a photographing operation. For example, it can be installed and operated in various places for self-driving cars, smart factories, intelligent CCTV, intelligent crime analysis, and AI-based disease prediction. In addition, the photographing device may be installed in a space where children are active, for example, in an arbitrary space such as a daycare center or kindergarten. Of course, the imaging device may be installed in the home to observe infants and nanny. The photographing device includes a general CCTV (Closed Circuit Television) camera or an IP (Internet Protocol) camera as a surveillance camera. In addition, the photographing apparatus 100 may include a PTZ (Pan-Tilt-Zoom) camera capable of pan, tilt, and zoom operations as well as a fixed camera. Of course, since the photographing apparatus 100 may include a camera mounted on a user terminal device such as a smart phone or a tablet PC, the embodiment of the present invention will not be particularly limited to any one form.

A photographing device installed at an industrial site may be linked to an edge device. In other words, as the 4th industrial revolution era enters, the amount of data to be processed by a network device such as the communication network 110 or the video overfit normalization device 120 increases, so that a lot of burden on arithmetic processing can be felt. To this end, by installing an edge device around the imaging device to primarily perform an image analysis operation, the amount of data processing can be drastically reduced. The edge device may perform various operations in connection with the video overfitting normalizer 120, for example, by changing the setting data for analysis (eg, observation object, setting of hyperparameters for preventing overfitting, etc.) A corresponding analysis result may be provided to the video overfitting normalization apparatus 120 . Alternatively, the edge device may operate as the video overfitting normalization device 120 according to an embodiment of the present invention.

Furthermore, the user device 100 may further include various devices such as a smart phone or tablet PC, a laptop computer, a desktop computer, a wearable device worn on the wrist, and a storage medium such as a USB for storing a captured image such as a video. have. Strictly, a user terminal device or an image display device equipped with a photographing device may be preferable. As described above, the user device 100 according to an embodiment of the present invention includes various devices.

The communication network 110 includes both wired and wireless communication networks. For example, as the communication network 110 , a wired or wireless Internet network may be used or interlocked. Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), Wibro network, etc. is meant to include Of course, the communication network 110 according to the embodiment of the present invention is not limited thereto, and may be used, for example, in a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 110 is a wired communication network, the access point in the communication network 110 can connect to a switching center of a telephone company, etc., but in the case of a wireless communication network, it accesses the SGSN or GGSN (Gateway GPRS Support Node) operated by the communication company to transmit data. or by accessing various repeaters such as a Base Transmissive Station (BTS), NodeB, and e-NodeB to process data.

The communication network 110 may include an access point (AP). Here, the access point includes a small base station, such as a femto or pico base station, which is often installed in a building. Femto or pico base stations are classified according to the maximum number of user equipment 100, etc. that can be accessed in the classification of small base stations. Of course, the access point may include a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the user device 100 . The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-distance communication may be performed in various standards such as Bluetooth, Zigbee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra-wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point extracts the location of the data packet, designates the best communication path for the extracted location, and passes the data packet along the designated communication path to the next device, such as the video overfit normalizer 120, etc. . The access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The video overfitting normalization apparatus 120 improves overfitting problems that occur in the video classification process, for example, when learning a deep learning network. Here, "overfitting" means overfitting learning data in machine learning. In general, since the training data is a subset of the actual data, the error decreases with respect to the training data, but the error increases with respect to the actual data. That is, the (program) model is over-adapted only to the training data and the generalization performance is poor. Therefore, it can be seen that the video overfitting normalization apparatus 120 according to an embodiment of the present invention performs normalization or generalization more precisely to solve this problem. In other words, when a video is provided from the user device 100 via the communication network 110, the video overfit normalization apparatus 120 according to an embodiment of the present invention may perform a deep learning operation on the video, Above all, before performing such a deep learning operation, a regularization operation to prevent overfitting may be performed as a kind of preprocessing operation.

As can be seen in FIG. 2 , the video overfitting normalizer 120 probabilistically randomizes video frames 200 and 201 in the learning step ((b) of FIG. 2) to prevent overfitting that occurs in video classification. Drop the whole of to a meaningless value, such as (0, gray). Of course, (b) of FIG. 2 is shown using the symbol of the multiplexer symbolically in the sense of learning data. In other words, it can be seen that one, that is, the entire pixel value of a unit video frame is converted into a meaningless value. Through this transformation, all pixel values of a unit video frame may have the same value. For example, converting to 0, unifying gray to a value of 127, or furthermore, the average value of red (R), green (G), and blue (B) pixel values of a unit video frame It can also be expressed as a luminance value or the like. Therefore, for example, when learning by checking the pixel values of such video frames, the corresponding video frame may not be used for learning.

More specifically, the moving image overfitting normalization apparatus 120 according to an embodiment of the present invention may perform an overfitting determination operation in units of a specified number of video frames from the received video frames. Here, among video frames that are converted to meaningless values according to probability, when n is a live video frame that is not converted to a meaningless value, n can be regarded as the number of frames that should exist at least in one video sequence. have. For example, when the length of a sequence of a specified number of video frames is 7 and n is 2, at least two video frames are not dropped and RGB values should be maintained. In other words, the number of live video frames (P_live) that are not dropped should be 2.

Therefore, in the embodiment of the present invention, it is necessary to first define hyperparameters before further explaining the operation of the video overfitting normalization apparatus 120 . Hyperparameters are distinguished from normal parameters. Hyperparameter refers to a value directly set by the user when modeling (program, etc.). Changes may be made freely through a user interface or the like. Therefore, there is no set optimal value. On the other hand, parameters are called 'parameters' in Korean and have a different meaning from hyperparameters. A parameter is a variable determined within the model. Its value is determined by the data. Therefore, it can be seen that hyperparameters and parameters are distinguished by whether the user sets them or not.

Accordingly, there are hyperparameters to be adjusted for the operation according to the embodiment of the present invention. p means the probability of being dropped every frame. n is the number of frames that should at least exist in one video sequence. n has been described above. pad_value means the value to be filled for the dropped area. As in (0, 0, 0), RGB pixel values are filled with 0, or RGB values are filled with specific gray values as in (127, 127, 127), or in the form of a mean value. can also be filled with The average value is to fill in the pad values (eg, r_mean, g_mean, b_mean) after calculating the R, G, and B average values of the entire training data in advance. In order to represent the drop of a specific video frame, for example, since pixel values can be expressed in various forms, the embodiment of the present invention will not be limited to any one form. For example, it may be possible to perform by tagging a specific frame.

Also, N may be defined as the length of a video frame. For example, as seen in (a) of FIG. 2 , N becomes 7, and n becomes 2. It can be seen that there is one video frame converted into a meaningless value for every two video frames. In the video frames 200 and 201 that are converted to corresponding meaningless values, pixel values of all pixels are converted to pad_value values, respectively, and an operation may be performed or may be skipped during operation. Also, P_d means a frame drop probability. P_live may mean a minimum video percentage (%) that must be alive in the entire frame length.

Accordingly, the video overfitting normalization apparatus 120 according to an embodiment of the present invention may perform the following operation to drop a video frame. Creates a drop frame mask with a probability of p being 0. Of course, such a mask may be generated and used. A mask is created with the same size as the size of an arbitrary video frame (eg, 20 × 20 pixels), and after overlapping pixels, the pixel values of the corresponding pixels are calculated and converted into pad_value. For example, the mask may have the form df_mask = [0, 0, 1, 1, 0, 0, 1, 1, 0, 0] when P_d is 50%.

And, if the following condition is not satisfied, the video frame may be passed, that is, passed.

If sum(df_mask) < N × p_live,

1) Random_index = N × p_live - sum(df_mask) randomly among the indices where the value of df_mask is 0

2) Df_mask[random_index] = 1 (random_index = 0, df_mask = [1, 0, 1, 1, 0, 0, 1, 1, 0, 0]

Accordingly, the RGB value of the frame at the position where df_mask is 0 is replaced with pad_val.

In the above condition, sum(df_mask) < N × p_live indicates that the proportion of the minimum number of frames not to be dropped is p_live, and N is the number of frames, so N × p_live is the minimum number of frames that should not be dropped. Since sum(df_mask) is the number of frames that have not been dropped, it can be summarized as if a drop mask was randomly generated from p_d, but dropped beyond the minimum number of frames that should not be dropped. If so, it is the process of restoring the number of parts that have already been specified as drops (eg 0 in the mask) that should not be dropped again (eg 0 → 1 in the mask).

Therefore, 1) above means randomly setting frames according to the frame set to 0 in the mask, that is, according to the minimum number of frames to be saved among the frames decided to be dropped, and 2) above is the random frame It means restoring the minimum number of frames that should not be dropped by bringing them back to life (eg, converting 0 to 1 in the mask).

Furthermore, the video overfitting normalization apparatus 120 according to an embodiment of the present invention executes a drop (video) frame with a relatively high probability (eg, p_d: 0.5 to 0.8) in an environment where overfitting is easy to occur under the above assumption, and , in an environment where overfitting is relatively difficult to occur, drop frames are executed with a relatively low probability (eg p_d: 0.2~0.5). To this end, the video overfitting normalization apparatus 120 may determine an overfitting state with respect to a specified number of video frames among the received video frames. For example, when pixel data of a corresponding video frame is received, property information of the video frame and the like are also received, so it is possible to determine whether an environment in which overfitting is likely to occur by checking this.

In addition, the video overfitting normalizer 120 has to save a relatively large number of frames (eg, p_live: 0.5 to 0.8) in the problem of identifying the class from continuous motion due to the nature of the video, and use fragmentary frame information. It is not necessary to save a relatively large number of frames (eg p_live: 0.2~0.5) for a problem that is easy to understand. Typically, when a similar scene is repeated over several frames, it is not necessary to save a relatively large number of frames, but when the scene rapidly changes to a new scene, it is necessary to save a relatively large number of frames.

In this process, the video overfitting normalization apparatus 120 may dynamically adjust the p_d value. For example, the p-d value may be adaptively adjusted according to the environment of the received video frame. Adaptive means that a value changes according to a specific environment. In relation to dynamic p_d adjustment, the video overfitting normalization apparatus 120 according to an embodiment of the present invention can determine the degree of overfitting during learning in order to dynamically adjust p_d, and the degree of overfitting is a learning loss value (train). loss) and the difference in the ratio between the validation loss (validation loss). It is the same as the formula below. Based on this, the video overfitting normalization apparatus 120 may analyze a pixel value of a video frame and calculate a learning loss value and a verification loss value.

Overfitting = validation loss (val_loss[t]) / training loss ( train_loss[t])

As the overfitting value increases as learning proceeds (overfitting > threshold, for example, threshold = 3), the video overfitting regularizer 120 increases p_d (eg, p_d = 0.4 if overfitting < 3., else p_d = 0.6).

For example, in order to determine the above learning loss or verification loss, for example, for example, a learning or verification data set for learning or verification is stored in advance, and the loss value (or loss rate) is calculated by comparing the data. you will be able to calculate

The loss value is the result of passing the loss function that sets the network so that the developer can achieve the desired objective. When training a deep learning network, data is usually divided into training data and validation data. The reason is that deep learning networks are prone to overfitting the training data, so the loss value for the training data can be reduced almost indefinitely, but It is rather easy to diverge about actual test data that has not been used.

Therefore, it is necessary to check the learning loss value and the validation loss value at the same time to monitor whether the learning is progressing properly. In addition, if the validation loss does not decrease compared to the very low learning loss, overfitting can be considered. If the overfitting becomes large, an operation to increase p_d may be performed in order to strongly regulate the overfitting.

As described above, the video overfit normalization apparatus 120 according to the embodiment of the present invention may analyze the characteristics of the video frame in units of 7 or 10, for example, when 60 video frames per second are received, and at this time Based on the probability value determined based on the analysis result, it is possible to determine the percentage of the minimum frame that is alive, that is, does not convert to a meaningless value. Accordingly, a frame drop operation is performed on an arbitrary frame. It can be seen that the frame drop unifies the pixel values of the corresponding frame into meaningless values. After that, 7 or 10 video frames in which the overfitting prevention operation is performed may be provided with data to a deep learning network, such as a S/W module for deep learning. Of course, the SW module for deep learning here may be configured in the video overfitting normalization device 120, but may be configured as a separate device, so it will not be particularly limited to any one form.

3 is a block diagram illustrating the detailed structure of the video overfitting normalization apparatus of FIG. 1 .

As shown in FIG. 3 , the video overfit normalization apparatus 120 according to an embodiment of the present invention includes a communication interface unit 300 , a control unit 310 , a frame drop processing unit 320 , and a portion of the storage unit 330 . or all inclusive.

Here, “including some or all” means that some components such as the storage unit 330 are omitted to configure the video overfit normalization device 120, or some components such as the frame drop processing unit 320 are controlled As it means that it can be configured by being integrated with other components, such as 310, it will be described as including all in order to help a sufficient understanding of the invention.

The communication interface unit 300 communicates with the user device 100 via the communication network 110 of FIG. 1 . A captured image received from the user device 100 such as a smartphone or tablet PC or a video file captured by a camera and stored in advance may be received. For example, when there is a request for analyzing a video file from a person in charge of a police station, the communication interface unit 300 may receive the video file and transmit it to the controller 310 .

The communication interface unit 300 may perform various operations, such as modulation/demodulation, muxing/demuxing, encoding/decoding, and scaling for converting resolution, in the process of performing communication with the user device 100 , which are those skilled in the art. Since it is self-explanatory, further explanation will be omitted.

The control unit 310 is responsible for overall control operations of the communication interface unit 300, the frame drop processing unit 320, and the storage unit 330 of FIG. 3 constituting the video overfit normalization apparatus 120 of FIG. 1 . Representatively, in order to prevent overfitting of a moving picture according to an embodiment of the present invention, the control unit 310 may control the frame drop processing unit 320 to execute a program mounted therein. Of course, in this process, the control unit 310 may selectively operate the frame drop processing unit 320 . In other words, when a video is received through the communication interface unit 300 , metadata such as additional information of the video can be received together with the corresponding video. The drop processing unit 320 may be controlled. Of course, since it is preferable to initially perform this determination operation before the frame drop processing unit 320 performs the frame drop processing, the above operation of the control unit 310 will not be particularly limited.

In addition, when the captured image or pre-recorded video of the user device 100 of FIG. 1 is received, the control unit 310 temporarily stores it in the storage unit 330 and then calls it and provides it to the frame drop processing unit 320 , , data generated in a specified format by image processing by the frame drop processing unit 320 may be temporarily stored in the storage unit 330 and then retrieved and systematically classified and stored in the DB 120a of FIG. 1 . For this, the communication interface unit 300 may be controlled.

The frame drop processing unit 320 may largely perform a learning operation such as an image analysis operation and artificial intelligence deep learning. More precisely, a frame drop operation can be performed. For this, a SW module that performs each operation may be included. The image analysis operation performs an operation for frame drop processing before the learning phase. To this end, it is possible to determine the degree of overfitting of the received video. The degree of overfitting can be determined for each of a specified number of video frames among the received frames. In this case, the degree of overfitting can be determined through pixel analysis of the initial video frame, or additional information or subtitles received in relation to the video frame can be determined. The degree of overfitting may be determined by checking metadata including information. As mentioned above, the frame drop processing unit 320 may determine various states prior to frame drop processing, and such state determination may be set in advance by a user, an administrator, or a programmer.

In other words, if only the environment where overfitting is easy to occur is judged, if the class needs to be identified from continuous motion due to the nature of the video, or if various overfitting degrees are judged to make a decision that is appropriate for each situation, it is dynamically , in other words, it may be possible to make judgments adaptively according to the situation. The degree of overfitting determined in this way affects the selection and change of probability values. For example, if the degree of overfitting is severe, frame drop is made with a high probability. If the probability value is high, the number of dropped frames may be small. Of course, the reverse case is also possible, so the embodiment of the present invention will not be particularly limited to any one form.

The frame drop processing unit 320 may, for example, perform an operation for frame drop processing within a range of 7 or 10 video frames received per second, that is, for every specified number of video frames, with respect to 60 video frames received per second. To this end, the degree of overfitting of a corresponding video frame is determined, a probability value is calculated according to the degree of overfitting, and drop processing of an arbitrary video frame is performed according to the calculated probability value. For example, according to the calculated probability value, the number of video frames to be saved among the 7 videos may be determined to be two. Therefore, the first two video frames are saved, and then frame drop processing is performed for the third video frame. In other words, an operation such as making all pixel values of the corresponding video frame to 0 or converting them to the same value may be performed. In addition, since 2 video frames need to be saved, the 4th and 5th video frames are saved to be used for training in the next learning network, and the next 6th video frame is dropped again.

In this way, the frame drop processing unit 320 may determine the sequence length among the received video frames and determine the number of frames that should be present at least in one video sequence. of video frames are provided as a learning network, that is, a learning SW. For drop processing of an arbitrary video frame, in other words, it has been described that a drop frame mask is created and used to make a pixel value 0 or convert it to the same value. To perform masking is to convert a specific value assigned to a mask on an input video frame into a desired value by calculating pixel values and masking values corresponding to each other more precisely. In this way, the operation speed can be quickly increased by masking through the drop frame mask. Of course, since the conversion method may be various, the embodiment of the present invention will not be particularly limited to any one form. Above all, the frame drop processing unit 320 may include a drop mask generation unit (eg, a SW module for generating a mask), through which an operation may be performed.

In addition, the frame drop processing unit 320 may perform various operations, and since other detailed information has been sufficiently described above, those contents will be replaced.

The storage 330 may temporarily store various types of data processed under the control of the controller 310 . For example, under the control of the controller 310 , the storage 330 may receive and store a captured image of a moving picture. The storage unit 330 may include various hardware memories such as ROM, RAM, and HDD, and it may be possible to permanently store data. For example, the storage 330 may receive metadata when receiving a photographed image, and the metadata may include various information such as properties related to the photographed image, and the metadata may be stored in the storage unit. It may be stored in 330 .

In addition to the above, the communication interface unit 300, the control unit 310, the frame drop processing unit 320, and the storage unit 330 of FIG. 3 may perform various operations, and other details have been sufficiently described above, so the contents I want to replace them with

The communication interface unit 300, the frame drop processing unit 320, and the storage unit 330 according to an embodiment of the present invention are composed of hardware modules physically separated from each other, but each module has a You will be able to save the software and run it. However, since the software is a set of software modules, and each module can be formed of hardware, it will not be particularly limited to the configuration of software or hardware. For example, the storage unit 330 may be a hardware storage (storage) or a memory (memory). However, since it is possible to store information in software (repository), it will not be particularly limited to the above.

Meanwhile, as another embodiment of the present invention, the control unit 310 may include a CPU and a memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), a command interpreter and a registry, and the memory may include a RAM. The control circuit performs a control operation, the operation unit performs an operation operation of binary bit information, and the instruction interpretation unit converts a high-level language into a machine language and a machine language into a high-level language, including an interpreter or compiler. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the video overfit normalization apparatus 120, the program stored in the frame drop processing unit 320 is copied, loaded into a memory, that is, RAM, and then executed, thereby speeding up data operation processing. can be increased quickly.

Furthermore, in the embodiment of the present invention, for convenience of explanation, in FIG. 1, for example, a device operating in the form of a server was designated as the video overfitting normalization device 120, but a smartphone operating in a standalone form, Various types of user devices such as a laptop computer, a desktop computer, and a tablet PC may also operate as the video overfit normalization device 120 , so it is not particularly limited to any one type. Above all, in an embodiment of the present invention, a program for performing deep learning network learning is operated, and it is more preferable that it can be used for this purpose.

4 is a flowchart illustrating a driving process of the video overfitting normalization apparatus of FIG. 1 .

Referring to FIG. 4 together with FIG. 1 for convenience of explanation, the moving image overfitting normalization apparatus 120 according to an embodiment of the present invention receives a video frame of a captured image (S400). In general, a moving picture sequentially receives video frames of 60 frames per second (or 45 frames, 80 frames, etc.), but it will not be particularly limited thereto. This is determined according to the performance of the CPU, etc., and if the driving frequency is 60 Hz, it can be seen that 60 video frames are received per second.

In addition, the video overfitting normalization apparatus 120 determines the degree of overfitting with respect to the video frame for every specified number (eg, 7 to 10) among the received video frames, and a probability value (p) adjusted according to the determined degree of overfitting. Based on this, learning is performed by dropping the pixel value of an arbitrary video frame to a meaningless value during data operation processing (S410).

Since the maximum value of the probability value corresponds to 1, it is possible to specify a specific range within the range of 1 and prescribe the form of a video frame to be dropped by matching it. Here, it can be seen that defining is related to how many videos should be saved for the next deep learning and how many videos should be dropped in the sequence length of the video. Of course, it is preferable that such an operation has a certain pattern, but it will not be particularly limited thereto. In other words, when the sequence length is 7 and the number of frames that should exist in one sequence is 2, the input 1st and 2nd video frames are saved, that is, they are provided to the learning network as they are, and the 3rd video frame is dropped, The 4th and 5th video frames are revived, and the 6th video frame is dropped to have a constant pattern of 2-1-2-1. However, when the corresponding video frames are to be regarded as a class of continuous motion due to the characteristics of the video, they may have a non-uniform, that is, asymmetric pattern of 3-1-2-1. will not be limited. Of course, as mentioned above, since the length of the sequence may be variously adjusted, the number of frames to be dropped and not dropped may be determined accordingly.

In addition to the above, the video overfitting normalization apparatus 120 of FIG. 1 can perform various operations, and since other details have been sufficiently described above, those contents will be replaced.

On the other hand, even though it has been described that all components constituting the embodiment of the present invention are combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, all of the components may be implemented as one independent hardware, but a part or all of each component is selectively combined to perform some or all of the combined functions in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily inferred by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, read and executed by the computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, memory, etc. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: user device 110: communication network
120: video overfitting normalization device 300: communication interface unit
310: control unit 320: frame drop processing unit
330: storage

Claims (8)

a communication interface unit for receiving a video frame of a captured image; and
The degree of overfitting is determined for each specified number of video frames among the received video frames, and pixel values of any video frame are calculated based on the probability value (p) adjusted according to the determined degree of overfitting. Including; a control unit to drop (drop) to the learning is made;
The controller determines the degree of overfitting (val_loss(t)/train_loss(t)) with a difference between a ratio of a training loss value and a validation loss value for the specified number of video frames. Sum regularizer. According to claim 1,
The controller is configured to determine, respectively, the number of video frames to be maintained for the learning and the video frames to be dropped from the specified number of video frames based on the probability value. delete 3. The method of claim 2,
The controller is configured to increase the number of video frames dropped from the specified number of video frames as the degree of overfitting increases. 3. The method of claim 2,
The controller is configured to reduce the number of dropped video frames when a class of the specified number of video frames is to be determined through continuous operations due to the characteristics of the video. According to claim 1,
The controller is configured to change the pixel values of the dropped video frames to meaningless values so that all pixel values have the same value. 7. The method of claim 6,
The controller is configured to perform a calculation operation by masking a previously generated masking video frame on the dropped video frame in order to change a pixel value of the dropped video frame. receiving, by the communication interface unit, a video frame of a captured image; and
The control unit determines the degree of overfitting for each specified number of video frames among the received video frames, and calculates the pixel value of any video frame based on the probability value (p) adjusted according to the determined degree of overfitting during data operation processing. Including the step of controlling the learning by dropping it to a meaningless value,
determining, by the controller, the number of video frames to be maintained for the learning and the video frames for the drop from the specified number of video frames based on the probability value; and
determining, by the control unit, the degree of overfitting (val_loss(t)/train_loss(t)) as a difference in a ratio between a training loss value and a validation loss value for the specified number of video frames; cast
Driving method of the video overfitting normalization device further comprising.

Images