KR20210056899A - Methods and systems for real-time data reduction - Google Patents

Abstract

A computing system for decimating video data includes: a processor; a persistent storage system coupled to the processor; and a memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to decimate a batch of frames of video data by: receiving the batch of frames of video data; mapping, by a feature extractor, the frames of the batch to corresponding feature vectors in a feature space, wherein each of the feature vectors has a lower dimension than a corresponding one of the frames of the batch; selecting a set of dissimilar frames from the plurality of frames of video data; and storing the selected set of dissimilar frames in the persistent storage system, wherein the size of the selected set of dissimilar frames is smaller than the number of frames in the batch of frames of video data.

Description

Systems and methods for real-time data reduction {METHODS AND SYSTEMS FOR REAL-TIME DATA REDUCTION}

The present disclosure relates to systems for real-time data reduction and methods for reducing real-time data, and includes the field of collecting video data for training machine learning systems such as computer vision systems and the like.

Machine learning systems generally compute predictions or answers based on underlying statistical models learned on a huge collection of training data (collection) representing a range of inputs that such systems will encounter.

In one example, some autonomous vehicles use cameras to detect surrounding objects. One technique for object detection is supervised machine learning, which utilizes deep learning (eg, Deep Neural Networks (DNNs)). In such approaches, installed on autonomous vehicles. Cameras capture images of a street scene in real time, and feed the images into one or more statistical models (eg, deep neural networks) that are trained to automatically detect (detect) and classify objects within the scene. This is a real-time, semantic, substantially real-time environment around the vehicle, such as the location of pedestrians, road hazards, curves, road markings, and other vehicles in the vicinity of an autonomous vehicle. Autonomous driving algorithms with (semantic) expressions are provided, and as a result, autonomous driving algorithms provide information about detected objects, such as slowing down and stopping the vehicle to avoid collisions with pedestrians crossing the vehicle's path. Can be used to control the vehicle.

In order to generate training data for training statistical models of computer vision systems, a large (e.g., thousands of) image collections are subject to various conditions (e.g., different types of traffic, weather, road types, parking lots). Fields, parking structures, etc.) while driving the vehicle. Images thus collected can then be annotated manually by human annotators (e.g. by manually specifying other masks or bounding (bounding) boxes labeled with objects appearing inside the masked area. ).

It is an object of the present disclosure to provide methods and systems for reducing captured data in real time for the purpose of training machine learning systems.

According to an embodiment of the present disclosure, a computing system for decimating video data includes: a processor; A permanent storage system coupled to the processor; And when performed by the processor, causing the processor to: receive a batch of frames of video data; Mapping, by a feature extractor, the frame arrangement into feature vectors corresponding in the feature space, each having a lower dimension than the corresponding frame arrangement; Selecting a set of dissimilar frames from the frames of the plurality of video data based on dissimilarities between the corresponding feature vectors; And a memory for storing instructions that cause the placement of frames of video data to be removed by performing the step of storing a selected dissimilar frame set whose size is less than the number of frames in the frame arrangement of video data in the persistent storage system. do.

The instructions that cause the processor to select a frame set, when executed by the processor, cause the processor to: randomly select a first reference frame from a frame arrangement of video data; Discarding the first frame set from the frame arrangement, the first frame set having corresponding feature vectors within a similarity threshold distance of the first feature vector corresponding to the first reference frame; Selecting a second reference frame from a plurality of frames of video data, wherein the second reference frame has a second feature vector, and a distance between the first feature vector and the second feature vector may be greater than the similarity threshold distance; And the second frame set is discarded from the frame arrangement, wherein the second frame set has corresponding feature vectors within the similarity threshold distance of the second feature vector, and the selected set of dissimilar frames includes first and second reference frames. In addition, the first and second frame sets may be excluded.

Feature extractors may include neural networks. The neural network may include a convolutional neural network (CNN).

Computing systems and permanent storage systems can be mounted on the vehicle, and the frame arrangement of video data can be part of a stream of video data captured by a video camera installed on the vehicle, and the video camera captures images of the environment around the vehicle. Is configured to

The frame arrangement of video data may be captured during a first time period having a length corresponding to the capture period, and the stream of video data is a video corresponding to the video data captured during a second time period having a length corresponding to the capture period. A second frame arrangement of data, the second time period can immediately follow the first time period, and the computing system is configured to remove the arrangement of frames of video data within a time corresponding to the capture period. Can be configured to map.

The memory causes the processor, when executed by the processor, to remove frames from the selected set of dissimilar frames based on the uncertainty metric, by: locating each frame of the selected set of dissimilar frames to instances of each object class of a plurality of object classes. Support with an object detector including a convolutional neural network (CNN) to calculate a plurality of bounding box sets that identify portions of a depicted frame, wherein each bounding box of the bounding box set has an associated confidence score; Calculate an uncertainty score based on the bounding box sets and associated confidence scores for each frame of the selected set of dissimilar frames; And the set of frames having uncertainty scores that do not satisfy the uncertainty threshold is removed from the set of selected dissimilar frames, but the set of selected dissimilar frames can be stored in a permanent storage system, and an uncertainty score that does not satisfy the uncertainty threshold. It is possible to further store instructions including instructions for excluding frame sets having s.

The object detector may further include a long-short term memory (LSTM) neural network.

The instructions for calculating the uncertainty score, when executed by the processor, cause the processor to, for each frame: identify the highest two related reliability scores that correspond to the same part of the frame and correspond to different object classes, and ; Then, the two highest related reliability scores are compared, but the uncertainty score is caused to be high if the difference between the two highest related reliability scores is small, and the uncertainty score is to be low if the difference between the two highest related reliability scores is large. It may contain commands to do.

The feature extractor may include a convolutional neural network (CNN) of the object detector.

According to an embodiment of the present disclosure, a computing system for removing video data includes: a processor; A permanent storage system coupled with a processor; And a memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to: receive a batch of frames of video data; An object detector comprising a convolutional neural network (CNN) for calculating sets of a plurality of bounding boxes that identify portions of a frame that describe each frame of an arrangement of frames of video data and instances of each object class of a plurality of object classes. Provided by, but each bounding box of the set of bounding boxes has an associated confidence score; Calculate an uncertainty score based on the sets of bounding boxes and associated confidence scores for each frame of the batch of frames of video data; A set of uncertain frames is selected from the arrangement of frames of the video data, and the uncertainty score of each frame of the set of uncertain frames satisfies the uncertainty threshold, and the selected set of uncertain frames is stored in a permanent storage system. The number is less than the number of frames in the arrangement of frames of video data, thereby causing the arrangement of frames of video data to be eliminated.

The object detector may further include a long-short term memory (LSTM) neural network.

The instructions for calculating the uncertainty score, when executed by the processor, cause the processor to, for each frame: identify the two highest related confidence scores that correspond to the same part of the frame and correspond to different object classes. and; And compare the two highest related confidence scores, where the uncertainty score can be high if the difference between the two highest related confidence scores is small, and the uncertainty score can be low if the difference between the two highest related confidence scores is large. It may contain instructions that cause it to occur.

The computing system and the permanent storage system can be mounted on the vehicle, and the placement of the video data can be part of a stream of video data captured by a video camera installed on the vehicle, while the video camera is configured to capture images of the vehicle's surroundings. It is composed.

The arrangement of frames of video data may be captured during a first time period having a length corresponding to the capture period, but the stream of video data corresponds to the video data captured during a second time period having a length corresponding to the capture period. A second arrangement of frames of video data, the second time period immediately following the first time period, and the computing system to remove the arrangement of frames of video data within a time corresponding to the capture interval. Can be configured.

According to an embodiment of the present disclosure, a computing system for removing video data includes: a processor; A permanent storage system coupled to the processor; And a memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to: receive a batch of frames of video data; By the feature extractor, mapping the arrangement of frames of the video data to corresponding feature vectors in the feature space, each feature vectors having a lower dimension than a corresponding frame of the frames of the arrangement; Calculating a plurality of dissimilarity scores based on the feature vectors, each dissimilarity score corresponding to any one of the frames of the batch; An object detector comprising a convolutional neural network (CNN) for calculating sets of a plurality of bounding boxes that identify portions of a frame that describe each frame of an arrangement of frames of video data and instances of each object class of a plurality of object classes. Provided by, but each bounding box of the set of bounding boxes has an associated confidence score; Calculating a plurality of uncertainty scores based on the sets of bounding boxes and associated confidence scores, each uncertainty score corresponding to any one of the frames of the batch; Calculating a total score for each frame by aggregating the associated dissimilarity score of the dissimilarity scores and the related uncertainty score of the uncertainty score; Select a set of frames in which the total score of each selected frame satisfies the total frame threshold from the arrangement of frames of video data; The selected set of frames is stored in the permanent storage system, and the number of the selected set of frames is smaller than the number of frames in the arrangement of frames of video data, thereby causing the arrangement of frames of video data to be removed.

The instructions for calculating the uncertainty score, when executed by the processor, cause the processor to, for each frame: identify the two highest related confidence scores that correspond to the same part of the frame and correspond to different object classes. and; And compare the two highest related confidence scores, where the uncertainty score can be high if the difference between the two highest related confidence scores is small, and the uncertainty score can be low if the difference between the two highest related confidence scores is large. It may contain instructions that cause it to occur.

The computing system and the permanent storage system can be mounted on the vehicle, and the placement of the video data can be part of a stream of video data captured by a video camera installed on the vehicle, while the video camera is configured to capture images of the vehicle's surroundings. It is composed.

The arrangement of frames of video data may be captured during a first time period having a length corresponding to the capture period, but the stream of video data corresponds to the video data captured during a second time period having a length corresponding to the capture period. A second arrangement of frames of video data, the second time period immediately following the first time period, and the computing system to remove the arrangement of frames of video data within a time corresponding to the capture interval. Can be configured.

The feature extractor may include a convolutional neural network (CNN) of the object detector.

The computing system according to an embodiment of the present disclosure may select a subset of frames of video data based on at least one of dissimilarity in a feature space between frames or uncertainty in object classification. Accordingly, unnecessary data can be removed in the learning of the machine learning model, and thus the performance of the computing system can be improved.

The accompanying drawings together with the specification illustrate exemplary embodiments of the present disclosure, and together with the detailed description, serve to explain the principles of the present disclosure.
1 is a block diagram depicting a video capture and data reduction system according to an embodiment of the present disclosure.
2 is a flowchart illustrating a method of reducing captured data according to an embodiment of the present disclosure.
3 is a schematic block diagram depicting a feature space similarity-based frame selection module that discards frames based on similarity according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method of selecting a frame by discarding frames based on similarity according to an embodiment of the present disclosure.
5 is a schematic block diagram depicting an uncertainty-based frame selection module for discarding frames based on uncertainty according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of selecting a frame by discarding frames based on an uncertainty score according to an embodiment of the present disclosure.
7 is a flowchart of a method of selecting frames in series, based on both similarity and uncertainty, according to an embodiment of the present disclosure.
8 is a flowchart of a method of selecting frames in parallel, based on both similarity and uncertainty, according to an embodiment of the present disclosure.
9 is a block diagram illustrating a computing system according to an embodiment of the present disclosure.

In the detailed description that follows, only specific embodiments of the present disclosure are shown and described by way of example. As will be appreciated by those of ordinary skill in the art, the present disclosure may be implemented in various forms, and should not be construed as being limited to the embodiments described below. In the drawings and the discussion below, like reference numbers refer to like elements.

In general, supervised learning of machine learning models involves a large collection of labeled training data. In comparative methodologies for the collection of learning data for computer vision systems for autonomous vehicles, cameras can be installed in vehicles and video data (e.g. 30 or 60 frames per second (fps)). ), each frame generally corresponds to one static image) can be captured during driving through various regions and under various driving conditions. The captured video can then be reviewed by human annotators, which can then read the video data, specific preselected classes of objects (e.g. For example, you can label, such as drawing boxes around instances of cars, trucks, pedestrians, walls, trees, signs, traffic signals, etc.).

In some situations, all video captured while driving is saved. However, storing all the video data captured while driving requires huge in-vehicle data storage, which can be expensive. Also, manually labeling huge amounts of data is extremely expensive because human annotators can have to see and label millions of frames.

In some situations, the driver (or other human operator (operator)) may manually activate or deactivate the camera to selectively record video clips of portions of the drive. However, this can be inefficient because the delay between the driver's recognizing a scene worth recording and activating the camera can cause important data to be lost. In addition, scenes manually selected by the driver may be affected by the driver's propensity and do not substantially reflect the types of scenes that would be most useful in providing additional data that could improve the performance of the learned machine learning model. May not.

In general, it is beneficial to collect data that is not similar to data that has already been collected (eg, data similar to previously captured data may be redundant). Continuing the above example, video clips and images depicting the diversity of objects (objects) and not similar to previously collected video clips are more than video clips depicting substantially the same objects and configurations shown previously. It could be more beneficial. Such more “salient” clips generally have a greater impact on improving the performance of trained machine learning models.

Accordingly, aspects of embodiments of the present disclosure relate to performing data reduction or removal (decimation) by selecting more salient portions of data to keep in order to train a machine learning model. Some aspects of embodiments of the present disclosure selectively maintain portions of the data stream and discard other portions of the data stream (e.g., by removing (decimating) the data stream) in resource-limited environments ( For example, it enables intelligent, automatic and continuous real-time data capture within environments with limited data storage capacity or data transmission bandwidth). Reducing or removing data by selecting portions of the data stream can reduce storage costs and human annotation costs, while maintaining the benefit of obtaining additional data to improve the performance of the learned algorithms.

Hereinafter, embodiments of the present disclosure will be mainly described in connection with reducing or reducing video data in the context of video cameras installed in vehicles and capturing data on an environment near the vehicle, but embodiments of the present disclosure are not limited thereto. It can also be applied to reduce stored data in other data collection contexts.

According to an embodiment of the present disclosure, an automated image and/or video capture methodology continuously monitors a scene (scene) captured by a video camera, and reconstructs a machine learning model using selected critical video frames. Select and store only important clips and images (eg frames of video) that are predicted by the system to improve the performance of the machine learning model when training.

1 is a block diagram depicting a video capture and data reduction system 100 depicted as aboard a vehicle 105, in accordance with an embodiment of the present disclosure. The video capture and data reduction system 100 of FIG. 1 includes one or more video cameras 110. In some embodiments, the video cameras 110 may capture video (including a stream or sequences of two-dimensional static (still) frames) at a frame rate such as 30 frames per second (fps) or the like. The video cameras 110 capture images generally continuously while operating, that is, while the vehicle 105 is driving. Computing system 130 is a buffer memory 150 for storing video data (including a stream of frames) received from the video cameras 110; for example, a dynamic random-access memory or a dynamic random access memory (DRAM) ).

In some embodiments, the video capture and data reduction system 100 is an object detection system (e.g., an object detection and/or scene segmentation system running on the same computing system 130 or another computing system mounted on the same vehicle). Works with The object detection system can be used to detect (detect) and classify objects viewed by the video cameras 110, and detection (detections) by object detection can be used as an input to the autonomous driving system of the vehicle.

In some embodiments, computing system 130 may perform active learning to select salient portions of the data stream (e.g., salient frames) to frames of video data stored in buffer memory 150 active learning) and novelty detection processes, in which the salient portions of the data stream, after annotation or labeling, are detected as a learning set containing these selected and labeled salient portions. It is expected to improve the performance of the object detection system when the system is re-learned. The computing system 130 writes a subset of the selected frames to the persistent storage system 170. Examples of suitable persistent storage system 170 include one or more flash memory units and/or which may be operated by a redundant array of independent disks or a controller such as a RAID controller, a network-attached storage controller, a storage area network controller, etc. It may include a mass storage system that may include one or more hard disk drives. As another example, the persistent storage system 170 may include a computer network device (e.g., cellular, WiFi, and/or Ethernet) for transmitting the received data to persistent storage not mounted on the vehicle 105. Yes (e.g., it may not be practical to transmit all the captured data over the network, but it may be feasible to transmit the removed (reduced) data). Frames that are not selected (eg, not written to the persistent storage system 170) are considered unnecessary, and may be discarded (eg, overwritten in the buffer memory).

2 is a flowchart illustrating a method for reducing captured video data according to an embodiment of the present disclosure. In operation 210, the computer system 130 receives a batch of data. Following the example described above, in some embodiments, the batch of data corresponds to a batch of video frames (eg, still images) captured by the video camera during the capture period I. The video cameras 110 can capture data at a constant rate f _s (eg, 30 fps to 60 fps), so each capture period I includes f _s *I frames. The length of the capture section I is the size T of the buffer memory 150 (e.g., _{the amount of space allocated to store one batch of video data of f s} *I frames) and the processing of the batch is the next batch of data. It may rely on the processing speed of techniques for selecting frames for storage that allow it to be completed prior to receiving it from video cameras 110 in operation 210. Therefore, certain parameters for the length of the capture period I and the size of the buffer memory 150 are techniques for selecting frames (e.g., algorithmic temporal complexity of processes for selecting frames may place restrictions on the batch size. And method 200 (e.g., performing selection of image frames for storage (storage)) may depend on details including the performance of computing system 130 .

In operation 230, computing system 130 stores the current batch of frames in buffer memory 150. Computing system 130 may store these frames as received from one or more video cameras 110 (eg, storing individual still (still) frames or a stream of video including still images). In some embodiments, the buffer memory 150 allows multiple batches (e.g., to enable the system to process the current frames in real time while still receiving batches of data and storing the received batches in the buffer memory 150). For example, it may contain enough space to store at least two data batches).

In operation 250, computing system 130 determines that a subset of a batch of frames of video data is more significant (e.g., a machine learning model is applied to the selected data). If it is learned based on the machine learning model, it is more likely to improve the performance of the machine learning model). This may also be referred to herein as reducing data, filtering data, or decimating data. The subset will be referred to herein as a collection of selected frames, where the number of selected frames is less than the number of frames in the arrangement of input frames (e.g., the size of the set of selected frames is the set of arrangements of input frames. Smaller than the size of). Embodiments of the present disclosure for performing prediction (eg, determining the saliency of data) will be described in detail below.

In operation 270, computing system 130 stores or writes selected frames to persistent storage system 170. Persistent storage system 170 stores selected frames permanently or non-transitory (e.g., during and after the end of the trip, during and after the end of the trip for which the video cameras 110 of the vehicle 105 collect data) .

In operation 290, computing system 130 determines whether to continue capturing data. This determination may be based, for example, on whether computing system 130 has received a command to stop or stop capturing data. If computing system 130 chooses to continue capturing data, computing system 130 can continue with operation 210 to receive from video cameras 110 a batch of frames of the next video data, At this time, the arrangement of frames of the next video data corresponds to a time period (having the same length as the length of the capture section I) immediately following the arrangement of the frames of the previous video data. If computing system 130 decides to stop capturing data, the method ends.

According to some embodiments of the present disclosure, video cameras 110 continuously capture images. Thus, video data is continuously received, and all batches of video data are captured during the capture period I. Accordingly, each iteration of the method 200 including selecting frames from the batch for storage in operation 250 is completed within a time not longer than the capture period I, and thus the computational resources of the computing system 130 are captured. It becomes possible to process the arrangement of frames of the next video data corresponding to the frames captured during the second time period having the same length as the period I.

Frame selection based on frame similarity in feature space

According to some embodiments of the present disclosure, computing system 130 selects frames by selecting different dissimilar frames in a feature space (eg, in operation 250 of FIG. 2 ). In some embodiments, this is equivalent to discarding frames similar to the'reference' frames in the feature space. In general, presenting many learning examples that are substantially identical to each other does not cause a significant improvement in the performance of the learned machine learning system because similar examples are redundant. Accordingly, aspects of embodiments of the present disclosure are related to increasing the efficiency of collection of training data by removing overlapping frames and maintaining only dissimilar frames.

3 is a schematic block diagram of a feature space similarity-based frame selection module 300 for discarding frames based on similarity according to an embodiment of the present disclosure. 4 is a flowchart illustrating a method for selecting a frame in which frames are discarded based on similarity in a feature space according to an embodiment of the present disclosure.

3 and 4, in operation 410, the computing system 130 (eg, stored in the buffer memory 150) a batch of N frames of video data (302, a batch of N frames). of video data). In operation 430, the computing system 130 supplies the arrangement 302 of frames of data to the feature extractor 310 of the feature space similarity-based frame selection module 300, and the feature extractor 310 Feature vectors 330 are extracted from and each feature vector corresponds to one of the frames of data. This may also be referred to as mapping frames to a feature space (or a latent space), where the feature space may have a lower rank than the original image space of the frames of data (or feature vectors are input It can have a lower dimension than the image). For example, each frame may have an image resolution of 2048*2048 pixels, but each feature vector may have a dimension of 256*256 pixels. For example, in this alignment (array), the feature vector will be 64 times smaller than the corresponding frame.

As such, feature vectors compress or summarize the contents of each of the frames of data so that two frames that are substantially similar in content are mapped to similar feature vectors, even if the contents of the corresponding pixels in the frames are significantly different. It is possible to be considered as. For example, an image of a scene at an instant may be substantially, semantically, similar to the image of the scene one second later, where other identical vehicles, pedestrians, and stationary objects will be visible in both images. Because it can. However, if two images are compared at the pixel level, the two images can be very different, because the position of each of the objects (objects) in the images may have moved (e.g., a common object between the two images). All of them can be transformed according to the movements of objects, along with the movement of the vehicle 105 along the way). In addition, comparing two frames of data (two frames of data) in the feature space with a rank lower than the input image space is less computationally intensive (e.g., the value to be compared is more Less), thus making the real-time performance of data selection more manageable (eg, by adjusting the size of the feature vectors).

ois. "Xception: Deep learning with depthwise separable convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.)을 포함할 수 있다. 몇몇 실시 예들에서, 특징 추출기(310)는 다수의 훈련된 잔여(residual) 모듈들(또는 잔여 계층들)을 포함하는 잔류 신경망(residual neural network)을 더 포함하고, 잔류 신경망은 백본 신경망의 출력을 제공받고, 그리고 최종 잔여 모듈의 출력은 입력 프레임의 추출된 특징들 또는 추출된 특징 벡터로 취해진다.In some embodiments of the present disclosure, the feature extractor 310 running on the computing system 130 extracts feature vectors 330 from the frames by supplying the frames to the neural network. The feature extractor 310 identifies instances of objects (objects) in the image, and the initial stages of the object detector (detector) neural network, which are learned to compute bounding boxes that identify the locations of those objects. Can respond to. The object (object) detector neural network may include a convolutional neural network (CNN) combined with a long-short term memory (LSTM). In this case, the extracted features correspond to the activations (or outputs) of the intermediate layer of the network prior to computing the final result (eg, prior to calculating the positions of the bounding boxes). For example, continuing the above-described example of a computer vision system comprising a deep CNN trained to compute labeled bounding boxes containing instances of objects of various classes in the input (input) image, feature extractor 310 It can respond to an appropriate'backbone' CNN. The backbone CNN is a standard learning corpus such as ImageNet (see, for example, Deng, Jia, et al. "ImageNet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009.) It can be pre-trained on the (standard training corpus). Examples of backbone neural networks are MobileNetV2 (see, for example, Sandler, Mark, et al. "MobileNetV2: Inverted residuals and linear bottlenecks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.), MobileNetV3 (Ref. For example, Howard, Andrew, et al. "Searching for MobileNetV3." arXiv preprint arXiv:1905.02244 (2019).), MnasNet (see, for example, Tan, Mingxing, et al. "MnasNet: Platform-aware neural" architecture search for mobile." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.), and Xception (see, for example, Chollet, Fran

ois. "Xception: Deep learning with depthwise separable convolutions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.). In some embodiments, the feature extractor 310 further includes a residual neural network including a plurality of trained residual modules (or residual layers), the residual neural network receiving the output of the backbone neural network. Is provided, and the output of the final residual module is taken as the extracted features of the input frame or the extracted feature vector.

In some embodiments, feature extractor 310 corresponds to a portion of a neural network of an object detection system associated with video capture and data reduction system 100. For example, if the object detection system includes a backbone neural network such as CNN and MobileNetV3, the feature extractor 310 may include the same backbone neural network.

In other embodiments of the present disclosure applied to other contexts such as speech processing or natural language processing, other feature extractors may be applied to perform feature extraction in a manner suitable for a domain. For example, in speech processing, a recurrent neural network operating on spectral features (eg, speech signals in a frequency domain) may be used as a component of the feature extractor.

In operation 450, computing system 130 selects one or more frames of reference from a placement of reference frames based on dissimilarities between feature vectors using similarity ranking module 350. In some embodiments, the computing system 130 randomly selects a seed frame from the frames 302 of data in the buffer memory 150, and dissimilar or distant from each other (e.g., , A set of frames that satisfy a dissimilar threshold) are identified based on feature vectors corresponding to the frames. In some embodiments, the k-Center-Greedy technique (see, eg, Sener, Ozan, and Silvio Savarese. “Active learning for convolutional neural networks: A core-set approach.” arXiv preprint arXiv: 1708.00489 (2017). ), etc. may be used to identify a set of reference frames. For example, in an embodiment using the k-Center-Greedy approach, an anchor frame F is randomly selected, and added as a seed to the set of selected frames S={F}. Then, additional frames are selected based on having the lowest similarity scores for the selected frames. For example, the second reference frame F1 may be selected by finding a frame that is least similar to the seed frame F. After adding the second frame, the selected frames S include both F and F1 (S={F, F1}). The third frame F2 can be selected by picking the frame with the lowest average similarity for the members in S (eg, the lowest average similarity for both F and F1). Frames can continue to be added in this way until K frames have been selected. According to another embodiment, the seed frame is randomly selected and similarity distances for the seed frame are calculated for all remaining frames around the frame, and scoring for the remaining frames is based on the calculated similarity distances. And it is done. The frame of reference will have the highest score (eg, most similar to itself).

For example, in some embodiments of the present disclosure, during each iteration of the process of selecting reference frames, one of the frames of the arrangement of data is randomly selected. The feature vector of that frame is compared to feature vectors of other reference frames (a set of reference frames can be initialized by randomly selecting a first reference frame) to calculate one or more similarity scores. In some embodiments, the similarity score from the comparison of two feature vectors uses, for example, an L1 norm (or Manhattan distance) or an L2 norm (or Euclidean distance) between two feature vectors. Is calculated. If the similarity score of the feature vector of the current frame fails to satisfy the dissimilarity threshold (eg, has a feature vector that is too similar to any one of the current reference frames), then it is discarded. Otherwise, the current frame is added to the selected collection of reference frames. Consequently, all reference frames are dissimilar in that their similarity scores indicate that they are at least a threshold distance (or score) apart from each other in the feature space, and the computing system 130 is substantially similar to the reference frames ( For example, in the feature space, sets of frames (with similarity scores indicating that they are located within a threshold distance of at least one reference frame) are discarded so that the selected reference frames exclude discarded frames.

The number of reference frames can be configured based on the desired degree of data reduction. For example, if a period (interval) contains 300 frames and you want a data reduction of 90% (e.g., keep 30 frames), then the number of reference frames can be set to 30 . In operation 470, the computing system 130 discards the remaining frames (eg, frames not selected as reference frames or'non-reference' frames). Accordingly, the reference frames are output as frames 352 selected by the feature space similarity-based frame selection module 300.

Accordingly, according to an embodiment of the present disclosure, the feature space similarity-based frame selection module 300 selects representative or reference data different (dissimilar) from other data in the arrangement, and unnecessary ( It concerns one method 400 of how to reduce the captured data by discarding the redundant) data.

Frame selection based on frame uncertainty

According to an embodiment of the present disclosure, the computing system 130 selects frames (for example, operation 250 of FIG. 2) and frames for outputting high uncertainty by a learned object detector (object detector). By making choices, and therefore additional learning examples will help train the object detector to differentiate between competing possible index fingers (eg, to distinguish between two classes such as'pedestrian' vs'cyclist').

5 is a schematic block diagram of an uncertainty-based frame selection module 500 for discarding frames based on uncertainty according to an embodiment of the present disclosure. 6 is a flowchart illustrating a frame selection method for discarding frames based on an uncertainty score according to an embodiment of the present disclosure.

5 and 6, in operation 610, computing system 130 receives a batch 502 of M frames of input frames (eg, stored in buffer memory 150). In operation 630, the computing system 130 supplies each of the M frames to the K-class object detector 510, and the K-class object detector 510 provides the bounding boxes 530 for each frame. It is configured to output K sets (for convenience, Fig. 5 shows a set of bounding boxes 531 for frame i and a set of bounding boxes 539 for frame i+M, as well as frame i and frame i+M. It shows a state in which frames from frame i to frame i+M are sequentially supplied to the K-class object detector 510 in order to calculate sets of bounding boxes for all frames in between), at this time, a set of bounding boxes Each corresponds to one of K different classes that the K-class object detector 510 is trained to detect. Within each set of bounding boxes, for example, for the jth class of the K classes, each bounding box corresponds to the position of the detected instance of the jth class. Each of the bounding boxes is also associated with a probability or confidence score in which the bounding box describes an instance of class j (see, eg, Aghdam, Hamed H., et al. "Active Learning for Deep Detection Neural Networks." Proceedings. of the IEEE International Conference on Computer Vision. 2019.).

ding, and Joachim Denzler. "Active learning for deep object detection." arXiv preprint arXiv:1809.09875 (2018)에 설명되어 있고, 그 전체 개시 내용은 여기에서 참조로서 포함된다. 예를 들어, '1-vs-2' 또는 '마진 샘플링(margin sampling)'은 두 개의 가장 높은 점수를 갖는 클래스들 c₁ 및 c₂에 대한 메트릭으로서 사용될 수 있다:In operation 650, the computing system 130 calculates an uncertainty score for each frame, using the uncertainty measurement module 550, based on the sets of bounding boxes for each of the K classes and the associated probabilities. Calculate. In some embodiments, a pixel-wise approach is used to compare the two highest reliability detections for each pixel of the segmentation map. In some embodiments, uncertainty scores are computed only for the selected sets of bounding boxes, and the sets are aggregated together (one box proposal will have one associated class probability). Techniques for calculating uncertainty metrics are, for example, Brust, Clemens-Alexander, Christoph K

ding, and Joachim Denzler. "Active learning for deep object detection." arXiv preprint arXiv:1809.09875 (2018), the entire disclosure of which is incorporated herein by reference. For example, '1-vs-2'or'marginsampling' can be used as a metric for the _{two highest scored classes c 1} and c _2:

는 바운딩 박스(또는 이미지) x가 클래스 c_i의 인스턴스를 묘사하는 확률 또는 신뢰도 점수를 나타낸다. 특정 바운딩 박스(또는 이미지) x에 대한(또는 서로 다른 두 클래스들의 겹치는 바운딩 박스들, 또는 특정한 픽셀에 대한) 두 개의 가장 높은 점수를 갖는 클래스들 c₁ 및 c₂ 사이의 신뢰도 점수

의 차가 작으면, 그렇다면 예시(example)는 결정 경계에 가까울 것(예를 들어, 더 불확실함)이고, 그러므로 더 높은 불확실성 점수를 가질 수 있으며, 이는 학습을 위한 더 현저한 또는 쓸모 있는 예시에 대응한다(예를 들어, 예시는 수동적으로 바운딩 박스 x의 올바른 클래스로 주석이 달릴 수 있고, 레이블링된 예시는 기계 학습 시스템을 재학습하는 데 사용될 수 있도록 함). 이러한 검출 메트릭들은 다양한 클래스들의 집합들에서, 프레임 x 내 모든 바운딩 박스들(또는 검지들) D에 대해 모든 v_1vs2들의 합을 계산하는 것과 같은 다양한 기법들에 따라 집계될 수 있다:In Equation 1,

Represents the probability or confidence score that the bounding box (or image) x _{describes an instance of class c i.} Confidence score between _{classes c 1} and c ₂ with the two highest scores for a particular bounding box (or image) x (or overlapping bounding boxes of two different classes, or for a particular pixel)

If the difference of is small, then the example will be closer to the decision boundary (e.g., more uncertain), and therefore may have a higher uncertainty score, which corresponds to a more salient or useful example for learning. (For example, examples can be manually annotated with the correct class of bounding box x, so labeled examples can be used to retrain machine learning systems). These detection metrics can be aggregated according to various techniques, such as calculating the sum of _{all v 1vs2} for all bounding boxes (or detections) D in frame x, in sets of various classes:

Another option is to calculate the average of the index fingers in frame x, for example:

Another option is to calculate the maximum of the index fingers in frame x, for example:

In operation 670, if the uncertainty meets the uncertainty threshold (e.g., if a sufficiently high uncertainty for the frame is present at the output of the K-class object detector 510), then the computing system 130 You can choose to save it as a useful example for your learning. If the uncertainty score does not satisfy the uncertainty threshold, the frame is discarded.

Accordingly, according to an embodiment of the present disclosure, the uncertainty-based frame selection module 500 has a high uncertainty in the K-class object detector 510 (for example, the K-class object detector 510 is well By selecting the data that is not working), and the K-class object detector 510 discarding data with low uncertainty (e.g., inputs for which the learned K-class object detector 510 already works well). It concerns one method 600 for reducing the captured data.

According to some embodiments of the present disclosure, the computing system 130 separates the feature space similarity-based frame selection module 300 and the uncertainty-based frame selection module 500, and corresponding methods, (as described above). Or using various combinations such as serial (eg, sequentially) or parallel, etc. to select frames from an arrangement of frames (eg, in operation 250 of FIG. 2).

For example, in some embodiments, the feature space similarity-based frame selection module 300 and the uncertainty-based frame selection module 500 may be used serially. In such embodiments, the feature space similarity-based frame selection module 300 removes redundant frames in the arrangement, thus reducing the number of frames provided to the uncertainty-based frame selection module 500, and thus the uncertainty-based frame. The computational load of the selection module 500 is also reduced. In some circumstances, such as the processes used by the uncertainty-based frame selection module 500 have super-linear algorithm time and/or spatial complexity, reducing the size of the set of input frames 502 This has a significant impact on whether the system is capable of operating in real time. In some embodiments, the similarity threshold may be configured such that the number of selected frames causes the uncertainty-based frame selection module 500 to complete frame selection within a limited time (eg, within a period of capture interval I).

In some embodiments, the selected frames 352 of the output of the feature space similarity-based frame selection module 300 are provided to the uncertainty-based frame selection module 500 as an input arrangement 502 of M frames, which The number of frames (eg, selected frames 552) to be stored in the persistent storage system 170 is further reduced. More specifically, FIG. 7 is a flowchart of a method 700 for selecting frames based on both similarity and uncertainty in order according to an embodiment of the present disclosure. As shown in FIG. 7, frames in operation 7300 are based on similarity in a feature space (e.g., feature space similarity-based frame selection module 300 shown in FIG. 3 and shown in FIG. 4 ). The placement of frames (e.g., placement of raw video data captured by video cameras 110 during capture interval I) to select (according to method 400) is processed by computing system 130 do. Computing system 130 then selects a subset of frames based on uncertainty in operation 7600 to select frames based on both a similarity metric (e.g., based on dissimilarity) and an uncertainty metric. By doing (eg, further reducing the data) the resulting frames (selected according to the (non)similarity of the frames in the feature space) are processed.

In some embodiments, rather than selecting frames, the feature space similarity-based frame selection module 300 and the uncertainty-based frame selection module 500 each receive similarity scores and uncertainty scores from video cameras 110. Compute for all frames of data. The similarity score of each frame is combined with or aggregated together with the corresponding uncertainty score of the frame to calculate the total score for each of the frames, wherein frames having a total score satisfying the threshold are selected and stored in the persistent storage system 170. Frames that are stored and have a total score that do not satisfy the threshold are discarded.

8 is a flowchart of a method 800 of selecting frames based on both dissimilarity and uncertainty in parallel according to an embodiment of the present disclosure. As shown in FIG. 8, in operation 8300, computing system 130 may perform such dissimilarity metrics (e.g., how many frames are) in substantially the same manner as described above with reference to FIGS. To calculate a per-frame score based on whether or not they are dissimilar to selected reference frames in the batch set of frames), the batch of frames of data is scored based on the similarity in the feature space. In various embodiments, the Euclidean (L2) distance, the Manhattan (L1) distance, or the cosine distance (or cosine similarity) is used to calculate the similarity score in the feature space. Further, in operation 8600, the computing system 130 calculates scores per frame based on uncertainty for a batch of frames of data (the same batch as the batch scored based on dissimilarity in the feature space), Based on the uncertainty, scores are scored in substantially the same manner as described above with reference to FIGS. 5 and 6. In operation 8700, the computing system 130 calculates the sum of the dissimilarity score d(x _i ) and the uncertainty score u(x _i ) _{for each frame to calculate the total score o(x i) for the frame.} Serve. In some embodiments, this may be a linear combination of the two scores, for example:

Wherein α is a parameter that controls the relative weights of dissimilarity scores and uncertainty scores. However, embodiments of the present disclosure are not limited thereto, and other techniques for aggregating two scores may be used instead.

In operation 8800, computing system 130 determines whether the total score for the frame satisfies the total frame threshold. If the total frame threshold is satisfied, the frame is selected for storage in the persistent storage system 170. If the total frame threshold is not satisfied, the frame is discarded.

In some embodiments, the feature extractor 310 of the feature space similarity-based frame selection module 300 may correspond to a portion of the neural network of the object detector of the uncertainty-based frame selection module 500. For example, if the K-class object detector 510 system includes a backbone neural network such as CNN and MobileNetV3, the feature extractor 310 of the feature space similarity-based frame selection module 300 may include the same backbone neural network. have.

9 is a block diagram illustrating a computing system 130 according to an embodiment of the present disclosure. As shown in FIG. 9, the computing system 130 according to an embodiment includes a processor 132 and a memory 134, wherein the processor 132 includes: (eg, having one or more cores) General-purpose central processing unit (CPU); A graphical processing unit (GPU), a field programmable gate array (FPGA); A neural processing unit (NPU) or a neural network processor (NNP) (eg, a processor having a structure tailored to perform inference using a neural network); Alternatively, it may be a neuromorphic processor. For example, the parameters of the neural network (e.g., weights and biases) and the neural network structure may be stored in a non-transitory memory connected to the processor, in which the processor loads the parameters and the neural network structure from the memory. To perform inference. As another example, in the case of an FPGA, the FPGA can be configured in a non-transitory manner with a network structure and weights using a bitfile. The memory 134 may include a buffer memory 150 for storing an arrangement of data (eg, arrangement of frames of video data received from one or more video cameras 110 ). Computing system 130 may further include a coprocessor 136, which includes: a general-purpose CPU (with one or more cores); GPU; FPGA; NPU, NNP; Alternatively, it may include a pneumotropic processor. For example, the coprocessor 136 may be configured to independently perform (and accelerate or offload) operations associated with elements such as the feature extractor 310 and the K-class object detector 510. As described above, computing system 130 may be configured to store data selected by computing system 130 in persistent storage system 170. Further, the memory 134, when executed by the processor 132 and/or the coprocessor 136, stores instructions for implementing the modules and methods described above according to embodiments of the present disclosure.

Accordingly, aspects of embodiments of the present disclosure relate to systems and methods for data reduction. While the present disclosure has been described in connection with certain exemplary embodiments, the present disclosure is not limited to the disclosed embodiments, but rather, vice versa, various modifications and equivalent arrangements included within the scope of the appended claims and their equivalents. It will be understood that it is intended for handling.

100: video capture and data reduction system

Claims (20)

In a computing system for removing (decimation) video data:
Processor;
A persistent storage system coupled to the processor; And
A memory for storing instructions, wherein the instructions, when executed by the processor, cause the processor to:
Receive a batch of frames of video data;
Mapping the frames of the arrangement to corresponding feature vectors in a feature space, by a feature extractor, each of the corresponding feature vectors having a lower dimension than a corresponding one of the frames of the arrangement;
Select a set of dissimilar frames from the frames of the video data based on dissimilarities between corresponding ones of the feature vectors; And
The selected set of dissimilar frames is stored in the persistent storage system, wherein the size of the selected set of dissimilar frames is smaller than the number of frames in the arrangement of the frames of the video data. A computing system that allows to eliminate (reduce) the placement of frames. The method of claim 1,
Instructions that cause the processor to select the set of frames, when executed by the processor, cause the processor to:
Randomly select a first reference frame from the arrangement of the frames of the video data;
Discarding a first set of frames from the frames of the arrangement, the first set of frames having corresponding feature vectors within a similarity threshold distance of a first feature vector corresponding to the first reference frame;
Selecting a second reference frame from the frames of the video data, the second reference frame having a second feature vector, and a distance between the first feature vector and the second feature vector is greater than the similarity threshold distance; And
Discarding a second set of frames from the frames of the arrangement, causing the second set of frames to have corresponding feature vectors within the dissimilarity threshold distance of the second feature vector,
The selected set of dissimilar frames includes the first reference frame and the second reference frame, and excludes the first set of frames and the second set of frames. The method of claim 1,
The feature extractor is a computing system including a neural network. The method of claim 3,
The neural network is a computing system including a convolutional neural network (CNN). The method of claim 1,
The computing system and the permanent storage system are on-boadr on a vehicle, and
The arrangement of the frames of video data is part of a stream of video data captured by a video camera mounted on the vehicle, and the video camera is configured to capture images of surrounding environments of the vehicle. The method of claim 5,
The arrangement of the frames of the video data is captured over a first time period having a length corresponding to the capture period,
The stream of video data includes a second arrangement of frames of video data corresponding to video data captured during a second time period having the length corresponding to the capture period, wherein the second time period comprises the first Right after the time period, and
The computing system is configured to map the frames of the batch to remove the batch of frames of the video data within a time corresponding to the capture interval. The method of claim 1,
The memory further stores instructions for removing frames from the selected set of dissimilar frames based on an uncertainty metric, the instructions causing the processor to:
Each frame of the selected set of dissimilar frames is supplied to an object detector including a CNN in order to calculate sets of a plurality of bounding boxes that identify portions describing instances of each object class of a plurality of object classes of the frame. Provided that the bounding box of each of the set of bounding boxes has an associated confidence score;
Calculating an uncertainty score for each frame of the selected set of dissimilar frames based on the sets of bounding boxes and the associated confidence score; And
Removing from the selected set of dissimilar frames a set of frames having uncertainty scores that failed to satisfy an uncertainty threshold,
The computing system further comprising instructions for causing the selected set of dissimilar frames stored in the persistent storage system to exclude the set of frames having uncertainty scores that failed to meet the uncertainty threshold. The method of claim 7,
The object detector further includes a long-short term memory (LSTM) neural network. The method of claim 7, wherein the instructions for calculating the uncertainty score, when executed by the processor, cause the processor to:
Identify two highest associated confidence scores corresponding to the same portion of the frame and corresponding to different object classes; And
Comparing the two highest associated confidence scores,
The uncertainty score is high when the difference between the two highest associated confidence scores is small, and
A computing system comprising instructions that cause the uncertainty score to be low when the difference between the two highest associated confidence scores is large. The method of claim 7,
The feature extractor comprises the CNN of the object detector. In a computing system for removing (decimation) video data:
Processor;
A permanent storage system coupled to the processor; And
A memory for storing instructions, wherein the instructions, when executed by the processor, cause the processor to:
Receive a batch of frames of video data;
Each frame of the arrangement of the frames of the video data is converted into a convolutional neural network (CNN) to calculate sets of a plurality of bounding boxes that identify a portion describing an instance of each of the object classes of the plurality of object classes of the frame. Supply to an object detector including, wherein the bounding box of each set of bounding boxes has an associated confidence score;
Calculating an uncertainty score for each frame of the batch of frames of the video data based on the sets of bounding boxes and the associated confidence scores;
Selecting a set of uncertain frames from the arrangement of the frames of the video data, wherein an uncertainty score of each frame of the set of uncertain frames satisfies an uncertainty threshold; And
The set of the selected uncertain frames is stored in the persistent storage system, wherein the number of the selected uncertain frames is smaller than the number of frames in the arrangement of the frames of the video data, thereby removing the arrangement of the frames of the video data. The computing system that causes it to occur. The method of claim 11,
The object detector further includes a long-short term memory (LSTM) neural network. The method of claim 11, wherein the instructions for calculating the uncertainty score, when executed by the processor, cause the processor to:
Identifying two highest associated confidence scores corresponding to the same part of the frame and corresponding to different object classes; And
Identify the two highest associated confidence scores,
The uncertainty score is high when the difference between the two highest associated confidence scores is small, and
A computing system comprising instructions that cause the uncertainty score to be low when the difference between the two highest associated confidence scores is large. The method of claim 11,
The computing system and the permanent storage system are on-board on a vehicle, and
The arrangement of the frames of video data is part of a stream of video data captured by a video camera mounted on the vehicle, and the video camera is configured to capture images of surrounding environments of the vehicle. The method of claim 14,
The arrangement of the frames of the video data is captured over a first time period having a length corresponding to the capture period,
The stream of video data includes a second arrangement of frames of video data corresponding to video data captured during a second time period having the length corresponding to the capture period, wherein the second time period comprises the first Right after the time period, and
The computing system is configured to map the frames of the batch to remove the batch of frames of video data within a time corresponding to the capture interval. In a computing system for removing (decimation) video data:
Processor;
A permanent storage system coupled to the processor; And
A memory for storing instructions, wherein the instructions when executed by the processor cause the processor to:
Receive an arrangement of frames of video data;
By a feature extractor, mapping the arrangement of the frames of the video data to corresponding feature vectors in a feature space, each of the feature vectors having a lower dimension than a corresponding one of the frames of the arrangement;
Calculating a plurality of dissimilarity scores based on the feature vectors, each of the dissimilarity scores corresponding to any one of the frames of the arrangement;
A convolutional neural network (CNN) is used for each frame of the arrangement of the frames of the video data to calculate sets of a plurality of bounding boxes that identify portions depicting instances of each object class of a plurality of object classes of the frame. Supply to the containing object detector, wherein each of the bounding boxes of each set of bounding boxes has an associated confidence score;
Calculating a plurality of uncertainty scores based on the sets of bounding boxes and the associated confidence scores, each of the uncertainty scores corresponding to any one of the frames of the batch;
Calculating a total score for each frame by aggregating an associated dissimilarity score among the dissimilarity scores and an associated uncertainty score among the uncertainty scores;
Selecting a set of frames from the arrangement of the frames of the video data, wherein the total score of each of the selected frames satisfies a total frame threshold; And
Storing the set of selected frames in the persistent storage system, wherein the number of the set of selected frames is smaller than the number of frames in the arrangement of frames of the video data, thereby causing the arrangement of frames of the video data to be removed. Computing system. The method of claim 16,
The instructions for calculating the uncertainty scores, when executed by the processor, cause the processor to:
Identifying two highest associated confidence scores corresponding to the same part of the frame and corresponding to different object classes; And
Identify the two highest associated confidence scores,
The uncertainty score is high when the difference between the two highest associated confidence scores is small, and
A computing system comprising instructions that cause the uncertainty score to be low when the difference between the two highest associated confidence scores is large. The method of claim 16,
The computing system and the permanent storage system are on-boadr on a vehicle, and
The arrangement of the frames of video data is part of a stream of video data captured by a video camera mounted on the vehicle, and the video camera is configured to capture images of surrounding environments of the vehicle. The method of claim 18,
The arrangement of the frames of the video data is captured over a first time period having a length corresponding to the capture period,
The stream of video data includes a second arrangement of frames of video data corresponding to video data captured during a second time period having the length corresponding to the capture period, wherein the second time period comprises the first Right after the time period, and
The computing system is configured to map the frames of the batch to remove the batch of frames of the video data within a time corresponding to the capture interval. The method of claim 16,
The feature extractor comprises the CNN of the object detector.

Images