TWI749870B - Device of handling video content analysis - Google Patents

Abstract

A computing device for handling video content analysis, comprises a preprocessing module, for receiving a first plurality of frames and for determining whether to delete at least one of the first plurality of frames according to an event detection, to generate a second plurality of frames according to the determination for the first plurality of frames; a first deep learning module, coupled to the preprocessing module, for receiving the second plurality of frames and for determining whether to delete at least one of the second plurality of frames according to a plurality of features of the second plurality of frames, to generate a third plurality of frames according to the determination for the second plurality of frames; and a second deep learning module, coupled to the first deep learning module, for receiving the third plurality of frames, to generate a plurality of prediction outputs of the third plurality of frames.

Description

Device for processing video content analysis

The present invention relates to a device used in a multimedia system, in particular to a device for processing video content analysis.

Video content analysis aims to identify (e.g., detect or determine) temporal events and/or spatial events (e.g. targets) in the frames of the video or stream. Video content analysis has been applied to solve problems in several application areas, such as detection (such as tamper detection and anomaly detection), video tracking (such as person re-identification (person re-ID)), traffic Monitoring (such as number of people/vehicle count), etc. In the prior art, several methods for video content analysis have been proposed, but these methods achieve better performance at the cost of high computational complexity. When the computing power of the hardware is insufficient, it is difficult to realize the hardware calculation of all frames of the video by these methods. Therefore, a method with low computational complexity for processing video content analysis is still a problem to be solved urgently.

The present invention provides a method and device for processing video content analysis to solve the above-mentioned problems.

The present invention discloses a computing device for processing video content distribution Video content analysis, including: a preprocessing module, used to receive the first plurality of frames, and according to an event detection (event detection), determine whether to delete the first plurality of frames At least one frame to generate a second plurality of frames according to the decision for the first plurality of frames; a first deep learning module (deep learning module) coupled to the preprocessing module for receiving The second plurality of frames, and based on the plurality of characteristics of the second plurality of frames, determine whether to delete at least one frame of the second plurality of frames, so as to generate the first plurality of frames according to the decision for the second plurality of frames Three plural frames; and a second deep learning module, coupled to the first deep learning module, for receiving the third plural frames to generate plural prediction outputs of the third plural frames.

10: Computing device

100: preprocessing module

110: The first deep learning module

120: The second deep learning module

20: Computing device

202: first buffer

204: second buffer

206: adaptability buffer

30: Flow chart

300~318: steps

400: Frame

410: Centerline

420: Orbit

500: frame

502: Track

504: Fastener

506: Bolt

508: Marking Line

60: Flow chart

600~618: steps

70: Flow Chart

700~718: steps

80: computing device

802: The third deep learning module

804: Fourth Deep Learning Module

90: Flow chart

900~918: Steps

Figure 1 is a schematic diagram of a computing device according to an embodiment of the present invention.

Figure 2 is a schematic diagram of a computing device according to an embodiment of the present invention.

Figure 3 is a flowchart of a process for detecting abnormalities in railway components according to an embodiment of the present invention.

Figure 4 is a schematic diagram of a frame according to an embodiment of the present invention.

Figure 5 is a schematic diagram of a frame according to an embodiment of the present invention.

Figure 6 is a flowchart of a process for pedestrian re-identification according to an embodiment of the present invention.

Figure 7 is a flow chart of a process for traffic monitoring according to an embodiment of the present invention.

Figure 8 is a schematic diagram of a computing device according to an embodiment of the present invention.

Figure 9 is a flowchart of a process according to an embodiment of the present invention.

FIG. 1 is a schematic diagram of a computing device 10 according to an embodiment of the present invention. The computing device 10 includes a preprocessing module (preprocessing module) 100 and a first deep learning module (deep learning module) 110 and the second deep learning module 120. In detail, the preprocessing module 100 receives the first plurality of frames, and determines whether to delete at least one frame of the first plurality of frames according to event detection. According to the decision for the first plurality of frames, the preprocessing module 100 generates a second plurality of frames. That is, the preprocessing module 100 processes (eg, checks) the first plurality of frames to determine whether the first plurality of frames contains at least one event based on event detection. If the frames of the first plurality of frames include at least one event, the preprocessing module 100 does not delete the frames of the first plurality of frames, and generates the frames of the second plurality of frames according to the frames of the first plurality of frames. If the frames of the first plurality of frames do not contain any events, the preprocessing module 100 deletes the frames of the first plurality of frames and does not generate the frames of the second plurality of frames. In other words, the second plurality of frames contains events.

The first deep learning module 110 is coupled to the preprocessing module 100 and receives a second plurality of frames. According to the first plurality of characteristics of the second plurality of frames, the first deep learning module 110 determines whether to delete at least one frame of the second plurality of frames, and according to the decision for the second plurality of frames, generates a third plurality of frames frame. That is, the first deep learning module 110 processes (for example, checks) the second plurality of frames to extract the first plurality of features of the second plurality of frames. According to the first plurality of characteristics, the first deep learning module 110 determines whether the events of the second plurality of frames belong to the first target event. If at least one event of the frames of the second plurality of frames belongs to the first target event, the first deep learning module 110 does not delete the frames of the second plurality of frames, and generates a third plurality of frames based on the frames of the second plurality of frames. Frame of frame. If at least one event of the second plurality of frames does not belong to the first target event, the first deep learning module 110 deletes the second plurality of frames, and does not generate the third plurality of frames. In other words, the frames of the third plurality of frames are the frames of the second plurality of frames including at least one event, and at least one event belongs to the first target event.

The second deep learning module 120 is coupled to the first deep learning module 110 and receives a third plurality of frames. The second deep learning module 120 generates a plurality of prediction outputs of a third plurality of frames (e.g., prediction Test results). That is, the second deep learning module 120 processes the third plurality of frames to extract the second plurality of features of the third plurality of frames. According to the second plurality of characteristics, the second deep learning module 120 determines whether the events of the third plurality of frames belong to the second target event, so as to generate a plurality of prediction outputs. If at least one event of the third plurality of frames belongs to the second target event, the second deep learning module 120 generates a prediction output of a plurality of prediction outputs, wherein the prediction output indicates that at least one event belongs to the second target event. If at least one event of the third plurality of frames does not belong to the second target event, the second deep learning module 120 generates a prediction output of a plurality of prediction outputs, wherein the prediction output indicates that at least one event does not belong to the second target event.

In an embodiment, the first target event and the second target event may be different. For example, when the computing device 10 is used to process the abnormal detection of railway components, the first target event is the perfectly photographed component, and the second target event is the normal state of the component. In an embodiment, the first target event and the second target event may be the same. For example, when the computing device 10 is used to process pedestrian re-identification (person re-identification, person Re-ID), the first target event and the second target event are the same target pedestrian. It should be noted that the computational complexity of the first deep learning module 110 is less than the computational complexity of the second deep learning module 120. Therefore, even if the first target event and the second target event are the same, the determination of the second deep learning module 120 may be more accurate than the determination of the first deep learning module 110.

In an embodiment, the computing device 10 may further include an adaptive buffer. The adaptability buffer is coupled to the second deep learning module 120, and is used to store a plurality of prediction outputs. According to the number of at least one frame containing the event, the size of the adaptability buffer is determined, and at least one frame is included in the first plurality of frames.

FIG. 2 is a schematic diagram of a computing device 20 according to an example of the present invention. The computing device 20 includes a preprocessing module 100, a first deep learning module 110, a second deep learning module 120, a first buffer 202, a second buffer 204, and an adaptability buffer 206. The preprocessing module 100 generates a second plurality of frames and transmits the second plurality of frames to the first buffer 202. The first buffer 202 is coupled to the preprocessing module 100 and the first deep learning module 110 for storing the second plurality of frames. The first deep learning module 110 receives the second plurality of frames from the first buffer 202. The first deep learning module 110 generates a third plurality of frames, and transmits the third plurality of frames to the second buffer 204. The second buffer 204 is coupled to the first deep learning module 110 and the second deep learning module 120 for storing the third plurality of frames. The second deep learning module 120 receives the third plurality of frames from the second buffer 204. The second deep learning module 120 generates a plurality of prediction outputs, and transmits the plurality of prediction outputs to the adaptability buffer 206. The adaptability buffer 206 is coupled to the second deep learning module 120 for storing a plurality of prediction outputs.

It should be noted that, according to the frame rate, the successive (successive) frames in the first plurality of frames may contain the same event, but the second deep learning module 120 may generate different prediction outputs related to the same event (Ideally, the predicted output associated with the same event should be the same). Therefore, the predicted output related to the same event can be averaged to prevent the execution of an incorrect operation (such as a false alarm) based on the incorrect predicted output generated by the second deep learning module 120.

In addition, according to the number of at least one frame containing the event, the size of the adaptability buffer 206 is determined (for example, changed), in which at least one frame is included in the first plurality of frames. For example, the preprocessing module 100 evaluates the first plurality of frames, and calculates the relative speed of the event (that is, the number of at least one frame containing the event). According to the relative speed (ie, the number), the preprocessing module 100 generates a result, and transmits the result to the adaptability buffer 206. If the number of at least one first frame containing a new event is less than the number of at least one second frame containing a current event, the size of the adaptability buffer 206 is reduced. If it contains something new The number of at least one first frame of the file is more than the number of at least one second frame containing the current event, and the size of the adaptability buffer 206 is increased. At least one first frame and at least one second frame are in the first plurality of frames.

In an embodiment, the first buffer 202, the second buffer 204, and the adaptable buffer 206 may be file systems or memory systems.

In one embodiment, when no event is detected, the preprocessing module 100 and the first deep learning module 110 can be suspended, and the second deep learning module 120 can continue to process (for example receive) storage in the first At least one frame of the third plurality of frames in the second buffer 204 is used to generate at least one prediction output of the plurality of prediction outputs.

In one embodiment, the time point when the preprocessing module 100 executes the operation, the time point when the first deep learning module 110 executes the operation, and the time point when the second deep learning module 120 executes the operation may be the same. In other words, the preprocessing module 100, the first deep learning module 110, and the second deep learning module 120 can operate in parallel (ie, simultaneously).

In one embodiment, the first plurality of frames may be generated (eg captured) by a video recorder (eg, camera). In an embodiment, the first plurality of frames may be generated (for example, captured) by different video recorders (for example, different cameras).

In one embodiment, the first plurality of frames are generated for streaming or video (such as video clips). In other words, when the video recorder is running (for example, operating or capturing), the preprocessing module 100 can immediately receive and process the first plurality of frames. In addition, after the video recorder stops running (such as operating or capturing), the preprocessing module 100 can receive and process the first plurality of frames.

In an embodiment, the first plurality of frames may be color images (for example, RGB images). In an embodiment, the first plurality of frames may be grayscale images.

In one embodiment, event detection includes at least one of motion detection and object detection. In one embodiment, motion detection may include scene subtraction. The preprocessing module 100 can use scene subtraction to perform motion detection to determine whether to delete at least one of the first plurality of frames. For example, the preprocessing module 100 subtracts the frames of the first plurality of frames from adjacent frames of the first plurality of frames to generate a matrix of the same size as the frame. The preprocessing module 100 sums all the elements of the matrix to generate a value. Then, if the value is greater than a critical value, the preprocessing module 100 decides not to delete the first plurality of frames. If the value is less than the critical value, the preprocessing module 100 decides to delete the first plurality of frames.

In one embodiment, target detection includes feature extraction. The preprocessing module 100 can use feature extraction to perform target detection to determine whether to delete at least one of the first plurality of frames. Feature extraction may include at least one traditional computer vision method (such as a low-level computer vision method), such as edge detection (such as Hough transform). For example, if according to feature extraction, the preprocessing module 100 determines that the frames of the first plurality of frames include at least one target, the preprocessing module 100 does not delete the first plurality of frames. If based on the feature extraction, the preprocessing module 100 determines that the frames of the first plurality of frames do not contain any objects, and the preprocessing module 100 deletes the first plurality of frames.

In an embodiment, if the preprocessing module 100 determines that the first plurality of frames are not deleted, the preprocessing module 100 processes the first plurality of frames. That is, after performing the decision for the first plurality of frames, the preprocessing module 100 processes the first plurality of frames that are not deleted to generate the second plurality of frames. exist In one embodiment, the operations of processing the frames of the first plurality of frames include noise reduction, downscaling operation, adaptive histogram equalization, and image quality enhancement (image At least one of quality enhancement and cropping operation.

In an embodiment, the first time interval between the preprocessing module 100 receiving the first plurality of frames and generating the second plurality of frames is smaller than the first deep learning module 110 receiving the second plurality of frames and generating the third plurality of frames The second time interval between frames. In an embodiment, the second time interval between the first deep learning module 110 receiving the second plurality of frames and generating the third plurality of frames is less than the second time interval between the second deep learning module 120 receiving the third plurality of frames and generating the third plurality of frames The third time interval between prediction outputs. In other words, the preprocessing module 100 has the fastest operating speed, and the operating speed of the first deep learning module 110 is faster than the operating speed of the second deep learning module 120. Therefore, the first deep learning module 110 and the second deep learning module 120 will not be idle to wait for the input generated by the previous module (that is, the preprocessing module 100 and the first deep learning module 110) (that is, the first deep learning module 110). Two plural frames and a third plural frames).

In one embodiment, the first time interval between receiving the first plurality of frames and generating the second plurality of frames by the preprocessing module 100 is equal to or less than the fourth time interval between consecutive frames of the first plurality of frames ( That is, the capture time interval between two consecutive frames of the first plurality of frames). In one embodiment, the second time interval between the first deep learning module 110 receiving the second plurality of frames and generating the third plurality of frames is equal to or less than the fourth time interval between consecutive frames of the first plurality of frames Interval (ie, the capture time interval between two consecutive frames of the first plurality of frames). That is, the operation speed of the preprocessing module 100 and the first deep learning module 110 is equal to or faster than the frame rate, and the operation of the preprocessing module 100 and the first deep learning module 110 is real-time ).

In an embodiment, the third time interval between the second deep learning module 120 receiving the third plurality of frames and generating the plurality of prediction outputs may be equal to or less than the fourth time interval between consecutive frames of the first plurality of frames Interval (ie, the capture time interval between two consecutive frames of the first plurality of frames). In an embodiment, the third time interval between the second deep learning module 120 receiving the third plurality of frames and generating the plurality of prediction outputs may be greater than the fourth time interval between consecutive frames of the first plurality of frames ( That is, the capture time interval between two consecutive frames of the first plurality of frames). In other words, the operation speed of the second deep learning module 120 may or may not be faster than the frame rate, and the operation of the second deep learning module 120 may or may not be real-time.

In an embodiment, the computational complexity of the preprocessing module 100 is less than the computational complexity of the second deep learning module 120. In an embodiment, the computational complexity of the first deep learning module 110 is less than the computational complexity of the second deep learning module 120. In one embodiment, according to different methods, the preprocessing module 100 and the first deep learning module 110 are used to determine whether to delete the received frames (that is, at least one frame of the first plurality of frames and the second plurality of frames). At least one frame). That is to say, although the computational complexity of the second deep learning module 120 is higher than the computational complexity of the preprocessing module 100 and the first deep learning module 110, the second deep learning module 120 processes (for example, receives) The number of frames (for example, the number of the third plurality of frames) is less than the number of the first plurality of frames and the second plurality of frames. Therefore, the computational complexity of the hardware is reduced.

In an embodiment, the number of the first plurality of frames received by the preprocessing module 100 is higher than the number of the second plurality of frames received by the first deep learning module 110. In an embodiment, the number of the second plurality of frames received by the first deep learning module 110 is higher than the number of the third plurality of frames received by the second deep learning module 120.

In one embodiment, the first deep learning module 110 and the second deep learning module 120 are convolutional neural networks (CNNs). In one embodiment, the convolutional neural network includes at least one of a convolutional layer, a max pooling layer, an activation function layer, and a fully connected layer . In one embodiment, the convolutional neural network further includes at least one identity mapping (for example, an identity mapping shortcut).

In an embodiment, the loss functions (for example, objective functions) of the first deep learning module 110 and the second deep learning module 120 may be the same or different. In an embodiment, the loss function may be a cross-entropy loss function. In an embodiment, the size of the kernel maps of different convolutional layers may be different. In an embodiment, when the first deep learning module 110 and/or the second deep learning module 120 are trained, a dropout method can be used to reduce the impact caused by overfitting. In one embodiment, after each layer of the first deep learning module 110 and the second deep learning module 120, batch normalization may be used.

In an embodiment, when the first deep learning module 110 and the second deep learning module 120 are trained, an adaptive moment estimation optimizer (Adaptive Moment Estimation optimizer, ADAM optimizer) is used to update the first deep learning The parameters of the module 110 and the second deep learning module 120.

In one embodiment, the plurality of prediction outputs are one-hot vectors. In an embodiment, the plurality of prediction outputs are used to indicate whether the events of the third plurality of frames are anomaly. In one embodiment, the plurality of prediction outputs are used to track the events of the third plurality of frames. In an embodiment, the plurality of prediction outputs are used to count the number of events in the third plurality of frames.

In an embodiment, the event of the above-mentioned embodiment may be a target (for example, an object, a railway element, a person or a vehicle), a car accident, or a traffic jam.

According to the present invention, the computing device 10 and/or 20 can be used in different application fields to solve different problems. The following embodiments are used to illustrate the operation of the computing devices 10 and 20.

FIG. 3 is a flowchart 30 of a process for detecting abnormalities of railway components according to an embodiment of the present invention. The flowchart 30 can be used to implement the computing device 10 and/or 20 for the process of detecting abnormalities of railway components, and includes the following steps.

Step 300: Start.

Step 302: A preprocessing module (for example, the preprocessing module 100) receives one frame of the first plurality of frames.

Step 304: According to a motion detection, the preprocessing module determines whether the camera is moving when a camera captures the frame of the first plurality of frames. If not, go to step 306; otherwise, go to step 308.

Step 306: The preprocessing module deletes the first plurality of frames.

Step 308: According to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames.

Step 310: A first deep learning module (such as the first deep learning module 110) determines whether an element of the second plurality of frames is perfectly captured by the camera. If not, go to step 312; otherwise, go to step 314.

Step 312: The first deep learning module deletes the frame of the second plurality of frames.

Step 314: According to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames.

Step 316: A second deep learning module (for example, the second deep learning module 120) generates a prediction output of the third plurality of frames.

Step 318: End.

According to the flowchart 30, a preprocessing module receives a frame of the first plurality of frames, and according to a motion detection, determines whether the camera is moving when a camera captures the frame of the first plurality of frames. When the camera captures the first plurality of frames, if the camera is determined not to be moving, the preprocessing module deletes the first plurality of frames. Otherwise, according to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames. A first deep learning module receives the frame of the second plurality of frames, and determines whether an element (such as a target element) of the frame of the second plurality of frames is perfectly captured by the camera (for example, in the second plurality of frames) In each frame, the component is complete). If the element is determined to be imperfectly captured, the first deep learning module deletes the frame of the second plurality of frames. Otherwise, according to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames. A second deep learning module receives the frame of the third plurality of frames, and generates a prediction output of the frame of the third plurality of frames (for example, according to the state of the element). That is to say, when the computing device 10 and/or 20 is used for detecting abnormality of a railway element, according to the flowchart 30, the first plurality of frames are processed.

In one embodiment, the operation of generating the third plurality of frames by the first deep learning module can be replaced by the operation of transmitting the second plurality of frames to the second deep learning module by the first deep learning module. In other words, if the elements of the second plurality of frames are determined to be perfectly captured, the third plurality of frames is the second plurality of frames.

In one embodiment, with the camera, the first plurality of frames are generated (eg captured). In an embodiment, a camera with a light source can be set on the bottom of a train or on a track (such as a railway). Detection device. On different paths, trains or monitoring devices can have different speeds. The path can be straight or curved. In an embodiment, the components may be fasteners, bolts, etc. installed on the track.

In one embodiment, motion detection includes scene subtraction. The preprocessing module uses scene subtraction to perform motion detection to determine whether the camera is moving when the camera captures the first plurality of frames. For example, the preprocessing module subtracts the frames of the first plurality of frames from adjacent frames of the first plurality of frames to generate a matrix having the same size as the frame. The preprocessing module sums all the elements of the matrix to generate a value. Then, if the value is greater than a critical value, the preprocessing module determines that when the camera captures the first plurality of frames, the camera is moving. If the value is less than the critical value, the preprocessing module determines that when the camera captures the frames of the first plurality of frames, the camera is not moving, and deletes the frames of the first plurality of frames.

In one embodiment, the preprocessing module may perform the following steps to generate the second plurality of frames: according to the track positioning operation, crop the first plurality of frames; process the first plurality of frames; and according to the cropping Operation and processing operations to generate a second plurality of frames.

Figure 4 is a schematic diagram of a frame 400 according to an embodiment of the present invention. The three cases (a) to (c) with the center line 410 are used to illustrate the possible scenarios of the present invention, but are not limited thereto. The frame 400 can be used to implement the first plurality of frames received by the preprocessing module in Figure 3. The frame 400 includes a track 420, and in the three cases, the track 420 can be a part of a track with different curves. The center line 410 is in the middle of the frame 400. In the case (a), the center line 410 and the position of the track 420 are compared, and the track 420 is in the middle of the frame 400. In case (b), compared to the positions of the center line 410 and the track 420, the track 420 has more pixels on the left side of the frame 400 than on the right side of the frame 400. In the case (c), compared with the positions of the center line 410 and the track 420, the track 420 has more pixels on the right side of the frame 400 than on the left side of the frame 400. In other words, because of the shaking of the train, the different speed of the train and the different curve of the track, the track The road is difficult to be in the middle of each of the first plural frames. Therefore, the preprocessing module needs track positioning operation to track the track shift in order to find the trimming coordinates.

In one embodiment, the orbit positioning operation may include binarization, centroid calculation, and moving average. In detail, according to a comparison, the preprocessing module binarizes the frames of the first plurality of frames by using a threshold value (because the track has higher brightness compared to other components), and calculates the first plurality of frames The centroid coordinates of the frame of the frame. Then, the preprocessing module uses a moving average to smooth the centroid coordinates with the previously calculated centroid coordinates of the first plurality of frames. The moving average is used to ensure that the coordinates of the center of mass are correct, and to ensure that the coordinates of the center of mass are not affected by uncertain factors (such as part of the rusty track).

In one embodiment, the operation of the preprocessing module to process the frames of the first plurality of frames may include at least one of noise reduction, downscaling, and adaptability histogram equalization. In one embodiment, the noise reduction can be Gaussian smoothing operation. In an embodiment, the preprocessing module performs a dimensionality reduction operation on the first plurality of frames to reduce computational complexity.

In an embodiment, the first deep learning module processes (eg, checks) the frames of the second plurality of frames to extract at least one first feature of the frames of the second plurality of frames. According to at least one first feature, the first deep learning module determines whether the elements of the second plurality of frames are perfectly captured by the camera.

In one embodiment, the second deep learning module processes (eg, checks) the frames of the third plurality of frames to extract at least one second feature of the third plurality of frames. According to at least one second feature, the second deep learning module determines whether the component is abnormal, and the at least one second feature represents the condition of the component in the third plurality of frames. According to the decision, the second deep learning module generates a prediction output, where the prediction output Indicates whether the component is abnormal. For example, according to whether the angle between the marking on the bolt of the fastener and the track is greater than N degrees, the second deep learning module can determine whether the fastener is abnormal.

Figure 5 is a schematic diagram of a frame 500 according to an embodiment of the present invention. The frame 500 can be used to implement the third plurality of frames received by the second deep learning module in Figure 3. The frame 500 includes a rail 502, a fastener 504, a bolt 506, and a marking 508. The bolt 506 is used to fix the fastener 504 to the rail 502. The marking line 508 is located on the bolt 506. Although different rail systems can be used for different applications, each element in Figure 5 can be used for any basic rail system with different materials and different shapes. Three cases (a) to (c) are used to illustrate the possible scenarios of the present invention, but they are not limited thereto.

In case (a), the angle between the marking line 508 and the track 502 is 0 degrees, and can be regarded as an ideal angle. Therefore, according to the angle, the second deep learning module can determine that the fastener 508 of the frame 500 is normal. In case (b), there is an angle A1 between the marking line 508 and the track 502. According to the angle A1, the second deep learning module can determine whether the fastener 508 of the frame 500 is abnormal. For example, if the angle A1 is greater than N degrees, the second deep learning module may determine that the fastener 508 of the frame 500 is abnormal, and generate a prediction output indicating that the fastener 508 of the frame 500 is abnormal. If the angle A1 is less than N degrees, the second deep learning module can determine that the fastener 508 of the frame 500 is normal, and generate a prediction output used to indicate that the fastener 508 of the frame 500 is normal. In case (c), there is an angle A2 between the marking line 508 and the track 502. According to the angle A2, the second deep learning module can determine whether the fastener 508 of the frame 500 is abnormal. For example, if the angle A2 is greater than N degrees, the second deep learning module may determine that the fastener 508 of the frame 500 is abnormal, and generate a prediction output indicating that the fastener 508 of the frame 500 is abnormal. If the angle A2 is less than N degrees, the second deep learning module can determine that the fastener 508 of the frame 500 is normal, and generate a prediction output used to indicate that the fastener 508 of the frame 500 is normal.

In one embodiment, when the first deep learning module and the second deep learning module are trained At this time, the contents of the training database of the first deep learning module and the second deep learning module are divided into normal data and abnormal data. For example, for the first deep learning module, the normal data is an image that contains a complete fastener, and the abnormal data is an image that contains incomplete fasteners or does not contain any fasteners. For the second deep learning module, the normal data is an image containing a complete fastener, in which the angle between the marking on the bolt of the fastener and the track is less than N degrees. The abnormal data is an image containing a complete fastener, where the angle between the marking line and the track is greater than N degrees. In other words, the training database between the first deep learning module and the second deep learning module is not shared with each other, and the labels (such as labels) of the first deep learning module and the second deep learning module operate differently of.

In one embodiment, the predicted output is a one-hot vector. For example, the prediction output can be a vector [0 1] or a vector [1 0]. The vector [0 1] indicates that the determining element of the second deep learning module is abnormal, and the vector [1 0] indicates that the determining element of the second deep learning module is normal.

In one embodiment, after the second deep learning module generates a prediction output indicating that the component is abnormal, other devices can perform corresponding operations (for example, repairing the component) based on the prediction output and the location of the component.

In one embodiment, the first deep learning module and the second deep learning module are convolutional neural networks. In one embodiment, the convolutional neural network includes at least one of a convolutional layer, a maximum pooling layer, an activation function layer, and a fully connected layer, wherein the activation function layer is a leaky rectified linear unit (ReLU) Function layer. In an embodiment, the sizes of the kernel maps of different convolutional layers may be different. In an embodiment, the loss functions of the first deep learning module and the second deep learning module are cross-entropy loss functions. In one embodiment, the convolutional neural network further includes at least one identity map (for example, identity map shortcut).

FIG. 6 is a flowchart 60 of a process for pedestrian re-identification according to an embodiment of the present invention. The flowchart 60 can be used to implement the computing device 10 and/or 20 for processing pedestrian re-identification, and includes the following steps.

Step 600: Start.

Step 602: A preprocessing module (for example, the preprocessing module 100) receives one frame of the first plurality of frames.

Step 604: According to a target detection, the preprocessing module determines whether the frame of the first plurality of frames contains at least one person. If not, go to step 606; otherwise, go to step 608.

Step 606: The preprocessing module deletes the first plurality of frames.

Step 608: According to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames.

Step 610: A first deep learning module (such as the first deep learning module 110) determines whether at least one first feature of the second plurality of frames matches a plurality of features in a database. If not, go to step 612; otherwise, go to step 614.

Step 612: The first deep learning module deletes the frames of the second plurality of frames, and stores the at least one first feature in the database.

Step 614: According to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames.

Step 616: A second deep learning module (for example, the second deep learning module 120) generates a prediction output of the third plurality of frames.

Step 618: End.

According to the flowchart 60, a preprocessing module receives one frame of the first plurality of frames, and according to A target detection (for example, person detection) determines whether the frame of the first plurality of frames contains at least one person. If the frame of the first plurality of frames is determined to not contain any pedestrians, the preprocessing module deletes the frame of the first plurality of frames. Otherwise, according to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames. A first deep learning module receives the frame of the second plurality of frames, and determines whether at least one first feature of the frame of the second plurality of frames (for example, at least one first feature of at least a group of people) matches A plurality of features in a database. If at least one first feature of the frame of the second plurality of frames is determined to not match the plurality of features in the database, the first deep learning module deletes the frame of the second plurality of frames , And store the at least one first feature in the database. Otherwise, according to the frames of the second plurality of frames, the first deep learning module generates one frame of the third plurality of frames. A second deep learning module receives the frame of the third plurality of frames, and generates a prediction output of the frame of the third plurality of frames. The predictive output indicates whether at least one group of people is a pedestrian in the database (for example, a target pedestrian). That is, when the computing device 10 and/or 20 is used for pedestrian re-identification, according to the flowchart 60, the first plurality of frames are processed.

In one embodiment, the operation of generating the third plurality of frames by the first deep learning module can be replaced by the operation of transmitting the second plurality of frames to the second deep learning module by the first deep learning module. That is, if the first deep learning module determines that at least one first feature matches a plurality of features in the database, the third plurality of frames is the second plurality of frames.

In one embodiment, by using different video recorders (for example, different cameras), the first plurality of frames can be generated (for example, captured).

In one embodiment, target detection includes feature extraction. The preprocessing module can use feature extraction to perform target detection to determine whether the first plurality of frames contains at least one person. feature The extraction includes at least one traditional computer vision method (such as a low-level computer vision method), such as edge detection (such as Hough transform).

In one embodiment, before performing target detection, according to motion detection, the preprocessing module may determine whether to delete the first plurality of frames. Motion detection can include scene subtraction. The preprocessing module can use scene subtraction to perform motion detection to determine whether the frames of the first plurality of frames are the same as their neighboring frames. In other words, the preprocessing module can perform target detection and motion detection to determine whether to delete the first plurality of frames.

In one embodiment, the preprocessing module may perform the following steps to generate the frames of the second plurality of frames: according to the target positioning operation, crop the frames of the first plurality of frames; process the frames of the first plurality of frames; and according to the cropping Operation and processing operations to generate a second plurality of frames.

In one embodiment, the target positioning operation may include feature extraction. The preprocessing module uses feature extraction to locate at least one position of at least one group of people, so as to tailor at least a certain bounding box of the at least one group of people. Feature extraction includes at least one traditional computer vision method (such as a low-level computer vision method), such as edge detection (such as Hough transform).

In one embodiment, the operation of the preprocessing module to process the frames of the first plurality of frames may include at least one of noise reduction, downscaling operation, and image quality enhancement (such as color contrast enhancement). In one embodiment, the noise reduction can be Gaussian smoothing operation. In an embodiment, the preprocessing module performs a dimensionality reduction operation on the first plurality of frames to reduce computational complexity.

In one embodiment, the database contains a plurality of characteristics of a plurality of persons. In one implementation In an example, the first deep learning module processes (eg, checks) the frames of the second plurality of frames to extract at least one first feature of the frames of the second plurality of frames. The first deep learning module determines whether at least one first feature matches a plurality of features in the database.

In an embodiment, the second deep learning module processes (for example, checks) the frames of the third plurality of frames to extract at least one second feature of the third plurality of frames (for example, at least one second feature of at least one group of people). feature). The second deep learning module determines whether at least one second feature matches a plurality of features in the database to generate a prediction output. The predictive output indicates whether at least one group of people is a pedestrian in the database (for example, a target pedestrian). It should be noted that the computational complexity of the first deep learning module is less than the computational complexity of the second deep learning module. Therefore, the at least one second feature extracted by the second deep learning module may be different from the at least one first feature extracted by the first deep learning module. Compared with the decision of the first deep learning module, the decision of the second deep learning module can be more accurate.

In one embodiment, if the prediction output used to indicate that at least one group of people is a pedestrian in the database (for example, a target pedestrian) is generated, the third plurality of frames are marked for storage in the database. According to the new database (for example, a frame containing a third plurality of frames), the parameters of the first deep learning module and the second deep learning module are trained. That is, if the second deep learning module determines that at least one second feature matches a plurality of features, the plurality of features in the database are refined according to the at least one second feature. Therefore, according to the present invention, the first deep learning module and the second deep learning module can be more robust.

In one embodiment, the predicted output is a one-hot vector.

In one embodiment, after the second deep learning module generates a predictive output indicating that at least one group of people is a pedestrian (for example, a target pedestrian) in the database, the predictive output and the video recorder are used according to the predictive output Other devices (or other modules of the computing device 10 and/or 20) can track the trajectory of at least a group of people.

In one embodiment, the first deep learning module and the second deep learning module are convolutional neural networks. In an embodiment, the convolutional neural network includes at least one of a convolutional layer, a maximum pooling layer, an activation function layer, and a fully connected layer. In an embodiment, the sizes of the kernel maps of different convolutional layers may be the same or different. In an embodiment, the loss functions of the first deep learning module and the second deep learning module may be the same or different.

FIG. 7 is a flowchart 70 of a process for traffic monitoring according to an embodiment of the present invention. The flowchart 70 can be used to implement the computing device 10 and/or 20 for processing traffic monitoring, and includes the following steps.

Step 700: Start.

Step 702: A preprocessing module (for example, the preprocessing module 100) receives one frame of the first plurality of frames.

Step 704: According to an event detection, the preprocessing module determines whether the frame of the first plurality of frames contains at least one event. If not, go to step 706; otherwise, go to step 708.

Step 706: The preprocessing module deletes the first plurality of frames.

Step 708: According to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames.

Step 710: A first deep learning module (such as the first deep learning module 110) determines whether at least one first feature of the second plurality of frames matches a plurality of features in a database. If not, go to step 712; otherwise, go to step 714.

Step 712: The first deep learning module deletes the frames of the second plurality of frames, and stores the at least one first feature in the database.

Step 714: According to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames.

Step 716: A second deep learning module (for example, the second deep learning module 120) generates a prediction output of the third plurality of frames.

Step 718: End.

According to the flowchart 70, a preprocessing module receives a frame of the first plurality of frames, and determines whether the frame of the first plurality of frames contains at least one event according to an event detection. If the frame of the first plurality of frames is determined to not contain any event, the preprocessing module deletes the frame of the first plurality of frames. Otherwise, according to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames. A first deep learning module receives the frame of the second plurality of frames, and determines whether at least one first feature of the frame of the second plurality of frames (for example, at least one first feature of at least one event) is consistent with A plurality of features in a database are matched. If the at least one first feature of the frame of the second plurality of frames is determined to not match the plurality of features in the database, the first deep learning module deletes the second plurality of frames Frame, and storing the at least one first feature in the database. Otherwise, according to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames. A second deep learning module receives the frame of the third plurality of frames, and generates a prediction output of the frame of the third plurality of frames. The prediction output indicates whether the at least one event is a target event in the database. That is, when the computing device 10 and/or 20 is used for traffic monitoring, according to the flowchart 70, the first plurality of frames are processed.

In one embodiment, the operation of generating the third plurality of frames by the first deep learning module can be replaced by the operation of transmitting the second plurality of frames to the second deep learning module by the first deep learning module. In other words, if the first deep learning module determines at least one first feature and the complex number in the database The features match, and the frame of the third plurality of frames is the frame of the second plurality of frames.

In one embodiment, with a video recorder (such as a camera), the first plurality of frames can be generated (such as captured). In one embodiment, by using different video recorders (for example, different cameras), the first plurality of frames can be generated (for example, captured).

In one embodiment, the event detection includes target detection and motion detection to determine whether the first plurality of frames includes at least one event.

In one embodiment, target detection includes feature extraction. The preprocessing module can use feature extraction to perform target detection to determine whether the first plurality of frames contains at least one target. Feature extraction includes at least one traditional computer vision method (such as a low-level computer vision method), such as edge detection (such as Hough transform). In one embodiment, motion detection includes scene subtraction. The preprocessing module can use scene subtraction to perform motion detection. In other words, the preprocessing module can perform target detection and motion detection to determine whether to delete the first plurality of frames.

In one embodiment, the preprocessing module may perform the following steps to generate the frames of the second plurality of frames: according to the target positioning operation, crop the frames of the first plurality of frames; process the frames of the first plurality of frames; and according to the cropping Operation and processing operations to generate a second plurality of frames.

In one embodiment, the target positioning operation may include feature extraction. The preprocessing module can use feature extraction to locate at least one position of the at least one target, so as to tailor at least a certain bounding box of the at least one target. Feature extraction may include at least one traditional computer vision method (such as a low-level computer vision method), such as edge detection (such as Hough transform).

In one embodiment, the operation of the preprocessing module to process the frames of the first plurality of frames may include at least one of noise reduction, downscaling operation, and image quality enhancement (such as color contrast enhancement). In one embodiment, the noise reduction can be Gaussian smoothing operation. In an embodiment, the preprocessing module performs a dimensionality reduction operation on the first plurality of frames to reduce computational complexity.

In one embodiment, the database contains a plurality of characteristics of a plurality of events. In an embodiment, the first deep learning module processes (eg, checks) the frames of the second plurality of frames to extract at least one first feature of the frames of the second plurality of frames. The first deep learning module determines whether at least one first feature matches a plurality of features in the database.

In one embodiment, the second deep learning module processes (for example, checks) the frames of the third plurality of frames to extract at least one second feature of the third plurality of frames (for example, at least one second feature of the at least one event) feature). The second deep learning module determines whether at least one second feature matches a plurality of features in the database to generate a prediction output. The predictive output indicates whether at least one event is a target event in the database. It should be noted that the computational complexity of the first deep learning module is less than the computational complexity of the second deep learning module. Therefore, the at least one second feature extracted by the second deep learning module may be different from the at least one first feature extracted by the first deep learning module. Compared with the decision of the first deep learning module, the decision of the second deep learning module can be more accurate.

In one embodiment, if the prediction output used to indicate that at least one event is a target event in the database is generated, the third plurality of frames are marked for storage in the database. According to the new database (for example, a frame containing a third plurality of frames), the parameters of the first deep learning module and the second deep learning module are trained. In other words, if the second deep learning module determines at least one second feature and a plurality of features Matching, according to at least one second feature, a plurality of features in the database are refined. Therefore, according to the present invention, the first deep learning module and the second deep learning module can be more robust.

In an embodiment, the at least one event may be one of a vehicle, a pedestrian, a car accident, or a traffic jam. In one embodiment, the plurality of prediction outputs are one-hot vectors.

In one embodiment, after the second deep learning module generates a prediction output indicating that at least one event is a target event in the database, according to the prediction output and the location of the video recorder, other devices (or computing device 10 And/or other modules of 20) can perform operations related to traffic monitoring. Operations related to traffic monitoring can be counting the number of vehicles, searching for specific vehicles, improving (for example, rescheduling) a traffic plan, or reporting a car accident.

In one embodiment, the first deep learning module and the second deep learning module are convolutional neural networks. In an embodiment, the convolutional neural network includes at least one of a convolutional layer, a maximum pooling layer, an activation function layer, and a fully connected layer. In an embodiment, the sizes of the kernel maps of different convolutional layers may be the same or different. In an embodiment, the loss functions of the first deep learning module and the second deep learning module may be the same or different.

FIG. 8 is a schematic diagram of a computing device 80 according to an embodiment of the present invention. The computing device 80 includes a preprocessing module 100, a first deep learning module 110, a second deep learning module 120, a third deep learning module 802, and a fourth deep learning module 804. In detail, the third deep learning module 802 is coupled to the preprocessing module 100, and receives the second plurality of frames from the preprocessing module 100. The third deep learning module 802 determines whether to delete at least one frame of the second plurality of frames, and generates a fourth plurality of frames according to the decision for the second plurality of frames. The fourth deep learning module 804 is coupled to the third deep learning module 802, and from the The three-deep learning module 802 receives the fourth plurality of frames. The fourth learning depth learning module 804 generates the new plurality of prediction outputs of the fourth plurality of frames. The above-mentioned embodiments for the first deep learning module 110 and the second deep learning module 120 can be correspondingly applied to the third deep learning module 802 and the fourth deep learning module 804, which will not be repeated here. Therefore, the combination of the third deep learning module 802 and the fourth deep learning module 804 and the combination of the first deep learning module 110 and the second deep learning module 120 receive the same frame (ie, the second plurality of frames), To generate predictive output for different applications.

In an embodiment, a plurality of prediction outputs and a new plurality of prediction outputs are applied to solve problems in different application fields. For example, when the computing device 80 is used for traffic monitoring, the plurality of prediction outputs and the new plurality of prediction outputs can be used to calculate the number of vehicles and search for specific vehicles, respectively.

In one embodiment, the computing device 80 may further include a buffer between the modules, where the buffer may be a file system or a memory system.

In one embodiment, the computing device 80 may further include M combinations of two deep learning modules, wherein the M combinations are coupled to the same preprocessing module (for example, the preprocessing module 100), and are connected in parallel with each other . That is, different combinations receive the same frame (for example, a second plurality of frames) to generate prediction output for different applications.

The operation of the communication device 10 in the above-mentioned embodiment can be summarized as a flowchart 90 in FIG. 9. The flowchart 90 can be implemented in the communication device 10 and includes the following steps:

Step 900: Start.

Step 902: A preprocessing module receives one frame of the first plurality of frames.

Step 904: According to an event detection, the preprocessing module determines whether to delete the first plural number Frames of this frame. If yes, go to step 906; otherwise, go to step 908.

Step 906: The preprocessing module deletes the first plurality of frames.

Step 908: According to the frame of the first plurality of frames, the preprocessing module generates a frame of the second plurality of frames.

Step 910: According to at least one feature of the second plurality of frames, a first deep learning module determines whether to delete the second plurality of frames. If yes, go to step 912; otherwise, go to step 914.

Step 912: The first deep learning module deletes the frame of the second plurality of frames.

Step 914: According to the frame of the second plurality of frames, the first deep learning module generates a frame of the third plurality of frames.

Step 916: A second deep learning module generates a prediction output of the third plurality of frames of the frame.

Step 918: End.

The "delete" described in the above operation can be replaced with a "drop" operation. The "determine" described in the above operation can be replaced with "identify", "distinguish", "decide", "confirm" or "discriminate" .

Those with ordinary knowledge in the art can combine, modify, and/or change the above-mentioned embodiments according to the spirit of the present invention, but are not limited thereto. The aforementioned preprocessing modules, deep learning modules, statements, functions, modules, and/or processes (including recommended steps) can be implemented through devices, which can be hardware, software, firmware (for hardware devices and computer commands) Combination with data, and computer commands and data belong to the read-only software on a hardware device, an electronic system, or a combination of the above devices.

Examples of hardware may include analog circuits, digital circuits, and/or hybrid circuits. For example, hardware may include application-specific integrated circuit(s) (ASIC(s)), field programmable gate array(s), FPGA(s) ), programmable logic device(s), coupled hardware components, or a combination of the above devices. In one embodiment, the hardware includes a general-purpose processor (s), a microprocessor (microprocessor (s)), a controller (controller (s)), and a digital signal processor (s). (s), DSP(s)), or a combination of the above devices.

An embodiment of the software may include a collection of program codes, a collection of instructions, and/or a collection of functions, which can be retained (eg stored) in a storage unit, such as a computer-readable medium. Computer readable media can include Subscriber Identity Module (SIM), Read-Only Memory (ROM), flash memory, and random access memory (Random). -Access Memory, RAM), CD-ROM/DVD-ROM/BD-ROM, magnetic tape, hard disk, optical data storage device, non-volatile storage device -volatile storage device), or a combination of the above devices. The computer readable medium (such as a storage unit) may be internally (such as integrated) or externally (such as separate) coupled to the at least one processor. At least one processor including one or more modules can (for example, be configured to) execute software in a computer-readable medium. The collection of program codes, the collection of instructions, and/or the collection of functions can enable at least one processor, module, hardware, and/or electronic system to perform relevant steps.

In summary, the present invention provides a computing device for processing video content analysis (such as railway component abnormality detection, pedestrian re-identification, and traffic monitoring). Modules with low computational complexity Group deletes unimportant frames of video. Then, the module with high computational complexity processes the remaining frames (that is, important frames) of the video. Therefore, the computational complexity of the hardware is reduced.

The foregoing descriptions are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

10: Computing device

100: preprocessing module

110: The first deep learning module

120: The second deep learning module

Claims (18)

A computing device (computing device) for processing video content analysis (video content analysis), including: a preprocessing module (preprocessing module) for receiving the first plurality of frames, and according to an event detection (event detection), determining whether to delete at least one frame of the first plurality of frames, so as to generate a second plurality of frames according to the decision for the first plurality of frames; a first deep learning module (deep learning module) , Coupled to the preprocessing module, used to receive the second plurality of frames, and according to the plurality of characteristics of the second plurality of frames, determine whether to delete at least one frame of the second plurality of frames, according to the use In the determination of the second plurality of frames, a third plurality of frames are generated; and a second deep learning module, coupled to the first deep learning module, is used to receive the third plurality of frames to generate The plurality of prediction outputs of the third plurality of frames. The computing device according to claim 1, further comprising: an adaptive buffer, coupled to the second deep learning module, for storing the plurality of prediction outputs, wherein according to the information including an event A number of at least one frame, a size of the adaptability buffer are determined, and the at least one frame is included in the first plurality of frames. The computing device according to claim 1, wherein the first plurality of frames are generated for a streaming or a video. The computing device according to claim 1, wherein the event detection includes at least one of a motion detection and an object detection. The computing device according to claim 1, wherein if a frame of the first plurality of frames is determined not to be deleted, the preprocessing module processes the frame of the first plurality of frames. The computing device according to claim 5, wherein the operation of processing the frames of the first plurality of frames includes a noise reduction, a downscaling operation, and an adaptability histogram equalization ( At least one of adaptive histogram equalization, image quality enhancement, and cropping operation. The computing device according to claim 1, wherein a first time interval between the preprocessing module receiving the first plurality of frames and generating the second plurality of frames is less than the first deep learning module receiving the first time interval A second time interval between two plural frames and generating the third plural frames. The computing device according to claim 1, wherein a second time interval between the first deep learning module receiving the second plurality of frames and generating the third plurality of frames is less than that received by the second deep learning module A third time interval between the third plurality of frames and the generation of the plurality of prediction outputs. The computing device according to claim 1, wherein a first time interval between receiving the first plurality of frames and generating the second plurality of frames is equal to or less than the continuation of the first plurality of frames (successive) A fourth time interval between frames. The computing device according to claim 1, wherein the first deep learning module receives the first deep learning module A second time interval between two plurality of frames and generating the third plurality of frames is equal to or less than a fourth time interval between consecutive frames of the first plurality of frames. The computing device according to claim 1, wherein a computational complexity of the preprocessing module is less than a computational complexity of the second deep learning module. The computing device according to claim 1, wherein a computational complexity of the first deep learning module is less than a computational complexity of the second deep learning module. The computing device according to claim 1, wherein the first deep learning module and the second deep learning module are convolutional neural networks (CNNs). The computing device according to claim 13, wherein the convolutional neural network includes a convolutional layer, a max pooling layer, an activation function layer, and a fully connected layer At least one of layer). The computing device according to claim 1, wherein the plurality of prediction outputs are one-hot vectors. The computing device according to claim 1, wherein the plurality of prediction outputs are used to indicate whether an event of the third plurality of frames is anomaly. The computing device according to claim 1, wherein the plurality of prediction outputs are used to track an event of the third plurality of frames. The computing device according to claim 1, wherein the plurality of prediction outputs are used to count a number of an event in the third plurality of frames.

Images