JP7452094B2 - Moving object extraction device, moving object extraction method and program - Google Patents

Description

The present invention relates to a moving object extraction device, a moving object extraction method, and a program.

2. Description of the Related Art In recent years, techniques have been known that perform various types of processing based on moving images captured by a camera. For example, a technique for reducing blur in moving images has been disclosed (see, for example, Patent Document 1). Additionally, a technology has been disclosed that extracts the trajectory of a moving object from a moving image and measures the number of moving objects that straddle measurement lines set in the moving image based on the trajectory of the moving object (for example, patented (See Reference 2).

JP2019-53588A JP 2017-33408 Publication

However, it is desired to provide a technique that can suppress a decrease in tracking accuracy of a moving object while suppressing a decrease in detection accuracy of a moving object.

In order to solve the above problem, according to one aspect of the present invention, the detection of a first object in an image frame and the detection of a second object that is a smaller object than the first object are performed using a first amount of calculation. a second model that detects the first object in an image frame with a second amount of calculations smaller than the first amount of calculations, and a video composed of a plurality of image frames. the first model or the second model for each image frame such that the frequency of execution of object detection by the second model is greater than the frequency of execution of object detection by the first model; an image frame distribution unit that distributes the moving image; and a trajectory of the first object in the moving image based on the result of object detection by the first model and the result of object detection by the second model. a first movement trajectory extraction unit that extracts the trajectory of the second object in the video image based on a result of object detection by the first model; and a second movement trajectory extraction unit that extracts the trajectory of the second object in the video image. , a moving object extraction device is provided.

According to another aspect of the present invention, a moving image composed of a plurality of image frames is acquired, and a first object is detected in an image frame by a first amount of calculation, and the first object is detected. a first model that detects a second object that is a smaller object ; and a second model that detects the first object in an image frame using a second amount of calculation that is smaller than the first amount of calculation. The first model or the second model is configured to perform the above-mentioned operation for each image frame so that the frequency of execution of object detection by the second model is greater than or equal to the frequency of execution of object detection by the first model. distributing the moving image; and extracting a trajectory of the first object in the moving image based on a result of object detection by the first model and a result of object detection by the second model. , extracting a trajectory of the second object in the moving image based on a result of object detection by the first model .

According to another aspect of the present invention, the computer is configured to detect a first object in an image frame and detect a second object that is smaller than the first object using a first amount of calculation. a first model; a second model that detects the first object in an image frame using a second calculation amount smaller than the first calculation amount; and a moving image composed of a plurality of image frames. the first model or the second model for each image frame such that the frequency of execution of object detection by the second model is greater than the frequency of execution of object detection by the first model. an image frame distribution unit that distributes images; and extracting a trajectory of the first object in the moving image based on a result of object detection by the first model and a result of object detection by the second model. A moving object comprising: a first movement trajectory extraction section; and a second movement trajectory extraction section that extracts a trajectory of the second object in the video image based on the result of object detection by the first model. A program for functioning as an extraction device is provided.

As described above, according to the present invention, a technique is provided that makes it possible to suppress a decrease in tracking accuracy of a moving object while suppressing a decrease in detection accuracy of a moving object.

FIG. 3 is a diagram showing an example of an image frame captured by a camera according to an embodiment of the present invention. FIG. 1 is a block diagram showing an example of a functional configuration of a moving object extraction system according to an embodiment of the present invention. It is a flow chart which shows an example of operation of a moving object extraction system concerning the same embodiment. FIG. 3 is a diagram showing an example of vehicle detection by a vehicle detection unit. FIG. 6 is a diagram illustrating an example of vehicle detection in an image frame one frame before. FIG. 3 is a diagram illustrating an example of vehicle tracking by a movement trajectory extraction unit. It is a figure which shows the example of the measurement of the number of vehicles which straddled a measurement line. FIG. 3 is a diagram for explaining possible improvements in the moving object extraction system according to the embodiment. FIG. 3 is a diagram for explaining specific functions of an image frame distribution section and a movement trajectory extraction section. FIG. 3 is a diagram for explaining an example of distributed processing. FIG. 2 is a diagram showing a hardware configuration of an information processing device as an example of a moving object extraction device according to the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

Further, in this specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers after the same reference numeral. However, if there is no particular need to distinguish between a plurality of components having substantially the same functional configuration, only the same reference numerals are given. Furthermore, similar components in different embodiments may be distinguished by using different alphabets after the same reference numerals. However, when it is not necessary to particularly distinguish between similar components of different embodiments, only the same reference numerals are given.

(0. Overview)
Embodiments of the present invention relate to a technique for extracting a trajectory of a moving object in a moving image based on a moving image captured by a camera. A moving image is composed of a plurality of image frames that are sequentially captured in time series. Hereinafter, the number of image frames per unit time will be referred to as frame rate. Note that the embodiment of the present invention will be described using a vehicle as an example of a moving object whose trajectory is extracted. However, moving objects are not limited to vehicles. For example, the moving object may be a vehicle other than a vehicle, or an object other than a vehicle (for example, a person).

FIG. 1 is a diagram showing an example of an image frame captured by a camera according to an embodiment of the present invention. Referring to FIG. 1, an image frame G0 constituting a moving image captured by a camera according to an embodiment of the present invention is shown. The camera according to the embodiment of the present invention is provided at a position where it can image a vehicle traveling on a road plane as shown in FIG. The camera is configured to include an image sensor, and images an imaging range using the image sensor. The type of camera is not particularly limited. For example, the camera may be a visible light camera or an infrared light camera.

In the image frame G0, vehicles A1 to A4 and a plurality of other vehicles are shown as examples of vehicles. A line for measuring the number of passing vehicles (ie, measurement line L0) is set in advance in the image frame G0. In the embodiment of the present invention, an example will be mainly described in which the number of vehicle trajectories that straddle the measurement line L0 is measured. However, measuring the number of vehicle trajectories is only one example of the use of vehicle trajectories. That is, the use of the vehicle trajectory extracted in the embodiment of the present invention is not particularly limited.

Furthermore, in the embodiment of the present invention, an example will be mainly described in which the number of vehicle trajectories that straddle the measurement line L0 is measured for each direction in which the vehicle trajectory straddles the measurement line L0. That is, in the embodiment of the present invention, the number of vehicle trajectories that straddle the measurement line L0 in the direction D1 (from one to the other of two regions divided by the measurement line L0) and the measurement line L0 in the direction D2 (the other An example will be mainly explained in which the number of vehicle trajectories that span the vehicle trajectory and the number of vehicle trajectories that straddle the vehicle trajectory and the vehicle trajectory that spans the vehicle trajectory from the vehicle trajectory to the vehicle trajectory that straddles the vehicle trajectory from the vehicle trajectory to the vehicle trajectory that straddles the vehicle trajectory from the vehicle trajectory to the vehicle trajectory that straddles the vehicle trajectory from the vehicle trajectory to the vehicle trajectory that straddles the vehicle trajectory is measured separately. However, the number of vehicle trajectories that straddle the measurement line L0 may be measured regardless of the direction in which the vehicle trajectories straddle the measurement line L0.

The outline of the embodiment of the present invention has been described above.

(1. Details of embodiment)
Next, details of embodiments of the present invention will be described.

(1-1. Example of system functional configuration)
First, an example of the functional configuration of a moving object extraction system according to an embodiment of the present invention will be described with reference to FIG. 2.

FIG. 2 is a block diagram showing an example of the functional configuration of a moving object extraction system according to an embodiment of the present invention. As shown in FIG. 2, the moving object extraction system 1 according to the embodiment of the present invention includes a moving object extraction device 10 and a camera 20. The moving object extraction device 10 and the camera 20 are connected by wire or wirelessly, and each image frame captured by the camera 20 is continuously outputted to the moving object extraction device 10 in chronological order. The moving object extraction device 10 includes a control section 110 and a storage section 130.

The control unit 110 includes a processor, and its functions can be realized by the processor loading a program stored in the storage unit 130 into a RAM (Random Access Memory) and executing it. At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the control unit 110 may be configured by dedicated hardware or a combination of multiple pieces of hardware.

In the embodiment of the present invention, it is assumed that the control unit 110 includes two CPUs (Central Processing Units) and an AI (Artificial Intelligence) chip as an example of a processor. However, the processor included in the control unit 110 is not limited to these. For example, the control unit 110 may include a processor such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array). In the embodiment of the present invention, it is assumed that a plurality of processors having different characteristics are parallelized and distributed processing is performed by the plurality of processors. This allows high performance calculations to be performed.

The control unit 110 includes a vehicle detection unit 111, a movement trajectory extraction unit 115, and a measurement processing unit 116. The vehicle detection unit 111 includes an image frame distribution unit 112, a heavy throughput model M1 (first model), and a light throughput model M2 (second model).

The heavy throughput model M1 performs vehicle detection from image frames input from the image frame distribution unit 112 using a predetermined amount of calculation (first amount of calculation). Furthermore, the light throughput model M2 performs vehicle detection from the image frames input from the image frame distribution unit 112 with a smaller amount of calculation (second amount of calculation) than the amount of calculation of the heavy throughput model M1. The specific calculation amount of each of the heavy throughput model M1 and the light throughput model M2 is not limited. For example, the amount of computation may be the number of computations performed by the model from input to the model to output to the model.

The heavy throughput model M1 and the light throughput model M2 each output a value indicating the probability that a vehicle exists in each of a plurality of rectangular areas that constitute the image frame input from the image frame distribution unit 112 as a vehicle detection score. . As an example, a rectangular area with a vehicle detection score greater than a predetermined score is detected as an area where a vehicle exists (vehicle area).

Here, the size, position, number, etc. of the rectangular areas constituting the image frame are not particularly limited. Moreover, other shaped areas may be used instead of the rectangular area. Furthermore, each of the heavy-throughput model M1 and the light-throughput model M2 may be a trained neural network (for example, a trained deep learning neural network). A trained neural network can be generated by training the neural network using a combination of an image frame in which a vehicle is captured and a vehicle position (that is, a rectangular area where the vehicle is present) in the image frame as training data.

As a specific example of a configuration, each of the heavy-throughput model M1 and the light-throughput model M2 may be a neural network including repetition of convolutional layers and pooling layers and multi-stage full connections. That is, each of the heavy throughput model M1 and the light throughput model M2 may be configured by a CNN (Convolutional Neural Network). However, each of the heavy-throughput model M1 and the light-throughput model M2 may be configured by a neural network that does not have a convolution layer or a pooling layer. Further, the full connections of the heavy throughput model M1 and the light throughput model M2 do not need to be configured in multiple stages.

The storage unit 130 is a storage device that can store programs and data for operating the control unit 110. Further, the storage unit 130 can also temporarily store various data required during the operation of the control unit 110. For example, the storage device may be a non-volatile storage device. For example, the storage unit 130 may store the position of the measurement line L0 as an example of various data.

The functional configuration example of the moving object extraction system 1 according to the embodiment of the present invention has been described above.

(1-2. System operation example)
Next, an example of the operation of the moving object extraction system 1 according to the embodiment of the present invention will be described with reference to FIGS. 3 to 9.

FIG. 3 is a flowchart showing an example of the operation of the moving object extraction system 1 according to the embodiment of the present invention. As shown in FIG. 3, S11 to S16 are executed for each image frame that is input continuously in time series.

(Vehicle detection unit 111)
First, in the moving object extraction device 10, the vehicle detection unit 111 detects a vehicle position (that is, a vehicle region) from image frames that are continuously input in time series from the camera 20 (S11).

FIG. 4 is a diagram showing an example of vehicle detection by the vehicle detection unit 111. Referring to FIG. 4, a current image frame G1 is shown as an example of an image frame (current image frame) immediately after being input to the moving object extraction device 10 from the camera 20. For example, the vehicle detection unit 111 detects a vehicle region P11 from the current image frame G1 as the region where the vehicle A1 exists. Similarly, from the current image frame G1, the vehicle detection unit 111 detects a vehicle region P21 as a region where vehicle A2 exists, a vehicle region P31 as a region where vehicle A3 exists, and a region where vehicle A4 exists. The vehicle area P41 is detected as follows.

(Movement trajectory extraction unit 115)
Returning to FIG. 3, the explanation will be continued. Subsequently, the movement trajectory extraction unit 115 tracks the vehicle (that is, the position of the vehicle area) detected by the vehicle detection unit 111 (S12). Here, it is mainly assumed that the movement trajectory extraction unit 115 tracks the center point of a vehicle area in which the same vehicle is photographed. However, the position of the vehicle area to be tracked may be a point other than the center point of the vehicle area (eg, the upper left corner of the vehicle area, etc.).

The movement trajectory extraction unit 115 extracts a plurality of rectangular areas obtained by the vehicle detection unit 111 from each of the current image frame and an image frame input from the camera 20 in the past than the current image frame (past image frame). Calculate the feature amount extracted from each. Furthermore, when the degree of similarity between the feature amounts of the vehicles is greater than a threshold value, the movement trajectory extraction unit 115 considers the vehicles to be the same vehicle. Then, the movement trajectory extraction unit 115 integrates the vehicle positions that are considered to be the same in the current image frame and the past image frames (S13). For example, the movement trajectory extraction unit 115 associates vehicle positions that are considered to be the same in each of the current image frame and past image frames.

Here, the past image frame may be an image frame that is one frame before the current image frame (hereinafter also simply referred to as "one frame previous image frame"). However, the past image frame may be an image frame that is two or more frames before the current image frame.

FIG. 5 is a diagram showing an example of vehicle detection in an image frame one frame before. Referring to FIG. 5, a current image frame G2 is shown. For example, the vehicle detection unit 111 detects the vehicle region P10 as the region where the vehicle A1 exists from the image frame one frame before. Similarly, the vehicle detection unit 111 detects a vehicle area P20 as an area where vehicle A2 exists, a vehicle area P30 as an area where vehicle A3 exists, and a vehicle area P30 as an area where vehicle A4 exists, from the image frame one frame before. Vehicle area P40 is detected as the area.

Further, FIG. 5 shows an example of a vehicle trajectory extracted up to one frame before the current image frame. For example, a vehicle trajectory T10 is shown as an example of the trajectory of the vehicle A1 extracted up to one frame before the current image frame. Similarly, vehicle trajectory T20 is shown as an example of the trajectory of vehicle A2 extracted up to one frame before the current image frame, and vehicle trajectory T20 is shown as an example of the trajectory of vehicle A2 extracted up to one frame before the current image frame. A vehicle trajectory T30 is shown as an example of the trajectory, and a vehicle trajectory T40 is shown as an example of the trajectory of the vehicle A4 extracted up to one frame before the current image frame.

FIG. 6 is a diagram showing an example of vehicle tracking by the movement trajectory extraction unit 115. In FIG. 6, a current image frame G3 is shown, and in addition to vehicle areas P11 to P41 detected from the current image frame G3, vehicles detected from an image frame one frame before the current image frame are shown. Areas P10 to P40 are shown. Also shown are vehicle trajectories T10 to T40 extracted up to one frame before the current image frame.

For example, the movement trajectory extraction unit 115 extracts the feature amount of the vehicle A1 obtained by the vehicle detection unit 111 from the current image frame, and the feature amount of the vehicle A1 obtained by the vehicle detection unit 111 from the image frame one frame before. Compare. When the similarity between the feature amounts is greater than the threshold, the movement trajectory extraction unit 115 extracts the vehicle position in the current image frame (vehicle region P11) and the vehicle position in the previous image frame (vehicle region P10). ).

As a result, as shown in the current image frame G4, the trajectory of the vehicle A1 changes from the vehicle trajectory T10 extracted up to one frame before the current image frame to the vehicle trajectory extracted up to the current image frame. It is updated to T11.

Note that here, we mainly assume a case where a vehicle is detected from both the current image frame and the image frame one frame before. However, there may be cases where the position of the vehicle is not detected from at least one of the current image frame and the previous image frame. Even in such a case, tracking of the vehicle and integration of vehicle positions may be continued (that is, image frames separated by two or three frames or more may be correlated).

Similarly, the trajectory of the vehicle A2 is updated from the vehicle trajectory T20 extracted up to one frame before the current image frame to the vehicle trajectory T21 extracted up to the current image frame. The trajectory of the vehicle A3 is updated from a vehicle trajectory T30 extracted up to one frame before the current image frame to a vehicle trajectory T31 extracted up to the current image frame. The trajectory of the vehicle A4 is updated from a vehicle trajectory T40 extracted up to one frame before the current image frame to a vehicle trajectory T41 extracted up to the current image frame.

(Measurement processing unit 116)
Returning to FIG. 3, the explanation will be continued. Subsequently, the measurement processing unit 116 measures the speeds of the vehicles that are considered to be the same by the movement trajectory extraction unit 115 (S14). The speed of the vehicle may be measured in any manner. Further, the measurement processing unit 116 detects a stopped vehicle based on the measured speed (S15). For example, the measurement processing unit 116 detects a vehicle whose speed is zero as a stopped vehicle. Next, as described above, the measurement processing unit 116 measures the number of vehicles (that is, the number of vehicle trajectories) that straddle the measurement line L0 (FIG. 7) set in the video input from the camera 20. (S16). For example, the measurement processing unit 116 measures the number of vehicles (that is, the number of vehicle trajectories) that straddle the measurement line L0 (FIG. 7) for each direction that the vehicle trajectories straddle.

FIG. 7 is a diagram showing an example of measuring the number of vehicles that straddle the measurement line. FIG. 7 shows the current image frame G5, and shows vehicle trajectories T11 to T41 extracted up to the current image frame. Referring to FIG. 7, vehicle trajectories T11, T21, and T41 do not straddle measurement line L0. On the other hand, the vehicle trajectory T31 straddles the measurement line L0 in the direction D1. Therefore, the measurement processing unit 116 may add 1 to the number of vehicles that straddle the measurement line L0 in the direction D1.

(Points that can be improved)
Although an example of the operation of the moving object extraction system 1 according to the embodiment of the present invention has been described above, there are points that can be improved in the moving object extraction system 1 described above. Therefore, points that can be improved will be explained below.

FIG. 8 is a diagram for explaining possible improvements in the moving object extraction system 1 according to the embodiment of the present invention. The above describes the process of detecting a vehicle from each image frame. Specifically, the process of detecting a vehicle from each image frame may be performed by the heavy-throughput model M1 or the light-throughput model M2.

Here, since the heavy processing amount model M1 has a larger amount of calculation (than the amount of calculation of the light processing amount model M2), an object (hereinafter also referred to as a "small object") that is smaller than a certain size in the image frame. can also be detected with high precision. However, in the heavy processing amount model M1, since it takes a long time to process one image frame, the number of image frames that can be processed per unit time is relatively small. Therefore, the heavy throughput model M1 has the characteristic that the frame rate of the object trajectory does not improve much.

On the other hand, in the light processing amount model M2, since it takes only a short time to process one image frame, the number of image frames that can be processed per unit time is relatively large. Therefore, in the light processing amount model M2, the frame rate of the object trajectory can be maintained high. However, since the light processing amount model M2 has a smaller amount of calculations (than the amount of calculations of the heavy processing amount model M1), it has the characteristic that the detection accuracy of small objects does not improve much. In the following, an object that is larger than a certain size and appears in the image frame will also be referred to as a "large object."

Below, while explaining how objects are actually detected from image frames by the heavy-throughput model M1 and the light-throughput model M2, we will explain the characteristics of the heavy-throughput model M1 and the light-throughput model M2. , will be explained more specifically.

FIG. 8 is a diagram for explaining the characteristics of the heavy throughput model M1 and the light throughput model M2. Referring to FIG. 8, an image frame G1 similar to image frame G1 shown in FIG. 4 is shown.

Here, objects (for example, vehicles A1 to A4) that exist in an area (near-distance area R1) where the distance to the camera 20 (which captures the image frame G1) is smaller than a certain distance are objects of the same size. is also reflected in the image frame G1 at a size larger than a certain degree. Therefore, an object existing in the short distance region R1 can be detected even by a model with relatively low accuracy. Furthermore, since the distance from the camera 20 to the object captured in the short-distance region R1 is relatively small, the amount of movement per unit time in the image frame G1 becomes large. Therefore, moving images with a relatively high frame rate are required to track an object existing in the short distance region R1.

On the other hand, an object (for example, a vehicle A5) that exists in an area where the distance from the camera 20 is larger than a certain distance (long-distance area R2) has a size smaller than a certain level in the image frame G1 even if the object is the same size. Take a picture with Therefore, objects existing in the long-distance region R2 need to be detected by a relatively accurate model. Furthermore, since the object photographed in the long-distance region R2 has a relatively large distance from the camera 20, the amount of movement per unit time in the image frame G1 becomes small. Therefore, an object existing in the long-distance region R2 can be tracked even from a moving image with a relatively low frame rate.

(Improvement proposal according to this embodiment)
Considering the above, an object existing in the short-distance region R1 may be detected by the light-throughput model M2 with relatively low accuracy, and is detected with a relatively high execution frequency (i.e., a video with a relatively high frame rate). This means that it is desirable to have the sensor detected from the image. On the other hand, objects existing in the long-distance region R2 are preferably detected by the high-accuracy heavy-throughput model M1, and may be detected by a relatively low execution frequency (i.e., a moving image with a relatively low frame rate). (can be detected from the image).

More specifically, the image frame distribution unit 112 acquires a plurality of image frames (moving images) that are continuously input from the camera 20 in chronological order. Then, the image frame distribution unit 112 distributes the heavy-throughput model M1 or the light-throughput model to each image frame so that the frequency of execution of object detection using the light-throughput model M2 is equal to or higher than the frequency of execution of object detection using the heavy-throughput model M1. Distribute moving images to model M2. Furthermore, the movement trajectory extraction unit 115 extracts the trajectory of the object in the moving image based on the result of object detection by the heavy-throughput model M1 and the result of object detection by the light-throughput model M2.

According to this configuration, a large object is detected by the heavy throughput model M1 with relatively high accuracy, so that a decrease in the detection accuracy of a moving object can be suppressed. Furthermore, according to this configuration, a small object is detected by the light-throughput model M2 with a relatively high execution frequency, so that a decrease in tracking accuracy of the trajectory of a moving object can be suppressed.

FIG. 9 is a diagram for explaining specific functions of the image frame distribution section 112 and the movement trajectory extraction section 115. FIG. 9 shows image frames G11 to G16 in order from the oldest to the oldest as an example of a moving image (a plurality of image frames) input from the camera 20 to the image frame distribution unit 112.

When the first image frame G11 is input from the camera 20, the image frame distribution unit 112 outputs the first image frame G11 to the light-throughput model M2, thereby causing the light-throughput model M2 to detect a large object. Execute. Subsequently, when the second image frame G12 is input from the camera 20, the image frame distribution unit 112 outputs the second image frame G12 to the light-throughput model M2, so that the light-throughput model M2 receives the second image frame G12. Execute object detection.

At this time, the movement trajectory extraction unit 115 compares the similarity between the feature amount of the large object obtained from the first image frame G11 and the feature amount of the large object obtained from the second image frame G12 with a threshold value. do. When the degree of similarity between the feature amounts is greater than the threshold, the movement trajectory extraction unit 115 extracts the position of the large object obtained from the first image frame G11 (for example, the position of the vehicle area) and the position of the large object obtained from the second image frame G11. The position of the large object obtained from G12 (for example, the position of the vehicle area) is associated with the position of the large object obtained from G12.

Subsequently, when the third image frame G13 is input from the camera 20, the image frame distribution unit 112 outputs the third image frame G13 to the heavy throughput model M1, thereby adding a small amount to the heavy throughput model M1. Perform object and large object detection.

At this time, the movement trajectory extraction unit 115 compares the similarity between the feature amount of the large object obtained from the second image frame G12 and the feature amount of the large object obtained from the third image frame G13 with the threshold value. do. When the degree of similarity between the feature amounts is greater than the threshold, the movement trajectory extraction unit 115 extracts the position of the large object obtained from the second image frame G12 (for example, the position of the vehicle area) and the position of the large object obtained from the third image frame G12. The position of the large object obtained from G13 (for example, the position of the vehicle area) is associated with the position of the large object obtained from G13.

When the fourth image frame G14, the fifth image frame G15, and the sixth image frame G16 are input from the camera 20, the first image frame G11, the second image frame G12, and the third image frame The same process as that executed when the th image frame G13 is input from the camera 20 is executed again.

However, the movement trajectory extraction unit 115 compares the similarity between the feature amount of the small object obtained from the third image frame G13 and the feature amount of the small object obtained from the sixth image frame G16 with a threshold value. . Then, when the similarity between the feature amounts is greater than the threshold, the movement trajectory extraction unit 115 extracts the position of the small object (for example, the vehicle The position of the small object (for example, the position of the vehicle area) obtained from the sixth image frame G16 is associated with the position of the small object (for example, the position of the vehicle area).

In this way, a heavy throughput model is generated from each of the current image frame (first image frame) and the past image frame (first image frame) input to the heavy throughput model M1 from the image frame distribution unit 112. The feature amount of the small object ( second object) is obtained by M1. Then, when the degree of similarity between the degrees of similarity is greater than a threshold value, the movement trajectory extraction unit 115 performs a process of associating the positions of the small object ( second object) in each of the current image frame and the past image frame. executed. By repeating these steps, the trajectory of the small object is extracted.

Further, the light processing amount model M2 is used from each of the current image frame (second image frame) and the past image frame (second image frame) inputted to the light processing amount model M2 from the image frame distribution unit 112. The feature amount of the large object ( first object) is obtained. Then, when the degree of similarity between the degrees of similarity is greater than a threshold value, the movement trajectory extraction unit 115 performs a process of associating the positions of the large object ( first object) in each of the current image frame and the past image frame. executed. By repeating these steps, the locus of the large object is extracted.

In the example shown in FIG. 9, one repeating unit (predetermined unit) is configured such that the frequency of object detection by the light throughput model M2 is greater than or equal to the frequency of object detection by the heavy throughput model M1. ing. Then, the image frame distribution unit 112 distributes the moving image so that the repetition unit is repeatedly executed. More specifically, in the example shown in FIG. 9, one execution unit performs object detection from two consecutive image frames using the light-throughput model M2, and then performs object detection from one image frame using the heavy-throughput model M1. It is configured to perform object detection.

Here, one repeating unit may be configured in any manner. However, for example, as shown in FIG. 9, if one repeating unit includes only one image frame in which object detection is performed by the heavy-throughput model M1, this repeating unit will be repeatedly executed. Accordingly, object detection by the heavy-throughput model M1 is also performed at constant frame intervals. That is, detection of small objects by the heavy throughput model M1 is performed at regular frame intervals. Note that since detection of a large object is performed by both the heavy-throughput model M1 and the light-throughput model M2, it can be performed at constant frame intervals regardless of the configuration of the repeating unit.

(Distributed processing)
In an embodiment of the present invention, a case is assumed in which a plurality of arithmetic devices having mutually different characteristics are parallelized and distributed processing is performed by the plurality of arithmetic devices. This allows high performance calculations to be performed.

Here, it is assumed that the heavy-throughput model M1 is executed on a first computing device, and the light-throughput model M2 is executed on a second computing device that is physically different from the first computing device. It is desirable that the arithmetic device (first arithmetic device) on which the heavy throughput model M1 is executed has a higher arithmetic capacity than the arithmetic device (second arithmetic device) on which the light throughput model M2 is executed. As a result, each model is executed by a calculation device suitable for the required amount of calculation, so that a decrease in the frame rate of processable moving images can be suppressed. Note that "high computing power" as used herein does not necessarily mean that it is determined only by the speed of computing. For example, even if the calculation speed is device A > device B, if the amount of memory installed in device A is less than the amount that can run heavy processing model M1, heavy processing model M1 cannot operate, so there is insufficient memory for operation. It is also possible to operate with device B, considering that the loaded device B has "higher computing power".

In the following, it is mainly assumed that a processor is used as an example of a calculation device. However, the type of arithmetic device is not particularly limited, as the arithmetic device may be any device used for arithmetic operations. For example, the computing device may be a memory or the like. An example of distributed processing will be described below.

Note that in the following description, the plurality of processors includes a first CPU (hereinafter also referred to as "CPU #1"), a second CPU (hereinafter also referred to as "CPU #2"), and A case in which an AI chip is included will be explained. An AI chip is used as an example of a computing device that has higher computing power than a CPU. However, the types of processors are not limited to these. For example, the multiple processors may include GPUs, FPGAs, and the like.

FIG. 10 is a diagram for explaining an example of distributed processing. As shown in FIG. 10, each of CPU #1, CPU #2, and AI chip has an inference request using the heavy throughput model M1 (that is, a request for object detection using the heavy throughput model M1), or a light An inference request using the throughput model M2 (that is, a request for object detection using the light throughput model M2) is output. The inference request may be output from the image frame distributor 112.

The first inference request r1 (using the light throughput model M2) is output to the CPU #1, which has relatively low computing power. Subsequently, the second inference request r2 (using the light throughput model M2) is output to the CPU #2, which has relatively low computing power. Furthermore, the third inference request (using the heavy throughput model M1) is output to an AI chip with relatively high computing power. When the inference request is distributed to each processor in this way and each processor executes inference based on the inference request, one repetition unit (the first round of the device) is completed.

Inference requests r4 to r6 are similarly distributed to each processor, and when each processor executes inference based on the inference requests, the repetition unit (second device rotation) ends. Inference requests r7 to r12 are similarly distributed to each processor, inference requests following inference request r12 are similarly distributed to each processor, and inference based on the inference request is executed by each processor. As a result, the heavy-throughput model M1 and the light-throughput model M2 are executed by a processor suitable for the required amount of calculations, so that a decrease in the frame rate of processable moving images can be suppressed.

An example of the operation of the moving object extraction system 1 according to the embodiment of the present invention has been described above.

(2. Hardware configuration example)
Next, an example of the hardware configuration of the moving object extraction device 10 according to the embodiment of the present invention will be described. An example of the hardware configuration of the information processing device 900 will be described below as an example of the hardware configuration of the moving object extraction device 10 according to the embodiment of the present invention. Note that the example hardware configuration of the information processing device 900 described below is only an example of the hardware configuration of the moving object extraction device 10. Therefore, in the hardware configuration of the moving object extraction device 10, unnecessary configurations may be deleted from the hardware configuration of the information processing device 900 described below, or new configurations may be added.

FIG. 11 is a diagram showing a hardware configuration of an information processing device 900 as an example of the moving object extraction device 10 according to the embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) #1 (901-1), a CPU #2 (901-2), an AI chip 901-3, a ROM (Read Only Memory) 902, and a RAM (Random). Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

CPU #1 (901-1), CPU #2 (901-2), and AI chip (901-3) function as an arithmetic processing unit and a control device, and control overall operations within the information processing device 900 according to various programs. do. Further, CPU #1 (901-1), CPU #2 (901-2), and AI chip (901-3) may be microprocessors. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus and the like.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. It is composed of etc. By operating the input device 908, a user operating the information processing device 900 can input various data to the information processing device 900 and instruct processing operations.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 is configured with, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network. Further, the communication device 911 may support either wireless communication or wired communication.

The example hardware configuration of the moving object extraction device 10 according to the embodiment of the present invention has been described above.

(3. Summary)
As described above, according to the embodiment of the present invention, there is a heavy processing amount model M1 that detects an object in an image frame using a first amount of calculation, and a second amount of calculation that is smaller than the first amount of calculation. A moving object extraction device 10 is provided, which includes a light processing amount model M2 that detects an object in an image frame. Furthermore, the moving object extraction device 10 includes an image frame distribution section 112 and a movement trajectory extraction section 115.

The image frame distribution unit 112 acquires a moving image composed of a plurality of image frames, and sets the object detection frequency using the light throughput model M2 to be higher than the frequency of object detection using the heavy throughput model M1. The moving image is distributed to the heavy-throughput model M1 or the light-throughput model M2 for each image frame. Then, the movement trajectory extraction unit 115 extracts the trajectory of the object in the moving image based on the result of object detection by the heavy-throughput model M1 and the result of object detection by the light-throughput model M2.

According to this configuration, a large object is detected by the heavy throughput model M1 with relatively high accuracy, so that a decrease in the detection accuracy of a moving object can be suppressed. Furthermore, according to this configuration, a small object is detected by the light-throughput model M2 with a relatively high execution frequency, so that a decrease in tracking accuracy of the trajectory of a moving object can be suppressed.

Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea stated in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

For example, above, the case where the moving object extraction device 10 and the camera 20 are configured as separate bodies has been mainly described. However, the moving object extraction device 10 and the camera 20 may be configured to be integrated.

1 Moving object extraction system 10 Moving object extraction device 110 Control unit 111 Vehicle detection unit 112 Image frame distribution unit 115 Movement trajectory extraction unit 116 Measurement processing unit 130 Storage unit 20 Camera L0 Measurement line M1 Heavy throughput model (first model)
M2 light throughput model (second model)

Claims (9)

a first model that detects a first object in an image frame and a second object that is smaller than the first object using a first amount of calculation;
a second model that detects the first object in an image frame with a second amount of calculation that is smaller than the first amount of calculation;
A moving image composed of a plurality of image frames is acquired, and the first model is detected for each image frame so that the frequency of execution of object detection by the second model is higher than the frequency of execution of object detection by the first model. an image frame distribution unit that distributes the moving image to the model or the second model;
a first movement trajectory extraction unit that extracts a trajectory of the first object in the video image based on a result of object detection by the first model and a result of object detection by the second model;
a second movement trajectory extraction unit that extracts a trajectory of the second object in the video image based on a result of object detection by the first model ;
A moving object extraction device comprising: The image frame distribution unit is configured to distribute the moving image so that a predetermined unit configured such that the frequency of object detection by the second model is equal to or higher than the frequency of object detection by the first model is repeatedly executed. distribute,
The moving object extraction device according to claim 1. the first model is executed on a first computing device;
the second model is executed on a second computing device that is physically different from the first computing device;
the first computing device has higher computing power than the second computing device;
The moving object extraction device according to claim 1 or 2. The movement trajectory extraction unit is configured to extract information obtained by the first model from each of a current first image frame and a past first image frame that are input to the first model from the image frame distribution unit. a process of associating the positions of the second object in each of the current first image frame and the past first image frame when the degree of similarity between the feature amounts of the second object is greater than a threshold; Extracting the trajectory of the second object by repeating
The moving object extraction device according to any one of claims 1 to 3. The movement trajectory extraction unit includes:
The second image frame is extracted from each of the second image frame input to the second model from the image frame distribution unit and a past image frame that is one or more frames before the second image frame. When the degree of similarity between the feature amounts of the first object obtained by the model is greater than a threshold, the positions of the first object in each of the second image frame and the past image frame are The process of associating
The feature amount of the first object obtained by the first model from the current first image frame input to the first model from the image frame distribution unit and the feature amount of the first object from the second image frame of the first object in each of the current first image frame and the second image frame when the similarity with the feature amount of the first object obtained by the model No. 2 is larger than a threshold value. The process of associating positions with each other,
Extracting the trajectory of the first object by repeating
The moving object extraction device according to any one of claims 1 to 4. The moving object extraction device includes a measurement processing unit that measures the number of trajectories of the first object and the second object that straddle a measurement line set in the moving image.
The moving object extraction device according to any one of claims 1 to 5. The measurement processing unit measures the number of trajectories that straddle the measurement line in each direction in which the trajectories straddle the measurement line.
The moving object extraction device according to claim 6. Obtaining a moving image composed of multiple image frames;
a first model that detects a first object in an image frame and a second object that is smaller than the first object using a first amount of calculation; and a first model that is smaller than the first amount of calculation. a second model that detects the first object in an image frame using a second amount of calculation; distributing the moving image to the first model or the second model for each image frame so that
extracting a trajectory of the first object in the moving image based on a result of object detection by the first model and a result of object detection by the second model;
extracting a trajectory of the second object in the video image based on a result of object detection by the first model;
A moving object extraction method, including: computer,
a first model that detects a first object in an image frame and a second object that is smaller than the first object using a first amount of calculation;
a second model that detects the first object in an image frame with a second amount of calculation that is smaller than the first amount of calculation;
A moving image composed of a plurality of image frames is acquired, and the first model is detected for each image frame so that the frequency of execution of object detection by the second model is higher than the frequency of execution of object detection by the first model. an image frame distribution unit that distributes the moving image to the model or the second model;
a first movement trajectory extraction unit that extracts a trajectory of the first object in the video image based on a result of object detection by the first model and a result of object detection by the second model;
a second movement trajectory extraction unit that extracts a trajectory of the second object in the video image based on a result of object detection by the first model ;
A program for functioning as a moving object extraction device.

Images