Disclosure of Invention
The invention provides a video retrieval method and a video retrieval device, which are used for improving the retrieval speed.
The technical scheme of the invention is realized as follows:
a method of video retrieval, the method comprising:
collecting and storing a video sequence;
carrying out motion detection on a video sequence, and calculating and storing a motion description value of each motion fragment when one motion fragment is detected;
when a motion segment is searched in a video sequence, calculating the motion description value of the motion segment to be searched, matching the calculated motion description value with the motion description value of each stored motion segment one by one, and if the calculated motion description value is matched with the motion description value of each stored motion segment, outputting the stored motion segment.
The motion description value is a motion region value or a motion trail value.
When the motion description value is a motion region value,
the calculating of the motion description value of the motion segment includes: determining a motion region description image of the motion segment, dividing the motion region description image into n blocks, wherein n is a positive integer,
for each block, if the block is located in a motion region, the block is valued as "1"; otherwise, taking the block as '0', and arranging the values of the n blocks according to a preset rule to obtain a motion region value of the motion segment; or,
for each block, if the block is located in a motion region, the block is valued as "0"; otherwise, the block is set to be 1, and the values of the n blocks are arranged according to a preset rule to obtain the motion region value of the motion segment.
Said dividing the motion region description image into n blocks further comprises: the n blocks are numbered from 1 to n,
arranging the values of the n blocks according to a preset rule as follows: arranging the values of the n blocks from low order to high order according to the sequence of the numbers from small to large; or,
and arranging the values of the n blocks from high order to low order according to the sequence of the numbers from small to large.
The numbering of n blocks from 1 to n is: numbering n blocks from 1 to n in sequence according to the probability of foreground occurrence of each block from high to low; or, the n blocks are numbered from 1 to n in sequence according to the probability of the foreground appearing in each block from low to high.
The dividing of the motion region description image into n blocks is: the motion region description image is divided into n equal-sized blocks, or,
and dividing the motion region description image into n blocks according to the principle that the area of each block in the region with high foreground occurrence probability is smaller than the area of each block in the region with low foreground occurrence probability.
When the motion description value is a motion trajectory value,
the calculating of the motion trail value of the motion segment comprises: determining a motion area description image of the motion fragment, dividing the motion area description image into n blocks, calculating the gravity center of each foreground in the motion area description image, and connecting the gravity centers of every two adjacent foregrounds to obtain a motion track, wherein n is a positive integer;
for each block, if the block is located on the motion track, the block is valued as "1"; otherwise, taking the block as '0', and arranging the values of the n blocks according to a preset rule to obtain a motion track value of the motion fragment; or,
for each block, if the block is located on the motion track, the block is valued as "0"; otherwise, the block is set to be 1, and the values of the n blocks are arranged according to a preset rule to obtain the motion track value of the motion segment.
The motion description values are represented in a binary representation,
the one-to-one matching of the calculated motion description values with the stored motion description values of the motion segments includes: and performing bitwise AND operation on the calculated motion description values and the stored motion description values of all the motion segments, and determining matching if the obtained result is greater than a preset value.
A video retrieval apparatus, the apparatus comprising:
the acquisition module acquires a video sequence and stores the video sequence into a cache;
the motion description value calculation module is used for carrying out motion detection on the collected video sequence, calculating the motion description value of the motion fragment when one motion fragment is detected, and storing the corresponding relation between the motion description value of the motion fragment and the motion fragment index into a database; receiving a motion description image of a motion segment to be retrieved, which is input by a user, calculating a motion description value corresponding to the motion description image, and sending the motion description value to a retrieval module;
and the retrieval module is used for receiving the motion description values, matching the motion description values with the motion description values of all the motion fragments stored in the database one by one, and reading the motion fragments from the cache according to the matched motion fragment indexes and providing the motion fragments for the user if the motion description values are matched with the motion description values of all the motion fragments stored in the database.
The motion description value calculation module includes:
the motion area determining module is used for carrying out motion detection on the collected video sequence, determining a motion area description image of the video sequence when detecting a motion fragment, and sending the motion area description image to the block dividing module;
the motion region value calculation module is used for calculating the motion region value of the motion region description image according to the motion region description image obtained by the motion region division module;
a motion region value calculation module which, for each block in the received motion region description image, takes the block as "1" if the block is located in the motion region; otherwise, the block is valued as '0'; or, for each block, if the block is located in the motion region, the block is valued as "0"; otherwise, the block is set to be 1, the values of the n blocks are arranged according to a preset rule to obtain a motion region value of the motion segment, and the corresponding relation between the motion region value and the motion segment index is stored in a database or the motion region value is sent to a retrieval module.
The motion description value calculation module includes:
the motion area determining module is used for carrying out motion detection on the collected video sequence, determining a motion area description image of a motion segment when one motion segment is detected, and sending the motion area description image to the block dividing module;
the block division module is used for dividing the received motion area description image into n blocks and sending the motion area description image after block division to the motion track determination module; dividing the received motion trail description image into n blocks, and sending the motion trail description image after block division to a motion trail value calculation module;
the motion track determining module is used for calculating the gravity center of each foreground in the received motion area description image, connecting the gravity centers of every two adjacent foregrounds to obtain a motion track mask image, and sending the motion track mask image to the motion track value calculating module;
a motion track value calculation module, which is used for taking the value of each block in the received motion track mask image or motion track description image as 1 if the block is positioned on the motion track; otherwise, the block is valued as '0'; or, for each block, if the block is located on the motion trajectory, the block is valued as "0"; otherwise, the block is set to be 1, the values of the n blocks are arranged according to a preset rule to obtain a motion track value, and the corresponding relation between the motion track value and the motion fragment index is stored in a database or the motion track value is sent to a retrieval module.
The retrieval module comprises:
the retrieval matching module is used for receiving the motion description values of the motion fragments to be retrieved, which are represented by binary and sent by the motion description value calculation module, respectively carrying out bitwise AND operation on the motion description values and the motion description values of the motion fragments, which are stored in the database and represented by binary, and outputting the AND operation result to the judgment module;
and the judging module is used for receiving the operation result sent by the retrieval matching module, and returning a matching instruction to the retrieval matching module if the operation result is greater than a preset value.
Compared with the prior art, the method has the advantages that the motion description value of the motion fragment is calculated and stored when one motion fragment is detected by carrying out motion detection on the collected video sequence; when a motion segment is searched from a video sequence, calculating the motion description value of the motion segment to be searched, matching the calculated motion description value with the stored motion description value of each motion segment one by one, and if the calculated motion description value is matched with the stored motion description value of each motion segment, outputting the motion segment in the database. The invention digitalizes the motion detection result of the video sequence, can realize the retrieval of the video data only by using the relational database mature in the prior art without applying a complex multimedia database, is simple to realize and improves the retrieval speed.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Fig. 2 is a flowchart of video retrieval according to an embodiment of the present invention, as shown in fig. 2, the specific steps are as follows:
step 201: collecting and storing a video sequence in a scene; at the same time, the video sequence is subjected to motion detection, and each time a motion segment is detected, a motion area description image of the motion segment is determined.
The starting point of a motion segment is the frame where the start of the motion is detected and the ending point is the frame where the motion ends. There may be multiple motion segments in a video sequence, for example: if the video sequence has 1000 frames in total, wherein no motion is detected in 1-500 frames, motion is detected in 501-700 frames, no motion is detected in 701-800 frames, and motion is detected in 801-1000 frames, two motion segments exist in the video sequence: 501-700 frames and 801-1000 frames, wherein each motion segment corresponds to a motion area description image for describing the motion area of the foreground in the motion segment.
In general, the size of the motion region description image is the same as the size of each frame of image in the video sequence.
If no foreground exists in the video sequence or the foreground does not move, no moving area description image exists.
Step 202: according to a preset block division method, each motion area description image is divided into n blocks, and the n blocks are numbered as 1, 2, 1.
The block division method of the motion portrayal image may be different for different scenes.
n is a positive integer, the value of n can be determined empirically, and usually n < image resolution. When n is larger, n is 2mWherein m is a positive integer. The size of the n blocks may be the same, as shown in FIG. 3-1; or different, as shown in fig. 3-2, the area of the block may be larger for the lower probability of foreground occurrence, and the area of the block may be smaller for the higher probability of foreground occurrence, as shown in fig. 3-2, where the blocks 1-16 are the higher probability of foreground occurrence. Adjacent blocks may overlap, as shown in fig. 3-3, with blocks 2, 4, 5 overlapping.
When numbering the blocks, the blocks may be numbered sequentially from left to right and from top to bottom according to their positions on the motion region description image, or according to the importance of the blocks, for example: the blocks in the region with the higher foreground occurrence probability can be numbered adjacently, as shown in fig. 3-2, the blocks 1-16 with the higher foreground occurrence probability are numbered adjacently.
For n blocks on the motion area description image, according to the low to high serial numbers of the n blocks, respectively executing steps 203-206:
step 203: judging whether the block is a motion area, if so, executing step 204; otherwise, step 205 is performed.
Step 204: the value of the block is set to "1", and the process proceeds to step 206.
Step 205: the value of the block is set to "0".
In practical applications, when a block is a motion region, the value of the block may also be set to "0", and, when the block is not a motion region, the value of the block may also be set to "1".
Step 206: judging whether the number of the current block is n, if yes, executing step 208; otherwise, step 207 is performed.
Step 207: and adding 1 to the number of the block and returning to the step 203.
Step 208: and arranging the values of the n blocks from high to low in sequence according to the sequence from low to high of the serial numbers of the n blocks to obtain a binary number of n bits, taking the binary number of the n bits as a motion region value, and storing the corresponding relation between the index of the motion segment and the motion region value in a monitoring database.
The index of the motion segment may be the start time + the end time of the motion segment, or the start time + the duration.
In practical applications, the values of the n blocks may be arranged from lower to higher according to the numbers of the n blocks from lower to higher.
Step 209: when a motion segment meeting a set motion area is searched in a video sequence under a certain scene, the block division method under the scene is determined according to the recorded corresponding relation between the scene identification and the block division method, and the set motion area description image is subjected to block division to obtain the motion area value of the motion segment.
Step 210: and matching the motion region value determined in the step 209 with the motion region value of each motion segment stored in the monitoring database, and if the motion region value is matched with the motion region value of each motion segment stored in the monitoring database, searching the motion segment in the stored video sequence according to the index of the motion segment.
Here, the motion region values determined in step 209 are respectively and-operated with the motion region values of the motion segments stored in the monitoring database by bit, and if the and-operation result of one bit is 1, the match is considered; or, the motion region value determined in step 209 is respectively and-operated with the motion region value of each motion segment stored in the monitoring database according to bit, all the obtained result bits are added, and if the obtained value is greater than the preset value, the match is considered.
In practical applications, in order to avoid a long motion segment, a maximum length Lmax of the motion segment may be preset, and if the detected length L of the motion segment is greater than Lmax, L is divided into a plurality of sections at intervals of Lmax, and a motion region value is calculated for the motion segment in each section.
Fig. 4-1 shows a motion region description image of a motion segment obtained by motion detection of a video sequence, as shown in fig. 4-1, which contains 6 human bodies, respectively representing positions of a person at different times from left to right. Fig. 4-2 shows a motion region mask image obtained by dividing a motion region description image into n blocks and taking a value for each block, as shown in fig. 4-2, a shaded portion in the figure is a motion region, and if n blocks are numbered from top to bottom from left to right as 1, 2,.. and n, respectively, a motion region value is 0x0063e6fe3c380000, where 0x represents a 16-system.
Fig. 2 shows a scheme for retrieving a video sequence according to a motion region value by identifying each motion segment in the video sequence with a motion region value, and in practical applications, each motion segment in the video sequence may also be identified with a motion track value to retrieve the video sequence according to the motion track value.
Fig. 5 is a flowchart of video retrieval according to a second embodiment of the present invention, as shown in fig. 5, the specific steps are as follows:
steps 501 to 502 are the same as steps 201 to 202.
Step 503: and carrying out gravity center calculation on each foreground in each motion area description image to obtain the gravity center of each foreground.
As shown in fig. 4-1, each human body, i.e., the foreground, corresponds to an irregular figure, the center of gravity of the irregular figure is the center of gravity of the human body, and a total of 6 human body centers of gravity can be obtained.
Step 504: and connecting the gravity centers of the adjacent foregrounds to obtain a motion trail mask image.
For example: for 6 human body barycenters obtained according to the figure 4-1, the motion trail of the human body can be obtained by connecting every two adjacent human body barycenters in a straight line, and the figure 4-3 is a motion trail mask image.
For n blocks on the motion track mask image, respectively executing steps 505-508 according to the sequence number from low to high:
step 505: judging whether the block is a motion track area, if so, executing step 506; otherwise, go to step 507.
Step 506: the value of the block is set to "1", and the process proceeds to step 508.
Step 507: the value of the block is set to "0".
In practical applications, when a block is a motion trajectory region, the value of the block may also be set to "0", and, when the block is not a motion trajectory region, the value of the block may also be set to "1".
Step 508: judging whether the number of the current block is n, if yes, executing step 510; otherwise, step 509 is performed.
Step 509: the block number is incremented by 1 and the process returns to step 505.
Step 510: and arranging the values of the n blocks from high to low in sequence according to the sequence from low to high of the serial numbers of the n blocks to obtain an n-bit binary number, taking the n-bit binary number as a motion track value, and storing the corresponding relation between the index of the motion segment and the motion region value in a monitoring database.
The index of the motion segment may be the start time + the end time of the motion segment, or the start time + the duration.
The motion trajectory value as represented in fig. 4-3 is 0xc3663c000000, where 0x represents 16-ary.
In practical applications, the values of the n blocks may be arranged from lower to higher according to the numbers of the n blocks from lower to higher.
Step 511: when a motion segment meeting a set motion track is searched in a video sequence under a certain scene, a block division method under the scene is determined according to the corresponding relation between the recorded scene identification and the block division method, and the set motion track description image is subjected to block division to obtain a motion track value of the motion segment.
Fig. 6-1 is a schematic diagram of block division of a motion trail description image of a motion segment to be retrieved, and fig. 6-2 is a motion trail mask image constructed according to fig. 6-1, and a motion trail value of the motion segment is obtained according to fig. 6-2.
Step 512: and matching the motion track value determined in the step 511 with the motion track value of each motion segment stored in the monitoring database, and if the motion track value is matched with the motion track value of each motion segment stored in the monitoring database, searching the motion segment in the stored video sequence according to the index of the motion segment.
Here, the motion trajectory values determined in step 511 are respectively and-operated with the motion trajectory values of each motion segment stored in the monitoring database according to bits, and if the and-operation result of one of the bits is 1, the match is considered; or, the motion trajectory values determined in step 511 are respectively and-operated with the motion trajectory values of each motion segment stored in the monitoring database according to bits, all the obtained result bits are added, and if the obtained values are greater than the preset values, the matching is considered.
Fig. 7 is a block diagram of a video retrieval apparatus according to an embodiment of the present invention, as shown in fig. 7, which mainly includes: a video capture module 71, a motion description value calculation module 72, and a retrieval module 73, wherein:
the video capture module 71: and acquiring a video sequence under a scene, and storing the video sequence into a cache.
The motion description value calculation module 72: sequentially taking out the video sequence from the cache, carrying out motion detection on the video sequence, calculating a motion description value of the motion segment when detecting one motion segment, and storing the corresponding relation between the motion description value of the motion segment and the motion segment index into a monitoring database; the motion description image of the motion segment to be retrieved, which is input by the user, is received, the motion description value of the motion description image is calculated, and the motion description value is sent to the retrieving module 73.
The retrieval module 73: receiving the motion description value of the motion segment to be retrieved sent by the motion description value calculation module 72, matching the motion description value with the motion description value of each motion segment stored in the monitoring database one by one, and if the motion description value is matched with the motion description value of each motion segment stored in the monitoring database, reading the motion segment from the cache according to the motion segment index on the match and providing the motion segment to the user.
Fig. 8 is a block diagram of the motion description value calculating module 72 according to an embodiment of the present invention, and as shown in fig. 8, the motion description value calculating module 72 may include: a motion region determining module 721, a block dividing module 722, and a motion region value calculating module 723, wherein:
the motion region determination module 721: the video sequence is fetched from the buffer, motion detection is performed on the video sequence, and each time a motion segment is detected, a motion area description image of the motion segment is determined, and the motion area description image and the motion segment index are output to the block division module 722.
The block division module 722: receiving the motion region description image and the motion segment index from the motion region determining module 721, dividing the motion region description image into n blocks, and outputting the divided motion region description image and motion segment index to the motion region value calculating module 723; receiving a motion description image and a scene identification of a motion segment to be retrieved, which are input by a user, determining a block division method corresponding to the input scene identification according to a correspondence between the stored scene identification and the block division method, dividing the input motion region description image into n blocks according to the block division method, and sending the divided motion region description image and a retrieval flag to a motion region value calculation module 723, wherein n is a positive integer.
The motion region value calculation module 723: receiving a motion region description image sent by the block dividing module 722, and taking the value of each block in the motion region description image as "1" if the block is located in the motion region; otherwise, the block is valued as '0'; or, for each block, if the block is located in the motion region, the block is valued as "0"; otherwise, the block is set to be "1", the values of the n blocks are arranged according to a predetermined rule to obtain a motion region value, if the motion segment index sent from the block dividing module 722 is received, the corresponding relationship between the motion region value and the motion segment index is stored in the monitoring database, and if the retrieval flag sent from the block dividing module 722 is received, the motion region value is sent to the retrieval module 73.
Fig. 9 is a block diagram of the motion description value calculation module 72 according to the second embodiment of the present invention, and as shown in fig. 9, the motion description value calculation module 72 may include: a motion region determining module 821, a block dividing module 822, a motion trajectory determining module 823, and a motion trajectory value calculating module 824, wherein:
the motion region determination module 821: the video sequence is fetched from the buffer, motion detection is performed on the video sequence, and each time a motion segment is detected, a motion area description image of the motion segment is determined, and the motion area description image and the motion segment index are output to the block division module 822.
The block division module 822: receiving the motion area description image and the motion segment index from the motion area determination module 821, dividing the motion area description image into n blocks, and sending the divided motion area description image and motion segment index to the motion trajectory determination module 823; receiving a motion description image and a scene identification of a motion segment to be retrieved, which are input by a user, determining a block division method corresponding to the input scene identification according to a correspondence between the stored scene identification and the block division method, dividing the input motion trajectory description image into n blocks according to the block division method, and sending the block-divided motion trajectory description image to a motion trajectory value calculation module 824, or further constructing a motion trajectory mask image according to the block-divided motion trajectory description image, and sending the block-divided motion trajectory mask image to the motion trajectory value calculation module 824, wherein n is a positive integer.
The motion trajectory determination module 823: receiving the motion area description image and the motion fragment index sent by the block dividing module 822, calculating the gravity center of each foreground in the motion area description image, linearly connecting the gravity centers of every two adjacent foregrounds to obtain a motion track mask image, and sending the motion track mask image and the motion fragment index to the motion track value calculating module 824; receiving the motion area description image and the search flag sent from the block dividing module 822, calculating the gravity center of each foreground in the motion area description image, linearly connecting the gravity centers of every two adjacent foregrounds to obtain a motion trajectory mask image, and sending the motion trajectory mask image and the motion fragment trigger to the motion trajectory value calculating module 824.
The motion trajectory value calculation module 824: receiving a motion track mask image sent by the motion track determining module 823 or the block dividing module 822, and taking a value of each block in the motion track mask image as "1" if the block is located on a motion track; otherwise, the block is valued as '0'; or, for each block, if the block is located on the motion trajectory, the block is valued as "0"; otherwise, the block is set to be "1", the values of the n blocks are arranged according to a predetermined rule to obtain a motion trail value, if the motion trail mask image comes from the motion trail determination module 823, the corresponding relationship between the motion trail value and the motion fragment index is stored in the monitoring database, and if the motion trail mask image comes from the block division module 822, the motion trail value is sent to the retrieval module 73.
Fig. 10 is a block diagram of the search module 73 according to an embodiment of the present invention, and as shown in fig. 10, the search module 73 may include: a retrieve matching module 731 and a decision module 732, wherein:
the search matching module 731: receiving the motion description value of the motion segment to be retrieved in binary representation sent from the motion description value calculation module 72, performing bitwise and operation on the motion description value and the motion description value representing each motion segment in binary representation stored in the motion description value calculation module 72, respectively, outputting the result of the and operation to the judgment module 732, and if receiving the matching indication output by the judgment module 732, reading the motion segment from the cache according to the matching motion segment index and providing the motion segment to the user.
The determination module 732: the and operation result sent from the search matching module 731 is received, and if the result is greater than the predetermined value, a matching instruction is returned to the search matching module 731.
The above-described embodiments of the process and method are merely exemplary and not intended to limit the present invention, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.