JP5976237B2 - Video search system and video search method - Google Patents

Description

  The present invention relates to a video search system, and more particularly to a video search system capable of easily displaying a search result in a search.

  A video search system in which images (moving images, still images, intermittent quasi-moving images in which still images are continuous in time series) recorded by a video camera or the like are recorded in a recording device, and a user searches and browses them. There is. Such a video search system is used for various purposes. For example, in a facility visited by an unspecified number of people such as a hotel, a building, a convenience store, or a road, it is used as a monitoring system for the purpose of crime prevention or accident prevention. It's being used.

  In a recording apparatus used as a monitoring system, image data (manual recording) that starts recording at a time point specified by the user while monitoring, or image data that starts or ends recording by a time setting specified by the user (manual recording) Schedule recording), image data recorded on the basis of output information from an object detection device or the like (alarm recording), or image data recorded constantly. Current monitoring systems often have a search function for more easily finding a desired video from a large number of videos recorded in the recording device. This search function narrows down and displays an image that the user wants to browse by using the date and time, the state of the sensor, and the like as search input information.

  Further, as a technique related to the present invention, a face is detected from an accumulated image, and a database is created by extracting image feature quantities of several hundred dimensions from visual information of the face image. A similar face image search system that compares classified databases and key images to be searched and displays them in order of similarity is known (see, for example, Non-Patent Document 1).

Japanese Patent No. 4706535 Japanese Patent No. 4759988 JP 2012-68717 A JP 2013-153304 A

“Similar face image search system”, Hitachi review, Hitachi review company, February 1, 2013, Vol. 95, No. 2, p. 42-43

  In a conventional video search system, videos of a plurality of cameras installed can be recorded and browsed arbitrarily. When checking whether a specific person shot and recorded with one camera is being shot with another camera, other images are taken into account while taking into account the position and date / time of the camera that shot the currently viewed image. The search operation must be repeatedly performed until the presence of the same person is confirmed on the recorded image of the apparatus, which is a very complicated operation.

  For example, when confirming whether a scooter photographed by the camera A at 21:00 is also photographed by another image device, first, the image of the camera B closest to the camera A is confirmed. At that time, since the moving direction and speed of the scooter can be grasped from the image of the camera A, the shift of the passing time of the scooter is estimated from the installation positions of the camera A and the camera B. Here, based on the distance between the two devices and the estimated speed of the scooter in the vicinity of the camera A, it is assumed that the video of the camera B passes at 21:02. However, by taking into account behavior such as a stop at a nearby convenience store before reaching camera B, the search time is specified from 21 o'clock to 21:10 and the video is confirmed with a longer time range.

  If the corresponding scooter does not exist in the recorded video of camera B within this range specification, consider the case where the search time range is further expanded or the moving means is changed from a scooter to walking or a car. Check the details of the face and clothes of the person on the scooter of camera A, or determine that this scooter is not photographed by camera B, and newly check the images of camera C and camera D with different passage routes There are several ways to do this. However, each method is a cumbersome task and is not always easy to find. In this way, it is understood that it is very complicated for the user to continue the work until the scooter is searched without knowing at what timing the scooter is searched.

  Alternatively, there is a method of performing similar face image search using a target person as a key for a plurality of camera videos at once. In order to perform a large-scale high-speed search as in Non-Patent Document 1, it is necessary to detect faces from all videos in advance and create a database of feature values. However, if a person appears backwards or wears a helmet, search omissions occur. Although there is a method of performing a similarity search by creating a database of feature quantities of the entire person image, the search target is not necessarily a pedestrian. It is cumbersome to extract a dedicated feature amount for a car, a motorcycle, etc. and make it into a database.

  The present invention has been made in view of such conventional circumstances, and searches for the same object picked up by a plurality of cameras based on a plurality of object information acquired from images, and displays the corresponding object. Accordingly, an object of the present invention is to provide a video search system in which a user can browse results without performing complicated work.

  In one aspect of the present invention, a video search system includes a video search server that records and collects video of a plurality of cameras and video analysis information thereof, an image captured by the video search server according to video search conditions such as time, and the like. A search terminal that displays video analysis information is provided, and an arbitrary object selected by the user is automatically searched from videos of cameras other than the captured camera and displayed on the search terminal.

  This video search system acquires two or more pieces of feature information and imaging range environment information from an object video selected by the user, and uses the feature information of the object and camera installation information to select the object video selected by the user. Other than the camera (selected video source) that captured the selected object, one or more other cameras that are estimated to have captured the selected object are selected, or the object feature information for the selected camera is selected. In addition, the estimated time of imaging is calculated using the camera installation information.

  Then, the same object as the selected object is searched using the feature information within the estimated range time sequentially from the video of the camera with the highest priority. If the same object is found as a result of the search, the feature information is added and updated with respect to the feature information of the selected object according to the acquired environment information. Further, using the search result, the feature information of the selected object, and the installation information of the camera, a screen showing an image taken by each camera of the selected object and a moving path is automatically generated.

  According to the present invention, in the video search system, the same object picked up by a plurality of cameras is searched for based on a plurality of object information acquired from the image, and related information such as the corresponding object and a moving route is displayed. it can.

1 is a configuration diagram of a video search system according to Embodiment 1 of the present invention. The flowchart which shows an example of the procedure of the process of the said video search system. The flowchart which shows an example of the detailed procedure of a process of the said video search system. An example of the detected object information and tracking object table used with the said video search system. An example of the object management data 113 used in the video search system is shown. An example of the key image used for operation | movement description of the said video search system. The figure which shows an example of the position of the camera in the monitoring area | region, and the position of the object which should be searched. The figure explaining the camera estimation step 204 of the said video search system. The figure which shows an example of the display image of the candidate object display step 301. FIG. The figure which shows an example of the position of the camera in a monitoring area | region, and the position of all the applicable objects. The figure which shows the other example of the display image of an object route image display step. The figure which shows the other example of the display image of an object route image display step. FIG. 10 is a sequence diagram of video search in the video search system according to the second embodiment.

  Embodiments according to the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a hardware configuration of the video search system according to the first embodiment. The video search system of this example includes a plurality of n cameras 1-1 to 1-n, a video search server 2, and a search terminal 3.

  The cameras 1-1 to 1-n image respective monitoring target areas. The cameras 1-1 to 1-n convert the captured optical images into video signals, and input the video signals to the image input I / F 21. The cameras 1-1 to 1-n may be connected to the video search server 2 via an IP network, or may have a built-in microphone to transmit an audio signal to the server 2.

  The video search server 2 includes an image input I / F 21, an image memory 22, a CPU 23, a program memory 24, a work memory 25, a recording device 26, an imaging information table 27, an image output I / F 28, and a data bus 29 for connecting them. Prepare. Further, the recording device 26 includes a plurality of n individual recording areas 112-1 to 112-n and object management data 113. The cameras 1-1 to 1-n are connected to the image input I / F 21 and the search terminal 3 is connected to the image output I / F 28.

  The search terminal 3 receives input of information (hereinafter referred to as search condition data) such as camera ID and imaging date / time from the user, and transmits the search condition data to the video search server. Search results are also displayed. Also, the search result received from the video search server 2 is displayed.

In the video search server 2, the image input I / F 21 converts the input video signal into image data in a format (for example, width 640 pixels, height 480 pixels) handled by the monitoring device 2, and via the data bus 29. Send to image memory 22.
The image memory 22 temporarily stores the image data sent from the image input I / F 21.

The CPU 23 performs video analysis and video search according to the operation program stored in the program memory 24.
In the video analysis, an image stored in the image memory 22 is analyzed using the work memory 25, and information such as an object existing in the imaging field of view of the camera 101 is obtained. Then, information when the object is detected (hereinafter, detected object information) is temporarily stored in the work memory 25. For example, the background difference method based moving object detection similar to that described in the object confirmation step 202 described later is performed, and tracking processing is performed by applying a Kalman filter or the like to the object position. At the beginning of this tracking, combined with Haar-like features and CoHOG (Co-occurrence Histograms of Oriented Gradients) detectors, classifiers of objects using the classifiers learned by adaBoost, separation of overlapping objects, trust Perform sex assessment.
The video search will be described in the object search step 207.

  The work memory 25 holds image data being analyzed and detected object information. This detected object information is information indicating, for example, the detected time, the area where the object exists, and the like.

FIG. 4 shows an example of detected object information and a tracking object table. The detected object information and the tracking object table are provided for each video source (camera).
As the detected object information, the result of the object detection (labeling) process for each frame is held for a predetermined time. The detected time is represented by a frame ID, and the area where the object exists (detected object area) is represented by the coordinates of the upper left and lower right corners of the circumscribed rectangle of the object. In addition, information such as an intra-frame label and an object area (number of pixels) useful for tracking processing can be held.
The tracking object table is data in which objects that are considered to be the same that are continuously detected are aggregated, and for each same object, an appearance frame ID and a leaving frame ID, an intra-frame label, a tracking label, and an object feature. Parts (category, contour shape, speed history, etc.) are retained. The tracking label is an ID uniquely given to an integrated or tracking object. The appearance frame ID and the leaving frame ID correspond to the times when tracking starts and ends, respectively. If tracking is in progress, the leaving frame ID becomes the current frame. The intra-frame label is the intra-frame label of the object in the frame for which tracking has been completed (or the most recent). The type indicates the tracking state or the reason added to the table. Tracking failure (0), tracking in progress (1), tracking by detected coordinates (2), or after verification by object feature (category etc.) (3) Or an object (4) designated from the key image by the user, category determination failure (4 + number of failures), or the like. The tracking object table is additionally written when an object is detected even once, and a type indicating a tracking failure is written when a predetermined number of frames have passed without being detected again. If the consistency of objects becomes uncertain due to the combination or separation of multiple objects during tracking, the records on the tracking object table are divided before and after that and handled as different objects.

In the recording device 26, image data temporarily stored in the image memory 22 and the detected object information temporarily stored in the work memory 25 are written according to a recording method set in advance by the user, and the cameras 1-1 to 1 are written. The recording areas 112-1 to 112-n corresponding to −n are held. Note that, regardless of the presence or absence of detected object information, it is desirable that the video be recorded with information that can specify the shooting time for each frame.
The object management data 113 is characteristic of the present embodiment, and holds feature information (detailed in the object feature collection step 203 in FIG. 2) and route information of the tracked or searched object. FIG. 5 shows an example of the object management data 113.

  The imaging information table 27 holds the installation positions of the cameras 1-1 to 1-n, the angle of view information, the actual distance information of the imaging range, and map information that covers the positions of all the cameras. These pieces of information can be obtained by communicating with the camera 1 in addition to being given manually.

The image output I / F 28 is a network interface for performing socket communication with the search terminal 3, and receives search condition data and transmits search result images and the like.
As a video search, the CPU 23 collates images corresponding to the search condition data in accordance with a search program stored in the program memory 24. If an image exists, the CPU 23 checks the corresponding image data from the recording device 26 and if necessary. The detected object information is read out and transmitted to the search terminal 3 via the image output I / F 28.

FIG. 2 is a flowchart illustrating a procedure for displaying, in the search system of the present example, a result of searching an object existing in the image selected by the user in time series from video data captured by a plurality of cameras.
In the key image setting step 201, the user performs an operation for designating an image on which an object to be automatically searched is displayed on the search screen displayed on the search terminal 3, whereby the main image (hereinafter, key key) in this processing is displayed. Image) is set. FIG. 6 shows an example of the key image. This key image is an image (frame) as it is with the photographing viewing angle of the camera 1.

  The object confirmation step 202 is a step for confirming whether a search target object exists in the key image set in the key image setting step 201. As an example, a foreground object or a moving object can be detected by detecting a difference with a known background image or by performing a difference process between temporally adjacent frames, and a candidate whose detection size is within a predetermined range is set as a search target object candidate. . If there is one candidate for the search target object, it becomes the search target object as it is, and the process proceeds to step 203. If there is no object, a notification is made on the screen to return to a state where the image on which the object to be automatically searched by the user can be specified again, and the key image setting step 201 is performed again. In addition, when a plurality of search target object candidates exist in the key image, the user is allowed to select one search target object by setting a screen state in which any one object is designated.

In the object feature collection step 203, the video search server 2 automatically collects all the features that can be acquired for the search target object set in the object confirmation step 202. Check the images before and after the key image, and use the image where the object exists between the frame when the search target object enters the imaging field of the camera and the frame when the object leaves, for example, the target In addition to “size”, “speed”, “direction of travel (when entering / leaving)”, “category” such as automobile, motorcycle, person, etc., “contour shape”, “existing time”, “sound” In addition, features (hereinafter referred to as object features) such as “detailed features specific to the category” such as the color of the vehicle body, the license plate, the number of headlights, the position of the light source, the color of the clothes, and the face image are extracted. As these features, HOG (Histograms of Oriented Gradients) features or their derivatives (such as spatio-temporal CoHOG) can be used. Details of the extraction will be described later.
It also collects the environmental conditions (weather, time, etc.) of the images used for feature extraction. This is used in the subsequent steps to determine the reliability of the obtained object feature.
The feature information collected in this way is stored in the recording device 26 as object management data 113.

  In the camera estimation step 204, using the object feature collected in the object feature collection step 203 and the imaging information table 27, a camera that is estimated to have captured the search target object other than the current camera is selected. For example, there is a possibility that the object to be searched passes by comparing the “traveling direction (when entering and leaving)” or “speed” of the object feature with the installation location of each camera described in the imaging information table 27. Wash out a camera. Then, the search order for the selected camera is determined. As for the search order, the direction prioritized from the current camera is determined in advance from the road information in the imaging information table 27 and the direction of the camera. In addition, at least two cameras are estimated at a time before the time when the key image was taken (hereinafter referred to as advance time) and a time after that (hereinafter referred to as subsequent time). The user may determine empirically (manually), for example, to search for an object only in a time zone in one direction such as after the fact.

Here, with the five cameras A to E installed in the monitoring area as shown in FIG. 7, the search target object is shown in the video of the camera A, and the left (west) direction as indicated by the arrow. Consider the case where the “traveling direction” information is acquired. Camera A is photographing a certain crossroad, and this “traveling direction” information indicates that the crossroad has traveled straight from east to west. At this time, the number of cameras through which the search target object may pass is estimated to be B, C, and E, for example, according to the theory of a weighted graph.
FIG. 8 shows the relationship between the estimated movement path of the search target object and the estimated camera. In FIG. 8, the solid line arrow indicates the estimated movement path at the posterior time, and the broken line arrow indicates the estimated movement path at the previous time. From FIG. 8, it can be seen that only the camera B is estimated to be a posteriori, and two cameras C and E are estimated to be a priori.

The estimation of the camera may simply be performed by referring to the imaging information table 27 and specifying a predetermined number of cameras within a predetermined distance from the camera A in the closest order.
In the method using the graph, first, the imaging information table 27 is referred to, and one of the one or more roads photographed by the camera A is identified that matches the “traveling direction” information. Next, using the adjacency matrix or adjacency list of the graph included in the map information in the imaging information table 27, all the nodes that start within the predetermined cost starting from the node corresponding to the camera A (or until the predetermined number is reached) are obtained. Search for. This graph is a graph in which a camera installation point and a main intersection are nodes, and a weight corresponding to the time required for passing is given to each side. If the starting node is a crossroad, there are a maximum of four sides. However, the “adjacent node” to be traced first is limited to one by the “traveling direction” information and prior / subsequent designation. The sum of the weights of the edges from the departure node to the arrival node is the cost. This is to search for a natural route that mimics the behavioral psychology of a person who wants to select a route with a short travel time. is there. In the example of FIG. 8, cameras B, C, and E are search results.
In creating such a graph, for example, a numerical map issued by the Geographical Survey Institute can be used as road information. The road centerline defined in the transport facility subpackage of this numerical map includes a spatial attribute that represents the location of the road as a broken line. A directed graph may be created so that different weights are given depending on the traveling direction. It is desirable to simplify the graph by deleting roads (sides) and intersections (nodes) that have little influence on the cost calculation between the target camera installation points.

Next, the search order is determined. In the simplest method, among the estimated three units, the order of the installation distance from the camera A that has obtained the key image is the closest, or if a graph is used, the order of the cost when searching for nodes is the lowest.
In the slightly improved method, the weight according to the angle formed by the “traveling direction” indicated by the arrow and the estimated direction of the location of each camera is set on the basis of the position of the camera A from which the key image is obtained. Of distance and cost. In the example of FIG. 8, a plurality of cameras (cameras C and E) are estimated as the prior time. In this case, the angle formed by the half line extending in the opposite direction (that is, the direction of arrival) to the “traveling direction” information and the half line in the direction of each camera is calculated. If the angle is 0, the angle is 1 and the angle is 180. Gives a weight that decreases monotonously as it gets closer. As a result, by preferentially implementing the camera existing on the right (east) side of the camera A of the key image, it becomes possible to preferentially select a camera that does not feel uncomfortable in the movement path.

In a more improved method, when searching for a node using a graph, a probability is allocated according to a branch at the node. In the example of FIG. 8, when the route from the camera A is traced, a branch to the path to the camera C and the path to the camera E is made. Therefore, as an example, the probabilities of the cameras C and E are allocated to 0.5 and 0.5, respectively. . This distribution may be held for each node as graph information in the imaging information table 27, or may be automatically distributed according to object features such as “traveling direction” information.
Here, it is assumed that it is determined to search for images of each camera in the order of camera B, camera C, and camera E, with both pre- and post-events as search targets.

  Returning to FIG. 2, in the search camera confirmation step 205, it is confirmed whether there are one or more cameras to be searched in the camera estimation step 204. Here, since there are three cameras, cameras B, C, and E, the process proceeds to step 206. When there is no camera to be searched in the camera estimation step 204, the search ends and the process proceeds to step 211 (described later).

  The unsearched camera confirmation step 206 is a step of confirming whether the search target object has been searched using the camera image determined in the camera estimation step 204. Initially, since all the estimated video images of the camera have not been searched, the process proceeds to the object search step 207.

In the object search step 207, an object that is considered to be the same is searched from the objects existing in the estimated camera image by comparing the characteristics with the object to be searched (image search). First, the search is started using the video of the camera B. As shown in FIG. 7, the following description will be made assuming that the key image capturing time is 21:15.
The search target object enters the camera B from the key image capturing time and the object feature “category” and “speed” collected in the object feature collecting step 203 and the camera installation information stored in the image capturing information table 27. Calculate time. As a result of the calculation, it is assumed that the estimated imaging time of the camera B is 21:17. A search is executed using a video that avoids an oversight due to an error by adding a certain width, for example, 5 minutes before and after this time. However, the time before the imaging time 21:15 with the camera A is excluded.

  Therefore, in the camera B, the video for the 7 minutes from 21:15 to 21:22 is set as the video to be searched (hereinafter, section video). In the same way as the object features collected in the object feature collection step 203, features are extracted from the objects present in the section video. Regarding the presence / absence of an object present in the section video, feature extraction is performed only on the frame in which the object exists based on the detection information (detected object region, category, etc.) added at the same time when it is recorded in the recording device 26. Alternatively, object detection may be performed on the entire section video using difference processing or the like, and feature extraction may be performed on the image of the object region obtained as a result. If the past tracking process result is held in the object management data 113, the object can be adopted.

  If the object feature obtained from the object in the currently referenced frame is compared with the object feature of the search target object and a similarity equal to or greater than a preset threshold is obtained, the current object and the search target It is determined that the objects are the same object. In the object feature matching method, a threshold value may be provided for each feature amount, and determination may be made based on a plurality of results, or one total value is calculated from all feature amounts (weak classifiers). It is also possible to prepare an expression to be determined and determine from the obtained total value. In addition, the reliability of the collation result is improved by providing a priority for the object feature according to the environmental state. For example, the time of the key image or the time of the section image is in the night of 21:00 and night, and the illumination intensity within the imaging range of the camera stored in the imaging information table 27 is not more than a certain illuminance. In this case, it is determined that the time is low contrast time, and the priority of collation based on features such as “object color” and “contour shape” that lower the feature extraction accuracy in a low contrast environment is lowered. By increasing the priority of the features related to the specific “headlight”, a matching process is performed so that the accuracy does not fluctuate due to environmental changes. Also, assuming that the objects present in the section video are present in the screen for a certain period of time and at least several frames, if it is determined that the matching results are consistently the same, the object is considered to be the same as the search target object. Hesitate.

Also, in order to perform this step at high speed, if the feature is significantly different from the search target object, for example, if the “category” is different, the feature extraction process for the current object is aborted, and feature extraction of the next existing object is performed. Move on to processing. In this description, it is assumed that even if an object that is determined to be the same is found, the entire section video is searched. However, when the object is found, the object search step 208 is ended and the search target object is displayed to the user. When the user operates the different object, the object search step 207 is restarted to check the unsearched video in the section video. Thus, the processing time may be shortened.
Here, it is assumed that there are two objects determined to be identical to the search target object, and the process proceeds to the next step.

  The corresponding object confirmation step 208 is a step of confirming whether or not the same object as the search target object exists in the object search step 207. In this description, since two objects correspond in the previous step, the process proceeds to the next step. If no camera is found, the process returns to the search camera confirmation step 205. If there is an unsearched camera, the object search is performed again.

The object number confirmation step 209 is a step of confirming whether the corresponding object is one body. If there is only one, go to the next step. Further, since there are two bodies in this description, the process proceeds to “1”, and will be described with reference to the flowchart of FIG.
In candidate object display step 301, as shown in FIG. 9, a notification that a plurality of objects have been searched from the collation result and an image that makes it easy to confirm the details of each object such as the largest image taken in a series of images are provided. It is displayed on the search terminal 3.
In the search object selection step 302, from the screen displayed in the candidate object display step 301, the object of the image selected by the user or the information not applicable is acquired.
In the selected object presence / absence step 303, if any one of the objects is selected from the object information on the screen selected by the user, the process proceeds to “2”, and if not applicable, the process proceeds to “3” (FIG. 2). (Return to the flowchart)
Here, the confirmation process for the user is performed only when there are a plurality of objects determined to be the same as the search target object, but the confirmation process for the user is similarly performed even in the case of integration. Thus, although the number of image confirmations and operations by the user increases, a means for preventing a continuous search based on an erroneous collation determination result may be used.

In the object feature update step 210, the object feature of the object searched in the object search step 207 is added or updated to the object management data 113. If the search target object (corresponding object) is determined to be relevant in the relevant object confirmation step 208, the relevant object and the object of the key image become members of the same object collection. As shown in FIG. 5, the same object collection includes a set of ID, the number of members, route information, and object features and integrated information for the number of members.
At this time, information such as “traveling direction”, “speed”, and “time”, which is generated every time the same object appears on the camera, is included in each “object feature” and held. On the other hand, if “contour shape”, “clothing pattern”, etc. match the currently held feature information, they can be integrated to generate more reliable feature information or Updates that increase the reliability can increase the priority in collation determination.

Furthermore, it is desirable to determine whether to update the feature information according to the environmental state, and not to update the feature information that is assumed to be low in reliability. For example, “contour shape” and “object color” information acquired in a low contrast state has low reliability, and is excluded from the feature information to be updated. By updating information in this manner, highly accurate object collation can be performed with stable object feature information.
After the end of this step, the process returns to the camera estimation step 204.

In order to search for a video from another camera, the camera estimation step 204 is performed again. Here, the video search of the camera B, which is the highest priority camera, has been completed, but since the search was performed at the posterior time with respect to the key image, the camera C that is estimated to be entered at the previous time next is searched. As the camera to be used, the search camera confirmation step 205, the unsearched camera confirmation step 206, and the object search step 207 are performed in the same manner as described above. As a result of performing the object search step 207, the corresponding object is determined to be “none” in this example, and the process returns from the corresponding object confirmation step 208 to the unsearched camera confirmation step 206 again.
This time, a search camera confirmation step 205, an unsearched camera confirmation step 206, and an object search step 207 are performed as a camera to be searched for the camera E estimated to enter at the previous time. Here, if there is one corresponding object, the object information corresponding to the feature information that passes through the corresponding object confirmation step 208 and the object number confirmation step 209 and is stored again in the object management data 113 in the object feature update step 210 is detected. After updating and adding the feature information of the object searched in the search step 207, the process returns to the camera estimation step 204 again.

FIG. 10 shows the standing position of the search target object at each camera collected at this time and the time information thereof.
When all the video searches of the cameras B, C, and E estimated in the initial camera estimation step 204 have been completed, but a sufficient video is not found, the user is assumed to have newly captured the search target object. The unsearched camera and the video section can be selected. For example, a camera is selected using the object feature and imaging information table 27, starting from the camera B for the posterior time image and starting from the camera E for the previous time image.
In this example, as shown in FIG. 7, there is no camera that picks up the previous and subsequent images of each camera in the monitoring area, so that no camera is selected.

In the search camera confirmation step 205, since the camera to be searched is 0, the process proceeds to the object route generation step 211.
In the object route generation step 211, a probable movement path of the search target object is generated using the object feature and the imaging information table 27 stored in the object management data 113. In this example, there are two corresponding objects that have already been searched, and a movement path is generated from each of the various information. If there is no search object at all, a movement path that does not pass through the imaging range of the camera is generated. It is generated using the imaging information table 27.

In the object route image display step 212, as shown in FIG. 12, the search target object searched by the video of each camera and the route through which the search target object has passed are displayed. In the example of FIG. 12, the moving path of the object and the thumbnail image of the search target object with each camera are displayed on the map image. You may make it reproduce | regenerate the image | video through which a search object passes by selecting a thumbnail image. In addition, when there are some candidates predicted at the time of generating the passage route, the prediction route exists by displaying it with a dotted line or displaying the color lightly according to the accuracy.
In this way, since a confirmation is requested from the user when a new similar object is found, the route can be narrowed down interactively.

  In this example, the camera to be searched is automatically estimated. However, in consideration of the case where the moving route is narrowed down in advance, the user can efficiently specify the object to which the search should be prioritized. A search may be performed. In addition, an explanation has been given of performing an arbitrary object search on an existing recorded image. However, if it is estimated in real time that the search target object passes within the range of another camera, the frame rate and image quality can be increased. Means for recording the object image in detail may be added. In addition, for example, when a search object is not found, a forced search that performs an object search within a certain range of time for all the arranged cameras in the order of the installation position from the camera of the key image. A mode or the like may be provided.

In the second embodiment, details of an operation for searching for a video from a compressed encoded video having no object tracking or object feature information in the object search step 207 of the video search system of the first embodiment will be described.
FIG. 13 is a sequence diagram of video search in this example. As basic functional units for performing video search, there are a video decoding unit, an object detection / tracking unit, a categorization / feature amount extraction unit, and a similarity determination unit. Among them, the video decoding unit has the largest processing load. is there. In FIG. 13, the CPU time is emphasized.

The video decoding unit (decoder) decodes video encoded by the H.264 method. This decoder can specify the decoded video quality, and if it is a recorded video of VGA size, it can be decoded at several tens of times the frame rate at the time of shooting by compromising the quality. Also, instead of completely decoding the entire screen, for example, arbitrarily specifying a region obtained by dividing the screen into 16 regions, generating a reconstructed image only in that region, and omitting all motion compensation etc. that refer to the outside of the region You can also. In H.264, an area with no motion is a skip macroblock, and the decoder that has received a predetermined instruction outputs the presence or absence of motion in units of macroblocks for each frame and the entire screen except for the I slice. In addition, an arbitrary number of processors can be specified and executed by a multiprocessor, or can be executed by a GPU.
In this example, the object detection / tracking unit, the categorization / feature amount extraction unit, and the similarity determination unit are executed as one integrated process, and when the processing for one frame is completed, a system call that releases the execution right ( If it is a POSIX system, execute sched_yield). Alternatively, each unit may be fork (prefork) as a child process or a user level thread.
Also, in practice, one or more relational database (RDB) server processes are running to provide object feature registration, search, and the like. Since the decoder and RDB server may wait for I / O, these are assumed to be separate processes and allow the OS to manage CPU time. The decoder, the object detection / tracking unit, the categorization / feature amount extraction unit, and the similarity determination unit may be executed by the search terminal 3.

The operation will be described.
First, the decoder is activated (in response to a pick-up message) in accordance with the designation of the video file or playback start point from the user, and stands by. The reproduction start point corresponds to the head of the video section obtained in the object search step 207 of the first embodiment.
At almost the same time, the object detection / tracking unit is activated in response to a request from the user (in response to a pick-up message), and requests the decoder to play back (or a new decoded frame). Also, periodic monitoring of the shared memory is started by software polling or the like. Also, a log text file is opened, and information such as a video file to be searched for video, a playback start point, and the current date and time is written.
In response to this, the decoder starts video decoding. The designated playback start point can be changed to the immediately preceding IDR frame. The decoded video is written on the shared memory as image data for each frame together with information such as reproduction position (elapsed time from the top of the video).
Next, the object detection / tracking unit creates a background frame. At first, the decoded frames are simply added, and when the number of frames is as good as 4, 8, 16, it is used as a temporary background frame. To obtain a background frame.

  When a background frame is prepared, the object detection / tracking unit starts object detection and object tracking processing by labeling the background difference every time it senses writing of a decoded frame by the decoder. The tracking process is performed on a plurality of objects in parallel. The tracking result is accumulated as a tracking object table (FIG. 4), and when a predetermined condition is satisfied, it is passed to the categorization / feature amount extraction unit together with a partial image of the object (in the case of another process, asynchronously). . The predetermined condition is that the size of the object on the image exceeds a predetermined value, the tracking is finished before the predetermined value is exceeded, and the details will be described later. Tracking objects are distinguished by tracking labels.

  Upon receiving the tracking result, the categorization / feature amount extraction unit scales the size of the tracking object image as necessary. One purpose of scaling is to make the object image size independent of the distance between the camera and the object. Based on the camera parameters of the imaging information table 114, the object distance is estimated from the position on the image. Do. If the object image size is large in spite of this scaling, it can be seen that the image may include a plurality of objects. For other scalings, the image size is always constant.

Thereafter, the discriminator corresponding to the category of the search target object is used to determine the category. For example, if you are looking for a person, you only need to try the person identifier. The appearance of the vehicle varies depending on the type of vehicle, and there is a large difference in how it is projected during the day, at night with lighting, and at night when it is dark (only the headlights etc. are reflected). However, basically, only the classifiers that have successfully identified the search target object need be tried. Discrimination is a process of scanning a place where the value of the discriminator becomes high while changing the position (or scale) in the partial image, and the scan range is initially widened. When the partial image includes the shadow of the object, the scanning range can be narrowed down for images taken in the same time zone with the same camera by learning the identification position (relative position in the partial image). Threshold processing is performed at the position where the value of the discriminator is maximum, and if it is true, it is determined that it matches the category to be searched, and subsequently, a feature quantity for similarity search is extracted. If false, the fact that the category is not applicable and the value of the discriminator before threshold processing are returned to the object detection / tracking unit.
In the feature amount extraction, the object image is cut out again at the position determined by the classifier, and the feature amount specific to the category is extracted. When the category is a person, a face discriminator is further applied to detect a face area, feature amounts are extracted from the face area, and clothes color and the like are extracted from other than the face area (that is, body). If the feature quantity is HOG and the classifier is also HOG-based, the calculation performed by the classifier can be reused. When the extraction is completed, the feature amount is passed to the similarity determination unit together with the frame ID (reproduction position) at the time of extraction.

The similarity determination unit calculates the feature amount of the search target object extracted in advance and the dissimilarity (weighted norm) of the feature amount from the categorization / feature amount extraction unit, and the similarity / dissimilarity determination result Return the dissimilarity value to the categorization / feature amount extraction unit. When they are similar (dissimilarity is less than the threshold), both are determined to be the same object (person), and are notified to the user together with a frame ID, a tracking label, and the like (disappearance message).
In the user side GUI, if the window does not exist, a confirmation screen window as shown in FIG. 9 is newly created. If waiting for the user's confirmation operation, an image to be displayed on the confirmation screen is added.

The categorization / feature amount extraction unit issues an SQL statement to the RDB server according to the received determination result and the degree of dissimilarity. For both the similar / dissimilar determination results, the category determination result and category reliability of the object feature of the object are inserted (inserted), a category sub-table is created, and category-specific feature values are inserted. When the determination results are similar, the presence / absence of a table of the same object collection is checked, and if it is found, it is created, and each data shown in FIG. 4 can be inserted or updated. Here, the integrated feature is obtained by integrating a plurality of object features to improve reliability, and includes reliability (dispersion of individual components of the object features) and the like. The integration information indicates a situation when the corresponding object features are integrated, and the integration reason indicates whether the user has not confirmed the visual confirmation. If registered in the RDB server in this way, similar searches can be performed on the RDB server without directly handling video data thereafter. In particular, the accuracy of the object search can be improved by feeding back the reliability of the integrated information to the weight when the similarity determination unit calculates the dissimilarity. Since only one search target object is handled at a time in this example, the reliability of the integrated information is not updated on the RDB, but is held inside the categorization / feature amount extraction unit and passed to the similarity determination unit. Also good.
The categorization / feature amount extraction unit notifies the received determination result to the object detection / tracking unit.

  When the object detection / tracking unit receives a non-applicable or dissimilar notification, the object detection / tracking unit determines whether it is necessary to perform category re-determination for the object. An upper limit (for example, 4) is set for the number of determinations for one tracked object, and the answer is negative when the upper limit is reached, when the discriminator value (score) is abnormally low, or when the dissimilarity is high. In other cases, for example, when the degree of dissimilarity is slightly higher than the threshold and it is determined as dissimilar, it is necessary. If the object is being tracked with reference to the tracking object table, the process waits until the tracking is completed. When the tracking is completed, the tracking time is referred to. If the tracking time is equal to or less than a predetermined value, the upper limit is changed to 3 or less for the number of determinations accordingly.

  After that, if the current determination is the second time, a partial image of the object is acquired at a position where the tracking period is internally divided by the first ratio, and if it is the third time, the partial image of the object is acquired at the position divided internally by the second ratio. It is passed to the categorization / feature amount extraction unit and re-determination is performed. Here, the ratios of the first and second are, for example, near the center of the tracking period (corresponding to a frame in which the size of the object is the largest), and just before the end of the tracking period (the last size of the object on the image larger than a predetermined value). And is separately learned using an optimization method. If the current determination is the fourth time, the tracking period is determined again at the maximum position estimated by third-order interpolation or the like of the discriminator value of the first to third determinations. The upper limit of the number of determinations is set smaller when a high-speed search is designated by the user.

When the object detection / tracking unit receives a similar notification or senses the end of reproduction (stop) of the decoder, a signal indicating the end of itself or the child process, etc., the object detection / tracking unit terminates the child process as necessary, and logs The information such as the last playback position and the current date is added to the text file for closing, and the process ends. The log file is a record indicating the unsettled state of the video search in the video file. When the video search indicates a midway, the search can be restarted from the interrupted point and the RDB registration of the entire video can be completed.
As described above, in the second embodiment, the similarity / dissimilarity determination result is notified to the user asynchronously, and the video search is continued for the remaining video without waiting for the user's confirmation result. The time for checking can be further shortened.

  It can be used for CCTV (Closed-Circuit Television) for crime prevention or police tracking of the criminal, tracking of people and vehicles, and consumer behavior analysis for marketing.

  Camera: 1-1 to 1-n, Video search server: 2, Image input I / F: 21, CPU: 23, Program memory: 24, Image memory: 22, Work memory: 25, Image output I / F: 28 Search terminal: 109, data bus: 29, recording device: 26, individual recording devices: 112-1 to 112-n, object management data: 113, imaging information table: 27.

Claims (3)

In accordance with a plurality of cameras that capture images of a region to be monitored, a video search server that collects video analysis information in the region while recording the captured images, and a video search condition that includes at least time information, In a video search system having a search terminal for displaying at least one of video captured by the video search server or video analysis information,
The video search server
It holds map information including camera installation information and roads,
Of the plurality of cameras, extracting a plurality of feature information related to the object from the image of the object arbitrarily selected by the search terminal and the position and time when the object was captured,
Using the feature information of the selected object, select one or more cameras whose relevance to the selected object is estimated, and for each camera, estimate using the feature information and camera installation information Calculate the appearance time,
A video having feature information similar to the feature information of the selected object is automatically searched from the video section corresponding to the estimated appearance time of the video of the one or more selected cameras, and the result is Displayed on search terminals ,
The video search server
Record the video or the feature information so that environmental information related to the sunshine or weather when shooting the video including the subject can be acquired later,
As a result of the search, when an object determined to be the same as the selected object is found, the feature information of the selected object is determined according to the determined object feature information and the acquired environment information. The video search system according to claim 1, wherein the video search system is updated or added .   The plurality of pieces of feature information relating to the object are any one of a size, a speed, a traveling direction, a category, a contour shape, a time of existence, and an image feature of the object. Video search system. A video search method for searching for an arbitrarily selected object from an image of a region to be monitored taken by a plurality of cameras.
A video search server that collects video analysis information in the area while recording the captured video, and at least a video captured by the video search server or video analysis information according to a video search condition including at least time information In a video search system having a search terminal for displaying one,
A first process in which the video search server holds map information including preinstalled camera installation information and roads;
A second process in which the video search server collects video analysis information in the area while recording the captured video;
A third step of recording the video or the feature information so that environmental information related to sunlight or weather when the video including the subject is captured can be acquired later;
A fourth step of extracting a plurality of feature information related to the object from the image of the object arbitrarily selected by the search terminal and the position and time when the object was picked up among the plurality of cameras;
The video search server uses the feature information of the selected object to select one or more cameras whose relevance to the selected object is estimated, and for each camera, the feature information and the installation of the camera A fifth step of calculating an estimated appearance time using information;
The video search server automatically searches for video having feature information similar to the feature information of the selected object from the video section corresponding to the estimated appearance time of the video of the selected one or more cameras. A sixth process of displaying the retrieved plurality of videos and video analysis information at once on the search terminal;
When an object determined to be the same as the selected object is found as a result of the search in the sixth process, the feature information of the selected object, the acquired object information, and the feature information of the selected object And a seventh process of updating or adding according to the above.

Images