CN105593850A - Video search device, video search method, and storage medium - Google Patents

Abstract

A video search device that: detects the movement path of at least one moving body in first and second videos and stores the same in a storage device, said first video comprising a plurality of frames captured at a first location and said second video comprising a plurality of frames captured at a second location; extracts an image feature value for each frame of a moving body selected from the at least one moving body detected in the first video, and stores the same in the storage device; selects, from among the extracted image feature values, a query image feature value to be used as a search query on the basis of the movement path of the selected moving body detected in the first video and on the basis of the movement path of the at least one moving body detected in the second video; searches, using the query image feature value, for the image feature value of the at least one moving body extracted from the second video; and outputs the search results.

Description

Video search device, method for retrieving image and storage medium

The application advocates in the Japanese publication Patent of putting down into application on 25 years (2013) December 9The priority of 2013-253897, is combined to its content in the application by reference.

Technical field

The present invention relates to a kind of video search technology.

Background technology

Be accompanied by the universal of security monitoring video camera, to search is wished from the image of taking in many places personage orThe demand of vehicle etc. has improved. But many existing security monitoring video camera systems are by security monitoring video camera, recordThe system that camera and player form, is difficult to the scene that search is wished from accumulated mass data.

To this, pay close attention to the system that has imported similar image retrieval technologies. If use similar image retrieval skillArt can be retrieved the frame of appear before one's eyes out specific personage or object from a large amount of image informations. Similar imageRetrieval refers to the figure of the feature similarity of the image, the outward appearance that obtain the retrieval and inquisition of being specified by user from databaseThe technology of picture. In the time calculating the similar degree of object, between differentiation of objects, (the significantly district from effective coverageTerritory) in extract be called as the numeric data of characteristic quantity and compare. In the time being applied to security monitoring video camera system,Extract characteristic quantity from personage's the marking area such as face or clothes. For example, in patent documentation 1, will be from taking a pictureThe image that machine is obtained is divided into piece, extracts the characteristic quantity of piece, as similar image retrieval according to color histogramInquiry.

On the other hand, known to extracting the successive frame detection dynamic object of self imaging and carrying out between frame dynamicallyThe corresponding technology of object. For example, frame can be divided into zonule, between frame, calculate each zonuleMotion vector. By observing motion vector and gathering the zonule of carrying out same movement, can follow the tracks of dynamicallyObject. Thus, as long as be present in frame, just can follow the tracks of dynamic object, therefore can be same from being included inThe object that in other frames in image, search subscriber is specified.

Patent documentation 1: TOHKEMY 2011-18238 communique

Summary of the invention

The problem that invention will solve

In patent documentation 1, use the inquiry of being specified by user to retrieve, therefore unsuitable in inquiryIn situation, likely cannot obtain the result for retrieval of wishing. In addition, below shown in patent documentation 1Method, search and the similar piece of inquiry from the frame of the front and back of the frame of the image that comprises inquiry, is used themAll in database, carry out similar image retrieval, but in the method, even on same dynamic objectMarking area, the inquiry of specifying in the case of characteristic quantity and user has a great difference, can not be chosen asRetrieval and inquisition, therefore the improvement of result for retrieval is limited.

For the means of dealing with problems

In order to address the above problem, the invention provides a kind of video search device, it possess processor, with upperState the storage device that processor connects, it is characterized in that: from the multiple frame structures by take gained in the first placeThe first image becoming and the second image being made up of multiple frames of taking gained in the second place detect respectivelyThe mobile route of more than one moving body is also stored in above-mentioned storage device, extracts from above-mentioned the first imageThe image spy of each above-mentioned frame of the selected moving body in the above-mentioned more than one moving body detectingThe amount of levying is also stored in above-mentioned storage device, according to above-mentioned selected the moving detecting from above-mentioned the first imageThe movement of the mobile route of kinetoplast and the above-mentioned more than one moving body that detects from above-mentioned the second imagePath, the query image feature using as retrieval and inquisition in the image feature amount of selection said extractedAmount, is used above-mentioned query image characteristic quantity, and retrieval is from above-mentioned more than one the moving of above-mentioned the second Extraction of ImageThe image feature amount of kinetoplast, exports the result of above-mentioned retrieval.

Invention effect

According to video search device of the present invention, from input many images carry out dynamic object tracking andThe detection of marking area, infers for determining to be suitable for each spot for photography according to put aside trace informationThe parameter of inquiry, if user has specified the object of wishing search, from the portable cord of this object automaticallyDecision is suitable for the query image of retrieval and carries out similar image retrieval, can alleviate thus user and select inquiryThe operation of image. In addition, only use the inquiry that is suitable for each spot for photography, inspection therefore can be improvedThe effect of Suo Sudu and reduction retrieval noise. According to the explanation of following embodiment, should be able to be much of thatSeparate above-mentioned problem, structure and effect in addition.

Brief description of the drawings

Figure 1A is the structure chart of the image retrieval system of embodiments of the invention 1.

Figure 1B is the hardware structure diagram of the image retrieval system of embodiments of the invention 1.

Fig. 2 is the structure of image database and the figure of data example that represents embodiments of the invention 1.

Fig. 3 is the figure of the action of the image retrieval system for embodiments of the invention 1 are described.

Fig. 4 is the processing that the video search device of explanation embodiments of the invention 1 is registered the image inputtedFlow chart.

Fig. 5 be the query argument carried out of the video search device of explanation embodiments of the invention 1 infer processingFigure.

Fig. 6 is the image that the video search device of embodiments of the invention 1 uses in order to judge marking areaThe key diagram of the variance ratio of characteristic quantity.

Fig. 7 is the structure of query argument savings portion and the figure of data example that represents embodiments of the invention 1.

Fig. 8 is that the video search device of explanation embodiments of the invention 1 is inferred inquiry according to put aside dataThe flow chart of the processing of parameter.

Fig. 9 is that the inquiry determination section of explanation embodiments of the invention 1 uses trace information to determine retrieval and inquisitionThe figure of action.

Figure 10 is that the video search device of explanation embodiments of the invention 1 determines according to use trace informationRetrieval and inquisition carries out the flow chart of the processing of similar image retrieval.

Figure 11 is the figure of the processing sequential of the image retrieval system of explanation embodiments of the invention 1.

Figure 12 is the object being illustrated in the video search device retrieval image that uses embodiments of the invention 1Time the operation screen that uses the figure of configuration example.

Figure 13 be image retrieval system for embodiments of the invention 2 are described use marking areaThe figure of the correction of trace information.

Figure 14 is that the video search device of explanation embodiments of the invention 2 uses marking area correction to follow the tracks of letterThe flow chart of the processing of breath.

Figure 15 is the depth information of considering of image retrieval system for embodiments of the invention 2 are describedThe figure of the correction of trace information.

Figure 16 is that the video search device of explanation embodiments of the invention 2 uses marking area to trace informationAppend the flow chart of the processing of depth information.

Figure 17 is that the prompting that is present in the inquiry in different images in embodiments of the invention 3 is relevantKey diagram.

Figure 18 is that the video search device of explanation embodiments of the invention 3 is searched for new kind from different imagesThe flow chart of the processing of the marking area of class.

The figure of the video summarization of Figure 19 is use for embodiments of the invention 4 are described trace information.

Figure 20 is the trace information that represented use that the image retrieval system of embodiments of the invention 4 is carried outThe flow chart of the processing of video summarization.

Detailed description of the invention

Embodiment 1

<system architecture>

Figure 1A is the structure chart of the image retrieval system 100 of embodiments of the invention 1.

Image retrieval system 100 is different from this image of the object of the image for user is specifiedTime period (for example from the moment different from comprising the frame of object that user specifies frame) or from different shadowsPicture (for example, from take the image of gained in the place different from the image of the object that comprises user's appointment) retrievalSystem out, the object of this system is: the tracking letter that uses the dynamic object (moving body) in imageBreath, generates this to each spot for photography of the image of retrieving by the query image that is suitable for retrieval and looks intoAsk image, improve thus speed and the precision of retrieval.

Image retrieval system 100 possesses image storage device 101, input unit 102, display unit 103And video search device 104.

Image storage device 101 is storage mediums of preserving image data, can use the hard of built-in computerDisk drive or connect by networks such as NAS (network attached storage) or SAN (storage area network)Storage system and form. In addition, image storage device 101 can be also for example temporarily to preserve from cameraThe buffer storage of the image data of input constantly.

In addition, be kept at the image data in image storage device 101, as long as can be captured in trackingWhen dynamic object, utilizing, can be the data of arbitrary form. For example, the image data of preservation can be bothTaking the motion image data of gained with video camera, can be also to clap with the interval of being scheduled to by still cameraTake the photograph a succession of Still image data of gained.

Input unit 102 is that mouse, keyboard, touch-screen equipment etc. are for passing on and use to video search device 104The input interface of the operation at family. Display unit 103 is the output interfaces such as liquid crystal display, for show imageThe result for retrieval of indexing unit 104, use with user's conversational operation etc.

Video search device 104 is followed the tracks of dynamic object and detects significantly from each frame of given image dataRegion savings. If having specified, user wishes from the object of the frame search of savings, video search device104 use trace information to select to be suitable for using query image to retrieve in its a series of frame in front and backThe query image of each spot for photography of image, carry out similar image retrieval. Video search device 104Handled image, having imagined is the image that the ocean weather station observation of gained is taken in place more than position at.In addition, the object of searching object is the dynamic object arbitrarily such as personage or vehicle. Video search device 104Possess image input part 105, frame register 106, dynamic object tracking portion 107, trace information register108, marking area test section 109, marking area register 110, image database 111, query argumentInfer portion 112, query argument savings portion 113, inquiry input part 114, inquiry determination section 115 and similarImage retrieval portion 116.

Image input part 105 is read image data from image storage device 101, is transformed at video search and fillsPut the 104 inner data modes that use. Specifically, image input part 105 carries out animation decoding to be processed,Be decomposed into frame (Still image data form) by image (motion image data form). By obtainedFrame sends to frame register 106, dynamic object tracking portion 107 and marking area test section 109.

The information of the image of the frame extracting and extraction source is written to image database by frame register 106111. As the explanation of Fig. 2, will be explained below the data that are recorded in image database 111 in detail.

The dynamic object that dynamic object tracking portion 107 is detected in image, carries out right with the dynamic object of front frameShould, carry out thus the tracking of dynamic object. For example can use S.BakerandI.Matthews"Lucas-kanade20yearson:Aunifyingframework",InternationalJournalofComputerVision, vol.53, no.3, the method arbitrarily such as 2004 methods of recording, realizes goerThe detection and tracking of body. The trace information of the dynamic object obtaining by the coordinate information of the dynamic object of each frame andThe ID (following the tracks of ID) that gives uniquely each tracking forms.

Trace information register 108 is from the dynamic object of each frame of obtaining by dynamic object tracking portion 107Extracted region images characteristic quantity, registers in image database 111. Image feature amount is for example to use fixed lengthThe quantize data of gained of the information of the outward appearances such as the vector performance of degree the CF to image. SeparatelyOutward, trace information register 108 is extracted mobile according to the coordinate of the dynamic object of having given identical tracking IDThe characteristic quantity of line (i.e. the mobile route of this dynamic object), registers in image database 111.

Marking area test section 109 detects significant region from frame, obtains its coordinate. Marking area basisApplication program and difference, but for example if the image that comprises personage is face region, head zone, clothesThe pattern of look, clothes or having, if the image that comprises vehicle is wheel or front grid etc.Marking area test section 109 comprises the multiple detections for extracting the marking area corresponding with the kind of objectModule, in the case of the kind of the object that occurs in cannot being limited to image, also can make multiple detection mouldsPiece is action concurrently simultaneously.

Marking area register 110 is extracted image feature amount from the each marking area detecting, with detection resourcesTogether with the coordinate information of frame information and this marking area, register in image database 111. Image feature amountExtracting method can change accordingly with the kind of marking area, but for the marking area of identical type, mustMust extract image feature amount by same method. For example, for face region, can use shape facility amount,Can use Color Characteristic for clothes region, but for the face region A detecting in different frames,B, can not use Color Characteristic and B is used to shape facility amount A.

Image database 111 is for preserving image, frame, trace information, dynamic object and marking areaThe database of information. For the project that has been endowed image feature amount, can carry out similar image retrieval.Similar image retrieval is according to image feature amount and the merit of inquiry from closely also exporting to order array data far awayEnergy. In the time of the comparison of image feature amount, for example, can use the Euclidean distance between vector. Stepping on from frameThe registration process of note portion 106, trace information register 108 and marking area register 110, from looking intoAsk that reading of parameter estimation portion 112 and inquiry determination section 115 processed and from similar image retrieval portionWhen 116 retrieval process, there is the access to image database 111. For the structure of image database 111Make, as the explanation of Fig. 2, will describe in detail in the back.

Query argument is inferred portion 112 and is used trace information and the marking area of savings in image database 111Information, infer the parameter of the kind of the inquiry for determining the each spot for photography that is suitable for image. To push awayThe parameter of making is kept in query argument savings portion 113.

Query argument savings portion 113 preserves inquiry for determining the each spot for photography that is suitable for imageThe parameter of kind. For the structure of query argument savings portion 113, as the explanation of Fig. 7, will be detailed in the backDescribe in detail bright.

In the time that user specifies the object of wishing to search for from the image of savings image database 111, inquiryInput part 114 transmits the user's who provides by input unit 102 operation to video search device 104.

Inquire about object and the trace information thereof of determination section 115 users' appointments and put aside from query argumentThe parameter that portion 113 reads, determines optimum more than one inquiry to each spot for photography of image. InquiryBe the dynamic object that detects by dynamic object tracking portion 107 region image feature amount or by aobviousThe image feature amount of the marking area that work region detecting part 109 detects.

Similar image retrieval portion 116 is used the more than one query graph of selecting by inquiry determination section 115The characteristic quantity of picture, carries out similar image retrieval to image database 111 respectively. At the marking area of inquiryIn diverse situation, provide the result for retrieval of different scale. Therefore, similar image retrieval portion 116 examplesAs similar degree is carried out to standardization, after being merged, result for retrieval outputs to display unit 103.

Figure 1B is the hardware structure diagram of the image retrieval system 100 of embodiments of the invention 1.

For example can realize video search device 104 by common computer. For example, video search dressPut 104 and also can there is interconnective processor 121 and storage device 122. Storage device 122 is by appointingThe storage medium of meaning kind forms. For example, storage device 122 also can be driven by semiconductor memory and hard diskConstituting of moving device.

In this example, by carrying out by processor 121 handling procedure being stored in storage device 122123, realize the image input part 105 shown in Fig. 1, frame register 106, dynamic object tracking portion 107,Trace information register 108, marking area test section 109, marking area register 110, query argumentInfer portion 112, inquiry input part 114, inquiry determination section 115 and similar image retrieval portion 116 like thisFunction part. In other words, in this example, in fact according to above-mentioned handling procedure 123, by processor 121Carry out the performed processing of above-mentioned each function part. In addition, image database 111 and query argument savings portion113 are included in storage device 122.

Video search device 104 also comprises the Network Interface Unit (NIF) 124 being connected with processor, shadowCan be also the NAS being connected with video search device 104 via Network Interface Unit as storage device 101Or SAN. Or image storage device 101 also can be included in storage device 122.

Fig. 2 is the structure of image database 111 and the figure of data example that represents embodiments of the invention 1.In the configuration example of this expression sheet form, but data mode is arbitrarily.

Image database 111 is by shadow table 200, frame table 210, trace information table 220, goer body surface230 and marking area table 240 form. The list structure of Fig. 2 and the field structure of Ge Biao are to implement the present inventionNecessary structure, also can append table and field accordingly with application program.

Shadow table 200 has image id field 201, filename field 202 and spot for photography id field203. Image id field 201 is preserved the identiflication number of each image data. Filename field 202 is preserved from shadowThe filename of the image data reading in as storage device 101. In the direct situation from Camiera input imageUnder, also can omit filename. Spot for photography id field 203 is preserved the ID in the place of ocean weather station observation. ShadowBoth can manage by application program as the ID of data and the correspondence of spot for photography, also can pass through to shadowAppend the management table of spot for photography manages as database. In the situation that using fixed camera, also canSo that spot for photography ID is replaced with to Camera ID. As the example of Fig. 2, also can take ground to oneThe multiple image files of some registration. In this case, in the plurality of image file, for example, comprise set-up siteA fixing camera is taken the image data of gained in the different respectively time periods with taking direction.

Frame table 210 has frame id field 211, image id field 212 and view data field 213.Frame id field is preserved the identiflication number of the each frame extracting from image data. Image id field 212 is to preserve frameThe field of identiflication number of image of extraction source, the image of this identiflication number and management in shadow table 200The value correspondence that id field 201 is preserved. View data field 213 is binary numbers of the rest image of frameAccording to, be kept at the data that use when result for retrieval etc. is presented to display unit 103.

Trace information table 220 has id field 221, the dynamic object ID list field 222 of tracking and movesMoving-wire characteristic quantity field 223. Following the tracks of id field 221 preserves in order to follow by dynamic object tracking portion 107The each dynamic object of track and the identiflication number that uses. Dynamic object ID list field 222 possesses and has same followingThe list of the dynamic object ID of track ID. The ID of dynamic object is pipe in goer body surface 230 described laterThe identiflication number of reason. The sequential that portable cord characteristic quantity field 223 is preserved the coordinate of the dynamic object from imageThe portable cord characteristic quantity of change detection. Image size is according to image and difference, therefore according to the mark of dynamic objectNormalized coordinates calculates portable cord characteristic quantity.

Goer body surface 230 has dynamic object id field 231, follows the tracks of id field 232, frame ID wordSection 233, coordinate fields 234 and characteristic quantity field 235. Dynamic object id field 231 is preserved by movingThe identiflication number (being dynamic object ID) of each dynamic object that state object tracking portion 107 detects. Follow the tracks of IDField 232 is preserved in order same dynamic object to be mapped between frame in dynamic object tracking portion 107And the identiflication number (following the tracks of ID) using. This identiflication number is followed with management in trace information table 220The identiflication number correspondence that track id field 221 is preserved. 233 preservations of frame id field detect dynamic objectThe identiflication number of frame. The knowledge that the frame id field 211 of this identiflication number and management in frame table 210 is preservedBian Hao be not corresponding. Coordinate fields 234 is preserved the coordinate in the image of dynamic object. For example use dynamic object" horizontal coordinate in the upper left corner, the vertical coordinate in the upper left corner, the horizontal coordinate in the lower right corner, the square of boundary rectangleThe vertical coordinate in the lower right corner of shape " such form shows coordinate. Characteristic quantity field 235 is preserved from dynamicallyThe image feature amount that the image of object extracts. For example carry out represent images characteristic quantity with the vector of regular length.

In addition dynamic object ID nonrecognition dynamic object self, and the image of identification dynamic object. CauseThis comprises the image of same dynamic object in multiple frames, gives not to each of these imagesSame (unique) dynamic object ID. For example, as shown in Figure 2, with the following of trace information table 220In dynamic object ID list field 222 corresponding to track ID:1, preserve dynamic object ID:1,2 and 3Situation under, at least representing will be with dynamic object ID:1,2 and 3 by dynamic object tracking portion 107It is same goer that the image (they are included in respectively in different frames) of 3 dynamic objects of identification is judgedThe image of body.

Marking area table 240 has marking area id field 241, frame id field 242, coordinate fields 243And characteristic quantity field 244. Marking area id field 241 is preserved by marking area test section 109 and is examinedThe identiflication number of each marking area of measuring. The knowledge that frame id field 242 is preserved the frame that detects marking areaBian Hao not. The identiflication number of preserving in the frame id field 211 of this identiflication number and management in frame table 210Corresponding. Coordinate fields 243 is preserved the coordinate in the image of marking area. Characteristic quantity field 244 is preserved from aobviousThe image feature amount of work extracted region. Come according to the number of the kind of the determined marking area of system designerPrepare marking area table 240. In addition, also can not prepare marking area table, and only according to dynamic objectImage feature amount is retrieved.

The action of<each portion>

The overall structure of image retrieval system 100 has been described above. Below, at summary image retrieval systemOn the basis of 100 operating principle, the detailed action of each function part is described.

Fig. 3 is the figure of the action of the image retrieval system 100 for embodiments of the invention 1 are described.

Image retrieval system 100 is in the time of the object of retrieving in image, and a frame of for example show image, shouldThe object that frame is appeared before one's eyes carries out similar image retrieval as inquiry. The key diagram 301 of Fig. 3 represents that user selectsSelect the situation of the searching object 302 in incoming frame. Searching object 302 is the illustrated inputs that comprise in imageThe dynamic object of appearing before one's eyes in multiple frames of frame. By processing described later, select to be included in these multiple framesImage (being rest image) some as retrieval and inquisition of searching object 302.

The direction that the additional arrow of searching object 302 is represented to object (is for example personages at searching object 302Situation under, be the positive direction of health). If the direction difference of general object, characteristics of image quantitative changeChange. In addition, according to the frame of selecting, the distinctive district of the searching object 302 of also sometimes originally not appearing before one's eyes outTerritory (marking area). For example, the in the situation that of personage, in rearward situation, do not appear before one's eyes outBecome the face region of feature, therefore cannot use the retrieval of face feature. Multiple frames from image are searchedRope is suitable for the very spended time of operation of the inquiry of similar image retrieval, becomes and obtains institute till result for retrievalThe factor of the time increase needing, the reduction of retrieval precision.

In the present invention, by using the trace information of dynamic object, from same portable cord determine one withOn suitable inquiry, use this inquiry to carry out similar image retrieval. Key diagram 303 is used in image mobileThe form in path (portable cord) express the trace information of searching object 302. Specifically, key diagram303 shown curves according to the order in moment of taking each frame by the successive frame from Extraction of ImageThe coordinate of searching object 302 on picture couples together, and the direction indication retrieval of the arrow of the end of curve is rightResemble 302 directions that move, the profile of key diagram 303 is equivalent to the profile of each frame. In addition, key diagram 303Shown portable cord is from oblique upper to the searching object 302 of looking down the picture of taking like that spot for photographyMobile route. Therefore, the downside of picture is equivalent to front side (in the coverage of this spot for photographyApproach a side of camera), the upside of picture is equivalent to inboard (in the coverage of this spot for photographyFrom the side away from camera). In key diagram afterwards, if not special record is also used with above-mentionedSame method shows the portable cord of dynamic object.

In addition, the mobile route of image that is certain dynamic object at certain portable cord, following sayingIn bright, sometimes this dynamic object is recited as " dynamic object on portable cord ", by the figure of this dynamic objectPicture is recited as " image on portable cord ", and the marking area of the image on portable cord is recited as to " portable cordOn marking area ".

In key diagram 303, for example searching object 302 at place A, the B on portable cord, C, D placeImage as shown in key diagram 304, for take from different directions respectively same searching object 302 (In this example, be personage) image of gained, the outward appearance difference presenting in each image, therefore obtains differenceImage feature amount. The multiple images that obtain are like this all to use as the inquiry of similar image retrievalQuery candidate, can use obtained whole query candidate to carry out similar image retrieval. But, from upperState such successive frame and can obtain many query candidate, therefore owing to using these many query candidateIncrease retrieval time. In addition, (for example determine to show the suitable of result for retrieval for the integrated approach of result for retrievalThe method of order) also have problems.

On the other hand, in the present invention, use the portable cord information to each spot for photography savings, automaticallyDetermine suitable inquiry, reduce retrieval number of times. For example, shown in key diagram 305, (scheming in certain placeIn 3, be recited as place 1) locate to take the portable cord of each dynamic object of being appeared before one's eyes out in the image of gained. ?In this example, as shown in the arrow of portable cord, many dynamic objects moving inwards above from picture, because ofThis can think and in the image of gained is taken in this spot for photography, not comprise the positive figure of many dynamic objectsPicture. Therefore, the image that is difficult to place such be taken to gained is as object, for example, by personage's faceThe feature in what feature was such appear at object front is carried out similar image retrieval as inquiry.

In the incoming frame shown in key diagram 301, include the image towards the searching object 302 of certain direction.In this image, be not limited to comprise suitably in the inquiry of the image of place 1 place's shooting gained as retrievalFeature. But, in the frame before and after it, as shown in key diagram 303 and 304, comprise towards various sidesTo the image of searching object 302, their part is likely taken gained as retrieval at 1 place, placeImage inquiry and comprise suitable feature. Specifically, if in the successive frame that comprises incoming frame,With many dynamic objects in place 1 similarly, comprise the mobile searching object inwards above from picture302 image comprises in the inquiry of the image of place 1 place's shooting gained as retrieval in this imageThe possibility of suitable feature is high.

Therefore, image retrieval system 100, according to the trace information of input image, moves with the many of 1 place, placeState object similarly, search searching object 302 mobile moment inwards on picture, uses from this moment takingThe feature of the marking area beyond the image of the frame of gained front that extract, searching object 302 is carried out similarImage retrieval. For example, in the situation that searching object 302 is personage, not by positive face feature, butThe style and colour of clothes feature 306 at the back side is retrieved as inquiry. On the other hand, shown in key diagram 3072 places, spot for photography, the image of the dynamic object not only in the past moving towards the inside in shooting picture, also takesImage to the dynamic object moving above from the inside on picture, the possibility of the face of appearing before one's eyes out in the latter is high,Therefore retrieve face feature 308 as inquiry.

As the effect of the present embodiment, use trace information automatically to increase inquiry, therefore running cost reduces,By selecting the inquiry corresponding with spot for photography, can reduce retrieval time. In addition, to each spot for photographyOnly select to represent the inquiry in significant region, therefore with the situation phase that uses whole query candidate to retrieveRatio, can expect to alleviate retrieval noise.

In order to implement the present invention, first must carry out the tracking of dynamic object and significantly in the savings stage of imageThe detection in region also registers to database. In addition, putting aside after many images, must derive for forEach spot for photography generates the parameter of suitable inquiry. In the time of retrieval, use these register informations, savings informationGenerate more than one inquiry and retrieve. Below, illustrate respectively registration, the parameter of image derivation,The action of the each portion in retrieval.

Fig. 4 is that the video search device 104 of explanation embodiments of the invention 1 is registered locating of inputted imageThe flow chart of reason. Below, each step of key diagram 4.

(Fig. 4: step S401)

Image input part 105 is decoded to the image data of inputting from image storage device 101, and frame is doneFor rest image extracts.

(Fig. 4: step S402～S410)

Each portion in video search device 104 is to each frame execution step of extracting in step S401S402～S410。

(Fig. 4: step S403)

The image information of frame and extraction source is registered to image database 111 by frame register 106.

(Fig. 4: step S404)

Dynamic object detects from frame in dynamic object tracking portion 107.

(Fig. 4: step S405)

Whether dynamic object tracking portion 107 judges also to exist in previous frame and detects in step S404, if also there is trace information in previous frame (frame in a upper moment of present frame) in dynamic objectRegister 108 implementation step S407. On the other hand, in previous frame, do not exist in step S404 and detectIn the situation of the dynamic object going out, this dynamic object is emerging dynamic object in present frame, therefore followsTrack information register 108 performs step S406.

(Fig. 4: step S406)

Trace information register 108 is using the dynamic object newly detecting in step S405 as tracing object,Newly register in the trace information table 220 of image database 111.

(Fig. 4: step S407)

Trace information register 108 is extracted image feature amount from each dynamic object, by the characteristics of image extractingThe frame ID of amount, the tracking ID identical with the dynamic object of previous frame definite in step S405, present frame,And the coordinate of each dynamic object in present frame registers to respectively the characteristic quantity field of goer body surface 230235, follow the tracks of id field 232, frame id field 233 and coordinate fields 234. In addition, trace information is stepped onNote portion 108 is appended to obtained dynamic object ID the dynamic object ID list word of trace information table 220In section 222.

(Fig. 4: step S408)

Marking area test section 109 detects marking area from frame. Preparing multiple marking area detection mouldIn the situation of piece, carry out Check processing according to the number of detection module.

(Fig. 4: step S409)

Marking area register 110 is extracted characteristics of image from the marking area detecting among step S408Amount, registers in the marking area table 240 of image database 111.

Step S404～S407, step S408～S409 independently process, and therefore also can use multiple metersOperator resource is carried out concurrently.

Above, be the explanation relevant with the registration process of image. Then, illustrate and use data registered to push awayBe decided to be the processing of the parameter that determines suitable inquiry and use.

Fig. 5 is the inferring of the query argument carried out of video search device 104 of explanation embodiments of the invention 1The figure processing.

If a certain number of above image is put aside in image database 111, about each shooting groundPoint, can obtain many portable cords. In key diagram 501, as an example, represent about with the saying of Fig. 3The portable cord that the place 2 that bright Figure 30 7 is identical obtains. For each portable cord, portable cord characteristic quantity is kept atIn trace information table 220. Query argument is inferred portion 112 and is first carried out cluster for these portable cord characteristic quantities(clustering) process, in key diagram 502, find out the representative portable cord 502A representing with the arrow of thick lineAnd 502B. In clustering processing, can use the such common method of k-means method.

Then, query argument is inferred portion 112 and is obtained and belonging to each group (cluster) from image database 111Portable cord on the marking area that detects. Consequently, for each kind of marking area, detectedThe set of the image feature amount of the number of the marking area going out and the marking area detecting. Get rid of at thisThe marking area of the kind of the discontented predetermined number of number that the stage detects is selected in remaining marking areaBe best suited for the marking area of retrieval.

Be suitable for the method for marking area of retrieval as judgement, for example, can consider to use image feature amountThe method of variance ratio.

The video search device 104 of Fig. 6 embodiments of the invention 1 uses in order to judge marking areaThe key diagram of the variance ratio of image feature amount.

The variance ratio of image feature amount is the image feature amount of the marking area that detects in same portable cordVariance yields (portable cord internal variance) and portable cord between the ratio (variance of variance yields (variance between portable cord)Variance between ratio=portable cord/average portable cord internal variance). In key diagram 601, schematically represent that variance ratio is largeSituation under the example of variance of image feature amount of marking area of each portable cord. In this example, withIn one portable cord, the time fluctuation of the image feature amount of same object is few, between portable cord, i.e. different objectsBetween the difference of image feature amount large, therefore easily find object by the retrieval of feature value vector.

On the other hand, showing of the each portable cord in the situation that of schematically representing that variance ratio is little in key diagram 602The example of the variance of the image feature amount in work region. In this example, cannot separate the characteristic quantity of an objectSpace and with it the characteristic quantity space of different objects, therefore find and original object of wishing retrieval mistakenlyThe possibility of the different object of thing is high, is difficult to obtain effective result for retrieval.

Query argument is inferred portion 112 each marking area is obtained the variance ratio of image feature amount, selects variance ratioHigh marking area, registers in query argument savings portion 113.

In the example of Fig. 5, as shown in key diagram 501 and 502, many portable cords of obtaining are categorized asComprise from picture above towards the group of many portable cords of left back, comprise from left back towards aboveThe group of many portable cords. In this example, the representative portable cord 502A of each group and 502B are not actualIn many portable cords of obtaining one, but the generation generating according to many portable cords that are included in each groupTable property portable cord. In addition, also will be included in portable cord in each group and be recited as the representative portable cord of each groupSimilar portable cord.

The key diagram 503 and 505 of Fig. 5 is illustrated respectively in and represents portable cord 502A and represent portable cord 502BSimilar portable cord on the example of the relevant information of the marking area that detects. Specifically, at key diagramIn 503 and 505, as the example of the information relevant with marking area, demonstrate marking area kind,The example of the number of the marking area of the various species detecting, the image of marking area and characteristic quantityVariance ratio. In the example of Fig. 5, each dynamic object is personage, and therefore the kind of marking area comprises and " movesKinetoplast " (being dynamic object entirety), " face " and " style and colour of clothes ", but also can comprise other kinds.

In the example of Fig. 5, the similar portable cord that represents portable cord 502A comprise many from picture beforeFacing to the portable cord of left back, therefore detecting kind is the many remarkable of " moving body " and " style and colour of clothes "Region, but do not detect the marking area that kind is " face ". In this example, the image of " style and colour of clothes "The variance ratio of characteristic quantity is larger than " moving body ", therefore " style and colour of clothes " is chosen as to the remarkable district that is suitable for retrievalThe kind 504 (being also recited as below " the effectively kind of marking area ") in territory. On the other hand, representative is mobileThe similar portable cord of line 502A comprises many left backs from picture towards portable cord above, therefore inspectionMeasuring kind is the marking area of enough numbers of " face ", the variance ratio maximum of its image feature amount, thereforeSelect " face " kind 506 as effective marking area.

More particularly, for example also can select to detect number and variance yields all exceed predetermined value etc., meet pre-The kind of fixed condition. For a group, multiple kinds meet in the situation of above-mentioned condition, both can selectSelect the whole of them, also such as one of selecting party difference maximum etc., and then dwindle according to other conditionsThe scope of kind.

Fig. 7 represents the structure of query argument savings portion 113 of embodiments of the invention 1 and data exampleFigure. At this, the configuration example of sheet form is shown, but data mode is arbitrarily.

Can with have parameter I D field 700, spot for photography id field 701, area coordinate field 702,The table that represents portable cord characteristic quantity field 703 and marking area kind field 704 constructs to show inquiry ginsengScalar product is held portion 113.

Parameter I D field 700 is preserved the identiflication number (being parameter I D) of each parameter. It is to above-mentioned movementThe ID that each group of line gives.

Spot for photography id field 701 is preserved the identiflication number (being spot for photography ID) of each spot for photography. ClapTake the photograph that the spot for photography id field 203 of the shadow table 200 in place ID and image database 111 preservesValue is corresponding. Area coordinate field 702 is preserved the seat of the distribution of the portable cord that represents the group who belongs to portable cordMark. Represent that portable cord characteristic quantity field 703 preserves the average characteristics amount of mobile line-group and (belong to mobile line-groupPortable cord characteristic quantity average of portable cord). Marking area kind field 704 is preserved and is passed through as Fig. 5With the explanation of Fig. 6 and the method kind that select, more than one effective marking area illustrating aboveClass.

Fig. 8 is that the video search device 104 of explanation embodiments of the invention 1 is inferred according to put aside dataThe flow chart of the processing of query argument. Below, each step of key diagram 8.

(Fig. 8: step S801～S809)

Query argument is inferred portion 112 and is performed step S801～S809 using each spot for photography as handling object.

(Fig. 8: step S802)

Query argument is inferred portion 112 and is obtained from the image of the spot for photography of handling object from image database 111The trace information extracting. Thus, for example obtain relevant with portable cord such shown in the key diagram 501 of Fig. 5Information.

(Fig. 8: step S803)

Query argument infer portion 112 according to portable cord characteristic quantity to the trace information of obtaining in step S802Carry out cluster. Thus, for example, as shown in Figure 5 many portable cords are categorized as to 2 groups, obtain representing eachGroup's representative portable cord 502A and 502B.

(Fig. 8: step S804～S808)

Query argument is inferred portion 112 using obtain in step S803 each group as handling object, carries out stepRapid S804～S808.

(Fig. 8: step S805)

Query argument is inferred the trace information that portion's 112 bases belong to the group of handling object, obtains on portable cordMarking area. For example, query argument is inferred portion 112 and is followed the tracks of ID and the corresponding goer of certain frame ID at certainThe coordinate (being kept at the value in coordinate fields 234) of body and the frame ID identical with it are corresponding significantlyThe Duplication of the coordinate (being the value of coordinate fields 243) in region is in situation more than predetermined value, is judged to beThis marking area is by the marking area on the portable cord of this tracking ID identification. Duplication is for example remarkableThe size of the lap of the scope of the scope of the coordinate in region and the coordinate of dynamic object is with respect to remarkable districtThe big or small ratio of the scope of the coordinate in territory. The each movement obtaining like this by the group's statistics for handling objectMarking area on line, can obtain marking area such shown in the key diagram 503 or 505 of for example Fig. 5.

(Fig. 8: step S806)

Query argument is inferred portion 112 each kind of marking area is derived to the variance that detects number and characteristic quantityBe worth, infer the kind of effective marking area by the method described in the explanation at Fig. 5 and Fig. 6. Thus,Obtain the kind 504 or 506 of example marking area as shown in Figure 5 etc.

(Fig. 8: step S807)

Query argument is inferred portion 112 parameter obtaining in step S806 is registered to query argument savings portionIn 113.

Above, be the pretreatment phase that trace information for using dynamic object makes similar image retrieval high efficiencyThe explanation of closing. Below, retrieval process of the present invention is described.

Fig. 9 is that the inquiry determination section 115 of explanation embodiments of the invention 1 uses trace information decision retrieval to look intoThe figure of the action of asking is the figure that illustrates in greater detail the concept map of Fig. 3.

For example, if user has specified the dynamic object (searching object 302 of Fig. 3) of searching object, asShown in key diagram 901, obtain the portable cord information of this object. For example, obtain and the key diagram of Fig. 3The 303 identical relevant information of portable cord. Then, inquiry determination section 115 is divided into the portable cord obtainingMore than one part portable cord. In the example of Fig. 9, obtain part portable cord by cutting apart901a～901e。

It is desirable to carry out making cutting apart of portable cord whole (or almost whole) on various piece portable cordImage be all from roughly the same direction take a dynamic object gained image (in other words, make according toTake the order in moment and will take the coordinate of multiple images of a dynamic object gained from roughly the same directionThe result of gained of coupling together becomes a part portable cord). Specifically, be for example conceived on portable cordShooting moment of each image, both can cut apart portable cord according to predetermined time interval, also can utilizeThe variation of the direction of portable cord (for example makes the side of advancing of the portable cord in an each place in part portable cordTo being included in predetermined scope) cut apart portable cord. Inquiry determination section 115 moves from the part obtaining like thisEach several part portable cord 901a～901e that the set 902 of moving-wire comprises etc. extract portable cord characteristic quantity, becomeThe state that can retrieve.

Then, inquiry determination section 115 is made the portable cord that respectively represents of putting aside in query argument savings portion 113For inquiry, the set of part portable cord is carried out to nearest portable cord search 903. Recently portable cord search be fromIn set, find the processing of the key element of the distance minimum between inquiry and feature value vector.

For example, at the image that will take to the place 2 at key diagram 307 gained, use query image to carry outIn the situation of retrieval, in portable cord search recently 903, respectively represent that portable cord 502A and 502B becomePortable cord inquiry, the part portable cord of the distance minimum of retrieval and each portable cord feature value vector. At Fig. 9Example in, by will represent portable cord 502A and 502B as the nearest portable cord search 903 of inquiry,Obtain respectively part portable cord 901a and 901d.

Represent the portable cord characteristic quantity of portable cord 502A and the portable cord characteristic quantity of part portable cord 901a itBetween distance little, mean represent the similar portable cord of portable cord 502A and part portable cord 901a similar.In the example of Fig. 9, represent portable cord 502A and part portable cord 901a be all equivalent to dynamic object fromThe motion of moving towards the inside above of picture.

Therefore, represent image and the part portable cord of the dynamic object on the similar portable cord of portable cord 502AThe image of the dynamic object on 901a is the image of taking each dynamic object gained from roughly the same directionPossibility high. This means the kind phase also comprising with the effective marking area relevant with the former in the latterThe possibility of the marking area of same kind is high. Represent between portable cord 502B and part portable cord 901dRelation also identical.

In the example of Fig. 9, as described above, effectively represent portable cord for the marking area of the style and colour of clothes502A selects part portable cord 901a, effectively represents that for the marking area of face portable cord 502B selectsPart portable cord 901d. In this case, as shown in key diagram 904, as retrieval and inquisition, inquiry certainlyBonding part 115 is by the image feature amount of the style and colour of clothes of the image extraction from part portable cord 901a, from partly movingThe image feature amount of the face that the image on moving-wire 901d extracts determines as retrieval and inquisition.

In addition, in part portable cord, exist multiple marking areas situation (for example component part portable cordIn multiple frames, comprise the situation of marking area) under, inquiry determination section 115 can be selected the some of them,Determined as retrieval and inquisition, but also can and then be selected the marking area conduct being more suitable for according to other conditionsRetrieval and inquisition. For example, inquiry determination section 115 can be selected place that the size of marking area is large or dynamicallyThe slow-footed places (for alleviating subject shake) of object etc., are determined as retrieval and inquisition. In addition,If there is the function of the reliability of output detections result in the detection module of marking area, also can makeUse this value, for example, the image feature amount of marking area high reliability is determined as retrieval and inquisition.

Figure 10 is that the video search device 104 of explanation embodiments of the invention 1 is determined according to use trace informationFixed retrieval and inquisition carries out the flow chart of the processing of similar image retrieval. Each step of Figure 10 is described below.

(Figure 10: step S1001)

Inquiry determination section 115 reads from image database 111 inspection that user specifies by inquiry input part 114The trace information of rope object 302. Thus, read such portable cord shown in the key diagram 901 of for example Fig. 9Information.

(Figure 10: step S1002)

Inquiry determination section 115 is according to the trace information generating portion portable cord collection obtaining in step S1001Close, extract the portable cord characteristic quantity of each several part portable cord. Thus, obtaining example part as shown in Figure 9 movesThe set 902 of line.

(Figure 10: step S1003)

What inquiry determination section 115 was read each spot for photography from query argument savings portion 113 respectively represents portable cordParameter. Thus, read representative portable cord 502A as shown in Figure 9 of example and the parameter of 502B.

(Figure 10: step S1004～S1008)

The each parameter execution step of inquiry determination section 115 to the representative portable cord of reading in step S1003S1004～S1008。

(Figure 10: step S1005)

Inquiry determination section 115, using the characteristic quantity that represents portable cord as inquiry, is searched from the set of part portable cordThe nearest portable cord of rope. This step is equivalent to the nearest portable cord search 903 of Fig. 9.

(Figure 10: step S1006)

Inquiry determination section 115 is chosen in the marking area on the nearest portable cord obtaining in step S1005, willThe retrieval and inquisition of the image feature amount that comprises this marking area determines as basis is read in step S1003Representative portable cord parameter specify spot for photography and the retrieval and inquisition in region. Thus, for example, as Fig. 9Key diagram 904 shown in, for representing that portable cord 502A determines the retrieval of the image feature amount that comprises the style and colour of clothesInquiry, for representing that portable cord 502B determines the retrieval and inquisition of the image feature amount that comprises face.

(Figure 10: step S1007)

Similar image retrieval portion 116 is used the retrieval and inquisition determining in step S1006, from image database111 obtain similar image searching result. Can use to this processing the technology of common similar image retrieval.

(Figure 10: step S1009)

If for each parameter that represents portable cord, the execution of step S1004～S1008 finishes, similarImage retrieval portion 116 is by the each spot for photography obtaining by step S1004～S1008 and each representativeThe result for retrieval merging of portable cord is presented at display unit 103. Each result for retrieval is by different types of remarkableRegion is as the result for retrieval of inquiry, and therefore similar image retrieval portion 116 marks similar degree in the time mergingStandardization. In addition, also can separately show result for retrieval for each spot for photography.

Figure 11 is the figure of the processing sequential of the image retrieval system 100 of explanation embodiments of the invention 1, toolSay, be that image registration process, the query argument of explanation image retrieval system 100 described above inferred bodyProcess, in retrieval process, user 1101, computer 1102, image database 111, query argument be long-pendingHold the figure of the processing sequential of portion 113. In addition, computer 1102 is meters of realizing video search device 104Calculation machine. In Figure 11, express distinctively image database 111 Hes in order to illustrate with computer 1102Query argument savings portion 113, but they also can be included in computer 1102. The step S1132 of Figure 11,S1133, S1134 are respectively that image registration process, query argument are inferred processing, relevant the locating of retrieval processReason. Each step of Figure 11 is described below.

[image registration process] (Figure 11: step S1003～S1112)

If user 1101 has inputted image (S1103) from image storage device 101 to computer 1102,, in computer 1102, frame image input part 105 being extracted by frame register 106 registers to imageDatabase 111 (S1104), image database 111 notification enrollment complete (S1105).

Then,, in computer 1102, extracted frame is detected and followed the tracks of in dynamic object tracking portion 107Interior dynamic object (S1106), trace information is registered to image database by trace information register 108111 (S1107), image database 111 notification enrollment complete (S1108). And then marking area detectsThe marking area (S1109) in extracted frame detects in portion 109, and marking area register 110 is by remarkable districtTerritory registers to image database 111 (S1110), and image database 111 notification enrollment complete (S1111).If the processing of whole frames finishes, notify image to register (S1112) to user 1101.

[query argument is inferred processing] (Figure 11: step S1113～S1119)

If user 1101 sends query argument and infers the request of processing to video search device 104(S1113),, in computer 1102, query argument is inferred portion 112 and is asked to image database 111The trace information (S1114) of each spot for photography, and obtain (S1115).

Query argument is inferred the method derivation that portion 112 illustrates above by the explanation as Fig. 5～Fig. 8Determine the needed parameter of inquiry (S1116), parameter is registered in query argument savings portion 113(S1117), query argument savings portion 113 notification enrollment complete (S1118). If for whole shootingsPlace, parameter estimation processing finishes, and completes (S1119) to user's 1101 notifier processes.

[retrieval process] (Figure 11: step S1120～S1131)

If user 1101 has specified the dynamic of searching object in the frame image database 111 from savingsObject (for example searching object 302) (S1120), in computer 1102, inquiry determination section 115 toImage database 111 is asked the trace information (S1122) of the dynamic object of (S1121) searching object, toQuery argument savings portion request (S1123) also obtains (S1124) parameter.

Inquiry determination section 115 uses looking into of the trace information of the dynamic object of searching object, each spot for photographyAsk parameter, the method that illustrates above by the explanation as Fig. 9～Figure 10 determines each spot for photographyInquiry (S1125), points out (S1126) to user 1101. If user 1101 confirms suggested inquiry,Send retrieval request (S1127), in computer 1102, similar image retrieval portion 116 use determineSimilar image retrieval (S1128) is carried out in fixed inquiry, obtains similar image retrieval knot from image database 111Really (S1129). Computer 1102 is incorporated in one by the result for retrieval obtaining according to multiple queries as requiredPlay (S1130) and point out (S1131) to user.

Figure 12 is illustrated in to use the video search device 104 of embodiments of the invention 1 to retrieve in imageThe figure of the configuration example of the operation screen using when object. In display unit 103, point out this picture to userFace. User uses input unit 102 operations to be presented at the cursor 1207 on picture, thus to video searchDevice 104 is given the instruction of processing.

The operation screen of Figure 12 has image selection key 1201, image display region 1202, query displayRegion 1203, retrieval button 1204 and result for retrieval viewing area 1205.

First user clicks image selection key 1201, selects to be thus recorded in image database 111Image arbitrarily. By the image display of selecting in image display region 1202. User will be presented at imageDynamic object arbitrarily 302 in viewing area 1202 is appointed as searching object.

The method that video search device 104 illustrates above by the explanation as Fig. 9～Figure 10, is referring toOn the portable cord of fixed dynamic object, search for suitable inquiry, be presented at query display region 1203.

User confirms shown inquiry, adjusts as required, and click is retrieved button 1204 and sentRetrieval request.

The similar image searching result of each inquiry is presented at result for retrieval viewing area by video search device 104Territory 1205.

More than the explanation relevant to embodiments of the invention 1. According to the present embodiment, user can be easilySpecify searching object, can carry out the similar image retrieval corresponding with spot for photography. In addition, by only usingBe defined in the inquiry of each spot for photography, can shorten retrieval time and alleviate retrieval noise.

Embodiment 2

In embodiment 1, the trace information that uses the dynamic object in image has been described, make similar image inspectionThe method of rope high efficiency. The tracking of dynamic object, only carries out correspondence according to the information approaching between frame conventionally,Therefore in the long-time static situation of object, or in the situation etc. that has veil between object and camera,Follow the tracks of sometimes and interrupt. In the situation that cannot carrying out following the tracks of for a long time, the inquiry obtaining from each portable cordCandidate's number reduce, therefore likely cannot give full play to the effect of embodiment 1.

Therefore, in embodiment 2, illustrated and used the marking area of savings in image database 111,Revise the method for trace information. Except the difference of following explanation, the image retrieval system of embodiment 2Each portion of 100 has identical with the each portion that has been endowed prosign of the embodiment 1 shown in Fig. 1～Figure 12Function, therefore omit their explanation.

Figure 13 be the image retrieval system 100 for embodiments of the invention 2 are described use remarkable districtThe figure of the correction of the trace information in territory.

For example, in key diagram 1301,3 that detect in being illustrated in certain image during certain are movedMoving-wire. Portable cord 1 and portable cord 3 are portable cords of different personage. On the other hand, portable cord 1 and movingMoving-wire 2 should be original same personage's a portable cord, but this personage passed through veil 1302 inSide has therefore disconnected midway. Therefore, as shown in the left side of Figure 13, the image database 111 before correctionTrace information table 220 and goer body surface 230 in, same personage's 2 portable cords are given and being followed respectivelyTrack ID:1 and 2, is recorded as different portable cords.

Therefore, the image retrieval system 100 of the present embodiment is using the marking area on each portable cord as inquiry,Using the marking area on the different portable cords in the given time of same spot for photography as object, carry out similarImage retrieval. If consequently finding similar degree on different portable cords is remarkable district more than predetermined valueTerritory, is judged to be the portable cord that these portable cords are same objects, revises trace information.

Delete from the revised trace information table 220 shown in the right side of Figure 13 following of existing correctionThe entry of track ID:2, appends the dynamic object ID of the entry of following the tracks of ID:2 to the entry of following the tracks of ID:1The content of list, merges into 1 by 2 portable cords thus. In addition, correspondingly, as Figure 13Shown in revised goer body surface 230 shown in right side, by the goer body surface 230 before revisingTracking ID:2 also change to follow the tracks of ID:1. The part that the thick frame of use of the table of Figure 13 surrounds is this processingThe revised position of result.

Figure 14 is that the video search device 104 of explanation embodiments of the invention 2 uses marking area correction to followThe flow chart of the processing of track information. Each step of Figure 14 is described below.

(Figure 14: step S1401～S1406)

Trace information register 108, for the each portable cord detecting in the given time, performs stepS1401～S1406。

(Figure 14: step S1402)

Trace information register 108 is read the marking area portable cord from image database 111.

(Figure 14: step S1403)

Marking area is carried out similar image retrieval by trace information register 108.

(Figure 14: step S1404)

Trace information register 108 judges on different portable cords in the given time whether have similar degreeFor marking area more than predetermined value, in the situation that existing, perform step S1405.

(Figure 14: step S1405)

Trace information register 108 merges portable cord, upgrades correspondingly following of image database 111Track information table 220, goer body surface 230.

The method of embodiment 1 is selected inquiry to every portable cord, therefore must suitably record the spy of portable cordThe amount of levying. But the feature of the portable cord obtaining by the tracking of dynamic object only represents in imageThe movement of two dimension, does not consider depth information (in other words, the distance from camera to dynamic object).Therefore, according to the method to set up of camera, even same portable cord, the marking area on this portable cordState also likely change.

Figure 15 is that the degree of depth of considering of the image retrieval system 100 for embodiments of the invention 2 are described is believedThe figure of the correction of the trace information of breath.

For example, in the key diagram 1501 and 1502 of Figure 15, be illustrated respectively in place 1 and place 2 is clappedTake the photograph portable cord, each the portable cord of the dynamic object (being personage) that the image of gained comprises in this exampleOn the example of multiple images. In this example, dynamic object ID:1～3 shown in key diagram 15013 images are near the starting point of portable cord, near intermediate point and near terminal, to take one dynamically respectivelyThe image of object gained. Marking area 1501A～1501C is respectively the image of dynamic object ID:1～3Marking area (being face in this example). Equally, dynamic object ID:11～13 shown in key diagram 15023 images be respectively near the starting point of portable cord, near intermediate point and near terminal, take one movingThe image of state object gained. Marking area 1502A～1502C is respectively the figure of dynamic object ID:11～13The marking area of picture.

The shape of these 2 portable cords is consistent, but almost do not have the portable cord shown in key diagram 1501 fromPoint to the variation of the depth direction of the position of the dynamic object of terminal (in other words, dynamic object and camera itBetween distance almost do not change), on the other hand, the dynamic object of key diagram 1502 is from the inside towards aboveMobile. This is according to learning below: as shown in the size 1503 of the marking area on portable cord, and significantly districtThe size of territory 1501A～C is identical (for example 10cm × 10cm) all, on the other hand, marking area 1502A,The size of B and C (for example respectively as 5cm × 5cm, 7cm × 7cm and 10cm × 10cm thatSample) change.

Even if this means the dynamic object as the changes in coordinates on the picture at each place shooting gainedThe shape of mobile route identical, as the variation of the three-dimensional coordinate in the space of each spot for photography dynamicallyThe actual mobile route of object also has a great difference. Under these circumstances, the remarkable district on each portable cordThe outward appearance in territory has a great difference sometimes, therefore in order to select more suitable retrieval and inquisition, it is desirable to make to be used asFor the mobile route of the dynamic object of the variation of the three-dimensional coordinate in the space of spot for photography (has been given the degree of depthThe portable cord of information).

Therefore, in the present embodiment, trace information register 108 is used the large of marking area on portable cordLittle by 1503, for the knowledge in advance 1504 of this marking area, give depth information to portable cord. Know in advanceKnowing 1504 is normal sizes of the marking area of each kind, for example, be the feelings of face in the kind of marking areaUnder condition, be 25cm × 25cm etc., in addition, the setting position of the camera that comprises each spot for photography place(particularly height), setting party are to information such as the focal lengths of the camera lens of (the particularly angle of depression) and camera.According to the size of the marking area of these information and actual photographed gained, can infer and comprise this marking areaDynamic object is with respect to the distance of camera.

Goer body surface 230 also can also have the depth characteristic amount word of the depth information of preserving dynamic objectSection 236. For example, the size of the marking area on portable cord is 10cm × 10cm, by itBe kept in depth characteristic amount field 236 with the ratio " 10/25 " of normal size 25cm × 25cm. FollowTrack information register 108 according to depth characteristic amount, to relevant in advance the knowing such as the setting position of above-mentioned cameraKnow, the portable cord of the mobile route that represents the dynamic object on picture is transformed to the three-dimensional that represents spot for photographyThe portable cord of the mobile route in space, the characteristic quantity of the portable cord after computational transformation. By the feature calculatingAmount, be kept at for example trace information table 220 consideration in the portable cord characteristic quantity 224 of the degree of depth.

Figure 16 is that the video search device 104 of explanation embodiments of the invention 2 uses marking area to trackingThe flow chart of the processing of information adding depth information. Each step of Figure 16 is described below.

(Figure 16: step S1601～S1607)

Trace information register 108 is for the each dynamic object execution step S1601～S1607 on portable cord.

(Figure 16: step S1602)

Trace information register 108 is by the method identical with the step 805 of Fig. 8, and it is aobvious that investigation detectsWhether the Duplication between the work coordinate in region and the coordinate of dynamic object, investigate thus on portable cord and existMarking area, if there is marking area, performs step S1603, if not, performs step S1604.

(Figure 16: step S1603)

Trace information register 108 bases in advance knowledge 1504 derive depth characteristic. Can use multiple aobviousThe reliability of depth characteristic is improved in work region.

(Figure 16: step S1604)

If do not detect marking area, trace information register 108 is according to the consecutive frame from front and backThe depth information that marking area is derived, carries out interpolation to depth characteristic.

(Figure 16: step S1605)

Trace information register 108 is appended obtained dark to the goer body surface 230 of image database 111Degree information.

(Figure 16: step S1607)

If trace information register 108 has obtained the depth information of the whole dynamical correlations on portable cord,The portable cord characteristic quantity of the degree of depth is considered in extraction, is appended in the trace information table 220 of image database 111.

The portable cord characteristic quantity extracting according to the processing by above determines method and the embodiment of retrieval and inquisition1 is identical, and therefore description thereof is omitted. By the revised trace information of such use, can improve based on movementThe retrieval precision of the nearest portable cord search 903 of line characteristic quantity. Thus, can be according to each spot for photographyImage determines suitable retrieval and inquisition, can improve the essence of the object retrieval described in embodiment 1 as its resultDegree.

Embodiment 3

In embodiment 1, use the marking area of the different frame on same portable cord to carry out similar image inspectionRope, improves retrieval precision thus. But, on this portable cord, do not exist and to become the feature of searching object thingIn the situation of marking area, the image sometimes obtaining by retrieval is limited. In embodiment 3, illustrate to useThe side of the marking area of the searching object thing that the family notice image different image specified from user comprisesMethod. Except the difference of following explanation, each portion of the image retrieval system 100 of embodiment 3 have withThe identical function of each portion that has been endowed prosign of embodiment 1 shown in Fig. 1～Figure 12, therefore omitsTheir explanation.

Figure 17 is the relevant explanation of the prompting that is present in the inquiry in different images of embodiments of the invention 3Figure.

For example, in key diagram 1701, illustrate that the retrieval of the Extraction of Image from take gained in place 1 is rightThe personage's of elephant portable cord. On this portable cord, there is personage's face and the remarkable district of the style and colour of clothes of searching objectTerritory. On the other hand, in key diagram 1702, illustrate from the place 2 different from place 1 and take gainedPersonage's Extraction of Image, identical with searching object portable cord. According to showing of the face on each portable cordThe image feature amount of work region 1701A and 1702A, judges that these portable cords are portable cords of same personage.In addition, according to the image on the portable cord of key diagram 1702, and then find out this personage's feature having (exampleAs suitcase) marking area 1702B. For example, if can (exist to the so different image of user notificationDifferent place is taken the image of gained or is taken the image etc. of gained in same place in the different time periods)Marking area, user can retrieve more image.

For example, in key diagram 1703, illustrate from all different place 3 clapping from place 1 and place 2Take the photograph the portable cord of the Extraction of Image of gained. In this example, this portable cord is and key diagram 1701 and 1702Shown figure picture personage's together portable cord, but as the marking area on this portable cord, do not detectAny one of face and the style and colour of clothes, and detect the marking area 1703A of suitcase. In this case, even willFrom the face of Extraction of Image in place 1 or the image feature amount of the marking area of the style and colour of clothes as retrieval and inquisition, also withoutMethod is the personage as searching object from the video search in place 3, if but by the marking area 1702B of suitcaseImage feature amount as retrieval and inquisition, can retrieve this personage.

Picture 1704 and 1705 is to show in order to be present in the marking area in different images to user notificationThe example of the picture in display unit 103. In picture 1704, show that user selects as searching objectPersonage's the frame of image. Display unit 103 so also can eject show by said method from other shadowsThe marking area that picture detects. In the example of picture 1704, respectively by pop-up window 1704A and1704B shows the conduct inspection of the Extraction of Image from take gained by other cameras (external camera 2)The suitcase of the personage's of rope object having, from taking by other cameras (external camera 4) in additionThe cap of the personage's as searching object that the image of gained detects having.

On the other hand, in picture 1705, with chart show the marking area that detects from different images itBetween relation property. In the example of picture 1705, show the node 1705A, the expression face that represent place 1Marking area node 1705B, represent the node 1705C of marking area of the style and colour of clothes, at edge in conjunction with jointPoint 1705A and 1705B, also at edge in conjunction with node 1705A and 1705C. This represents from place 1The image detection of taking the personage of the searching object of gained goes out marking area (for example marking area of face1701A) and the marking area of the style and colour of clothes.

And then in picture 1705, the marking area of demonstration expression place 2, face, the style and colour of clothes is aobvious respectivelyNode 1705D, 1705E, 1705F and the 1705G of the marking area of work region and suitcase, jointPoint 1705D is combined with node 1705E, 1705F and 1705G respectively at edge. And then, node 1705EBe combined with node 1705B at edge, node 1705F is combined with node 1705C at edge. They representThe marking area (for example marking area 1702A) of the face from certain portable cord of the Extraction of Image in place 2 withAnd the marking area of the style and colour of clothes respectively with marking area (for example marking area of the face of the searching object in place 11701A) and the marking area of the style and colour of clothes similar, and then detect marking area (for example marking area of suitcase1702B) as the marking area on this portable cord.

And then, in picture 1705, show respectively and represent the marking area of place 4, face and capNode 1705H, 1705I and the 1705J of marking area, node 1705H at edge respectively with node1705I, 1705J combination. And then node 1705I is combined with node 1705B at edge. Their represent fromShowing of the face of the marking area of the face on certain portable cord of the Extraction of Image in place 4 and the searching object in place 1Work region (for example marking area 1701A) is similar, and then the marking area that detects cap moves as thisMarking area on line.

User can be with reference to above-mentioned demonstration, the new marking area of specifying retrieval and inquisition to use. For example, existUser uses input unit 102 to specify in the situation of pop-up window 1704A or node 1705G, carries outSimilar image retrieval using the image feature amount of the marking area of suitcase as retrieval and inquisition. Thus, Neng GouquMust comprise the image of marking area 1703A of the suitcase in place 3 as result for retrieval. For example,, in place 3Take in the image of gained, the personage's of searching object face and the style and colour of clothes are not all shown as searchable degree,But in the situation that appearing before one's eyes out suitcase, the face detecting in place 1 in use or the image feature amount of the style and colour of clothesIn similar image retrieval, cannot obtain from the image in place 3 this personage's image. But, as described above,By using the image feature amount of the suitcase of obtaining in place 2 as retrieval and inquisition, can be from the shadow in place 3Picture is obtained this personage's image.

Figure 18 is that the video search device 104 of explanation embodiments of the invention 3 is new from different image searchThe flow chart of processing of marking area of kind. Each step of Figure 18 is described below.

(Figure 18: step S1801)

The dynamic object that inquiry determination section 115 is specified from user is selected inquiry to each spot for photography. This processingIdentical with the processing that the step S1006 of Figure 10 was former.

(Figure 18: step S1802～S1805)

Inquiry determination section 115 is for the query execution step that each spot for photography is selectedS1802～S1805。

(Figure 18: step S1803)

Similar image retrieval portion 116 is used the inquiry of selecting to specifying spot for photography to carry out similar image inspectionRope.

(Figure 18: step S1804)

If similar image retrieval portion 116 finds the aobvious of new kind on the portable cord under result for retrievalWork region, the such display methods of for example picture 1704 or 1705 by Figure 17 is to user notification.In the situation that user has specified some marking areas according to this notice, similar image retrieval portion 116 is usedThe retrieval and inquisition of the image feature amount that comprises specified marking area, the step S1007 of execution Figure 10.

Embodiment 4

In above embodiment, the purposes of the object of retrieval user appointment is described. On the other hand, sometimesUser does not imagine specific searching object, and wish rest in efficiently predetermined during in occur wholeObject. In embodiment 4, descriptive abstract ground shows the method for long image. Except following explanationBeyond difference, each portion of the image retrieval system 100 of embodiment 4 has and the reality shown in Fig. 1～Figure 12The identical function of each portion of prosign of having executed being endowed of example 1, therefore omits their explanation.

The figure of the video summarization of Figure 19 is use for embodiments of the invention 4 are described trace information.

Image database 111 is preserved the information of the dynamic object detecting in each frame, therefore for example canGeneration transverse axis is that to be dynamic object detect several chart 1901 for time (frame number), the longitudinal axis. If userBy using input unit 102 to operate cursor 1207, for example, select to exist the time period of many dynamic objects1905, the overlapping whole dynamic objects that detect in this time period 1905 that are presented in frame. But,Under this state, many dynamic objects mix existence, and identity is poor. Key diagram 1902 is to fill by demonstrationPut the example of the picture of 103 demonstrations. In this example, in a frame, demonstrate 4 personages at portable cordOn image, but show many images for each personage, therefore picture mixes and identity reduces.

Therefore, the image retrieval system 100 of the present embodiment uses the trace information of image database 111, pinEach portable cord is only shown to the image of a dynamic object. In the situation that dynamic object is overlapping, adjustOverlapping subject image is moved on portable cord, and not overlapping between object. Key diagram 1903 is theseThe example that passes through the picture that display unit 103 shows of embodiment. In this example, show moving of certain personageMoving-wire 1903A, only shows an image in the multiple images of the personage on this portable cord 1903A1903B. Equally, for each personage, show an image on portable cord, this portable cord, this is movedImage on line shows not overlappingly with other personages' that show image. Thus, eliminate the mixed of pictureAssorted, improve identity.

In addition, by the method for the decision inquiry described in use embodiment 1, each dynamic object is emphasized to showBecome the marking area of inquiry, can grasp more efficiently each object. Key diagram 1903 is by this enforcementOther examples of the picture that the display unit 103 of example shows. In this example, to certain personage, except movementBeyond image 1903B on line 1903A and portable cord, also eject the marking area showing on this portable cord1904A. For other personages too.

In addition, be not limited to the image that comprises marking area in all images on a portable cord. This realityExecuting routine image retrieval system 100 also can be at will show one that selects in multiple images of each personageTime, the image that preferential selection comprises marking area.

Figure 20 has represented use that the image retrieval system 100 of embodiments of the invention 4 is carried out to follow the tracks of letterThe flow chart of the processing of the video summarization of breath. Each step of Figure 20 is described below.

(Figure 20: step S2001)

Inquiry determination section 115 is read the spot for photography of user's appointment, the whole portable cord information in the time.

(Figure 20: step S2002～S2008)

Inquiry determination section 115 is for the each portable cord execution step obtaining in step S2001S2002～S2008。

(Figure 20: step S2003)

115 search of inquiry determination section are suitable for the marking area of the inquiry on portable cord. This processing with figureIn 10, the processing of explanation is identical.

(Figure 20: step S2004)

Inquiry determination section 115 is read dynamic object the frame that has marking area from image database 111Coordinate.

(Figure 20: step S2005)

Inquiry determination section 115 judge the dynamic object of reading in step S2004 coordinate scope whether withThe scope of the coordinate of the dynamic object having shown is overlapping, performs step S2006 overlapping in the situation that,In nonoverlapping situation, perform step S2007.

(Figure 20: step S2006)

Inquiry determination section 115 moves the coordinate of dynamic object on portable cord, turns back to step S2005.

(Figure 20: step S2007)

Video search device 104 makes the image of dynamic object overlapping on portable cord, is presented at display unit103。

By above processing, use the trace information of dynamic object and marking area to detect, user can be highRest in to effect the dynamic object and the marking area thereof that in the fixed time, occur.

In addition, the present invention is not limited to above-described embodiment, comprises various distortion examples. For example, for easy reasonSeparate ground explanation the present invention and describe above-described embodiment in detail, might not be limited to and possess illustrated whole knotsStructure. In addition, a part for the structure of certain embodiment can be replaced into the structure of other embodiment, in addition alsoCan append to the structure of certain embodiment the structure of other embodiment. In addition, can be to the structure of each embodimentA part carry out the appending/delete of other structures/replace.

Also can for example design etc. with integrated circuit, realize above-mentioned each structure, function, processing with hardwarePart or all of portion, processing unit etc. In addition, also can be by processor to realizing the journey of each functionOrder makes an explanation, carries out, and realizes above-mentioned each structure, function etc. thus with software. Can each function will be realizedThe information such as program, table, file be stored in memory, hard disk drive, SSD (solid-state drive) etc.The data storage medium of the nonvolatile of the embodied on computer readable such as storage device or IC-card, SD card, DVDIn.

In addition, be shown in the drawingsly considered to illustrate the necessary control line of embodiment and information wire, do not limitIn the whole control lines and the information wire that necessarily illustrate that application actual product of the present invention comprises. In fact, alsoCan consider entire infrastructure to be almost connected with each other.

Claims (15)

1. a video search device, the storage device that it possesses processor, is connected with above-mentioned processor,Above-mentioned video search device is characterised in that:

From the first image by forming at multiple frames of the first place shooting gained and by clapping in the second placeThe second image of taking the photograph multiple frames formations of gained detects respectively the mobile route of more than one moving body and depositsStore up in above-mentioned storage device,

Extract the selected movement the above-mentioned more than one moving body detecting from above-mentioned the first imageThe image feature amount of each above-mentioned frame of body is also stored in above-mentioned storage device,

According to the mobile route of the above-mentioned selected moving body detecting from above-mentioned the first image and fromState the mobile route of the above-mentioned more than one moving body that the second image detects, select the image of said extractedThe query image characteristic quantity using as retrieval and inquisition in characteristic quantity,

Use above-mentioned query image characteristic quantity, retrieval is from above-mentioned more than one the moving of above-mentioned the second Extraction of ImageThe image feature amount of kinetoplast,

Export the result of above-mentioned retrieval.

2. video search device according to claim 1, is characterized in that,

The image that comprises multiple above-mentioned moving bodys in above-mentioned the second image,

Above-mentioned video search device,

By the mobile route of the above-mentioned multiple moving bodys that detect from above-mentioned the second image, move road according to eachThe characteristic quantity in footpath is categorized as multiple groups, generates the delegated path of the mobile route that represents each group,

Cut apart by the mobile route to above-mentioned selected moving body, generate multiple parts path,

According to the characteristic quantity in the characteristic quantity of the delegated paths of above-mentioned multiple groups and above-mentioned multiple parts path, retrievalIn above-mentioned multiple parts path with the most similar part of some above-mentioned delegated paths path,

The characteristics of image of the above-mentioned selected moving body on the part path of selecting to obtain by above-mentioned retrievalAmount is as above-mentioned query image characteristic quantity.

3. video search device according to claim 1, is characterized in that,

Above-mentioned memory device stores with from the first moving body of above-mentioned the first Extraction of Image and the second moving bodyThe information that mobile route is relevant,

Above-mentioned video search device and then according to the image feature amount of above-mentioned the first moving body and above-mentioned secondThe image feature amount of moving body, be judged to be from the image of above-mentioned first moving body of above-mentioned the first Extraction of Image andIn the similar situation of image of above-mentioned the second moving body, by relevant to the mobile route of above-mentioned the second moving bodyInformation merges in the information relevant to the mobile route of above-mentioned the first moving body.

4. video search device according to claim 1, is characterized in that,

The image of the image of above-mentioned selected moving body and the moving body that detects from above-mentioned the second image allThe region that comprises the first predetermined kind, being included in above-mentioned the first kind in the above-mentioned image of selectingThe image in region and be included in the above-mentioned the first the image of the moving body detecting from above-mentioned the second imageThe image in the region of class is similar, and also comprises the second from the image of the moving body of above-mentioned the second Extraction of ImageIn the situation in the region of class, export the information relevant to the region of above-mentioned the second kind.

5. video search device according to claim 1, is characterized in that,

Also possess display unit,

Detect the image of multiple moving bodys from some images, and detecting the many of above-mentioned each moving bodyIn the situation of individual image, select one of multiple images of above-mentioned each moving body, above-mentioned by above-mentioned each moving bodyThe image of selecting shows not overlappingly with the above-mentioned image of selecting of other moving bodys.

6. a method for retrieving image of being carried out by video search device, above-mentioned video search device possesses placeReason device, with the storage device that above-mentioned processor is connected, above-mentioned method for retrieving image is characterised in that,

Comprise the following steps:

First step, from by take in the first place the first image that multiple frames of gained form and byThe second image that multiple frames formations of gained are taken in the second place detects respectively moving of more than one moving bodyMoving path is also stored in above-mentioned storage device;

Second step, extracts selected the above-mentioned more than one moving body that detects from above-mentioned the first imageThe image feature amount of each above-mentioned frame of the moving body of selecting is also stored in above-mentioned storage device;

Third step, according to the mobile road of the above-mentioned selected moving body detecting from above-mentioned the first imageThe mobile route of footpath and the above-mentioned more than one moving body that detects from above-mentioned the second image, in selectionState the query image characteristic quantity using as retrieval and inquisition in the image feature amount of extraction;

The 4th step, is used above-mentioned query image characteristic quantity, and retrieval is from above-mentioned one of above-mentioned the second Extraction of ImageThe image feature amount of individual above moving body; And

The 5th step, exports the result of above-mentioned retrieval.

7. method for retrieving image according to claim 6, is characterized in that,

The image that comprises multiple above-mentioned moving bodys in above-mentioned the second image,

Above-mentioned third step comprises the following steps:

The mobile route of the above-mentioned multiple moving bodys that detect from above-mentioned the second image is moved to road according to eachThe characteristic quantity in footpath is categorized as multiple groups, generates the delegated path of the mobile route that represents each group;

Cut apart by the mobile route to above-mentioned selected moving body, generate multiple parts path;

According to the characteristic quantity in the characteristic quantity of the delegated paths of above-mentioned multiple groups and above-mentioned multiple parts path, retrievalIn above-mentioned multiple parts path with the most similar part of some above-mentioned delegated paths path; And

The characteristics of image of the above-mentioned selected moving body on the part path of selecting to obtain by above-mentioned retrievalAmount is as above-mentioned query image characteristic quantity.

8. method for retrieving image according to claim 6, is characterized in that,

Above-mentioned memory device stores with from the first moving body of above-mentioned the first Extraction of Image and the second moving bodyThe information that mobile route is relevant,

Above-mentioned method for retrieving image is further comprising the steps of: according to the image feature amount of above-mentioned the first moving bodyWith the image feature amount of above-mentioned the second moving body, what be judged to be to detect from above-mentioned the first image above-mentioned first movesIn the similar situation of image of the image of kinetoplast and above-mentioned the second moving body, by with the moving of above-mentioned the second moving bodyThe information of moving path coherence merges in the information relevant to the mobile route of above-mentioned the first moving body.

9. method for retrieving image according to claim 6, is characterized in that, further comprising the steps of:

The image of the image of above-mentioned selected moving body and the moving body that detects from above-mentioned the second image allThe region that comprises the first predetermined kind, being included in above-mentioned the first kind in the above-mentioned image of selectingThe image in region and be included in the above-mentioned the first the image of the moving body detecting from above-mentioned the second imageThe image in the region of class is similar, and also comprises the second from the image of the moving body of above-mentioned the second Extraction of ImageIn the situation in the region of class, export the information relevant to the region of above-mentioned the second kind.

10. method for retrieving image according to claim 6, is characterized in that, further comprising the steps of:

Detect the image of multiple moving bodys from some images, and detecting the many of above-mentioned each moving bodyIn the situation of individual image, select one of multiple images of above-mentioned each moving body, above-mentioned by above-mentioned each moving bodyThe image of selecting shows not overlappingly with the above-mentioned image of selecting of other moving bodys.

The storage medium that the calculating function of 11. 1 kinds of nonvolatiles reads, controls computer for storageProgram, it is characterized in that,

The storage device that above-mentioned computer possesses processor, is connected with above-mentioned processor,

Said procedure makes above-mentioned processor carry out following steps:

First step, from by take in the first place the first image that multiple frames of gained form and byThe second image that multiple frames formations of gained are taken in the second place detects respectively moving of more than one moving bodyMoving path is also stored in above-mentioned storage device;

Second step, extracts selected the above-mentioned more than one moving body that detects from above-mentioned the first imageThe image feature amount of each above-mentioned frame of the moving body of selecting is also stored in above-mentioned storage device;

Third step, according to the mobile road of the above-mentioned selected moving body detecting from above-mentioned the first imageThe mobile route of footpath and the above-mentioned more than one moving body that detects from above-mentioned the second image, in selectionState the query image characteristic quantity using as retrieval and inquisition in the image feature amount of extraction;

The 4th step, is used above-mentioned query image characteristic quantity, and retrieval is from above-mentioned one of above-mentioned the second Extraction of ImageThe image feature amount of individual above moving body;

The 5th step, exports above-mentioned result for retrieval.

The storage medium that the calculating function of 12. nonvolatiles according to claim 11 reads, its featureBe,

The image that comprises multiple above-mentioned moving bodys in above-mentioned the second image,

Above-mentioned third step comprises the following steps:

The mobile route of the above-mentioned multiple moving bodys that detect from above-mentioned the second image is moved to road according to eachThe characteristic quantity in footpath is categorized as multiple groups, generates the delegated path of the mobile route that represents each group;

By the mobile route of the above-mentioned moving body of selecting is cut apart, generate multiple parts path;

According to the characteristic quantity in the characteristic quantity of the delegated paths of above-mentioned multiple groups and above-mentioned multiple parts path, retrievalIn above-mentioned multiple parts path with the most similar part of some above-mentioned delegated paths path;

The characteristics of image of the above-mentioned selected moving body on the part path of selecting to obtain by above-mentioned retrievalAmount is as above-mentioned query image characteristic quantity.

The storage medium that the calculating function of 13. nonvolatiles according to claim 11 reads, its featureBe,

Above-mentioned memory device stores with from the first moving body of above-mentioned the first Extraction of Image and the second moving bodyThe information that mobile route is relevant,

Said procedure also makes above-mentioned processor carry out following steps: according to the image spy of above-mentioned the first moving bodyThe image feature amount of the amount of levying and above-mentioned the second moving body, be judged to be to detect from above-mentioned the first image above-mentionedIn the similar situation of image of the image of one moving body and above-mentioned the second moving body, will with above-mentioned the second moving bodyThe relevant information of mobile route merge in the information relevant to the mobile route of above-mentioned the first moving body.

The storage medium that the calculating function of 14. nonvolatiles according to claim 11 reads, its featureBe,

Said procedure also makes above-mentioned processor carry out following steps: the image of above-mentioned selected moving body and fromThe image of the moving body that above-mentioned the second image detects all comprises the region of the first predetermined kind, is being included inThe image in the region of above-mentioned the first kind in the above-mentioned image of selecting and being included in from above-mentioned the second imageThe image in the region of above-mentioned the first kind in the image of the moving body detecting is similar, and from above-mentioned secondThe image of the moving body of Extraction of Image also comprises in the situation in region of the second kind, output and above-mentioned the secondThe relevant information in region of class.

The storage medium that the calculating function of 15. nonvolatiles according to claim 11 reads, its featureBe,

Said procedure also makes above-mentioned processor carry out following steps: detecting multiple movements from some imagesThe image of body, and detect in the situation of multiple images of above-mentioned each moving body, above-mentioned each moving body selectedOne of multiple images, the above-mentioned image of selecting of above-mentioned each moving body is shown with other moving bodysThe above-mentioned image of selecting is not overlapping.