JP2015062090A - Video processing system, video processing method, video processing device for portable terminal or server, and control method and control program of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To notify a user in real time of results of recognition of a plurality of recognition objects in a video while maintaining the accuracy of the recognition.SOLUTION: A video processing device: stores a plurality of recognition objects and m1 to mk first local feature amounts each consisting of first to i1 to ik-th dimensional feature vectors in association with each other; extracts n feature amounts from an image in a video to create n second local feature amounts each consisting of first to j-th dimensional feature vectors; selects the smaller number of dimension from the number of dimension i1 and the number of dimension j, and when determined that a predetermined ratio or more of each of the m1 to mk first local feature amounts consisting of feature vectors of first to the selected number of dimension correspond to the n second local feature amounts consisting of feature vectors of first to the selected number of dimension, recognizes the presence of the plurality of recognition object with the predetermined ratio or more of correspondence in the image in the video; and displays information indicating the plurality of recognition objects in the image in the video in which the plurality of recognition objects are present.

Description

  The present invention relates to a technique for identifying a plurality of objects existing in a video in real time.

  In the above technical field, Patent Document 1 discloses that a shelf model of a product is accurately obtained in a short time by performing color correction, size correction, and distortion correction of a photographic image of a sales floor and matching with a product master of each product. A well-crafted technique is described.

JP 2009-187482 A

  However, since the techniques described in the above documents require color correction, size correction, and distortion correction of photographic images on the sales floor, it is impossible to identify and notify a plurality of products in real time.

  The objective of this invention is providing the technique which solves the above-mentioned subject.

In order to achieve the above object, an apparatus according to the present invention provides:
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , First local feature storage means for storing m1, m2,..., Mk first local feature amounts each consisting of feature vectors from one dimension to i1, i2,.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. Second local feature generating means for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, Recognition means for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
Display means for displaying information indicating a plurality of recognition objects recognized by the recognition means on an image in which the plurality of recognition objects exist in the video;
It is characterized by providing.

In order to achieve the above object, the method according to the present invention comprises:
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , M1, m2,..., Mk first local feature amounts each comprising feature vectors from one dimension to i1, i2,. A control method for a video processing apparatus,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
It is characterized by including.

In order to achieve the above object, a program according to the present invention provides:
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , M1, m2,..., Mk first local feature amounts each comprising feature vectors from one dimension to i1, i2,. A video processing device control program,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
Is executed by a computer.

In order to achieve the above object, a system according to the present invention provides:
A video processing system having a video processing device for a mobile terminal and a video processing device for a server connected via a network,
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , First local feature storage means for storing m1, m2,..., Mk first local feature amounts each consisting of feature vectors from one dimension to i1, i2,.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. Second local feature generating means for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, Recognition means for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
Display means for displaying information indicating a plurality of recognition objects recognized by the recognition means on an image in which the plurality of recognition objects exist in the video;
It is characterized by providing.

In order to achieve the above object, the method according to the present invention comprises:
A mobile terminal video processing apparatus and a server video processing apparatus connected via a network, each having a plurality of recognition objects and m1, m2,. ..., mk local regions including mk feature points, m1, m2,..., each of feature vectors from one dimension to i1, i2,. , Mk first local feature values in association with each other, and a first local feature value storing means for storing the image data in association with each other.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
It is characterized by including.

  ADVANTAGE OF THE INVENTION According to this invention, a recognition result can be alert | reported with respect to the several recognition target object in an image | video in real time, maintaining recognition accuracy.

It is a block diagram which shows the structure of the video processing apparatus which concerns on 1st Embodiment of this invention. It is a figure explaining the video processing by the video processing apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the function structure of the video processing apparatus which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the local feature-value production | generation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the process of the collation part which concerns on 2nd Embodiment of this invention. It is a figure which shows the structure of the local feature-value production | generation data concerning 2nd Embodiment of this invention. It is a figure which shows the structure of local feature-value DB which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the video processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the video processing apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the local feature-value production | generation process which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence of the collation process which concerns on 2nd Embodiment of this invention. It is a figure explaining the video processing by the video processing apparatus which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the function structure of the video processing apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows the structure of the multiple collation result holding part which concerns on 3rd Embodiment of this invention. It is a figure which shows the structure of combination identification DB which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the video processing apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the process sequence of the video processing apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the process sequence of the video collation process which concerns on 3rd Embodiment of this invention. It is a figure explaining the video processing by the video processing system which concerns on 4th Embodiment of this invention. It is a sequence diagram which shows the video processing procedure of the video processing system which concerns on 4th Embodiment of this invention. It is a block diagram which shows the function structure of the video processing apparatus for portable terminals which concerns on 4th Embodiment of this invention. It is a block diagram which shows the structure of the encoding part which concerns on 4th Embodiment of this invention. It is a block diagram which shows the function structure of the video processing apparatus for servers which concerns on 4th Embodiment of this invention. It is a figure which shows the structure of link information DB which concerns on 4th Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the video processing apparatus for portable terminals which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of the video processing apparatus for portable terminals which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of the encoding which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of the encoding of the difference value which concerns on 4th Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the video processing apparatus for servers which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of the video processing apparatus for servers which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of local feature-value DB production | generation processing which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process sequence of the recognition target object / link information acquisition process which concerns on 4th Embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. However, the constituent elements described in the following embodiments are merely examples, and are not intended to limit the technical scope of the present invention only to them.

[First Embodiment]
A video processing apparatus 100 as a first embodiment of the present invention will be described with reference to FIG. The video processing device 100 is a device that recognizes a plurality of recognition objects in an image in a video in real time while maintaining recognition accuracy.

  As shown in FIG. 1, the video processing apparatus 100 includes a first local feature quantity storage unit 110, a second local feature quantity generation unit 120, a recognition unit 130, and a display unit 140. The first local feature quantity storage unit 110 includes a plurality of recognition objects 111-1 to 111-k and m 1 including m 1, m 2,..., M k feature points in the images of the plurality of recognition objects. , M2,..., Mk local regions, each of m1, m2,..., Mk first local features 112 each consisting of a feature vector from one dimension to i1, i2,. −1 to 112-k are stored in association with each other. The second local feature generation unit 120 extracts 121 n feature points from the image 101 in the video, and each of the n local regions 122 including each of the n feature points from 1 to j dimensions. N second local feature values 123 of the feature vectors are generated. The recognizing unit 130 selects a smaller number of dimensions among the number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity. The recognizing unit 130 adds m1, m2,..., Mk first local feature quantities consisting of feature vectors up to the selected dimension number to n second local feature quantities consisting of feature vectors up to the selected dimension number. It is determined whether or not a predetermined ratio or more corresponds to each. If it is determined that the recognition unit 130 corresponds, the recognition unit 130 recognizes that there are a plurality of recognition objects corresponding to the image 101 in the video corresponding to a predetermined ratio or more. The display unit 140 displays information 141 indicating a plurality of recognition objects recognized by the recognition unit 130 on an image including a plurality of recognition objects in the video.

  According to this embodiment, a recognition result can be notified to a plurality of recognition objects in a video in real time while maintaining recognition accuracy.

[Second Embodiment]
Next, a video processing apparatus according to the second embodiment of the present invention will be described. In the present embodiment, the video processing device as a mobile terminal recognizes a plurality of recognition objects in the image being captured, and performs video processing for displaying the recognition result on the recognition object in the video in real time. explain. In this embodiment, an example will be described in which the names of recognition objects are displayed in real time. In the present embodiment, processing for video captured by a mobile terminal will be described. However, video is similarly applied to video content reproduction processing and broadcast program viewing.

  According to this embodiment, while a user is viewing a video, it is possible to notify the user of the recognition results for a plurality of recognition objects in the video on the video in real time while maintaining the recognition accuracy.

<< Description of Video Processing According to this Embodiment >>
FIG. 2 is a diagram for explaining video processing by the video processing apparatus 200 according to the present embodiment. Although FIG. 2 shows one processing example, the present invention is not limited to this.

  In FIG. 2, when a user such as a tourist wants to know a building such as a tower or a building in the field of view, or wants to know a destination or his / her current location, the mobile terminal captures an image including a recognition target object. An example is given. On the display screen 210 of the video processing apparatus 200 as a portable terminal, as shown in the left diagram of FIG. 2, a video display area 211 being imaged and an instruction button display area 212 of the touch panel are displayed. Note that the building displayed in the video display area 211 is a still image (photograph) that is displayed as it is with the video being captured. In the present embodiment, real-time recognition processing is performed on the display video, and the names 222 to 224 of the plurality of buildings are displayed in the video display area 221 of the right display screen 220 in FIG. The The user can know the destination and his / her current location from the video display area 221 without, for example, sightseeing guidance.

《Functional configuration of video processing device》
FIG. 3 is a block diagram showing a functional configuration of the video processing apparatus 200 according to the present embodiment.

  The video processing apparatus 200 includes an imaging unit 310 that acquires video. The captured image is displayed on the display unit 360 and input to the local feature amount generation unit 320. The local feature value generating unit 320 generates a local feature value from the captured video (refer to FIG. 4A for details). In the local feature DB 330, local feature values generated by an algorithm similar to the local feature generating unit 320 in advance from individual recognition objects, such as a single unit such as the sky tree / building of FIG. 2, are associated with the recognition objects. Stored. The contents of the local feature DB 330 may be received by the local feature receiver 380 via the communication controller 390.

  The collation unit 340 collates whether there is data corresponding to the local feature stored in the local feature DB 330 in the local feature generated by the local feature generation unit 320 from the captured video. . If there is corresponding data, collator 340 determines that there is a recognition object in the captured video. Note that the correspondence between local feature amounts not only means that the same local feature amount exists, but may also include determining whether or not the order and arrangement can be acquired from the same object (FIG. 4G). reference).

  The verification result generation unit 350 generates data to be displayed on the display unit 360 from the verification result of the verification unit 340. Such data includes data such as names of recognition objects and recognition errors. The display unit 360 displays the collation result superimposed on the video imaged by the imaging unit 310. In addition, the data generated by the verification result generation unit 350 may be transmitted to the outside via the communication control unit 390. The operation unit 370 includes keys and a touch panel (such as the instruction button in FIG. 2) of the video processing device 200, and operates the operation of the video processing device 200 such as the imaging unit 310.

  Note that the video processing apparatus 200 according to the present embodiment is not limited to a video that is being captured, and can be applied to a video that is being played back or a video that is being broadcast. In that case, the imaging unit 310 may be replaced with a video reproduction unit or a video reception unit.

<< Local feature generator >>
FIG. 4A is a block diagram illustrating a configuration of the local feature value generation unit 320 according to the present embodiment.

  The local feature quantity generation unit 320 includes a feature point detection unit 411, a local region acquisition unit 412, a sub region division unit 413, a sub region feature vector generation unit 414, and a dimension selection unit 415.

  The feature point detection unit 411 detects a large number of characteristic points (feature points) from the image data, and outputs the coordinate position, scale (size), and angle of each feature point.

  The local region acquisition unit 412 acquires a local region where feature amount extraction is performed from the coordinate value, scale, and angle of each detected feature point.

  The sub area dividing unit 413 divides the local area into sub areas. For example, the sub-region dividing unit 413 can divide the local region into 16 blocks (4 × 4 blocks) or divide the local region into 25 blocks (5 × 5 blocks). The number of divisions is not limited. In the present embodiment, the case where the local area is divided into 25 blocks (5 × 5 blocks) will be described below as a representative.

  The sub-region feature vector generation unit 414 generates a feature vector for each sub-region of the local region. As the feature vector of the sub-region, for example, a gradient direction histogram can be used.

  The dimension selection unit 415 selects a dimension to be output as a local feature amount based on the positional relationship between the sub-regions so that the correlation between the feature vectors of the adjacent sub-regions is low (for example, dimension deletion or thinning is performed). To do). In addition, the dimension selection unit 415 can not only select a dimension but also determine a selection priority. That is, for example, the dimension selection unit 415 can select dimensions with priorities so that dimensions in the same gradient direction are not selected between adjacent sub-regions. Then, the dimension selection unit 415 outputs a feature vector composed of the selected dimensions as a local feature amount. Note that the dimension selection unit 415 can output the local feature amount in a state where the dimensions are rearranged based on the priority order.

<< Processing of local feature generator >>
4B to 4F are diagrams illustrating processing of the local feature value generation unit 320 according to the present embodiment.

  First, FIG. 4B is a diagram illustrating a series of processing of feature point detection / local region acquisition / sub-region division / feature vector generation in the local feature amount generation unit 320. Such a series of processing is described in US Pat. No. 6,711,293, David G. Lowe, “Distinctive image features from scale-invariant key points” (US), International Journal of Computer Vision, 60 (2), 2004. See Years, p. 91-110.

(Feature point detector)
421 in FIG. 4B is a diagram illustrating a state in which feature points are detected from an image in the video in the feature point detection unit 411 in FIG. 4A. Hereinafter, generation of a local feature amount will be described by using one feature point 421a as a representative. The starting point of the arrow of the feature point 421a indicates the coordinate position of the feature point, the length of the arrow indicates the scale (size), and the direction of the arrow indicates the angle. Here, as the scale (size) and direction, brightness, saturation, hue, and the like can be selected according to the target image. In the example of FIG. 4B, the case of six directions at intervals of 60 degrees is described, but the present invention is not limited to this.

(Local area acquisition unit)
For example, the local region acquisition unit 412 in FIG. 4A generates a Gaussian window 422a around the starting point of the feature point 421a, and generates a local region 422 that substantially includes the Gaussian window 422a. In the example of FIG. 4B, the local region acquisition unit 412 generates the square local region 422, but the local region may be circular or have another shape. This local region is acquired for each feature point.

(Sub-region division part)
Next, a state in which the scale and angle of each pixel included in the local region 422 of the feature point 421a is divided into sub regions 423 in the sub region dividing unit 413 is shown. FIG. 4B shows an example in which 4 × 4 = 16 pixels are divided into 5 × 5 = 25 subregions. However, the sub-region may be 4 × 4 = 16, other shapes, or the number of divisions.

(Sub-region feature vector generator)
The sub-region feature vector generation unit 414 generates and quantizes the histogram of each pixel in the sub-region in units of angular directions in eight directions to obtain the sub-region feature vector 424. That is, the direction is normalized with respect to the angle output by the feature point detection unit 411. Then, the sub-region feature vector generation unit 414 aggregates the frequencies in the eight directions quantized for each sub-region, and generates a histogram. In this case, the sub-region feature vector generation unit 414 outputs a feature vector constituted by a histogram of 25 sub-region blocks × 6 directions = 150 dimensions generated for each feature point. In addition, the gradient direction is not only quantized to 8 directions, but may be quantized to an arbitrary quantization number such as 4 directions, 8 directions, and 10 directions. When the gradient direction is quantized in the D direction, if the gradient direction before quantization is G (0 to 2π radians), the quantized value Qq (q = 0,..., D−1) in the gradient direction is, for example, Although it can obtain | require by (1), Formula (2), etc., it is not restricted to this.

Qq = floor (G × D / 2π) (1)
Qq = round (G × D / 2π) mod D (2)
Here, floor () is a function for rounding off the decimal point, round () is a function for rounding off, and mod is an operation for obtaining a remainder. Further, when generating the gradient histogram, the sub-region feature vector generation unit 414 may add up the magnitudes of the gradients instead of adding up the simple frequencies. Further, when the sub-region feature vector generation unit 414 aggregates the gradient histogram, the sub-region feature vector generation unit 414 assigns weight values not only to the sub-region to which the pixel belongs, but also to sub-regions adjacent to each other (such as adjacent blocks) according to the distance between the sub-regions. You may make it add. Further, the sub-region feature vector generation unit 414 may add weight values to gradient directions before and after the quantized gradient direction. Note that the feature vector of the sub-region is not limited to the gradient direction histogram, and may be any one having a plurality of dimensions (elements) such as color information. In the present embodiment, it is assumed that a gradient direction histogram is used as the feature vector of the sub-region.

(Dimension selection part)
Next, processing will be described in the dimension selection unit 415 in the local feature amount generation unit 320 according to FIGS. 4C to 4F.

  The dimension selection unit 415 selects (decimates) a dimension (element) to be output as a local feature amount based on the positional relationship between the sub-regions so that the correlation between feature vectors of adjacent sub-regions becomes low. More specifically, for example, the dimension selection unit 415 selects dimensions so that at least one gradient direction differs between adjacent sub-regions. In the present embodiment, the dimension selection unit 415 mainly uses adjacent sub-regions as adjacent sub-regions. However, the adjacent sub-regions are not limited to adjacent sub-regions, for example, from the target sub-region. A sub-region within a predetermined distance may be a nearby sub-region.

  FIG. 4C shows an example of selecting a dimension from a feature vector 431 of a 150-dimensional gradient histogram generated by dividing a local region into 5 × 5 block sub-regions and quantizing gradient directions into six directions 431a. FIG. In the example of FIG. 4C, dimensions are selected from feature vectors of 150 dimensions (5 × 5 = 25 sub-region blocks × 6 directions).

(Dimension selection of local area)
FIG. 4C is a diagram illustrating a state of a feature vector dimension number selection process in the local feature value generation unit 320.

  As shown in FIG. 4C, when selecting a half 75-dimensional gradient histogram feature vector 432 from the 150-dimensional gradient histogram feature vector 431, the dimension selection unit 415 selects adjacent left, right, upper and lower sub-region blocks. The dimension can be selected so that the same gradient direction dimension is not selected.

  In this example, when the quantized gradient direction in the gradient direction histogram is q (q = 0, 1, 2, 3, 4, 5), a block for selecting elements of q = 0, 2, 4 and , Q = 1, 3, and 5 are alternately arranged with sub-region blocks for selecting elements. In the example of FIG. 4C, when the gradient directions selected in the adjacent sub-region blocks are combined, there are six directions.

  In addition, when the dimension selection unit 415 selects the feature vector 433 of the 50-dimensional gradient histogram from the feature vector 432 of the 75-dimensional gradient histogram, only one direction exists between the sub-region blocks located at an angle of 45 degrees. The dimensions can be selected to be identical (the remaining one direction is different).

  In addition, when the dimension selection unit 415 selects the feature vector 434 of the 25-dimensional gradient histogram from the feature vector 433 of the 50-dimensional gradient histogram, the gradient direction selected between the sub-region blocks located at an angle of 45 degrees. Dimension can be selected so that does not match. In the example shown in FIG. 4C, the dimension selection unit 415 selects one gradient direction from each sub-region from the first dimension to the 25th dimension, selects two gradient directions from the 26th dimension to the 50th dimension, and starts from the 51st dimension. Three gradient directions are selected up to 75 dimensions.

  In this way, it is desirable that the gradient directions are selected uniformly so that the gradient directions do not overlap between adjacent sub-region blocks. At the same time, as in the example shown in FIG. 4C, it is desirable that dimensions be selected uniformly from the entire local region. Note that the dimension selection method illustrated in FIG. 4C is an example, and is not limited to this selection method.

(Local area priority)
FIG. 4D is a diagram illustrating an example of the selection order of feature vectors from sub-regions in the local feature value generation unit 320.

  The dimension selection unit 415 can determine the priority of selection so as to select not only the dimensions but also the dimensions that contribute to the features of the feature points in order. That is, the dimension selection unit 415 can select dimensions with priorities so that, for example, dimensions in the same gradient direction are not selected between adjacent sub-area blocks. Then, the dimension selection unit 415 outputs a feature vector composed of the selected dimensions as a local feature amount. Note that the dimension selection unit 415 can output the local feature amount in a state where the dimensions are rearranged based on the priority order.

  That is, the dimension selection unit 415 selects between 1 to 25 dimensions, 26 dimensions to 50 dimensions, and 51 dimensions to 75 dimensions, for example, by adding dimensions in the order of the sub-region blocks as indicated by 441 in FIG. 4D. You may do it. When the priority order indicated by 441 in FIG. 4D is used, the dimension selection unit 415 can select the gradient direction by increasing the priority order of the sub-region blocks close to the center.

  451 in FIG. 4E is a diagram illustrating an example of element numbers of 150-dimensional feature vectors in accordance with the selection order in FIG. 4D. In this example, 5 × 5 = 25 blocks are represented by numbers p (p = 0, 1,..., 25) in raster scan order, and the quantized gradient direction is represented by q (q = 0, 1, 2, 3, 4). , 5), the element number of the feature vector is 6 × p + q.

  461 in FIG. 4F is a diagram showing that the 150-dimensional order according to the selection order in FIG. 4E is hierarchized in units of 25 dimensions. That is, 461 in FIG. 4F is a diagram showing a configuration example of local feature amounts obtained by selecting the elements shown in FIG. 4E according to the priority order shown in 441 in FIG. 4D. The dimension selection unit 415 can output dimension elements in the order shown in FIG. 4F. Specifically, for example, when outputting a 150-dimensional local feature amount, the dimension selection unit 415 can output all 150-dimensional elements in the order shown in FIG. 4F. When the dimension selection unit 415 outputs, for example, a 25-dimensional local feature amount, the first line (76th, 45th, 83rd,..., 120th) element 471 shown in FIG. 4F is shown in FIG. 4F. Can be output in order (from left to right). For example, when outputting a 50-dimensional local feature amount, the dimension selecting unit 415 adds the element 472 in the second row shown in FIG. 4F in the order shown in FIG. To the right).

  Incidentally, in the example shown in FIG. 4F, the local feature amount has a hierarchical structure. That is, for example, in the 25-dimensional local feature value and the 150-dimensional local feature value, the arrangement of the elements 471 to 476 in the first 25-dimensional local feature value is the same. In this way, the dimension selection unit 415 selects a dimension hierarchically (progressively), and thereby, depending on the application, communication capacity, terminal specification, etc., the local feature quantity of an arbitrary number of dimensions, that is, the local size of an arbitrary size. Feature quantities can be extracted and output. Further, the dimension selection unit 415 can hierarchically select dimensions, rearrange the dimensions based on the priority order, and output them, thereby performing image matching using local feature amounts of different dimensions. . For example, when images are collated using a 75-dimensional local feature value and a 50-dimensional local feature value, the distance between the local feature values can be calculated by using only the first 50 dimensions.

  Note that the priorities shown in FIG. 4D from 441 to FIG. 4F are examples, and the order of selecting dimensions is not limited to this. For example, regarding the order of blocks, in addition to the example of 441 in FIG. 4D, the order as shown in 442 in FIG. 4D and 443 in FIG. 4D may be used. Further, for example, the priority order may be set so that dimensions are selected from all the sub-regions. Also, the vicinity of the center of the local region may be important, and the priority order may be determined so that the selection frequency of the sub-region near the center is increased. Further, the information indicating the dimension selection order may be defined in the program, for example, or may be stored in a table or the like (selection order storage unit) referred to when the program is executed.

  In addition, the dimension selection unit 415 may select a dimension by selecting one sub-region block. That is, 6 dimensions are selected in a certain sub-region, and 0 dimensions are selected in other sub-regions close to the sub-region. Even in such a case, it can be said that the dimension is selected for each sub-region so that the correlation between adjacent sub-regions becomes low.

  Further, the shape of the local region and the sub-region is not limited to a square, and can be an arbitrary shape. For example, the local region acquisition unit 412 may acquire a circular local region. In this case, the sub-region dividing unit 413 can divide the circular local region into, for example, nine or seventeen sub-regions into concentric circles having a plurality of local regions. Even in this case, the dimension selection unit 415 can select a dimension in each sub-region.

  As described above, as illustrated in FIGS. 4B to 4F, according to the local feature value generation unit 320 of this embodiment, the dimensions of the feature vectors generated while maintaining the information amount of the local feature values are hierarchically selected. The This processing enables real-time object recognition and recognition result display while maintaining recognition accuracy. Note that the configuration and processing of the local feature value generation unit 320 are not limited to this example. Naturally, other processes that enable real-time object recognition and recognition result display while maintaining recognition accuracy can be applied.

<Verification part>
FIG. 4G is a diagram illustrating processing of the matching unit 340 according to the present embodiment.

  FIG. 4G is a diagram illustrating a collation example for recognizing a building in the video of FIG. Local feature amounts generated according to the present embodiment from recognition objects (in this example, buildings including a sky tree, a XX building, a XX railway, and a ΔΔ gymnasium) are stored in the local feature amount DB 330. On the other hand, a local feature amount is generated from the video display area 231 in the display screen 230 of the video processing apparatus 200 as the mobile terminal in the left diagram according to the present embodiment. And it is collated whether each of the local feature-values stored in local feature-value DB330 is in the local feature-value produced | generated from the image | video display area 231. FIG.

  As illustrated in FIG. 4G, the matching unit 340 associates the local feature amounts 481 to 484 stored in the local feature amount DB 330 with the feature points that match the local feature amounts so as to be thin lines. Note that the matching unit 340 determines that the feature points match when a predetermined ratio or more of the local feature amounts match. And the collation part 340 will recognize that it is a recognition target object, if the positional relationship of the related feature point is a linear relationship. If recognition is performed using a set of corresponding feature points, recognition can be performed by size, direction difference (difference in viewpoint), or inversion. In addition, since recognition accuracy can be obtained if there are a predetermined number or more of associated feature points, recognition objects can be recognized even if a part of them is hidden from view. Here, the local feature amount matching conditions and the feature point conditions for recognition may be the same, or different conditions may be set when the recognition targets are different.

  In the matching process of the matching unit 340 according to the present embodiment, matching is performed based on the feature point coordinates and the local feature amount. The matching is performed from the local feature amount generated from the matching recognition target object and the image in the video. Recognition is possible only by the linear relationship of the arrangement order with the determined local feature quantity. On the other hand, although this embodiment has been described with a two-dimensional image, the same processing can be performed even if three-dimensional feature point coordinates are used.

(Local feature generation data)
FIG. 5 is a diagram showing a configuration of local feature value generation data 500 according to the present embodiment. These data are stored and held in the RAM 740 of FIG.

  In the local feature quantity generation data 500, a plurality of detected feature points 502, feature point coordinates 503, and local region information 504 corresponding to the feature points are stored in association with the input image ID 501. A selection dimension 508 including a plurality of sub-region IDs 505, sub-region information 506, a feature vector 507 corresponding to each sub-region, and a priority order in association with each detected feature point 502, feature point coordinates 503 and local region information 504. Is memorized.

  A local feature quantity 509 generated for each detected feature point 502 from the above data is stored.

(Local feature DB)
FIG. 6 is a diagram showing a configuration of the local feature DB 330 according to the present embodiment.

  In the local feature amount DB 330, local feature amounts of the respective recognition objects are stored in different feature points and / or different feature vector dimensions suitable for recognition of the recognition object.

  For example, since the local feature amount DB 331 has a low correlation with other recognition objects, the local feature amount is stored by 25 feature points / 25 dimensions. Further, the local feature DB 332 stores the local feature amount by 50 feature points / 25 dimensions because there is little correlation with other recognition objects. Further, the local feature DB 333 stores local feature values based on the number of feature points 100 / number of dimensions 50 with correlation with other recognition objects. In addition, since the local feature DB 334 has a high correlation with other recognition objects, the local feature is stored with 150 feature points / 50 dimensions.

  Each local feature value DB 331 to 334 stores the m-th local feature value from the first local feature value in association with the recognition object ID and the recognition object name. Note that m is a positive integer and may be a different number corresponding to the recognition object. In the present embodiment, the feature point coordinates used for the matching process are stored together with the respective local feature amounts.

<< Hardware configuration of video processing device >>
FIG. 7 is a block diagram showing a hardware configuration of the video processing apparatus 200 according to the present embodiment.

  In FIG. 7, a CPU 710 is a processor for arithmetic control, and implements each functional component of the video processing device 200 that is a portable terminal by executing a program. The ROM 720 stores initial data and fixed data such as programs and programs. The communication control unit 390 is a communication control unit, and in the present embodiment, communicates with other devices via a network. Note that the number of CPUs 710 is not limited to one, and may be a plurality of CPUs or may include a graphics processing unit (GPU) for image processing.

  The RAM 740 is a random access memory that the CPU 710 uses as a work area for temporary storage. In the RAM 740, an area for storing data necessary for realizing the present embodiment is secured. The input video 741 is an area for storing the input video input by the imaging unit 310. The feature point data 742 is an area for storing feature point data including feature point coordinates, scales, and angles detected from the input video 741. The local feature quantity generation table 500 is an area for storing the local feature quantity generation table shown in FIG. The collation result 743 is an area for storing a collation result including a plurality of recognition objects recognized from the collation between the local feature amount generated from the input video and the local feature amount stored in the local feature amount DB 330. The collation result display data 744 is an area for storing collation result display data for notifying the user of the collation result 743. In addition, when outputting a voice, collation result voice data may be included. The input video / collation result superimposition data 745 is an area for storing input video / collation result superimposition data displayed on the display unit 360 in which the collation result 743 is superimposed on the input video 741. The input / output data 746 is an area for storing input / output data input / output via the input / output interface 760. Transmission / reception data 747 is an area for storing transmission / reception data transmitted / received via the communication control unit 390.

  The storage 750 stores a database, various parameters, or the following data or programs necessary for realizing the present embodiment. The local feature DB 330 is an area in which the local feature DB shown in FIG. 6 is stored. The collation result display format 751 is an area in which a collation result display format used for generating a format for displaying the collation result is stored.

  The storage 750 stores the following programs. The portable terminal control program 752 is an area in which a portable terminal control program that controls the entire video processing apparatus 200 is stored. The local feature value generation module 753 is an area in which a local feature value generation module that generates a local feature value from an input video according to FIGS. 4B to 4F in the mobile terminal control program 752 is stored. The collation control module 754 is an area in which the collation control module that collates the local feature amount generated from the input video and the local feature amount stored in the local feature amount DB 330 in the portable terminal control program 752 is stored. The verification result notification module 755 is an area in which a verification result notification module for displaying the verification result to the user by display or voice is stored.

  The input / output interface 760 interfaces input / output data with input / output devices. The input / output interface 760 is connected with a display unit 360, a touch panel and keyboard as the operation unit 370, a speaker 764, a microphone 765, and an imaging unit 310. The input / output device is not limited to the above example. In addition, a GPS (Global Positioning System) position generation unit 766 acquires a current position based on a signal from a GPS satellite.

  FIG. 7 shows only data and programs essential to the present embodiment, and does not illustrate data and programs not related to the present embodiment.

《Processing procedure of video processing device》
FIG. 8 is a flowchart showing a processing procedure of the video processing apparatus 200 according to the present embodiment. This flowchart is executed by the CPU 710 in FIG. 7 using the RAM 740, and implements each functional component in FIG.

  First, in step S811, it is determined whether or not there is a video input for performing object recognition. As a function of the mobile terminal, reception is determined in step S821, and transmission is determined in step S831. Otherwise, other processing is performed in step S841.

  If there is video input, the process proceeds to step S813, and local feature generation processing is executed from the input video (see FIG. 9A). Next, in step S815, collation processing is executed (see FIG. 9B). In step S817, the result of the collation process is superimposed on the input video, and the video / collation result superimposed display process is executed. In step S819, it is determined whether or not to finish the object recognition process. The end is performed, for example, by a reset button in the instruction button display area 212 of FIG. If not completed, the process returns to step S813 to repeat the object recognition for video input.

  When receiving and downloading local feature DB data, the local feature DB data is received in step S823 and stored in the local feature DB in step S825. On the other hand, if it is data reception as another portable terminal, reception processing is performed in step S827. In addition, when uploading local feature DB data, the local feature generated from the input video is transmitted as local feature DB data in step S833. On the other hand, if it is data transmission as another portable terminal, transmission processing is performed in step S835. The data transmission / reception processing as a portable terminal is not a feature of the present embodiment, and thus detailed description thereof is omitted.

(Local feature generation processing)
FIG. 9A is a flowchart illustrating a processing procedure of local feature generation processing S813 according to the present embodiment.

  First, in step S911, the position coordinates, scale, and angle of the feature points are detected from the input video. In step S913, a local region is acquired for one of the feature points detected in step S911. Next, in step S915, the local area is divided into sub-areas. In step S917, a feature vector for each sub-region is generated to generate a feature vector for the local region. The processing from step S911 to S917 is illustrated in FIG. 4B.

  Next, in step S919, dimension selection is performed on the feature vector of the local region generated in step S917. The dimension selection is illustrated in FIGS. 4D to 4F.

  In step S921, it is determined whether the generation of local feature values and dimension selection have been completed for all feature points detected in step S911. If not completed, the process returns to step S913, and the process is repeated for the next one feature point.

(Verification process)
FIG. 9B is a flowchart showing the processing procedure of the collation processing S815 according to the present embodiment.

  First, in step S931, parameters p = 1 and q = 0 are set as initialization. Next, in step S933, the dimension number j of the local feature amount generated in step S813 is acquired.

  In the loop of steps S935 to S945, the collation of each local feature amount is repeated until p> m (m = number of feature points of recognition target object). First, in step S935, data of the dimension number j of the p-th local feature amount of the recognition target stored in the local feature amount DB 330 is acquired. That is, the j dimension is acquired from the first one dimension. Next, in step S937, the p-th local feature amount acquired in step S935 and the local feature amounts of all feature points generated from the input video are sequentially checked to determine whether or not they are similar. In step S939, it is determined whether or not the similarity exceeds the threshold value α from the result of collation between the local feature amounts. If so, in step S941, the local feature amount matches the input video and the recognition target object. A combination with the positional relationship of feature points is stored. Then, q, which is a parameter for the number of matched feature points, is incremented by one. In step S943, the feature point of the recognition target object is advanced to the next feature point (p ← p + 1). If all feature points of the recognition target object have not been matched (p ≦ m), the process returns to step S935. Repeat matching of matching local features. Note that the threshold value α can be changed according to the recognition accuracy required by the recognition object. Here, if the recognition object has a low correlation with other recognition objects, accurate recognition is possible even if the recognition accuracy is lowered.

  When collation with all the feature points of the recognition target object is completed, the process proceeds from step S945 to S947. In steps S947 to S953, it is determined whether or not the recognition target object exists in the input video. First, in step S947, it is determined whether or not the ratio of the feature point number q that matches the local feature amount of the feature point of the input video among the feature point number p of the recognition target object exceeds the threshold value β. If exceeded, the process proceeds to step S949, and it is further determined as a recognition object candidate whether the positional relationship between the feature point of the input video and the feature point of the recognition object is a relationship that allows linear transformation. . That is, the positional relationship between the feature point of the input image and the feature point of the recognition target that is stored as the local feature amount matches in step S941 is possible even by a change such as rotation, inversion, or change of the viewpoint position. It is determined whether it is a positional relationship that is impossible or impossible. Since such a determination method is geometrically known, detailed description thereof is omitted. In step S951, if linear conversion is possible based on the determination result of whether or not shaping conversion is possible, the process proceeds to step S953, where it is determined that the collated recognition target object exists in the input video. The threshold value β can be changed according to the recognition accuracy required by the recognition object. Here, accurate recognition is possible even if there are few matching feature points as long as the recognition object has a low correlation with other recognition objects or a feature can be determined even from a part. That is, even if a part is hidden and cannot be seen, or if a characteristic part is visible, the object can be recognized.

  In step S955, it is determined whether or not an unmatched recognition target remains in the local feature DB 330. If there is still a recognition object, the next recognition object is set in step S957, initialized to parameters p = 1 and q = 0, and the process returns to step S935 to repeat the collation.

  As is clear from the description of the collation processing, the processing for storing recognition objects in all fields in the local feature DB 330 and collating all the recognition objects with the mobile terminal is very heavy. Therefore, for example, before the object recognition from the input video, it is conceivable that the user selects the field of the object from the menu, searches the field from the local feature DB 330, and collates the field. The load can also be reduced by downloading only the local feature amount of the field used by the user (for example, a building in the example of FIG. 2) to the local feature amount DB 330.

[Third Embodiment]
Next, a video processing apparatus according to the third embodiment of the present invention will be described. The video processing apparatus according to the present embodiment is different from the second embodiment in that it recognizes what the entire video represents by combining a plurality of recognition objects recognized from the video. For example, the present location and the destination are recognized from the recognition of a plurality of buildings. Alternatively, a product is recognized from recognition of a plurality of parts. Other configurations and operations are the same as those of the second embodiment, and thus description of the same configurations and operations is omitted.

  According to the present embodiment, while the user is viewing the video, the recognition result of the recognition target object represented by the entire video can be notified to the user in real time on the video while maintaining the recognition accuracy.

<< Video processing according to this embodiment >>
FIG. 10 is a diagram for explaining video processing by the video processing apparatus 1000 according to the present embodiment.

  First, the upper part of FIG. 10 shows an image including an object to be recognized on a mobile terminal when a user such as a tourist wants to know a building such as a tower or a building in the field of view or wants to know a destination or his / her current location. The example which imaged is shown.

  On the display screen 1010 of the video processing apparatus 1000 as a portable terminal, a video display area 1011 during imaging and an instruction button display area 1012 on the touch panel are displayed as shown in the upper left diagram of FIG. Note that the building displayed in the video display area 1011 is the image being captured as it is, and is not a still image (photo).

  In this embodiment, real-time recognition processing is performed on the display image, and the name 1022 of each building is displayed in the image display area 1021 of the center display screen 1020 in the upper part of FIG.

  Furthermore, in the present embodiment, the current location of the user imaged by the video processing apparatus 1000 is recognized from the arrangement, distance, angle, and the like in the video of the plurality of recognized buildings, and the right side in the upper part of FIG. In the video display area 1031 of the display screen 1030, the current location mark 1033 on the map around the current location and the address 1032 of the current location are displayed in real time. Alternatively, sound may be output from a speaker. The user can know the destination and his / her current location from the video display area 1031 without, for example, sightseeing guidance.

  The lower part of FIG. 10 shows an example in which, when a product is imaged, each part in the video is recognized and a product having these parts is recognized and displayed.

  On the display screen 1040 of the video processing apparatus 1000 as a portable terminal, a video display area 1041 being imaged and an instruction button display area 1042 on the touch panel are displayed as shown in the left diagram in the lower part of FIG. Note that the product displayed in the video display area 1041 is a still image (photograph) that is displayed as it is with the video being captured.

  In the present embodiment, real-time recognition processing is performed on the display image, and the name 1052 of each component is displayed in the image display area 1051 of the center display screen 1050 in the lower part of FIG.

  Furthermore, in the present embodiment, the product imaged by the video processing apparatus 1000 is recognized from the arrangement, distance, angle, and the like of the recognized plurality of parts in the video, and the right display screen 1060 in the lower part of FIG. The product name 1062 is displayed in real time in the video display area 1061. Alternatively, sound may be output from a speaker. For example, the product can be known from the video display area 1061.

  The recognition target is not limited to the above example. Any video can be applied as long as the entire video can be recognized from the arrangement of the recognition objects.

《Functional configuration of video processing device》
FIG. 11 is a block diagram showing a functional configuration of the video processing apparatus 1000 according to the present embodiment. Note that the functional configuration of the video processing device 1000 is a configuration in which a configuration for displaying related information is added to the video processing device 200 of the second embodiment, and therefore, the same reference numerals are given to the same components, and the description will be omitted. Omitted.

  The video processing apparatus 1000 includes a multiple verification result holding unit 1110 that holds a plurality of verification results in the video by the verification unit 340 (see FIG. 12). Moreover, it has combination identification DB1120 for recognizing an image | video from the combination of a some collation result (refer FIG. 13).

  If there is an audio output, it is output from the speaker. Further, the combination information stored in the combination identification DB 1120 may be downloaded via the communication control unit 390 (not shown in FIG. 11).

(Multiple verification result holding part)
FIG. 12 is a diagram illustrating a configuration of the multiple matching result holding unit 1110 according to the present embodiment.

  The multiple matching result holding unit 1110 stores a position 1203, a direction (angle) 1204, and a size (distance) 1205 in the video in association with the recognition target object ID 1201 and the recognition target name 1202 of the matching result.

(Combination identification DB 1120)
FIG. 13 is a diagram showing a configuration of the combination identification DB 1120 according to the present embodiment.

  There are two types of combination identification DB 1120. One is a combination identification DB 1310, which stores a recognizable image even if the positional relationship among a plurality of recognition objects is unknown. The combination identification DB 1310 stores a collation video ID 1312 and a collation video name 1313 in association with a combination of a plurality of recognition objects 1311.

  The other is a combination identification DB 1320 that stores video recognition in consideration of the positional relationship between a plurality of recognition objects. The combination identification DB 1320 stores a collation video ID 1322 and a collation video name 1323 in association with combinations of a plurality of recognition objects and positions 1321.

<< Hardware configuration of video processing device >>
FIG. 14 is a block diagram showing a hardware configuration of the video processing apparatus 1000 according to the present embodiment. The difference of the configuration of the video processing apparatus 1000 from the configuration of FIG. 7 of the second embodiment is the configuration of combination verification of a plurality of recognition objects. Since other configurations are the same, the same reference numerals as those in FIG.

  A plurality of collation results shown in FIG. The combination matching result 1441 is an area for storing the combination matching result. And the combination identification DB 1120 of the storage 1450 is a bizarre in which the combination identification DB shown in FIG. 13 is stored. The combination identification DB 1452 is an area in which the portable terminal control program of this embodiment is stored (see FIG. 15). Video collation control module 1456 is an area in which video collation control module for collating video from a plurality of recognition objects is stored in portable terminal control program 1452 (see FIG. 16).

《Processing procedure of video processing device》
FIG. 15 is a flowchart showing a processing procedure of the video processing apparatus 1000 according to the present embodiment. This flowchart is executed by the CPU 710 in FIG. 14 using the RAM 740, and implements each functional component in FIG.

  In FIG. 15, the difference from the second embodiment shown in FIG. 8 is the addition of video collation processing using a plurality of recognition objects. Since the other processes are the same as those in FIG. 8, the same steps are denoted by the same step numbers and description thereof is omitted.

  After the matching process in step S815, in step S1501, a video matching process using a plurality of recognition objects is performed (see FIG. 16).

(Video verification process)
FIG. 16 is a flowchart showing a processing procedure of the video collation processing S1501 according to the present embodiment.

  First, in step S1601, a plurality of recognition objects (or positions) recognized from the video are acquired. In step S1603, with reference to the combination identification DB 1120, video is collated from combinations of a plurality of recognition objects (positions).

In step S1605, it is determined whether there is an image to be verified. If there is a collation video, the process proceeds to step S1607 to acquire information related to the collation video and return.
[Fourth Embodiment]
Next, a video processing system according to the fourth embodiment of the present invention will be described. Compared with the second embodiment and the third embodiment described above, the video processing system according to the present embodiment generates a local feature amount of video and transmits it to the server, and the recognition target object recognized by the server. And receiving the recognition result of the entire video. In communication between the mobile terminal and the server of this embodiment, capacity reduction in local feature generation of this embodiment is effective for real-time processing.

  According to the present embodiment, the load on the mobile terminal is reduced, and while the user is viewing the video, the recognition target object and the recognition result or the link information in the video are displayed on the video while maintaining the recognition accuracy. Can inform the user in real time.

<< Video processing according to this embodiment >>
FIG. 17 is a diagram for explaining video processing by the video processing system 1700 according to the present embodiment.

  The video processing system 1700 does not perform video input, local feature generation processing, and collation processing in a self-contained manner, unlike the video processing device that is a portable terminal shown in the second and third embodiments. That is, the video processing device 1710 that is a portable terminal performs video input and local feature generation processing, and the video processing device 1720 that is a collation server connected by the network 1770 performs processing with a large load such as collation. In such processing, the size of the local feature amount transferred over the network 1770 affects the verification speed and communication traffic. 4A to 4F according to the present embodiment, the capacity reduction of the local feature amount that maintains the accuracy can be realized, and the object recognition in the input video, the video recognition, and the superimposed display of the recognition result on the input video can be performed in real time. . In the present embodiment, the object recognition in the input video and the superimposed display of the recognition result on the input video can be performed at higher speed by the encoding process.

  A video processing system 1700 in FIG. 17 includes a video processing device 1710 that is a terminal device, and a video processing device 1720 that is a collation server connected to the video processing device 1710 via a network 1770. Also connected to the network 1770 is a service providing server 1730 that holds service information and provides the user with link information.

  The video processing device 1720, which is a collation server, includes a local feature DB 1721 used for collation processing of recognition objects, a combination identification DB 1722 used for video collation processing, and a link information DB 1723 used for providing link information. .

  FIG. 17 shows one processing example of the video processing device 1710 which is a terminal device. FIG. 10 shows an example in which when a product is imaged, each component in the video is recognized and a product having the plurality of components is recognized and displayed.

  On the display screen 1740 of the video processing apparatus 1710 as a portable terminal, as shown in the left figure, a video display area 1741 during imaging and an instruction button display area 1742 on the touch panel are displayed. Note that the product displayed in the video display area 1741 displays the video being captured as it is, not a still image (photograph).

  In this embodiment, real-time recognition processing is performed on the display video, and the name 1752 of each component is displayed in the video display area 1751 of the central display screen 1750.

  Furthermore, in the present embodiment, the product imaged by the video processing device 1710 is recognized from the arrangement, distance, angle, etc. of the recognized parts in the video, and the video display area of the right display screen 1760 is recognized. In 1761, a product name 1762 and link information 1763 to related information are displayed in real time. Alternatively, sound may be output from a speaker. From the video display area 1761, for example, product information can be known.

《Video processing procedure of video processing system》
FIG. 18 is a sequence diagram showing a video processing procedure of the video processing system 1700 according to the present embodiment. In step S1800, if necessary, the application according to the present embodiment is downloaded from the verification server.

  First, in step S1801, the mobile terminal and the verification server application are activated and initialized. In step S1803, the portable terminal captures an image using the imaging unit 310. Next, in step S1805, the portable terminal generates a local feature amount. In step S1807, the portable terminal encodes the generated local feature amount and the position coordinates of the feature point, and in step S1809, transmits them to the matching server via the network.

  In the collation server, in step S1811, the object in the video is recognized by collating the local feature quantity of the recognition target object in the local feature quantity DB 1721 with the received local feature quantity. In step S1813, the collation process is repeated until collation of a plurality of recognition objects is completed. Next, in step S1815, with reference to the combination identification DB 1722, video collation is performed using a plurality of recognition objects as a collation result. In step S1817, the collation server refers to the link information DB 1723, and acquires link information corresponding to the recognition target object. In step S1819, the picture recognition target object, the image comparison result, and the link information are returned to the portable terminal.

  In step S1821, the portable terminal superimposes and displays the received recognition target object and link information on the input video (corresponding to the product name 1762 and link information 1763 in FIG. 17). If a link destination is instructed in step S1823, the process advances to step S1825 to access a link destination server such as the service providing server 1730 in FIG. 17 based on the collation video ID.

  In step S1827, the service providing server reads related information (document, sound, or image) of the collation video from the related information DB. In step S1829, the related information of the collation video is downloaded to the mobile terminal. In the mobile terminal, the related information of the received verification video is superimposed and displayed on the input video, or the audio is played back, and the target of the input video is notified to the user.

  In the present embodiment, such a series of processing is realized in real time, and the user can see the display of the recognition object name, the collation video name, and related information on the input video.

<Functional configuration of video processing device for portable terminal>
FIG. 19A is a block diagram illustrating a functional configuration of a video processing device 1710 for a portable terminal according to the present embodiment. The functional configuration of the video processing device 1710 is a configuration in which the configuration related to the collation processing is eliminated from the video processing device 200 of the second embodiment, and instead, a local feature transmission configuration and a collation result reception configuration are added. Therefore, the same components as those in FIG. 3 are denoted by the same reference numerals, and description thereof is omitted.

  The video processing device 1710 includes an encoding unit 1930 that encodes the local feature amount and the feature point coordinates generated by the local feature amount generation unit 320 via the communication control unit 390 (see FIG. 19B). ).

  On the other hand, from the data received by the verification result / link information receiving unit 1950 via the communication control unit 390, the recognition target name recognized by the verification server, the verification video, and the like and link information are acquired. A display screen in which the acquired data is superimposed on the input video is generated and displayed on the display unit 360. If there is audio data, it is output from the speaker.

(Encoding part)
FIG. 19B is a block diagram showing an encoding unit 1930 according to this embodiment. Note that the encoding unit is not limited to this example, and other encoding processes can be applied.

  The encoding unit 1930 includes a coordinate value scanning unit 1931 that inputs the coordinates of feature points from the feature point detection unit 411 of the local feature quantity generation unit 320 and manipulates the coordinate values. The coordinate value scanning unit 1931 scans the image according to a specific scanning method, and converts the two-dimensional coordinate values (X coordinate value and Y coordinate value) of the feature points into one-dimensional index values. This index value is a scanning distance from the origin according to scanning. There is no restriction on the scanning direction.

  In addition, a sorting unit 1932 that sorts the index values of the feature points and outputs permutation information after sorting is provided. Here, the sorting unit 1932 sorts in ascending order, for example. You may also sort in descending order.

  Further, a difference calculation unit 1933 is provided that calculates a difference value between two adjacent index values in the sorted index value and outputs a series of difference values.

  And it has the difference encoding part 1934 which encodes the series of a difference value in order of a series. The sequence of the difference value may be encoded with a fixed bit length, for example. When encoding with a fixed bit length, the bit length may be specified in advance, but this requires the number of bits necessary to express the maximum possible difference value, so the encoding size is small. Don't be. Therefore, when encoding with a fixed bit length, the differential encoding unit 1934 can determine the bit length based on the input sequence of difference values. Specifically, for example, the difference encoding unit 1934 obtains the maximum value of the difference value from the input series of difference values, obtains the number of bits (expression bit number) necessary to express the maximum value, A series of difference values can be encoded with the obtained number of expression bits.

  On the other hand, it has a local feature encoding unit 1935 that encodes the local feature of the corresponding feature point in the same permutation as the index value of the sorted feature point. By encoding with the same permutation as the sorted index value, the coordinate value encoded by the differential encoding unit 1934 and the corresponding local feature amount can be associated one-to-one. In this embodiment, the local feature amount encoding unit 1935 encodes a local feature amount that is dimension-selected from 150-dimensional local feature amounts for one feature point, for example, one dimension with one byte, and the number of dimensions. Can be encoded.

<< Functional configuration of server video processing apparatus >>
FIG. 20 is a block diagram showing a functional configuration of the server video processing apparatus 1720 according to the present embodiment.

  The server video processing device 1720 includes a communication control unit 2010. The decoding unit 2020 decodes the encoded local feature amount and feature point coordinates received from the mobile terminal via the communication control unit 2010. Then, the collation unit 2030 collates with the local feature amount of the recognition target object in the local feature amount DB 1721. The multiple verification result holding unit 2040 holds a plurality of recognition objects of the verification result of the verification unit 2030 together with positions in the video. The video collation unit 2050 refers to the combination identification DB 1722 and collates videos from a plurality of recognition objects. The link information income unit 2060 acquires the link information acquired from the link information DB 1723 corresponding to the collation result video. Transmission data is generated in the transmission data generation unit 2070 from the recognition target of the verification unit 2030, the verification video of the video verification unit 2050, and the link information of the link information acquisition unit 2060. The transmission data is returned from the transmission unit 2080 to the portable terminal via the communication control unit 2010.

(Link information DB)
FIG. 21 is a diagram showing the configuration of the link information DB 1723 according to this embodiment.

  The link information DB 1723 which is a link information storage unit stores link information including a link destination address 2102 and a link destination display image 2103 in association with the collation video ID 2101. The link information DB 1723 may be prepared integrally with the combination identification DB 1722 and the local feature DB 1721.

<< Hardware configuration of video processing device for portable terminal >>
FIG. 22 is a block diagram showing a hardware configuration of a video processing device 1710 for a portable terminal according to the present embodiment. The hardware configuration of the video processing device 1710 for the portable terminal eliminates the configuration related to the collation processing from the video processing device 200 of the second embodiment, and instead, the local feature transmission configuration and the collation result reception configuration. Therefore, the same components as those in FIG. 7 are denoted by the same reference numerals, and the description thereof is omitted.

  The RAM 2240 is a random access memory that the CPU 710 uses as a work area for temporary storage. In the RAM 2240, an area for storing data necessary for realizing the present embodiment is secured. The combination matching result 2241 is an area for storing the combination matching result received from the matching server. The link information 2242 is an area for storing link information received from the verification server. The input video / collation result / link information superimposition data 2245 is an area for storing input video / collation result / link information superimposition data in which the input video / collation result / link information is superimposed.

  The storage 2250 stores a database, various parameters, or the following data or programs necessary for realizing the present embodiment. The display format 2251 is an area in which a display format used for generating a format for displaying a recognition target object and a video collation result / link information is stored. The storage 2250 stores the following programs. The portable terminal control program 2252 is an area in which a portable terminal control program for controlling the entire video processing apparatus 1710 is stored. The local feature value transmission module 2253 is an area in which a local feature value transmission module that encodes the generated local feature value and feature point coordinates and transmits them to the matching server in the mobile terminal control program 2252 is stored. The collation result receiving module 2254 is an area in which a collation result receiving module for receiving the collation result / link information and notifying the user by display or voice is stored. The link destination access module 2255 is an area for storing a link destination access module for accessing the link destination and downloading the related information when the displayed link information is instructed.

  Note that FIG. 22 shows only data and programs essential to the present embodiment, and data and programs not related to the present embodiment are not shown.

<< Processing procedure of video processing device for mobile terminal >>
FIG. 23 is a flowchart illustrating a processing procedure of the video processing device 1620 for the portable terminal according to the present embodiment. This flowchart is executed by the CPU 710 in FIG. 22 using the RAM 2240, and implements each functional component in FIG. 19A.

  First, in step S2311, it is determined whether or not there is a video input for performing object recognition. As a function of the mobile terminal, reception is determined in step S2321. Otherwise, other processing is performed in step S2331. Note that description of normal transmission processing is omitted.

  If there is video input, the process proceeds to step S2313 to execute local feature generation processing from the input video (see FIG. 9A). Next, in step S2315, local feature quantities and feature point coordinates are encoded (see FIGS. 24A and 24B). In step S2317, the encoded data is transmitted to the verification server.

  In the case of data reception, the process proceeds to step S2323, and it is determined whether the recognition result is received from the verification server or the related information is received from the link destination server. If it is a recognition result, it will progress to step S2325 and will superimpose and display the received recognition result and link destination information on an input image | video. On the other hand, if it is related information, it will progress to step S2327 and the related information from a link destination server will be displayed or audio | voice output.

(Encoding process)
FIG. 24A is a flowchart showing the processing procedure of the encoding processing S2315 according to the present embodiment.

  First, in step S2411, the coordinate values of feature points are scanned in a desired order. Next, in step S2413, the scanned coordinate values are sorted. In step S2415, a coordinate difference value is calculated in the sorted order. In step S2417, the difference value is encoded (see FIG. 24B). In step S2419, local feature amounts are encoded in the coordinate value sorting order. The difference value encoding and the local feature amount encoding may be performed in parallel.

(Difference processing)
FIG. 24B is a flowchart illustrating a processing procedure of difference value encoding processing S2417 according to the present embodiment.

  First, in step S2421, it is determined whether or not the difference value is within a codeable range. If it is within the range that can be encoded, the process proceeds to step S2427 to encode the difference value. Then, control goes to a step S2429. If it is not within the range that can be encoded (outside the range), the process proceeds to step S2423 to encode the escape code. In step S2425, the difference value is encoded by an encoding method different from the encoding in step S2427. Then, control goes to a step S2429. In step S2429, it is determined whether the processed difference value is the last element in the series of difference values. If it is the last, the process ends. When it is not the last, it returns to step S2421 again and the process with respect to the next difference value of the series of a difference value is performed.

<< Hardware configuration of video processing device for server >>
FIG. 25 is a block diagram showing a hardware configuration of the server video processing apparatus 1720 according to the present embodiment.

  In FIG. 25, a CPU 2510 is a processor for arithmetic control, and implements each functional component of the video processing device 1620 that is a collation server by executing a program. The ROM 2520 stores fixed data and programs such as initial data and programs. The communication control unit 2010 is a communication control unit, and in the present embodiment, communicates with other devices via a network. Note that the number of CPUs 2510 is not limited to one, and may be a plurality of CPUs or may include a GPU for image processing.

  The RAM 2540 is a random access memory that the CPU 2510 uses as a work area for temporary storage. The RAM 2540 has an area for storing data necessary for realizing the present embodiment. The received local feature value 2541 is an area for storing the local feature value including the feature point coordinates received from the mobile terminal. The read local feature value 2542 is an area for storing the local feature value when the feature point coordinates read from the local feature value DB 1721 are included. The plurality of recognition target object matching results 2543 is an area for storing a plurality of recognition target object matching results recognized from the matching between the received local feature value and the local feature value stored in the local feature value DB 330. The video recognition result 2544 is an area for storing a video recognition result of video recognition from a combination of a plurality of recognition objects with reference to the combination identification DB 1722. The link information 2545 is an area for storing link information retrieved from the link information DB 1723 corresponding to the recognition target object. Transmission / reception data 2546 is an area for storing transmission / reception data transmitted / received via the communication control unit 2010.

  The storage 2550 stores a database, various parameters, or the following data or programs necessary for realizing the present embodiment. The local feature DB 1721 is an area in which a local feature DB similar to that shown in FIG. 6 is stored. Since the verification server has sufficient processing capacity and storage capacity, local feature values in all fields may be stored. The combination identification DB 1722 is an area in which a combination identification DB similar to that shown in FIG. 13 is stored. The link information DB 1723 is an area in which the link information DB shown in FIG. 21 is stored.

  The storage 2550 stores the following programs. The verification server control program 2351 is an area in which a verification server control program for controlling the entire video processing apparatus 1720 is stored. The local feature amount module 2552 is an area in which the local feature amount module is stored from the image of the recognition target portion in the matching server control program 2551. The object recognition control module 2553 stores an object recognition control module that recognizes an object by collating the received local feature quantity with the local feature quantity stored in the local feature quantity DB 330 in the matching server control program 2551. It is an area to be done. The video recognition control module 2554 is an area in which a video recognition control module that recognizes video by referring to the combination identification DB 1722 from a combination of a plurality of recognition objects. The link information acquisition module 2555 is an area in which a link information acquisition module that acquires link information from the link information DB 723 corresponding to the recognized video is stored.

  In FIG. 25, only data and programs essential to the present embodiment are shown, and data and programs not related to the present embodiment are not shown.

<< Processing procedure of server video processing apparatus >>
FIG. 26 is a flowchart showing a processing procedure of the server video processing apparatus 1720 according to the present embodiment. This flowchart is executed by the CPU 2510 of FIG. 25 using the RAM 2540, and implements each functional component of FIG.

  First, in step S2611, it is determined whether or not a local feature DB is generated. In step S2621, it is determined whether a local feature amount is received from the mobile terminal. Otherwise, other processing is performed in step S2631.

  If it is generation of a local feature DB, the process proceeds to step S2613, and local feature DB generation processing is executed (see FIG. 27A). If the local feature amount is received, the process advances to step S2623 to execute object recognition / video recognition / link information acquisition processing (see FIG. 27B). In step S2625, the recognition target object and link information are transmitted to the mobile terminal.

(Local feature DB generation processing)
FIG. 27A is a flowchart showing a processing procedure of local feature DB generation processing S2613 according to the present embodiment.

  First, in step S2711, an image of a recognition object is acquired. In step S2713, the position coordinates, scale, and angle of the feature point are detected. In step S2715, a local region is acquired for one of the feature points detected in step S2713. Next, in step S2717, the local area is divided into sub-areas. In step S2719, feature vectors for each sub-region are generated to generate feature vectors for local medical treatment. The processing from step S2713 to S2719 is illustrated in FIG. 4B.

  Next, in step S2721, dimension selection is performed on the feature vector of the local region generated in step S2719. The dimension selection is illustrated in FIGS. 4D to 4F. However, in the generation of the local feature DB 1721, hierarchization is performed in dimension selection, but it is desirable to store all generated feature vectors.

  In step S2723, it is determined whether the generation of local feature values and dimension selection have been completed for all feature points detected in step S2713. If not completed, the process returns to step S2713 to repeat the process for the next one feature point. When all the feature points have been completed, the process proceeds to step S2725, and the local feature amount and the feature point coordinates are registered in the local feature amount DB 1721 in association with the recognition target object.

  In step S2727, it is determined whether there is another recognition object. If there is another recognition object, the process returns to step S2711 to acquire an image of the recognition object and repeat the process.

(Recognized object / link information acquisition process)
FIG. 27B is a flowchart showing the processing procedure of the recognition object / link information acquisition processing S2623 according to the present embodiment.

  First, in step S2731, the local feature amount of one recognition target is acquired from the local feature amount DB 1721. In step S2733, the local feature amount of the recognition target object is compared with the local feature amount received from the mobile terminal. Note that the matching process in step S2733 is basically the same as the matching process of FIG. 9B performed by the mobile terminal, and detailed description thereof is omitted.

  In step S2735, it is determined whether or not they match. If they match, the process proceeds to step S2737 to store the matched recognition object. In step S2739, it is determined whether all recognition objects in the video have been collated. If not, the process returns to step S2731, and the object recognition process in the video is repeated.

  If it is determined that all the corresponding recognition objects have been collated, the process proceeds to step S2741, and the video is recognized from the combination of a plurality of recognition objects with reference to the combination identification DB 1722. Next, link information corresponding to the recognized video is acquired from the link information DB 1622.

[Other Embodiments]
Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In addition, a system or an apparatus in which different features included in each embodiment are combined in any way is also included in the scope of the present invention.

  In addition, the present invention may be applied to a system composed of a plurality of devices, or may be applied to a single device. Furthermore, the present invention can also be applied to a case where a control program that realizes the functions of the embodiments is supplied directly or remotely to a system or apparatus. Therefore, in order to realize the functions of the present invention on a computer, a control program installed in the computer, a medium storing the control program, and a WWW (World Wide Web) server that downloads the control program are also included in the scope of the present invention. include.

Claims (19)

Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , First local feature storage means for storing m1, m2,..., Mk first local feature amounts each consisting of feature vectors from one dimension to i1, i2,.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. Second local feature generating means for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, Recognition means for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
Display means for displaying information indicating a plurality of recognition objects recognized by the recognition means on an image in which the plurality of recognition objects exist in the video;
A video processing apparatus comprising:   The video processing apparatus according to claim 1, further comprising an identification unit that identifies the video based on an image in which the plurality of recognition objects exist in the video. Link information storage means for storing link information for accessing related information related to the video in association with the video identified by the recognition means;
The video processing apparatus according to claim 1, wherein the display unit further displays the link information superimposed on an image in the video. Link information storage means for storing link information for accessing related information related to the video in association with the video identified by the recognition means;
Download means for accessing the related information according to the link information;
Further comprising
The video processing apparatus according to claim 1, wherein the display unit further displays the related information superimposed on an image in the video. The first local feature amount storage means includes the m1, m2,..., Mk first local feature amounts and the m1, m2,..., Mk features in the respective images of the plurality of recognition objects. Memorize the set of point coordinates and
The second local feature quantity generation means holds a set of the n second local feature quantities and the position coordinates of the n feature points in the image in the video,
The recognizing means is a predetermined ratio of a set of the n second local feature values and their position coordinates, and a set of the m1, m2,..., Mk first local feature values and their position coordinates. 5. The apparatus according to claim 1, wherein the plurality of recognition objects are recognized to exist in the image in the video when it is determined that the set is in a linear transformation relationship. The video processing apparatus according to the item.   The first local feature amount and the second local feature amount are a plurality of dimensions formed by dividing a local region including a feature point extracted from an image into a plurality of sub-regions, and comprising histograms of gradient directions in the plurality of sub-regions. The video processing device according to claim 1, wherein the video processing device is generated by generating a feature vector.   The first local feature amount and the second local feature amount are generated by deleting a dimension having a larger correlation between adjacent sub-regions from the generated plurality of dimension feature vectors. The video processing apparatus according to claim 6.   The plurality of dimensions of the feature vector are selected for each predetermined number of dimensions so that the dimension can be selected in order from the dimension that contributes to the feature of the feature point and from the first dimension according to the accuracy required for the local feature amount. The video processing apparatus according to claim 6, wherein the video image processing apparatus is arranged so as to make a round around the local area. Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , M1, m2,..., Mk first local feature amounts each comprising feature vectors from one dimension to i1, i2,. A control method for a video processing apparatus,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
A control method for a video processing apparatus, comprising: Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , M1, m2,..., Mk first local feature amounts each comprising feature vectors from one dimension to i1, i2,. A video processing device control program,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
A control program for causing a computer to execute. A video processing system having a video processing device for a mobile terminal and a video processing device for a server connected via a network,
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , First local feature storage means for storing m1, m2,..., Mk first local feature amounts each consisting of feature vectors from one dimension to i1, i2,.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. Second local feature generating means for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, Recognition means for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
Display means for displaying information indicating a plurality of recognition objects recognized by the recognition means on an image in which the plurality of recognition objects exist in the video;
A video processing system comprising: The video processing device for the portable terminal is:
The second local feature quantity generating means;
First transmitting means for encoding the n second local feature values and transmitting the encoded second local feature values to the server video processing apparatus via the network;
First receiving means for receiving information indicating a recognition object recognized by the server video processing device from the server video processing device;
With
The server video processing device is:
The first local feature storage means;
Second receiving means for receiving and decoding the n second local feature values encoded from the video processing device for the mobile terminal;
The recognition means;
Second transmission means for transmitting information indicating the recognition object recognized by the recognition means to the video processing device for the portable terminal via the network;
The video processing system according to claim 12, further comprising: A video processing apparatus for a portable terminal in the video processing system according to claim 11 or 12,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. Second local feature generating means for generating a quantity;
First transmitting means for encoding the n second local feature values and transmitting the encoded second local feature values to the server video processing apparatus via the network;
First receiving means for receiving, from the server video processing device, information indicating a plurality of recognition objects recognized by the server video processing device;
Display means for displaying the received information indicating the plurality of recognition objects on an image in which the objects are present in the video;
A video processing apparatus comprising: A method for controlling a video processing device for a portable terminal in the video processing system according to claim 11 or 12,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
A first transmission step of encoding the n second local feature amounts and transmitting the encoded second local feature amounts to the server video processing apparatus via the network;
A first receiving step of receiving, from the server video processing device, information indicating a plurality of recognition objects recognized by the server video processing device;
A display step for displaying the received information indicating the plurality of recognition objects on an image in which the objects exist in the video;
A control method for a video processing apparatus, comprising: A control program for a video processing device for a portable terminal in the video processing system according to claim 11 or 12,
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
A first transmission step of encoding the n second local feature amounts and transmitting the encoded second local feature amounts to the server video processing apparatus via the network;
A first receiving step of receiving, from the server video processing device, information indicating a plurality of recognition objects recognized by the server video processing device;
A display step for displaying the received information indicating the plurality of recognition objects on an image in which the objects exist in the video;
A control program for causing a computer to execute. A video processing apparatus for a server in the video processing system according to claim 11 or 12,
Generated for each of a plurality of recognition objects and m1, m2,..., Mk local regions each including m1, m2,..., Mk feature points in the images of the plurality of recognition objects. , First local feature storage means for storing m1, m2,..., Mk first local feature amounts each consisting of feature vectors from one dimension to i1, i2,.
Second receiving means for receiving and decoding the n second local feature values encoded from the video processing device for the mobile terminal;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, Recognition means for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
Second transmission means for transmitting information indicating a plurality of recognition objects recognized by the recognition means to the video processing device for portable terminal via the network;
A video processing apparatus for a server, comprising: 13. The video processing system according to claim 11 or 12, comprising a plurality of recognition objects and m1, m2,..., Mk feature points, respectively, in the images of the plurality of recognition objects. ..., corresponding to m1, m2, ..., mk first local features generated from feature vectors from 1 to i1, i2, ..., ik, respectively, generated for each of mk local regions. A method for controlling a video processing apparatus for a server provided with first local feature storage means for storing,
A second receiving step of receiving and decoding the encoded second local feature values from the video processing device for the mobile terminal;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A second transmission step of transmitting information indicating a plurality of recognition objects recognized in the recognition step to the video processing device for the portable terminal via the network;
A method for controlling a video processing apparatus for a server, comprising: 13. The video processing system according to claim 11 or 12, comprising a plurality of recognition objects and m1, m2,..., Mk feature points, respectively, in the images of the plurality of recognition objects. ..., corresponding to m1, m2, ..., mk first local features generated from feature vectors from 1 to i1, i2, ..., ik, respectively, generated for each of mk local regions. A control program for a video processing apparatus for a server provided with first local feature storage means for storing
A second receiving step of receiving and decoding the encoded second local feature values from the video processing device for the mobile terminal;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A second transmission step of transmitting information indicating a plurality of recognition objects recognized in the recognition step to the video processing device for the portable terminal via the network;
A control program for causing a computer to execute. A mobile terminal video processing apparatus and a server video processing apparatus connected via a network, each having a plurality of recognition objects and m1, m2,. ..., mk local regions including mk feature points, m1, m2,..., each of feature vectors from one dimension to i1, i2,. , Mk first local feature values in association with each other, and a first local feature value storing means for storing the image data in association with each other.
N feature points are extracted from the image in the video, and n second local features each consisting of a feature vector from the first dimension to the jth dimension for each of the n local regions including each of the n feature points. A second local feature generation step for generating a quantity;
The number of dimensions i1, i2,..., Ik of the first local feature quantity and the dimension number j of the second local feature quantity are selected, and the feature vector consisting of the feature vector up to the selected dimension number is selected. When it is determined that n second local feature amounts correspond to a predetermined ratio or more for each of the m1, m2,..., mk first local feature amounts including feature vectors up to the selected number of dimensions, A recognition step for recognizing that there are a plurality of recognition objects corresponding to the predetermined ratio or more in the image in the video;
A display step of displaying information indicating a plurality of recognition objects recognized in the recognition step on an image in which the plurality of recognition objects exist in the video;
A video processing method comprising:

Images