JP4607836B2 - Apparatus for automatically assigning metadata to image area, method for automatically assigning metadata, and program for automatically assigning metadata - Google Patents

Description

  The present invention relates to an apparatus for simplifying editing work by automatically assigning attributes of each scene of a video during shooting. In recent years, as the distribution and broadcasting of digital contents becomes easier by using an Internet line or the like, a device for processing and editing a large amount of video material in a short time is required. For example, a device for generating a replay video by performing live broadcast on a sports video or the like and cutting out a highlight scene or the like at the same time is applicable.

  Conventionally, there is a non-linear editing device as a typical device for video editing. In this device, the video acquired by the video camera and the video recorded on the tape are converted into digital data, which is taken into the hard disk of a personal computer and processed and edited. For example, to cut a marathon finish scene, play the video stored on the hard disk, manually set the start and end positions of the cutout, save the corresponding section as a separate data file, Create replay video by recording to However, in a non-linear editing apparatus, editing operations such as specifying a cut-out section and storing a replay video must be performed manually, which requires a great deal of labor and cost.

  In view of this, several methods have been proposed in which an attribute is assigned to each scene in a video, and attribute information is searched to automatically specify an extraction section.

  Non-Patent Document 1 proposes a method for recognizing an ID signal emitted from an infrared LED attached to a subject and indexing the stored video data in real time as “collaborative interaction recording by multiple sensors”. Has been. However, since the LED is used as a signal source, there is a problem that ID information cannot be received when there is no light receiver in the light emission direction or when there is an obstacle between the LED and the light receiver. .

  Patent Document 1 proposes a method for preventing falsification of an image by recording ID information such as an RF tag attached to a subject and storing it together with image data when recording a camera image. . Further, this Patent Document 1 describes that a signal is received only from a tag facing the camera by a directional antenna. However, there is no specific description on how to associate the directional antenna with the camera field of view. That is, there is a problem that even if ID information of the RF tag is recorded, it is not guaranteed whether or not the corresponding photographing target is reflected.

  Patent Document 2 proposes a method of making it easy to find a desired article by detecting the direction of an RF tag using a directional antenna and irradiating light beams such as LEDs in the same direction. This Patent Document 2 describes an outline of a directional antenna, but does not describe a technique for associating an antenna with a camera or associating an RF tag ID information with a mobile object.

  Patent Document 3 proposes a method of connecting an image region and tag information by causing only an RF tag that detects infrared light to emit radio waves. However, there is no description regarding a method for associating the ID information of the RF tag with the image area of the subject.

  In Patent Document 4, the height of a person having a portable terminal with a built-in reader / writer is measured by image processing, and the person is identified by the height data by reading the RF tag ID attached to the floor. In addition, a method is described in which a person's position is detected based on a tag ID, and a video of the person is captured. However, image processing is applied to measuring the height of a human body, and there is no mention of a method for extracting a human body region from an image.

  In Patent Document 5, the direction of a wireless tag attached to a subject is detected with a directional antenna integrated with a digital camera, and image information and position information are associated with each other and stored in a storage unit. It describes technologies that can be applied, such as immediately knowing where a picture was taken when address information is included in the tag. In this, there is a description that the direction of the tag is detected and the direction of the tag is detected. However, there is no description of how to control the tag, and the moving object detection result detected by the image processing is described as follows. It is not considered to be reflected in the radio signal processing when identifying the tag ID of each mobile unit.

  Therefore, in the conventional method or the combination of the conventional methods as described above, for example, when the performance of the antenna deteriorates due to the influence of multipath, fading, etc., the moving object equipped with the RF tag in the camera field of view is not used. Since it cannot be detected with high accuracy, there is a problem that it may not be possible to automatically assign an attribute to an image area occupied by a target moving object.

Also, according to “Motion Detection and Tracking by Local Correlation Calculation” described in Non-Patent Document 2, if an image processing method such as the local correlation method is used, the moving object is located in any region in the image. However, there is a problem that it is not possible to automatically give in real time what kind of attribute each mobile object has.
JP 2000-261551 A JP 2002-271229 A JP 2004-080347 A JP 2002-148349 A JP 2005-347795 A Tsuno, Ito, Matsuguchi, Sidney Fels, Utsumi, Suzuki, Nakahara, Iwasawa, Kogure, Mase, Iwata, "Collaborative Interaction Recording by Multiple Sensors", Interaction 2003 Proceedings Information Processing Society of Japan, pp.255-262 Morita, "Motion Detection and Tracking by Local Correlation", IEICE Transactions D-II, Vol.J84-D-II, No.2, pp.299-309, 2001

  The present invention accurately assigns metadata including attribute information obtained from an RF tag to an image area of a moving object obtained by image processing even when the antenna performance deteriorates due to multipath or fading. It is an object of the present invention to provide means that can be used.

  This makes it possible to automate the editing work by using the automatically assigned metadata when extracting the image area in which the target scene or the specific subject is shown from the acquired video.

  The present invention relates to an apparatus for automatically assigning metadata to an image area by associating an image area occupied by a moving body photographed by a video camera with information on an RF tag included in the moving body. Based on the detection result of the moving object obtained by the processing, the most important feature is to control the angle at which the radio wave is radiated from the directional antenna to the RF tag.

  The present invention has an input / output interface between an imaging device for imaging a moving body equipped with an RF tag and an RF tag input / output device having a directional antenna, and an image area of the moving body from an image captured by the imaging device. Is a metadata automatic assigning device that extracts metadata including unique information or attribute information obtained from the RF tag to an image area of a mobile object, and in particular, to the orientation of the mobile object obtained by image processing Radio waves are radiated from a directional antenna, and metadata including unique information or attribute information such as an ID obtained from an RF tag is assigned to an image area from the most probable direction calculated by the RF tag and a tag ID. And

  In the above invention, when calculating the orientation of the RF tag, the angle data of the radio wave is transmitted from the directional antenna, and the angle data placed on the radio wave having the strongest radio wave intensity at the time of reception is returned to the RF tag. The most probable orientation of the RF tag is obtained from the angle data.

  In the above invention, as another method for calculating the orientation of the RF tag, radio waves are radiated from the RF tag, and the directional antenna sequentially directs the directional beam toward the orientation of the moving body obtained by image processing. The radio wave intensity from each received RF tag is measured, and the automatic metadata adding apparatus obtains the most probable azimuth in which each RF tag is located based on the radio wave intensity of all RF tags.

  As described above, according to the present invention, even when the antenna performance deteriorates due to the influence of multipath or fading, the unique information or attribute obtained from the RF tag is transferred to the image area of the moving object obtained by image processing. Since metadata including information can be automatically assigned, it can be efficiently processed when the shot video is edited, distributed, and broadcast.

  FIG. 1 shows a block diagram of an apparatus according to an embodiment of the present invention. In FIG. 1, an automatic metadata adding device 10 is a device that includes a CPU, a memory, peripheral devices, a software program, and the like. The metadata automatic assigning device 10 includes a signal input / output interface (I / F) 11 for the video camera 20 and the RF tag reader / writer 30, a moving object detection unit 12 for detecting a moving object (also referred to as a moving object) from an input image, A moving body direction calculation unit 13 that calculates the direction of the detected moving body, an antenna direction control unit 14 that controls the antenna according to the calculated direction of the moving body, a metadata generation unit 15 that generates metadata, and an image area of the moving body A metadata adding unit 16 for adding metadata and a signal input / output interface 17 for outputting the result of adding the metadata to the outside are provided. Examples of the output destination of the result of adding the metadata include the display device 40, the display device 41 via the network 42, and a recording medium 43 such as a disk storage device. Hereinafter, processing performed by each unit in the block diagram shown in FIG. 1 will be described in detail.

[Video camera 20]
The video camera 20 forms an image at a constant sampling time interval (for example, 33 ms) from the light intensity detected by the CCD or the like, and further generates a moving image.

  One pixel of an image is given by a gradation value of a red component (R), a green component (G), and a blue component (B) in the case of a color image, and given by a luminance gradation value in the case of a monochrome image. . For example, the gradation values of the red component (R), green component (G), blue component (B), and luminance (I) of the pixel at coordinates (x, y) indicated by integers x and y are digital values, respectively. R (x, y), G (x, y), B (x, y), I (x, y). Note that the luminance (monochrome gray value) is used as the pixel value used in the following calculation.

[Moving object detection unit 12 (first method)]
A first method for detecting a moving body in the moving body detection unit 12 is a method of combining the background difference method and the local correlation method.

  The background difference method is a method in which an image that does not include a moving object is prepared in advance as a background image, and then a difference from an image that includes a moving object is obtained, and a portion having a large difference is used as a moving object. The background subtraction method can be processed in a shorter processing time, but has a drawback that it is vulnerable to noise such as brightness fluctuations.

  In the local correlation method, an image is first divided into grid-like blocks (for example, 8 × 8 pixels). Next, a correlation value between neighboring blocks is obtained between a frame at a certain sampling time and a frame acquired at the next sampling time, and it is determined whether the block has moved according to the magnitude. This processing has a large amount of calculation, so that the image size cannot be increased, and the distance between blocks that perform correlation calculation is limited, and it cannot follow a fast-moving object. High resistance to noise. Therefore, in the first method, a rough moving object region is obtained by the background subtraction method, and then a moving object region is determined more finely by the local correlation method.

  FIG. 2 is a diagram illustrating the relationship between tracking blocks, sub-partitions, and sub-partition templates. Prior to the description of the processing procedure, a sub-partition, a sub-partition template that is a collection of sub-partitions of the same object, and a tracking block will be described with reference to FIG.

  The acquired image 50 is an image taken by the video camera 20. The sub section 52 is composed of a square of vertical n pixels and horizontal n pixels. A hatched sub-partition template 53 shown in FIG. 2 is a collection of sub-partitions 52 and indicates a region of a moving object. The tracking block 51 is an area that is set so as to surround the sub-partition template 53, and is provided between the upper, lower, left, and right m pixels outside the sub-partition template 53. The search by the local correlation method is an area surrounded by the tracking block 51. Run in. In this embodiment, the image area of the moving object is represented by the sub-partition template 53.

  The background subtraction method is used to obtain a region where a moving object may exist, and then the local correlation method is used to calculate the effective sub-partition for detecting the moving object.

  FIG. 3 is a flowchart of detection processing of the moving body. A procedure for checking whether a moving object is included in a captured image and detecting an effective sub-part is shown.

  [S10]: The moving object detection unit 12 captures an image (bitmap) of the current frame obtained by the video camera 20 through the signal input / output interface 11 at a fixed period (every sampling time). The acquired current timing image is divided into two series and passed through a low-pass filter having different cut-off frequencies.

  [S11]: A background image of a slow-moving current frame in which the cutoff frequency of the low-pass filter is set low is obtained. This cut-off frequency can be obtained in accordance with the moving speed of the moving body, and can be set variably according to the target moving body. The background image is divided into sub-partitions (n × n pixel squares).

  [S12]: The cut-off frequency of the low-pass filter is set to be higher than that of the low-pass filter used in step S11, and an image from which influences such as flickering of a fluorescent lamp are removed is obtained while leaving a moving object. This cut-off frequency can also be variably set corresponding to the target moving body, similarly to the cut-off frequency in step S11. This image is the current image at the current sampling time. As in step S11, the image is divided into sub-partitions (n × n pixel squares).

  [S13]: In the current image (hereinafter referred to as current frame image) and the background image, one of the sub-sections (n × n pixel square) at the same position is extracted.

  [S14]: The sum T of the differences between the pixels in the sub-partitions at the same position in both images is calculated according to the following equation.

Here, I ^C (x, y) is the monochrome gray value of the pixel position (x, y) in the current image, I ^B (x, y) is the monochrome gray value of the pixel position (x, y) in the background image, R _sub represents a sub-partition. When the calculated value T is equal to or greater than a certain threshold value T _th , it indicates that the calculated value T is different from the background image, and it is determined as a moving sub-partition having movement, and the process proceeds to step S15. Remove the compartment.

[S15]: A local correlation method is used to find out from which sub-section of the previous frame image the moving sub-section determined to have moved in step S14. Correlation calculation is used to determine which sub-section S _q (x ′, y ′) of the previous frame image has moved from the sub-section S _p (x, y) of the current frame image focused as a moving sub-section. Explore. The correlation value C with the sub-partitions S _p (x, y) and S _q (x ′, y ′) is calculated by the following equation, and it is determined whether or not it has moved from a predetermined threshold.

In the above equation, I ^p (x, y) and I ^q (x ′, y ′) are the black and white grayscale values of the pixels in each sub-partition S _p (x, y), S _q (x ′, y ′). , R _p , R _q indicate that they are within the sub-partition region.

The presence or absence of movement is determined using the zero comparison method of Non-Patent Document 2 described above. The difference Q between the correlation value C ₀ between the sub-part of the previous frame at the same position as the sub-part of interest and the minimum value C _{1 of} correlation values obtained from the correlation calculation with all sub-parts within the scanning range is It is calculated by the following formula and compared with a threshold value Q _{th for} movement determination.

Q = C ₀ −C ₁
Q> Q _th
[S16]: If Q is larger than the threshold value _Qth , the sub-part of interest S _p (x, y) is determined as a sub-part moved from S _q (x ′, y ′), To do. If it is an effective sub-partition, the next step S17 is executed.

[S17]: The speed and direction of movement from the previous frame image of the effective sub-partition are calculated. The upper left point of the sub-part of the previous frame image is (x _LeftTop1 , y _LeftTop1 ), the upper left point of the sub-part of the current frame image of interest is (x _LeftTop2 , y _LeftTop2 ), and the sub-part movement vector (x _LeftTop2− x _LeftTop1 , y _LeftTop2− y _LeftTop1 ), the speed of movement and the direction of movement are as follows.

  [S18]: The processing from step S13 to S17 is repeated until the confirmation in all the sub-compartments is completed. Through a series of processing, all effective sub-parts, the moving speeds from the previous frame of each effective sub-partition, and the direction of the moving vector are obtained.

  Next, the image area of the moving object is extracted based on the effective sub-section obtained above. FIG. 4 is a flowchart of a candidate image region extraction process. An effective sub-partition template for confirming a moving object is generated from the set of effective sub-partitions obtained in the process of FIG. 3, and a moving tracking block candidate is set from the effective sub-partition template.

  [S20]: One of the effective sub-partitions is taken out and used as an effective sub-partition template.

  [S21, S22]: The next one of the effective sub-sections is taken out and compared to determine whether the moving speed and moving direction of both coincide. If they match, the process proceeds to step S23, and if they do not match, the process proceeds to step S24.

  [S23]: Combine both effective sub-partitions to generate a sub-partition template. This sub-partition template is set as an effective sub-partition template.

  [S24]: Set the effective sub-partition as a new effective sub-partition template.

  [S25]: Steps S21 to S24 are repeated until confirmation with all the valid sub-partitions is completed.

  [S26]: A region obtained by adding m pixels from the top, bottom, left and right edges of the sub-partition template to all the effective sub-partition templates defined in steps S23 and S24 is set as a movement tracking block candidate.

The area of the movement tracking block candidate is calculated as follows. Let x _sub-min , x _sub-max , y _sub-min , and y _sub-max be the upper, lower, left and right edges of the effective sub-partition template in step S26. If a region to which a predetermined pixel value m is given outside the region surrounded by the upper, lower, left, and right ends is a candidate (however, the pixel coordinate value is 0 or more), the region of the moving tracking block candidate is as follows.

The center position (x _trk-cen , y _trk-cen ) of the moving tracking block candidate is as follows.

  A moving body tracking block for determining the moving body is extracted from the area of the moving tracking block candidate obtained by the processing of FIG. 4, and the moving body is determined. The moving object is determined by calculating the moving vector of the moving object. FIG. 5 is a flowchart of the mobile object determination process.

[S30]: The movement vector from the previous frame image of the moving body existing in the movement tracking block candidate is calculated. If the moving tracking block center position (x _trk-cen1 , y _trk-cen1 ) of the _previous frame image and the moving tracking block candidate center position of the current frame image are (x _trk-cen2 , y _trk-cen2 ), the current frame image The moving speed v _trk ^(k) of the moving vector (x _trk-cen2 -x _trk-cen1 , y _trk-cen2 -y _trk-cen1 ⁾ and the moving direction vector (d _trk-x , d _{trk- y} ) ^(k) is as follows.

  [S31]: One of movement tracking block candidates of the current frame image is extracted.

  [S32]: One of the movement tracking blocks of the previous frame image is extracted.

  [S33, S34]: The movement vector of the current frame image is calculated, and the tracking condition is calculated from the movement vector of the previous frame image.

  The tracking conditions are as follows.

Here, v _trk ^(k-1) and (d _trk-x , d _trk-y ) ^(k-1) indicate the speed of movement of the movement vector of the previous frame image and the direction of the movement vector.

When the difference between the speeds of the moving vector of the moving tracking block candidate and the moving tracking block is not more than a certain value v _th and the length is not less than a certain value d _th , the moving objects of both moving tracking blocks are the same tracking object. I reckon. The length is obtained as the inner product of the movement direction vectors. If the tracking condition is satisfied, the process proceeds to the next step S35, and if the tracking condition is not satisfied, the process proceeds to step S39.

[S35, S36]: Set and confirm the number of continuations when the frame continuation condition is set as the tracking condition. Let FR be the number of consecutive frames, and add 1 to FR. If FR is the predetermined threshold value FR _th greater whether frames successive condition is satisfied, the process proceeds to step S37, if not satisfied, the processing proceeds to step S39.

  [S37]: Recognize that a moving object exists in the area of the moving tracking block candidate, replace the moving tracking block candidate with the moving tracking block of the current frame image, and hold it. Thereafter, the process proceeds to step S39.

  [S38]: When the tracking condition is not satisfied, the moving tracking block candidate is held as a new moving tracking block by regarding the moving frame as a new moving object newly appearing in the current frame image. The tracking condition FR = 1 is set, and the process proceeds to step S39.

  [S39]: Steps S32 to S38 are repeated until the confirmation of all the movement tracking blocks in the previous frame is completed. All the movement tracking blocks in which the moving body recognized by the current frame image acquired by this series of processes exists are obtained. Each moving tracking block has a different moving body.

[Moving object detection unit 12 (second method)]
FIG. 6 shows a block diagram of the moving object detection unit 12 that executes the second moving object detection method. In the method using the background subtraction method and the local correlation method described in the first method, when multiple moving objects overlap and the sub-partition straddles both objects near the boundary, the correlation value for which moving object However, it may become undetectable due to deterioration. Therefore, in the second moving object detection method, when the previous frame image and the current frame image are given, the following shows where the sub-partition determined as the moving object in the previous frame has moved to the current frame. The movement probability calculated based on the index is given at the maximum position. By using different indices, the detection performance near the boundary can be improved. Each index is obtained by the calculation unit and comparison unit indicated in brackets [].

(Indicator 1) Probability based on the similarity between the movement vector of the sub-partition of the previous frame and the movement vector of the sub-partition of the current frame. [Movement vector calculation unit 121]
(Indicator 2) Probability based on the similarity between the sub-part of the previous frame and the destination sub-part of the current frame. When the similarity is maximum, the probability is also maximum. [Local correlation value calculation unit 122]
(Indicator 3) Probability based on the degree of relation between the sub-part of the previous frame and its neighboring part, and the degree of relation between the sub-part of the current frame and its neighboring part. [Sub-partition feature quantity calculation unit 123, adjacent sub-partition relationship degree calculation unit 124, adjacent sub-partition relation degree comparison unit 125]
Details of the second moving body detection method shown in FIG. 6 will be described below.

[Movement vector calculation unit 121]
When the sub-partition of the previous frame (time t-1) moves to a sub-partition with the current frame (time t), the movement vector is calculated using the upper left end point of the sub-partition.

V = (v _x , v _y ) = (i _LT −i _LT ′, j _LT −j _LT ′)
However, (i _LT ′, j _LT ′) is the coordinates of the upper left pixel of the sub-partition of the previous frame, and (i _LT , j _LT ) is the coordinates of the upper left pixel of the sub-partition of the current frame.

Assume that the sub-section of the current frame of interest (the upper left pixel coordinates (i _LT , j _LT )) is at the following position at the previous sampling time and the previous sampling time.

Before sampling: (i _LT ″, j _LT ″)
Pre-sampling: (i _LT ', j _LT ')
In this case, the use of calculation expression of the movement vector described above, the movement vector V of the previous frame _{'= (v x', v} y '), the movement vector of the current frame _{V = (v x, v y} ) , the following equation It becomes.

V ′ = (v _x ′, v _y ′) = (i _LT ′ −i _LT ″, j _LT ′ −j _LT ″)
V = (v _x , v _y ) = (i _LT −i _LT ′, j _LT −j _LT ′)
The similarity between the vectors V ′ and V is evaluated by the following two values.

Difference in vector magnitude: A = | V | − | V ′ |
Inner product of unit vectors:

  The closer the difference value A is to 0 and the closer the inner product B is to 1, the higher the similarity of the movement vectors.

[Local correlation value calculation unit 122]
For the pixels at the same position included in the sub-partition, the difference in luminance value between the previous frame (time t-1) and the current frame (time t) is taken, and the sum of absolute values is calculated.

Where I _t-1 (i ′, j ′) is the luminance value of the pixel included in the sub-partition of the previous frame, I _t (i, j) is the luminance value of the pixel included in the sub-partition of the current frame, R _t-1 indicates that it is in the sub-partition area of the previous frame, and R _t indicates that it is in the sub-partition area of the current frame.

  The smaller the sum C, the greater the similarity between the sub-section of the previous frame and the current frame sub-section, and the greater the value, the smaller the similarity.

[Sub-zone feature value calculation unit 123]
The luminance value of the pixel included in the sub-part of interest is acquired, and the histogram H _sub obtained by integrating the number of appearances of the pixels included in the same class is used as the sub-part feature. The histogram _Hsub is expressed as follows.

H _sub [I] = N _I
I is the luminance value, N _I are included in interest sub-zone, the total number of pixels having luminance values I.

[Adjacent sub-partition relationship calculation unit 124]
Since the luminance value distribution in the local portion of the moving object region generally changes gently, adjacent sub-partitions have similar characteristics. For example, when a human torso is divided into sub-parts, the luminance distribution in each sub-part is almost constant. On the other hand, when one of the adjacent sub-partitions is included in the moving object and the other is included in the background area, the similarity between the two is low. In order to quantitatively express the relationship between adjacent sub-partitions, the degree of relation between adjacent sub-partitions is introduced. The degree of relationship between adjacent sub-partitions is calculated by overlapping histograms used for sub-partition features. The degree of relationship REL between adjacent sub-partitions of the sub-partition sub1 and the sub-partition sub2 is obtained by the following expression.

  min is a function that selects the smaller value.

FIG. 7 shows a histogram calculation example. Images and histograms of the previous frame are shown in (a) to (d), and images and histograms of the current frame are shown in (e) to (h). The two squares displayed superimposed on the image represent sub-sections (4 pixels × 4 pixels) and are located in the same part of the person photographed in the previous and current frames. When the histogram of each sub-section is obtained, it is as shown in (b), (c), (f), and (g) of FIG. (D) (h) is a result of plotting min (H _sub1 [I], H _sub2 [I]) in the above equation. The degree of relationship between adjacent sub-partitions corresponds to the areas (d) and (h). Since (d) and (h) have substantially the same value, it can be seen that the degree of relationship between the sub-partitions assigned to the same part on the object is maintained even if the object moves.

[Adjacent sub-partition relationship degree comparison unit 125]
When the sub-partition (p ′, q ′) of the previous frame (time t−1) moves to a sub-partition (p, q) having the current frame (time t), the sub-partition (p + a) adjacent to the target sub-partition , Q + b) (where a = −1, 0, 1, b = −1, 0, 1) is compared as follows, and an output value is calculated. A combination of values a and b indicating adjacent sections is given as follows.

・ In the case of four adjacent sections:
(A, b) = (0, −1), (−1, 0), (1, 0), (0, 1)
・ In the case of 8 adjacent blocks:
(A, b) = (0, −1), (−1, 0), (1, 0), (0, 1),
(-1, -1), (-1, 1), (1, -1), (1, 1)
Whether to use four adjacent sections or eight adjacent sections is determined in advance before executing the process.

  Hereinafter, the degree of relationship with the sub-partition (a, b) = (0, −1) in contact with the upper side of the target sub-partition will be described. Other adjacent sections can be described in the same procedure.

  First, the relationship REL (sub (p ', q'), sub) between adjacent sub-partitions between the sub-partition (p ', q') of the previous frame and the sub-partition (p ', q'-1) adjacent on the upper side. (P ′, q′−1)) is calculated. Next, the sub-partition (p ′, q ′) of the previous frame and the adjacent sub-partition of the sub-partition (p, q) that is the destination candidate of the current frame on the upper side The degree of interrelationship REL (sub (p ′, q ′), sub (p, q−1)) is calculated. Finally, the difference between the obtained two adjacent sub-partitions is calculated and set as an output value.

REL (sub (p ', q'), sub (p ', q'-1))
-REL (sub (p ', q'), sub (p, q-1))
When the degree of relationship between adjacent sub-partitions is obtained by the first method,
REL (sub (p ', q'), sub (p ', q'-1)) = 0
REL (sub (p ', q'), sub (p, q-1)) = 0
If the output value is N, the output value is N (the number of pixels in the sub-partition). The four sections adjacent to the sub-partition (p ′, q ′) are four sections in the upper and lower sides and the left and right sides. Therefore, an index D based on the degree of relationship between adjacent sub-partitions is expressed by the following equation.

  R ′ represents a set of four adjacent sections of the sub-partition (p ′, q ′) of the previous frame, and R represents a set of four adjacent sections of the sub-partition (p, q) of the current frame. The closer the index D is to zero, the higher the similarity between the subframe of the previous frame and the current frame subsection, and the lower the value is from zero, the lower the similarity.

[Movement probability calculation unit 126]
Using the three indices obtained by the above-described method, the similarity between the sub-partition (p ′, q ′) of the previous frame and the sub-partition (p, q) of the current frame is obtained. The most similar sub-partition in the current frame is considered to have the maximum movement probability, and is the destination of sub-partition (p ′, q ′). The similarity S _total between sub-partitions is expressed by the following formula using indices A, B, C, and D.

S _total (p ′, q ′, p, q)
= −α | A | + β (B−1) −γC−δ | D |
α, β, γ, and δ are weight coefficients given in advance.

The destination (p _dst , q _dst ) is
S _total (p ′, q ′, p _dst , q _dst )
= Max (S _total (p ′, q ′, p, q))
, 0 ≦ p <SizeX / n, 0 ≦ q <SizeY / n
Meet. When the sub-section (p, q) of the current frame is given, the processing time can be shortened by limiting the scanning range. For example, when scanning in the range of up, down, left and right r sections centering on the sub section of the previous frame, the value of the sub section (p, q) is set to p′−r ≦ p ≦ p ′ + r.
q′−r ≦ q ≦ q ′ + r
It is better to

When the moving objects overlap, the movement destinations of two different sub-parts (p1 ′, q1 ′), (p2 ′, q2 ′) of the previous frame are near the same sub-partition of the current frame near the boundary of the object. (P _dst , q _dst ). As described above, when there are a plurality of movement sources, the one having the higher similarity S _total is selected.

S _total (p1 ', q1', _pdst , _qdst )
≧ S _total (p2 ′, q2 ′, p _dst , q _dst )
→ The source is sub-partition (p1 ', q1'),
S _total (p1 ', q1', _pdst , _qdst )
<S _total (p2 ', q2', _pdst , _qdst )
→ The source is sub-partition (p2 ', q2'),
Even when the number of movement sources of the previous frame becomes three or more, processing can be performed by sequentially comparing the movement source selected by the above equation and the next movement source.

[Moving object tracking unit 127]
When the movement probability calculation unit 126 determines the movement source sub-partition of the sub-partition (p, q) of the current frame, the movement source ID value is inherited.

Attrib_SubBlock ^(t) [p _dst ] [q _dst ] [kID]
= Attrib_SubBlock ^(t-1) [p '] [q'] [kID]
kID is the attribute number of the mobile object ID storage destination. When the movement source of all sub-partitions in the current frame is determined and the mobile object ID is set in the sub-partition attribute storage unit 128, the area of each mobile object in the current frame Can be determined. For example, the image area (sub-partition template) of the object having the moving object ID = Z is represented by a collection of sub-parts satisfying Attrib_SubBlock [p] [q] [kID] = Z.

  Each time the sampling time elapses and a new frame image is acquired, the moving object can be tracked by repeating the above processing.

[Sub partition attribute storage unit 128]
As described in the first method of detecting the moving object, in this embodiment, the image is divided into sub-parts having a fixed shape (for example, a grid of 8 × 8 pixels), and a mobile object ID is assigned to each sub-part. This gives the position of the detected moving object. For example, when an image of horizontal 720 pixels × vertical 480 pixels is divided into 8 × 8 pixel sub-partitions, a total of 5400 sub-partitions of 90 horizontal sections × 60 vertical sections are provided. The area occupied by the moving object in the image is represented by a set of sub-partitions having the same moving object ID. The sub-partition is expressed by an array representation and stores various attribute values. For example, the k-th attribute value of the sub-part of the p-th row and the q-th column at time t is given by the following equation.

Attrib_SubBlock ^(t) [p] [q] [k]
0 ≦ p <SizeX / n, 0 ≦ q <SizeY / n, 0 ≦ k <k _max
(Here, SizeX and SizeY are the horizontal and vertical sizes of the frame image, n is the number of pixels on one side of the sub-partition, and k _max is the number of attributes.)
Examples of sub-partition attributes include the types shown in FIG.

  FIG. 9 is a flowchart of detection processing when the second moving object detection method for detecting a moving object based on the movement probability is used.

  [S40]: As an initial setting, various constants (image size SizeX, SizeY, number of pixels n on one side of sub-partition, etc.) are set in advance.

  [S41]: Shooting is started by the video camera 20.

  [S42]: Images taken by the video camera 20 are acquired at each sampling time.

  [S43]: The data stored in the current frame image buffer is moved to the previous frame image buffer, and the image acquired in step S42 is stored in the current frame image buffer.

  [S44]: One sub-partition (p ′, q ′) of the previous frame is extracted.

  [S45]: One sub-partition (p, q) of the current frame is extracted.

[S46]: sub-zone of the previous frame (p ', q') the upper left end pixel coordinates _{(i LT ', j LT'} ) and, the upper left end pixel coordinates (i _LT of the current frame sub-zone (p, q) , J _LT ), the movement vector V is calculated.

  [S47]: The movement vector V ′ of the sub-partition (p ′, q ′) of the previous frame is extracted from the sub-partition attribute storage unit 128, and the magnitude of the vector with the movement vector V of the current frame obtained in step S46 Difference value A and inner product B are calculated.

  [S48]: The local correlation value C is calculated from the pixels of the previous frame image buffer included in the sub-partition (p ′, q ′) and the pixels of the current frame image buffer included in the sub-partition (p, q).

[S49]: The histogram H _{p ′, q ′} [I] of the sub-partition (p ′, q ′) of the previous frame and the adjacent part (p ′ + a, q ′ + b) (from the sub-partition attribute storage unit 128) The histogram H _{p ′ + a, q ′ + b} [I] of a, b = −1, 0, 1) is taken out, and the degree of relationship REL (sub (p ′, q ′), sub (p ′ + A, q '+ b)) is calculated.

[S50]: Histogram H _{p, q} from the pixels of the current frame image buffer included in the sub-partition (p, q) and the adjacent section (p + a, q + b) (a, b = -1, 0, 1) of the current frame I], H _{p + a, q + b} [I] is obtained, and the degree of relationship REL (sub (p, q), sub (p + a, q + b)) between both sections is calculated.

  [S51]: Relationship REL (sub (p ′, q ′), sub (p ′ + a, q ′ + b)) of the previous frame and relationship REL (sub (p, q), sub (p + a, An index D based on the degree of relationship between adjacent sub-partitions is calculated from q + b)).

  [S52]: Steps S49 to S51 are repeated until the degree of relationship between adjacent sub-partitions for all adjacent sections is calculated.

[S53]: Using the indices A, B, C, and D, the similarity S _total (p ′, q ′, p, q) between the sub-partitions is obtained.

[S54]: for all sub-compartments of the current frame is scanned until the similarity is calculated S _total, repeat steps S45～S53.

[S55]: The sub-partition (p, q) of the current frame that maximizes the similarity S _total (p ′, q ′, p, q) between the sub-partitions is set as the movement destination (p _dst , q _dst ).

  [S56]: The mobile unit ID; Attrib_SubBlock (t−1) [p ′] [q ′] [kID] of the sub-partition (p ′, q ′) of the previous frame is extracted from the sub-partition attribute storage unit 128.

[S57]: from the sub-compartment attribute storage unit 128, the mobile ID of the destination sub-zone of the current frame (p _dst, q _dst); retrieving the _{Attrib_SubBlock (t) [p dst]} [q dst] [kID].

[S58]: the destination sub-zone of the current frame _{(Attrib_SubBlock (t) [p dst} ] [q dst] [kID]), if already mobile ID is stored, the process proceeds to step S59. If not stored, the process proceeds to step S61.

[S59]: The similarity between the sub-partitions of the movement destination sub-partition (p _dst , q _dst ) of the current frame is extracted from the sub-partition attribute storage unit 128, and is set as S _total-save .

[S60]: The similarity S _total (p ′, q ′, p, q) between the sub-partitions obtained in step S53 is compared with the similarity S _total-save extracted in step S59.
If S _total (p ′, q ′, p _dst , q _dst ) ≧ S _total-save , the process proceeds to step S61.
If S _total (p ′, q ′, p _dst , q _dst ) <S _total-save , the process proceeds to step S62.

[S61]: The sub-partition attribute storage unit 128 stores the source ID value Attrib_SubBlock (t−1) [p ′] [q ′] [kID], the sub-partition similarity S _total (p ′, q ′, p _dst , _qdst ) is set.

  [S62]: If all the sub-sections of the previous frame have been scanned, the process returns to step S42. If not scanned, the process returns to step S44.

  When information on the moving object is required, data is read from the sub-partition attribute storage unit 128.

[Moving object direction calculation unit 13]
The azimuth angle of the moving object is calculated from the detection result of the image processing. The angle detection method will be described with reference to FIGS.

  FIG. 10 is an external view schematically showing the positional relationship between the video camera 20 and the moving body. The moving body direction calculation unit 13 calculates the azimuth angle of the moving body with reference to the left end of the camera field of view (the left end of the image) as shown in the figure.

When the moving object detection unit 12 determines the image area of the moving object, the direction of the moving object is obtained as follows. In FIG. 11, it is assumed that the video camera 20 is facing the right direction in the figure. The viewing angle of the video camera 20 is 2α. The pixel value of the screen is 0 at the left end and x _{size at} the right end, and the center position of the moving object is the center position x _trk-cen of the tracking block. At this time, the azimuth angle β of the moving body detected on the screen as shown in FIG. 12 is obtained as follows.

  When there are a plurality of moving objects, the same processing is repeated for each moving object to calculate the azimuth angles of all the moving objects.

[Antenna direction control unit 14]
The antenna direction control unit 14 performs communication with the RF tag 33 attached to the moving body by matching the directivity of the antenna to the azimuth angle of the moving body obtained by the moving body direction calculating unit 13. Therefore, as the directional antenna 31, for example, an adaptive array antenna having a variable directivity pattern as shown in FIG. 13 is used. An adaptive array antenna weights signals received by a plurality of antenna elements and controls the directivity pattern by appropriately changing the weights.

  Since the directivity of the antenna is reversible in transmission and reception, the calculation method of the weighting factor of the adaptive array antenna at the time of reception is shown below.

The direction of arrival of radio waves from the RF tag 33 with respect to the directional antenna 31 shown in FIG. When the antenna elements are arranged in a straight line at equal intervals D, the propagation delay time τ _i (θ) to the i-th antenna when compared with the reference antenna is

It is represented by c is the propagation speed. If the modulation signal is m (n), the angular frequency of the carrier wave is ω, the i-th antenna output is x _i (n), and n is time,

It becomes.

  If the desired signal waveform is known and can be used as the reference signal d (n), the weighting factor can be determined by minimizing the mean square error with the array output signal y (n). Let u (n) be an error component such as interference or noise.

u (n) = d (n) -y (n)
The input signal x _i (n) and the weighting coefficient w _i (θ) of each antenna are expressed as vectors, and the combined output of each element is substituted into the above equation as y (n).

X (n) ^T = [x ₁ (n) x ₂ (n) ...]
W (θ) ^T = [w ₁ (θ) w ₂ (θ) ...]
y (n) = W (θ) ^H X (n)
u (n) = d (n) −W (θ) ^H X (n)
Note that the subscript T represents transposition and H represents complex conjugate transposition. At this time, the expected value J of the square of the error component is expressed by the following equation.

J = E {| d (n) -W (θ) ^H X (n) | ² }
The autocorrelation matrix of the input signal of each antenna is R _xx = E {x (n) x (n) ^H }, and the cross-correlation matrix of the reference signal and the input signal of each antenna is r _xd = E {x (n) d ^If defined as ^* (n)}, the optimum weight is obtained by partial differentiation of the expected value J with the weight vector W,
W (θ) = R _xx ⁻¹ r _xd
Given in.

As described above, the radio wave arrival direction θ can be associated with the weighting factor W (θ). Therefore, when radio waves are radiated from the antenna to the RF tag 33, when the range of the camera viewing angle is θ ₁ to θ ₂ and the direction of the antenna is swung by the step size Δθ, a weighting factor is set in advance.
θ ₁ → W (θ ₁ )
θ ₁ + Δθ → W (θ ₁ + Δθ)
θ ₁ +2 ・ Δθ → W (θ ₁ +2 ・ Δθ)
:
θ ₁ + k · Δθ → W (θ ₁ + k · Δθ)
:
θ ₂ → W (θ ₂ )
Calculate as follows. The weighting coefficient when the azimuth angle of the moving body obtained by the moving body direction calculation unit 13 is β is
β ≒ θ ₁ + n · Δθ
Wβ = W (θ ₁ + n · Δθ)
It is calculated as follows. By setting the weighting coefficient Wβ and transmitting, the directivity pattern of the antenna can be directed in the β direction.

[RF tag 33]
The RF tag 33 includes an IC chip and an antenna (coil), and is a device that can communicate with the RF tag reader / writer 30 in a non-contact manner by using wireless communication. In the case of an RF tag 33 of a type that obtains driving power from electromagnetic waves supplied from the RF tag reader / writer 30, it can operate without a built-in power source such as a battery. A battery built-in type RF tag 33 may be used. In the present system, when a person is the target of a moving object and the person is wearing the RF tag 33, it is desirable that the person be worn on the head or chest where the movement is relatively stable.

  In the memory in the RF tag 33, the name, characteristics (color, shape, ID number, etc.), specifications, or network address in which these are described, depending on the subject to which the RF tag 33 is attached. Is stored in advance. When the writable RF tag 33 is used, it is possible to rewrite the data content stored in the memory during shooting with the video camera 20 as necessary.

  Some of the battery built-in RF tags 33 are provided with a CPU (Central Processing Unit), and it is possible to implement a calculation for obtaining an orientation on the tag side as follows.

FIG. 14 is an explanatory diagram of a method for calculating the respective azimuth angles when the RF tags 33 of ID1 and ID2 are attached to two moving bodies. First, radio waves are radiated in order from the antenna direction control unit 14 to the azimuth angles β1 and β2 using the azimuth angles β1 and β2 obtained by the moving body direction calculation unit 13. At this time, the azimuth angle information is transmitted to the RF tag 33. The RF tag 33 receives the azimuth angle information β 1 and β 2, detects the received signal strength at that time, and stores it in the memory in the RF tag 33. The received signal strength of each RF tag 33 is
• When the antenna's directional beam is directed in the β1 direction, S _ID1 > S _ID2
• When the directional beam of the antenna is pointing in the β2 direction, S _ID1 <S _ID2
It becomes.

  The ID1 RF tag 33 is positioned in the β1 direction because the received signal strength is strong when the azimuth angle is β1, and the ID2 RF tag 33 is strong in the received signal strength when the azimuth angle is β2. From this, the result that it is located in the β2 direction is obtained. By transmitting the azimuth angle of the tag obtained together with the attribute information from the RF tag 33 to the RF tag reader / writer 30, the mobile object and the RF tag (ID value) can be linked.

[RF tag reader / writer 30]
At the time of data reading, the RF tag reader / writer 30 demodulates the signal modulated by the data of the RF tag 33 from the signal y (n) output from the directional antenna 31, and further performs AD conversion to perform digital conversion. Get the data. The digital data is output to the metadata generation unit 15 via an appropriate signal line in the signal input / output interface 11. In addition, when data is written, digital data is written into the memory in the RF tag 33 by performing the reverse operation of reading.

[Metadata generation unit 15]
Metadata is generated from the attribute information transmitted from the RF tag 33 together with the azimuth. In order to shorten the transmission time, only data may be sent from the RF tag 33, and the data may be sent in a predetermined order. In this case, the digital data output from the RF tag reader / writer 30 is converted into metadata by applying a predetermined rule. An example of conversion to metadata will be described with reference to FIG. For example, when this device is applied to a marathon, the RF tag 33 (ID1, ID2)
-Runner number number-Competition date-Start time-Numeric value and character string data such as events are stored. The numerical value / character string data is converted into metadata by the metadata generation unit 15 and sent to the metadata adding unit 16.

  As the metadata, variable values can be used instead of fixed values. For example, sensor data can be embedded in a frame image acquired by the video camera 20 by mounting a sensor for detecting temperature, blood pressure, and pulse on the RF tag 33 and sending the sensor to the RF tag reader / writer 30. Also, based on the ID obtained from the RF tag 33, various attribute information registered in advance corresponding to the ID can be obtained from other devices and used as metadata. Further, the ID of the RF tag 33 can be used as metadata as the unique information of the mobile object.

[Metadata giving unit 16]
The azimuth angle of the mobile body obtained by the mobile body direction calculation unit 13 is compared with the azimuth angle of the tag sent from the RF tag 33, and the mobile body having the same azimuth angle and the RF tag 33 are linked.

  FIG. 15 is a diagram illustrating an association between an image area and metadata in the present embodiment. Usually, the video camera 20 outputs image data (frame image) for one frame at an appropriate sampling period (for example, 33 ms). The frame number is associated with all the movement tracking block numbers existing in the frame image. In the movement tracking block number, the number of the RF tag 33 recognized as the number of the moving body to be tracked is described.

  Frame numbers 1 and 2 in FIG. 15 indicate a situation in which no moving object is detected. The frame number 3 is reached, the moving body is detected at the movement tracking block numbers 1 and 3, and the movement tracking block number 1 indicates that the RF tag of the tag 1 can be recognized. In the movement tracking block number 1, the moving body number 1 and the tag 1 of the moving body are obtained from the moving body direction calculation unit 13. The moving tracking block number 3 indicates that the moving object can be detected, but the tag number cannot be recognized. The tag 5 can be recognized at the frame number 5 and the movement tracking block number 2, and the tag 8 can be recognized at the frame number 8 and the movement tracking block number 3.

  The video with metadata generated in the metadata adding unit 16 is transmitted to the display device 41 via the signal input / output interface 17 and the network 42, for example. Note that a recording medium 43 such as streaming relay using the Internet, DVD, video, or CD-ROM is also conceivable as an image output destination.

[Overall processing flow]
Next, FIG. 16 shows a processing flow from the detection of the moving object by image processing until the ID of the RF tag 33 is given to the moving object.

  [S70]: The moving object detection unit 12 detects the moving object shown in the image.

  [S71]: The moving body direction calculation unit 13 calculates the azimuth from the position of the image area occupied by the moving body.

  [S72]: The antenna direction control unit 14 directs the direction of the antenna to the angle obtained in step S71. At this time, the azimuth angle (radiation angle data) is transmitted to the RF tag 33.

  [S73]: The tag side stores the received azimuth value and the signal strength at that time in association with each other.

  [S74]: Steps S72 and S73 are repeated until radio waves are radiated in all directions obtained in step S71.

  [S75]: The RF tag 33 selects the azimuth angle having the strongest signal intensity among the detected signals, and transmits it to the antenna side together with the attribute information of the tag.

  [S76]: The angle data and attribute information transmitted from the RF tag 33 are received on the antenna side.

  [S77]: The ID of the metadata converted by the metadata generation unit 15 is assigned to the moving object having the same angle data received from the RF tag 33 in step S76 and the image area.

[Other Examples]
In the above example, the method of receiving radio waves radiated from the antenna side with the RF tag 33 and comparing the signal strengths has been described. In the present embodiment, a method of receiving radio waves radiated from the RF tag 33 on the antenna side and comparing the signal strength will be described. As shown in FIG. 14, the case where the two moving bodies are each equipped with the RF tag 33 is taken as an example. First, ID numbers are sequentially transmitted from the two RF tags 33 with the antenna receiving direction fixed at β1. On the antenna side, at the same time as receiving the ID number, the signal strength is detected and stored in a memory or the like. Next, the ID number is transmitted from the RF tag 33 with the receiving direction of the antenna fixed at β2, and the signal strength is stored in the same manner. When the scanning in all directions is completed, the reception intensity of the radio wave emitted from each RF tag 33 is compared.

In the example of FIG. 14, the received signal strength of each tag is
• When the antenna's directional beam is directed in the β1 direction, S _ID1 > S _ID2
• When the directional beam of the antenna is pointing in the β2 direction, S _ID1 <S _ID2
It becomes.

  The ID1 RF tag 33 is located in the β1 direction because the signal strength is strong when the azimuth angle is β1, and the ID2 RF tag 33 is strong in signal strength when the azimuth angle is β2. The result that it is located in the β2 direction is obtained. By receiving the attribute information together with the ID number, the mobile object and the RF tag (ID value, attribute information) can be linked.

[Processing Flow of Other Examples]
The flow of processing in this embodiment is shown in FIG.

  [S80]: The moving object detection unit 12 detects the moving object shown in the image.

  [S81]: The moving body direction calculation unit 13 calculates the azimuth from the position of the image area occupied by the moving body.

  [S82]: The antenna direction control unit 14 directs the antenna reception direction to the angle obtained in step S81, and waits for a signal sent from the RF tag 33.

  [S83]: The ID number and attribute information are transmitted from the RF tag 33.

  [S84]: The ID number / attribute information sent from the RF tag 33 is stored, and the signal strength at the time of reception is stored.

  [S85]: Steps S82 to S84 are repeated until radio waves are radiated from all the RF tags 33.

  [S86]: A directional beam is directed in all directions obtained in step S81, and steps S82 to S85 are repeated until data from the RF tag 33 is received in each direction.

  [S87]: Metadata is added to the image area by associating the ID number of the RF tag 33 having the strongest signal strength with a moving object that matches the received orientation.

  The processing performed by the automatic metadata adding apparatus 10 described above can be realized by a computer and a software program. The program can be provided by being recorded on a computer-readable recording medium or provided via a network. Is possible.

It is a block diagram of the apparatus concerning the Example of this invention. It is a figure which shows the relationship between a tracking block, a subdivision, and a subdivision template. It is a detection process flowchart of a moving body. It is an extraction process flowchart of the candidate of a moving body image area | region. It is a determination process flowchart of a moving body. It is a block diagram of the mobile body detection part which performs a mobile body detection method. It is a figure which shows the example of calculation of a histogram. It is a figure which shows the example of a sub division attribute. It is a detection process flowchart of the moving body at the time of using the 2nd moving body detection method. It is the external view which showed typically the positional relationship of a video camera and a moving body. It is explanatory drawing of an angle detection method. It is explanatory drawing of an angle detection method. It is a figure which shows the example of a directional antenna. It is explanatory drawing of the method of calculating each azimuth angle of two moving bodies. It is a figure which shows correlation with an image area | region and metadata. It is the whole processing flowchart. It is a process flowchart of another Example.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Metadata automatic provision apparatus 11, 17 Signal input / output interface 12 Mobile body detection part 13 Mobile body direction calculation part 14 Antenna direction control part 15 Metadata production | generation part 16 Metadata provision part 20 Video camera 30 RF tag reader / writer 31 Directionality Antenna 32 Interface 33 RF tag 40, 41 Display device 42 Network 43 Recording medium

Claims (6)

It has an input / output interface between an imaging device that captures a moving body with an RF tag and an RF tag input / output device having a directional antenna, and extracts an image area of the moving body from an image captured by the imaging device. , An automatic metadata adding device for adding metadata including unique information or attribute information obtained from the RF tag to an image area of a mobile object,
A moving body detecting means for detecting an image area of the moving body by image processing from an image photographed by the photographing device;
Moving body direction calculating means for calculating the direction of the moving body from the detected image area of the moving body;
Antenna direction control means for controlling the directivity of the directional antenna in the RF tag input / output device to the direction of the moving object calculated by the moving object direction calculating means;
The RF tag input / output device reads specific information or attribute information emitted by the RF tag, and the direction of the mobile body calculated by the mobile body direction calculation means and the radio wave from the RF tag by the RF tag input / output device. Meta data that associates the image area of the moving object with the RF tag from the angle information obtained by reception, and assigns metadata including specific information or attribute information read from the RF tag to the image area of the moving object. An apparatus for automatically assigning metadata to an image area, comprising: a data attaching unit. The antenna direction control means places a data signal of the radiation angle of the radio wave on the radio wave transmitted from the directional antenna of the RF tag input / output device,
The metadata assigning means obtains radiation angle data when the RF tag is received at the strongest radio wave intensity from a radio wave signal transmitted by the RF tag, and uses the angle data as an image area of the mobile object. The apparatus for automatically assigning metadata to an image area according to claim 1, wherein the apparatus is used for association with the RF tag. The antenna direction control means receives the radio wave from the RF tag by sequentially controlling the directivity of the directional antenna of the RF tag input / output device to the direction of the moving object calculated by the moving object direction calculating means. Control
The metadata providing means sets a direction in which the radio wave intensity is strongest when radio waves are received from the RF tag as a direction of the RF tag, and angle information of the direction is obtained between the image area of the mobile object and the RF tag. The apparatus for automatically assigning metadata to an image area according to claim 1, wherein the apparatus is used for association. It has an input / output interface between an imaging device that captures a moving body with an RF tag and an RF tag input / output device having a directional antenna, and extracts an image area of the moving body from an image captured by the imaging device. , An automatic metadata assignment method executed by an automatic metadata assignment device for assigning metadata including unique information or attribute information obtained from the RF tag to an image area of a moving object,
A process of detecting an image area of a moving object by image processing from an image photographed by the photographing device;
Calculating the direction of the moving object from the detected image area of the moving object;
A process of radiating a radio wave including a signal of angle data in the direction from the directional antenna of the RF tag input / output device toward the calculated moving body;
The RF tag input / output device reads the unique information or attribute information emitted by the RF tag and the angle data when the RF tag is received with the strongest radio wave intensity, and calculates the direction of the calculated moving body and the RF Based on the angle data obtained from the RF tag by the tag input / output device, the image area of the moving object is associated with the RF tag, and the unique information read from the RF tag in the image area of the moving object or A method of automatically assigning metadata to an image area, comprising: a step of assigning metadata including attribute information. It has an input / output interface between an imaging device that captures a moving body with an RF tag and an RF tag input / output device having a directional antenna, and extracts an image area of the moving body from an image captured by the imaging device. , An automatic metadata assignment method executed by an automatic metadata assignment device for assigning metadata including unique information or attribute information obtained from the RF tag to an image area of a moving object,
A process of detecting an image area of a moving object by image processing from an image photographed by the photographing device;
Calculating the direction of the moving object from the detected image area of the moving object;
A process of sequentially controlling the directivity of the directional antenna of the RF tag input / output device in the direction of the calculated moving body and receiving radio waves from the RF tag;
The RF tag input / output device reads specific information or attribute information emitted by the RF tag, and the direction of the RF tag when the radio wave intensity is received from the RF tag is defined as the direction of the RF tag. Based on the direction of the body and the direction of the RF tag, the image area of the moving body is associated with the RF tag, and the image area of the moving body includes unique information or attribute information read from the RF tag. A method for automatically assigning metadata to an image area. It has an input / output interface between an imaging device that captures a moving body with an RF tag and an RF tag input / output device having a directional antenna, and extracts an image area of the moving body from an image captured by the imaging device. , An automatic metadata assignment program for causing a computer of an automatic metadata assignment device to assign metadata including unique information or attribute information obtained from the RF tag to an image area of a mobile object,
Said computer,
A moving body detecting means for detecting an image area of the moving body by image processing from an image photographed by the photographing device;
Moving body direction calculating means for calculating the direction of the moving body from the detected image area of the moving body;
Antenna direction control means for controlling the directivity of the directional antenna in the RF tag input / output device to the direction of the moving object calculated by the moving object direction calculating means;
The RF tag input / output device reads specific information or attribute information emitted by the RF tag, and the direction of the mobile body calculated by the mobile body direction calculation means and the radio wave from the RF tag by the RF tag input / output device. Meta data that associates the image area of the moving object with the RF tag from the angle information obtained by reception, and assigns metadata including specific information or attribute information read from the RF tag to the image area of the moving object. As a data giving means,
A program for automatically assigning metadata to image areas for functioning.

Images