CN101204088A - Metadata adding apparatus and metadata adding method - Google Patents

Abstract

It is possible to easily search for and extract images which have captured the same region, with a low load. A multi-angle information generation device (10) for grouping images captured by a plurality of imaging devices (20) includes: sensing meta data acquisition means (101) for acquiring sensor information concerning the imaging condition of the imaging devices (20); focus plane meta data derivation means (102) for deriving focus planes containing the images captured by the respective imaging devices (20); grouping judgment means (103) for grouping images according to the positional relationship of the respective focus planes; and multi-angle meta data recording means (104) for recording the grouping results as multi-angle information while correlating them with the aforementioned images.

Description

Metadata adding set and metadata adding method

Technical field

The present invention relates to metadata is added to the metadata adding set and the metadata adding method of the image that imaging device catches.

Background technology

Traditionally, proposed to come a lot of apparatus and method that the image of being caught is classified (classify) and managed according to theme.Among them, there is such seizure image processing apparatus, it utilizes according to the graphical analysis of object etc., and to the image of being caught classify (for example, seeing patent documentation 1).In this device, according to object, automatically to classifying and manage by catch resulting Still image data by digital camera etc.

In a lot of situations, produced for the needs that come according to object the image of being caught is classified.Except rest image, in the live sports cast of broadcasting from the video of the camera that is placed in a plurality of positions, for example, there is such situation, from a plurality of video datas, extract the video section relevant as expectation with specific conclusive moment, and, edit described video section, so that the video section of being edited as broadcasting at the video (multi-angle video) of the same target of different angles photographs and continuously.

Patent documentation 1:JP-A-2004-356984 (the 6th page, Fig. 1)

Summary of the invention

The problem that real name will solve

Yet, need very big processing load based on traditional classification of graphical analysis.Therefore, for classification the video that constitutes from each free a plurality of picture frame and extract the purpose of the video section of wherein catching same target, it is unpractiaca using such classification.For example, with the video of considering that 30 picture frames of each free per second constitute.Be scheduled to need the graphical analysis of 60 * 30 * 3=5400 frame under the situation of video from the visual classification and the extraction that have the 60 seconds length of taking separately by three cameras.

In addition, in the traditional classification based on graphical analysis, by different way under the situation of the image of captured object (that is, the angle of object and vary in size), trimming process is necessary therein.Therefore, the identification accuracy is bad often.In the example of superincumbent live sports cast, camera is placed different positions, and thus, always captured object in a different manner.And,, in graphical analysis, be difficult to classify and extract the arbitrary portion of video from this point.

For example, will consider such situation, wherein, in the broadcasting of baseball game, broadcast the scene that specific team member hits hommer continuously as the video of various angles.Under these circumstances, traditionally, need carry out editing, wherein, manually (that is, visually) search for each video, and, extract relative section, and it is connected to each other.

Consider conventional cases discussed above and made the present invention.The objective of the invention is to, such metadata adding set and method is provided, wherein, make it possible under low load and carry out in an easy manner by catching the search and the extraction of the image that the same area obtains.

Solve the means of described problem

The device that is used to add metadata of the present invention is metadata to be added to the metadata adding set of the image of being caught by imaging device, and comprises: the sensitive information acquiring unit is used to obtain the sensor information relevant with the capturing condition of imaging device; Focussing plane (focus-plane) lead-out unit is used for based on the sensor information of being obtained, and derivation is as the position of the focussing plane of the imaging surface of the image of being caught; And the metadata adding device, the position that is used for the focussing plane of will be derived is added the image of being caught to as metadata.According to this configuration, add the position of focussing plane to this image as metadata, and, based on the position of focussing plane relation and to image packets.Therefore, compare, can reduce to handle load with traditional technology of carrying out grouping by graphical analysis.Thus, make it possible under low load and carry out in an easy manner by catching the search and the extraction of the image that the same area obtains.

In addition, metadata adding set of the present invention comprises: grouped element, be used for based on the relation of the position between the focussing plane, and to image packets; And the interpolation information recording unit, the result who is used to write down grouping is as interpolation information, and is simultaneously that interpolation information is relevant with image.According to this configuration, derived the focussing plane that comprises the image of being caught, and, based on the position relation of focussing plane, and to image packets.Therefore, compare, can reduce to handle load with traditional technology of carrying out grouping by graphical analysis.Thus, make it possible under low load and carry out in an easy manner by catching the search and the extraction of the image that the same area obtains.

In addition, in metadata adding set of the present invention, the image packets that grouped element will have the focussing plane that intersects each other is same group.According to this configuration, can be by calculating to image packets.

In addition, in metadata adding set of the present invention, based on the table of the position relation between the storage focussing plane, it is same group that grouped element will have the image packets that is included in the focussing plane in the relation of position.According to this configuration, when before to have determined to be used for image classification be same group the position of focussing plane, can be to image packets under situation about not calculating.

The method of interpolation metadata of the present invention is metadata to be added to the metadata adding method of the image of being caught by imaging device, and has: the sensitive information obtaining step, obtain the sensor information relevant with the capturing condition of imaging device; Focussing plane is derived step, and based on the sensor information of being obtained, and derivation is as the position of the focussing plane of the imaging surface of the image of being caught; And metadata adds step, adds the position of the focussing plane of being derived to caught image as metadata.

In addition, metadata adding method of the present invention has: the grouping step, and based on the relation of the position between the focussing plane, and to image packets; And the information of interpolation recording step, the result who is used to write down grouping is as interpolation information, and is simultaneously that interpolation information is relevant with image.

In metadata adding method of the present invention, the image packets that the grouping step will have the focussing plane that intersects each other is same group.

In metadata adding method of the present invention, based on the table of the position relation between the storage focussing plane, it is same group that the grouping step will have the image packets that is included in the focussing plane in the relation of position.

Effect of the present invention

According to the present invention, add the position of focussing plane to image as metadata, and, based on the position of focussing plane relation and to image packets.Therefore, compare, can reduce to handle load, and precision that can be higher realizes identical purpose with traditional technology of carrying out grouping by graphical analysis.Thus, make it possible under low load and carry out in an easy manner by catching the search and the extraction of the image that the same area obtains.

Description of drawings

Fig. 1 is the figure of configuration that the internal configurations of the multi-angle information generation device in the embodiments of the invention is shown and comprises the multi-angle information generating system of this multi-angle information generation device.

Fig. 2 is the figure of the internal configurations of the imaging device that uses in the multi-angle information generating system that illustrates in an embodiment of the present invention.

Fig. 3 is the flow chart of the operating process of the imaging device that uses in the multi-angle information generating system that illustrates in an embodiment of the present invention.

Fig. 4 is the flow chart of process that the videograph operation of imaging device is shown.

Fig. 5 is the flow chart of process that the sensing metadata generating run of imaging device is shown.

Fig. 6 is the figure that illustrates the data structure of the sensing metadata that is generated.

Fig. 7 is the flow chart of process that the multi-angle information generating run of the multi-angle information generation device in the embodiments of the invention is shown.

Fig. 8 is the figure of diagram focussing plane.

Fig. 9 illustrates the flow chart that the focussing plane of multi-angle information generation device is derived the process of operation.

Figure 10 is the figure that illustrates the data structure of the focussing plane metadata that is generated.

Figure 11 is the flow chart of process that the multi-angle metadata generating run of multi-angle information generation device is shown.

Figure 12 is the figure of the judgement of intersecting of diagram focussing plane.

Figure 13 is the flow chart of process that the grouping decision of multi-angle information generation device is shown.

Figure 14 is the figure that illustrates the data structure of the multi-angle metadata that is generated.

Figure 15 is the figure of the judgement of the existence of diagram focussing plane in presumptive area.

The figure of Figure 16 rule of classification that to be diagram generate by the positional information of specifying a plurality of zones.

Figure 17 is the flow chart of process that the grouping decision of the multi-angle information generation device among the embodiment 2 is shown.

Figure 18 is the figure that illustrates the data structure of the multi-angle metadata that is generated.

Figure 19 is the figure of configuration that the internal configurations of the interpolation information generation device in the embodiments of the invention 3 is shown and comprises the interpolation information generating system of this interpolation information generation device.

Figure 20 is the figure of the internal configurations of the imaging device that uses in the interpolation information generating system that is illustrated in the embodiments of the invention 3.

Figure 21 is the flow chart of the operating process of the imaging device that uses in the interpolation information generating system that illustrates in an embodiment of the present invention.

Figure 22 is the flow chart of process that the videograph operation of imaging device is shown.

Figure 23 is the flow chart of process that the sensing metadata generating run of imaging device is shown.

Figure 24 is the figure that illustrates the data structure of the sensing metadata that is generated;

Figure 25 is the flow chart of process that the interpolation information generating run of the interpolation information generation device in the embodiments of the invention is shown.

Figure 26 is the figure of diagram focussing plane.

Figure 27 is the flow chart that the process of the focussing plane derivation operation of adding information generation device is shown.

Figure 28 is the figure that illustrates the data structure of the focussing plane metadata that is generated.

Figure 29 is the flow chart that the process of the interpolation metadata generating run that adds information generation device is shown.

Figure 30 is the figure of image that the combination of all frames is shown.

Figure 31 is the figure of the judgement of intersecting of diagram focussing plane.

Figure 32 is the flow chart that the process of the grouping decision of adding information generation device is shown.

Figure 33 is the figure that illustrates the data structure of the interpolation metadata that is generated.

The description of drawing reference numeral and mark

10: the multi-angle information generation device

20,1020: imaging device

30,1030: database

40: the multi-angle video searching apparatus

101,1101: the sensing metadata acquiring unit

102,1102: focussing plane metadata lead-out unit

103,1103: the grouping identifying unit

104: multi-angle metadata record unit

201: lens group

202、1202：CCD

203,1203: drive circuit

204,1204: the timing signal generation unit

205,1205: sampling unit

206,1206:A/D converting unit

207,1207: the video file generation unit

208,1208: the video address generation unit

209,1209: the video identifier generation unit

210,1210: the machine information transducer

211,1211: the sensing metadata generation unit

212,1212: record cell

1010: add information generation device

1040: video searching apparatus

1104: the metadata record unit

Embodiment

Hereinafter, will describe metadata adding set according to an embodiment of the invention in detail by the reference accompanying drawing.In embodiment 1 and 2, show the example that the metadata adding set is embodied as the multi-angle information generation device, and, in embodiment 3, show the example that the metadata adding set is embodied as the interpolation information generation device.

[embodiment 1]

Fig. 1 is the figure of configuration that the internal configurations of the multi-angle information generation device in the embodiments of the invention is shown and comprises the multi-angle information generating system of this multi-angle information generation device.Multi-angle information generating system shown in Fig. 1 comprises: multi-angle information generation device 10, and it is to dividing into groups by catching the image that obtains by a plurality of imaging devices; A plurality of imaging devices 20 (20a to 20n); Database 30; And multi-angle video searching apparatus 40.Hereinafter, will the example that the multi-angle information generating system divides into groups to the video by a plurality of image constructions be described.

Multi-angle information generation device 10 comprises sensing metadata acquiring unit 101, focussing plane metadata lead-out unit 102, grouping identifying unit 103 and multi-angle metadata record unit 104.

Sensing metadata acquiring unit 101 obtains the sensor information relevant with the capturing condition of imaging device 20.Sensing metadata acquiring unit 101 is via database 30, and obtains the relevant sensing metadata of position, orientation, inclination angle, subtended angle and focusing distance (focus distance) with each imaging device.In this embodiment, suppose by imaging device 20 generation sensing metadata.The internal structure of imaging device 20 and the details of sensing metadata will be described in the back.

Focussing plane metadata lead-out unit 102 is based on resulting sensing metadata, and derivation is as the focussing plane of the imaging surface of the image of being caught by imaging device 20, and, calculate rectangle based on sensing metadata, the seizure focussing plane in the real space of its indication imaging device 20 as coordinate figure.To describe the focussing plane metadata in the back in detail.

Grouping identifying unit 103 is based on the position of focussing plane relation and to image packets.In the focussing plane that uses each imaging device of being derived by focussing plane metadata lead-out unit 102, the grouping identifying unit is based on predetermined decision condition, and whether judgement has obtained described image by catching the same area.

Multi-angle metadata record unit 104 writes down the result of grouping as multi-angle information, wherein, this information is relevant with image, and, as the multi-angle metadata, will be judged as the information relevant and export and record in the database 30 with image by catching those images that the same area obtains.To describe the multi-angle metadata in the back in detail.

Multi-angle information generation device 10 is connected to the database 30 of storage from the video data of a plurality of imaging devices 20, based on the sensing metadata that obtains from imaging device, and produce as with the multi-angle metadata of relevant information by catching a plurality of video datas that same target obtains simultaneously, and these data are outputed to database 30.The multi-angle video searching apparatus 40 that is connected to database 30 can be searched for video data based on the multi-angle metadata.

Next, imaging device will be described.Fig. 2 is the figure of the internal configurations of the imaging device that uses in the multi-angle information generating system that illustrates in an embodiment of the present invention.Imaging device 20 comprises lens group 201, CCD 202, drive circuit 203, timing signal generation unit 204, sampling unit 205, A/D converting unit 206, video file generation unit 207, video address generation unit 208, video identifier generation unit 209, machine information transducer 210, sensing metadata generation unit 211 and record cell 212.

The timing signal synchronised ground driven CCD 202 that is generated with the timing signal generation unit 204 that is connected to drive circuit 203, and, will be by lens group 201 picture signal of the object images of incident output to sampling unit 205.

Sampling unit 205 comes picture signal is sampled with the sample rate that is exclusively used in CCD 202.A/D converting unit 206 will be converted to DID from the picture signal of CCD 202 outputs, and these data are outputed to video file generation unit 207.

Video address generation unit 208 is in response to from the signal of timing signal generation unit 204 and produce video address.Video identifier generation unit 209 sends and adds the identifier that video is relevant with the sensing metadata of describing later (for example, filename or ID).

Utilize GPS (global positioning system) receiver, gyro sensor (gyro sensor), aspect sensor, range sensor and subtended angle transducer to dispose machine information sensor 210.

Gps receiver receives radio wave from satellite, to obtain the distance with the three or more artificial satellites of its position previously known, thus, can obtain the three-dimensional position (latitude, precision, highly) of gps receiver self.When using this function, might obtain imaging device absolute position on earth.

Gyro sensor is commonly called three acceleration sensors, and uses terrestrial gravitation to detect along the degree of the axial acceleration of looking from this transducer, that is, and and as the gradient vertically of numerical value.When using this function, might obtain the gradient (azimuth, inclination angle) of imaging device.

Aspect sensor is commonly called electronic compass (electronic compass), and uses the magnetic of the earth to detect tellurian north, south, east or western direction.When with gyro sensor and aspect sensor combination, might detect imaging device absolute direction on earth.

Range sensor is the transducer that is measured to the distance of object.This transducer sends infrared ray or ultrasonic wave from imaging device towards object, and can be till receive reflection up to imaging device elapsed time, know the distance from the imaging device to the object, that is, be used for the focusing distance that obtains to focus on.

The subtended angle transducer can obtain subtended angle from focal length (focal length) and the height of CCD.Can obtain focal length by the distance of measuring between camera lens and the light receiving part, and the height of light receiving part is the value that is exclusively used in imaging device.

Based on output request from sensing metadata 211, machine information transducer 210 output and position, the orientation that will be used as benchmark, azimuth, inclination angle, subtended angle and the relevant sensitive information of focusing distance from the imaging device of GPS (global positioning system) receiver, gyro sensor (gyro sensor), aspect sensor, range sensor and subtended angle transducer.Sensing metadata generation unit 211 generates regularly according to the video address from video address generation unit 208, and obtains sensitive information from machine information sensor 210, produces sense data, and these data are outputed to record cell 212.In response to the signal from timing signal generation unit 204, machine information transducer 210 and sensing metadata generation unit 211 begin operation.

The generation of sensitive information and output and main purpose of the present invention are irrelevant, and, therefore, omit the detailed description of the operation of transducer.

Sampling rate (1/30 second) that can CCD is carried out obtaining of sensitive information, perhaps, can carry out obtaining of sensitive informations every several frames.

Indoor execution take or the idle situation of GPS transducer under, the positional information in input capture place manually.In the case, will be input to the machine information transducer by the positional information of unshowned input unit input.

Hereinafter, the sensing metadata generating run that description is had the imaging device of above-mentioned configuration.Fig. 3 is the flow chart of the operating process of the imaging device that uses in the multi-angle information generating system that illustrates in an embodiment of the present invention.

At first, when pushing of the predetermined switch of the formant of carrying out imaging device waited, receive seizure commencing signal (step S101).Subsequently, imaging device 20 beginning videograph processes (step S102), and imaging device 20 begins to generate the process (step S103) of sensing metadata.When timing signal generation unit 204 received the seizure end signal, imaging device 20 stopped videograph process and sensing metadata generative processes (step S104).

To be described in videograph process that begins among the step S102 and the sensing metadata generative process that in step S103, begins by the reference Figure 4 and 5.

Fig. 4 is the flow chart that is illustrated in the process of the videograph operation among the step S102.When obtaining seizure commencing signal (step S201), in response to operation directive command from timing signal generation unit 204, and beginning capture operation (step S202).In addition, in response to directive command, and generate video identifier (step S203) by video identifier generation unit 209 from timing signal generation unit 204.

Obtain the video electrical signal (step S204) from CCD 202,205 pairs of signals that obtained of sampling unit are carried out sampling (step S205), and A/D converting unit 206 is carried out the conversion (step S206) of DID.

In response to directive command, and generate video address (step S207) by video address generation unit 208 from timing signal generation unit 204, and, video file (step S208) generated by video file generation unit 207.In addition, add the video identifier (step S209) that generates by video identifier generation unit 209, and, final video file is recorded (step S210) in the record cell 212.

Fig. 5 is the flow chart that is illustrated in the process of the sensing metadata generating run among the step S103.When having obtained seizure commencing signal (step S301), sensing metadata generation unit 211 obtains the video address (step S302) that is generated by video address generation unit 208.Obtain the video identifier (step S303) that generates by video identifier generation unit 209.In addition, and obtain video address side by side, sensing metadata generation unit 211 sends the request that is used to export sensitive information to machine information transducer 210, with the information of the position, azimuth, inclination angle, subtended angle and the focusing distance that obtain camera.Can obtain the position of camera from gps receiver, can obtain azimuth and inclination angle, can obtain focusing distance from range sensor from gyro sensor, and, can obtain subtended angle (step S304) from the subtended angle transducer.

Next, sensing metadata generation unit 211 writes down focusing distance with video identifier that is acquired and video address, generate and output sensing metadata (step S305), and, with data record (step S306) in record cell 212.

Fig. 6 is the figure that illustrates the data structure of the sensing metadata that is generated.Video identifier is added to a series of video datas of the frame configuration by arbitrary number.By this video identifier, allow video data uniquely corresponding to sensing metadata.For each video address, record camera coordinates, azimuth, inclination angle, subtended angle and focusing distance.The least unit of video address is the sampling rate of CCD 202, that is, and and frame.For example, will be input to the video identifier of Fig. 6 as " 12345 " of the information of obtaining from video identifier generation unit 209.In addition, will be input to the video address as " 00:00:00:01 " of the information of obtaining from video address generation unit 208.Input is as the camera position " 1; 0; 0 of the information of obtaining from machine information sensor 210 when obtaining video address in video address " 00:00:00:01 " ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".Represent camera position by " x, y, z ", wherein x represents latitude, and y represents longitude, and z represents height (more than the sea level).Latitude, precision and the height of the value of actual input for obtaining by the GPS transducer.Yet, in this embodiment, obtained latitude x=1, precision y=0, and height z=0 in order to describe for simplicity, to suppose.The information that input is obtained from video address generation unit 208 in next video address " 00:00:00:02 ".Input in next video address " 00:00:00:02 ": as the camera position " 1; 0; 0 of the information of obtaining from machine information sensor 210 in the moment of obtaining video address ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".The information that input is obtained from video address generation unit 208 in next video address " 00:00:00:03 ".Input in video address " 00:00:00:03 ": as the camera position " 1; 0; 0 of the information of obtaining from machine information sensor 210 in the moment of obtaining video address ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".

Next, the multi-angle information generating run that description is had the multi-angle information generation device of above-mentioned configuration.Fig. 7 is the flow chart of process that the multi-angle information generating run of the multi-angle information generation device in the embodiments of the invention is shown.

At first, the sensing metadata acquiring unit 101 of multi-angle information generation device 10 obtains all sensing metadata (step S401) of one group of video being taken simultaneously by a plurality of imaging devices 20.Next, focussing plane metadata lead-out unit 102 is derived focussing plane metadata (step S402) based on the sensing metadata of being obtained.

Subsequently, focussing plane metadata lead-out unit 102 determines whether to have finished for all sensing metadata the derivation of focussing plane metadata.If do not finish, the then operation of the derivation focussing plane metadata among the repeating step S402.On the contrary, if finished the derivation of focussing plane metadata for all sensing metadata, then this process is transferred to the operation (step S403) that generates the multi-angle metadata subsequently.Next, grouping identifying unit 103 produces multi-angle metadata (step S404) based on the focussing plane metadata of obtaining from focussing plane metadata lead-out unit 102.

At last, the multi-angle metadata of obtaining from grouping identifying unit 103 (step S405) is exported in multi-angle metadata record unit 104 to database 30.

The operation of the derivation focussing plane metadata among the step S402 will be described by reference Fig. 8 and 9.Fig. 8 is the figure of diagram focussing plane.Focussing plane is meant the rectangular surfaces that is shown as picture zone, wherein, carrying out when catching, obtain focusing on or so-called focus, and the coordinate figure (being called boundary coordinate) at four angles that can be by rectangle is represented focussing plane.As shown in the drawing, come to determine distance by focusing distance (that is, focal length) from imaging device (camera) to focussing plane, and, determine the size of rectangle by the subtended angle of camera.The center of this rectangle is a focus.

To describe the flow chart of Fig. 9, its focussing plane that shows the multi-angle information generation device is derived the process of operation.At first, focussing plane metadata lead-out unit 102 obtains sensing metadata (step S501).

In any camera as shown in Figure 8 and the sensitive information of arbitrary timing be camera position (a, b, c), under the situation of orientation angles α degree, angle of inclination beta degree, subtended angle 2 γ degree and focusing distance L (rice), can obtain wherein camera position (a from orientation angles α degree and angle of inclination beta degree, b c) is set to the direction vector of the camera of reset condition.According to sensitive information, the direction vector of camera is (sin α cos β, cos α cos β, sin β).The direction vector of supposing resulting camera for (e, f, g).The camera direction vector be (e, f g) vertically pass focussing plane, and, be normal vector (step S502) thus to focussing plane.

Next, can be from the camera direction vector (e, f, g) and camera position (a, b, c) derive by camera position (a, b, c) and the linear equation of focus.When using intermediate parameters z, linear equation can be expressed as (ez, fz, gz).According to this linear equation, can derive and be positioned at that (a, b c) separate the coordinate of distance L as focus on this straight line and with camera position.The expression formula that is used to obtain is $L=\sqrt{{(\mathrm{ez}-a)}^2+{(\mathrm{fz}-b)}^2+{(\mathrm{gz}-c)}^2}.$ From then on expression formula derives intermediate parameters z.When having found the solution $L=\sqrt{{(\mathrm{ea}-a)}^2+{(\mathrm{fz}-b)}^2+{(\mathrm{gz}-c)}^2}$ The time, obtain $z=\{(\mathrm{ae}+\mathrm{bf}+\mathrm{cg})\&PlusMinus;\sqrt{{(\mathrm{ae}+\mathrm{bf}+\mathrm{cg})}^2-(e+f+g)(a^2+{\mathrm{<figure-callout\; id="2"\; label="b"\; filenames="S2006800191746E00301.png"\; state="\{\{state\}\}">b</figure-callout>}}^2+c^2-L^2)}\}/(e+f+g),$ And, by replacing at (ez, fz, the z that obtains in gz) and obtain focus (step S503).

Resulting focus is expressed as (h, i, j).Can from normal vector (e, f, g) and focus (h, i j) derive the focussing plane equation.The focussing plane equation is ex+fy+gz=eh+fi+gj (step S504).

According to subtended angle 2 γ degree, (c) distance to the boundary coordinate of focussing plane is L/cos γ for a, b from camera position.Boundary coordinate can be: be at the center camera position (a, b, c) and have a coordinate in the focussing plane that exists on the sphere of radius L/cos γ and on above, obtain.The center be in camera position (a, b, c) and the equation of sphere with radius L/cos γ for (x-a) ²+ (y-b) ²+ (z-c) ²=(L/cos γ) ²

The plane that use will be caught by camera (promptly, horizontal-shift does not appear (promptly, the height of the top on plane (z axle) is constant, and following height (z axle) is also constant) and focussing plane in length and wide ratio be those planes of fixing) feature, as the condition of finding the solution this equation.Because z constant (that is, the height of the top on plane (z axle) is constant, and following height (z axle) is also constant) is so z can be set to two value z1 and z2.By last, equation ex+fy+gz1=eh+fi+gj, ex+fy+gz2=eh+fi+gj, (x-a) have been obtained ²+ (y-b) ²+ (z1-c) ²=(L/cos γ) ², and (x-a) ²+ (y-b) ²+ (z2-c) ²=(L/cos γ) ²

When having found the solution described 4 equations, can derive the value x and the y that represent with z1 and z2 respectively.At first, consider that z is z1 or ex+fy+gz1=eh+fi+gj and (x-a) ²+ (y-b) ²+ (z1-c) ²=(L/cos γ) ²Situation.For simplicity, eh+fi+gj-gz1=A, (z1-c) are set ²=B and (L/cos γ) ²=C, and, subsequently, obtain x+fy+gz1=A and c ²+ (y-b) ²+ B=C.When from two equation cancellation x and be provided with A-ea=D, e ²(B-C)=E, e ²+ f ²=F ,-(2DF+2be ²)=G and e ²b ²During+E=H, obtain Fy ²+ Gy+H=0, and the value of y is $y=-G\&PlusMinus;\sqrt{G^2-4\mathrm{FH}}.$ Similarly, can obtain $x=(A-f(-G\&PlusMinus;\sqrt{G^2-4\mathrm{FH}})/2F).$ For simplicity, resulting x and y are set to X1, Y1, X2, Y2 respectively.

Next, be z2 or ex+fy+gz2=eh+fi+gj and (x-a) also at z ²+ (y-b) ²+ (z2-c) ²=(L/cos γ) ²Situation under obtain x and y.This method under the situation of the deriving method under the situation of z2 and z1 is identical, and the descriptions thereof are omitted.Resulting x and y are set to X3, Y3, X4, Y4 respectively.Therefore, described 4 boundary coordinates be (X1, Y1, Z1), (X2, Y2, Z1), (X3, Y3, Z2) and (X4, Y4, Z2).

Since the length of focussing plane and wide ratio be fix (here, long: wide=P: Q), so, can derive the length of top: the length=P on the right: Q and following length: the length=P on the left side: Q.Schematically, will (X1, Y1, Z1), (X2, Y2, Z1), (X3, Y3, Z2) and (X4, Y4, Z2) be set to upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and the bottom right (X4, Y4, Z2).

And,

Therefore, obtain $\sqrt{{(X1-X2)}^2+{(Y1-Y2)}^2}:\sqrt{{(X2-X4)}^2+{(Y2-Y4)}^2+{(Z1-Z2)}^2}=P:Q$ With $\sqrt{{(X3-X4)}^2+{(Y3-Y4)}^2}:\sqrt{{(X1-X3)}^2+{(Y1-Y3)}^2+{(Z1-Z2)}^2}=P:Q,$ And, can obtain two equations.Upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and (X4, Y4 Z2) are the value of representing by z1 and z2 in the bottom right.When the replacement that is used for abbreviation turns back to reset condition, therefore, can basis $\sqrt{{(X1-X2)}^2+{(Y1-Y2)}^2}:\sqrt{{(X2-X4)}^2+{(Y2-Y4)}^2+{(Z1-Z2)}^2}=P:Q$ With $\sqrt{{(X3-X4)}^2+{(Y3-Y4)}^2}:\sqrt{{(X1-X3)}^2+{(Y1-Y3)}^2+{(Z1-Z2)}^2}=P:Q$ Obtain being used for the simultaneous equations of z1 and z2, and, z1 and z2 can be obtained.The expression formula of z1 and z2 is very complicated, and thus, the descriptions thereof are omitted.When upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and (X4, Y4 when substituting resulting z1 and z2 in Z2), might obtain boundary coordinate in the bottom right.That resulting boundary coordinate is set to is upper left (k, l, m), upper right (n, o, p), the lower-left (q, r, s) and bottom right (t, u, v) (step S505).

At last, focussing plane metadata lead-out unit 102 adds the boundary coordinate information of 4 points calculating to be used for each video address sensing metadata, to produce the data (step S506) as the focussing plane metadata.

Hereinafter, the sensing metadata of using Fig. 6 is actually described the method that derives focussing plane.The sensing metadata of the Fig. 6 that uses in this description is camera position (1,0,0), orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and the focusing distance " 1 meter " that video address " 00:00:00:01 " is located.At first, orientation and inclination angle " 90 degree, 0 degree " are decomposed into x, y and z component with amplitude 1, and the vector of indication camera direction is (1,0,0) different with camera position (1,0,0).The vector of indication camera direction is the normal vector to focussing plane.

Next, according to normal vector (1,0,0) and camera position (1,0,0), might obtain such linear equation, wherein, normal vector is (1,0,0), and this straight-line pass camera position (1,0,0).This linear equation is y=0, z=0.According to linear equation and focusing distance=1 of y=0, z=0, on straight line and with the focusing distance of camera position (1,0,0) be that 1 coordinate (that is the coordinate of focus) is (0,0,0).

Next, according to the coordinate (0,0,0) and the normal vector (1,0,0) of focus, derive the equation of focussing plane.According to the coordinate (0,0,0) and the normal vector (1,0,0) of focus, the equation of focussing plane is x=0.

Because subtended angle is 90 degree, so the distance of the boundary coordinate on focussing plane is 1/cos45 °, that is, and .Can think that boundary coordinate is present in and has on the sphere that radius  and center be in camera position (1,0,0), and in focussing plane.Having the equation that radius  and center be in the sphere of camera position (1,0,0) is (x-1) ²+ y ²+ z ²=2.Can be from spherical equation (x-1) ²+ y ²+ z ²=2 and the focussing plane EQUATION x=0 derive y ²+ z ²=1.When the supposition camera screen size of catching has 4: 3 length-width ratio, obtain z=4/3y.When finding the solution y ²+ z ²=1 and during z=4/3y, can derive y=± 3/5 and z=± 4/5.Therefore, boundary coordinate (0,3/5,4/5), (0 ,-3/5,4/5), (0 ,-3/5 ,-4/5) and (0,3/5 ,-4/5).

Figure 10 is the figure that illustrates the data structure of the focussing plane metadata that is generated.For each video address, the boundary coordinate of record focussing plane and the equation of focussing plane.In Figure 10, the item of Dao Chuing " focussing plane boundary coordinate " and " focussing plane equation " are added to the video address shown in Fig. 6 " 00:00:00:01 " as mentioned above, and " (0; 3/5; 4/5), (0 ,-3/5,4/5), (0 ;-3/5 ;-4/5) and (0,3/5 ,-4/5) " is input in " focussing plane boundary coordinate ", and, " x=0 " is input in " focussing plane equation ".

Next, will be described in the operation of the generation multi-angle metadata among the step S404 by reference Figure 11.Figure 11 is the flow chart of process that the multi-angle metadata generating run of multi-angle information generation device is shown.At first, constant n is initialized as 1 (step S601), and grouping identifying unit 103 obtains the information (equation and boundary coordinate) (step S602) of focussing plane metadata of the n frame of all videos, and carry out grouping decision (step S603).Next, grouping identifying unit 103 outputs to multi-angle metadata record unit 104 (step S604) with the multi-angle metadata that is generated.Then, constant n is increased progressively 1 (step S605), and grouping identifying unit 103 judges whether there is next frame of video (n frame) (step S606).If there is next frame of video, then this process turns back to step S602, and repeats multi-angle metadata generating run.On the contrary, if there is no next frame of video then finishes multi-angle metadata generating run.

To be described in grouping decision among the step S603 by reference Figure 12 and 13.The grouping decision is based on operation predetermined decision condition, that from a plurality of video datas of catching simultaneously the video data that obtains by the seizure same target is divided into groups.In embodiment 1, the image classification that focussing plane is intersected each other is same group.That is, in embodiment 1, carry out as the decision condition that divides into groups " judgement of intersecting of focussing plane ".Figure 12 is the figure of the judgement of intersecting of diagram focussing plane.As shown in the drawing, the video data of the camera that focussing plane intersects each other (imaging device) is judged as the video data that obtains by the seizure same target, and the video data that focussing plane does not intersect each other is judged as by catching the video data that different objects obtains.

Figure 13 is the flow chart of process that the grouping decision of multi-angle information generation device is shown.At first, for the focussing plane metadata of being obtained to some extent, grouping identifying unit 103 judges that the intersecting lens of plane equations is whether in boundary coordinate (step S701).If the intersecting lens of plane equation in boundary coordinate, then adds the video identifier information of the correspondence of indication n frame and video address to the focussing plane metadata (step S702) that will generate as the multi-angle metadata.

Hereinafter, will use the focussing plane metadata of Figure 10 to describe the grouping decision method by reality.In the focussing plane metadata of Figure 10, import as " video identifier " " 012345 ", import as " focussing plane boundary coordinate " " (0; 3/5; 4/5), (0 ,-3/5,4/5), (0 ;-3/5 ;-4/5) and (0,3/5 ,-4/5) ", and, import as " focussing plane equation " " x=0 ".Here, suppose to have another focussing plane metadata, wherein, " video identifier " is " 543210 ", and " focussing plane boundary coordinate " is " (3/5,0; 4/5), (3/5,0,4/5), (3/5; 0 ;-4/5) and (3/5,0 ,-4/5) ", and " focussing plane equation " is " y=0 ".Because the focussing plane equation is " x=0 " and " y=0 ", so the equation of intersecting lens is " x=0, y=0 ".

Next, judge that the intersecting lens of plane equation is whether in boundary coordinate.Boundary coordinate " (0; 3/5,4/5); (0 ,-3/5; 4/5); (0 ;-3/5 ,-4/5) on two planes passing through " x=0 " and " y=0 ", (0,3/5,-4/5) "; and " (3/5,0,4/5), (3/5,0,4/5), (3/5,0 ,-4/5), (3/5,0,-4/5) " Biao Shi bounds-3/5≤x≤3/5 ;-3/5≤y≤3/5; in-4/5≤z≤4/5; the equation " x=0 of resulting intersecting lens, y=0 " be the x=0 between-4/5≤z≤4/5; and y=0, and, can be judged as in-3/5≤x≤3/5 ;-3/5≤y≤3/5; in the bounds of-4/5≤z≤4/5.Therefore, judge that two focussing planes intersect each other or obtain video data by catching same target.Subsequently, video identifier " 543210 " is added to generate as the multi-angle metadata, focussing plane metadata that wherein " video identifier " is " 012345 ".Video identifier " 012345 " added to generate as the multi-angle metadata, focussing plane metadata that wherein " video identifier " is " 543210 ".

Figure 14 is the figure that illustrates the data structure of the multi-angle metadata that is generated.Multi-angle information comprises: data ID, and it can be specified by catching the video data that same target obtains simultaneously; And video address, it can be appointed as each video address and the relative position of the video data that writes down.In Figure 14, add the item " multi-angle information " of deriving to the video address shown in Figure 10 " 00:00:00:01 " in the above, and " data ID:543210, video address 00:00:00:01 " is input to " multi-angle information ".

As mentioned above, in relevant with corresponding video data, record multi-angle metadata.Therefore, by using the multi-angle metadata, multi-angle video searching apparatus 40 can be searched for and extract by catching the video data that same target obtains simultaneously.

In this embodiment, the configuration example particle that imaging device separates with the multi-angle information generation device has been described.Replacedly, imaging device can comprise sensing metadata acquiring unit and focussing plane metadata lead-out unit.

In this embodiment, by using video identifier that video data is relevant with various metadata.Replacedly, can then it be multiplexed to video data, so that do not use video identifier with various metadata conversion for stream.

In grouping is judged, can carry out grouping in such a way and judge.According to the depth of field as the front and back scope of the object that as if will obtain to focus on, and expansion or shrink focusing distance.Subsequently, calculate focussing plane for each focusing distance.

Therefore, can improve work load under the situation of for example editing the multi-angle video significantly.

[embodiment 2]

Next, will describe such example, wherein, in grouping is judged, under other decision condition, carry out grouping and judge.Identical among the process of the configuration of multi-angle information generation device and multi-angle information generating system and multi-angle information generating run and the embodiment 1, and the descriptions thereof are omitted thus.

In embodiment 2, being used for image packets based on storage is the table of positional information of same group focussing plane, and the grouping of carries out image.That is, in embodiment 2, grouping identifying unit 103 merges the table of describing rule of classification, and, carry out based on this table " judgement of the existence of focussing plane in presumptive area ".Figure 15 is the figure of the judgement of the existence of diagram focussing plane in presumptive area.As shown in the drawing, to in the presumptive area that is set to the three-dimensional coordinate zone, exist the video data of focussing plane to be judged to be the video data that will be grouped into same group, and the video data that focussing plane is not present in the presumptive area is judged to be the video data that will be grouped into different groups.In the case, this judges whether intersect irrelevant with focussing plane.According to the grouping decision condition, can carry out the grouping of video data by specified number of areas (for example, near near the object by " middle section " in the ground, catch rod court or " right side area " obtain video data).

The figure of Figure 16 rule of classification that to be diagram generate by the positional information of specifying a plurality of zones.As shown in the drawing, when being provided with 4 kinds when regional, video data is categorized as 4 groups.In Figure 16, when the x coordinate was 0≤x≤1, for example, the y coordinate was 0≤y≤1, and the z coordinate is 0≤z≤1, and this zone be named as central authorities near.When the x coordinate was 2≤x≤3, the y coordinate was 2≤y≤3, and the z coordinate is 2≤z≤3 o'clock, and this zone be named as the right side near.

Figure 17 is the flow chart of process of the grouping decision of multi-angle information generation device under the Rule of judgment that illustrates among the embodiment 2.At first, for all resulting focussing plane metadata, grouping identifying unit 103 judges that the boundary coordinate on planes is whether in the zone of rule of classification (step S801).If described coordinate in the zone of rule of classification, then adds the video identifier information of correspondence etc. to the focussing plane metadata (step S802) that will generate as the multi-angle metadata.

To use the focussing plane metadata of Figure 10 and the rule of classification of Figure 16 to describe the grouping decision method by reality.In the focussing plane metadata of Figure 10, import as " video identifier " " 012345 ", import as " focussing plane boundary coordinate " " (0,3/5,4/5), (0 ,-3/5,4/5), (0 ,-3/5 ,-4/5) and (1,3/5 ,-4/5) ".Here, suppose there is another focussing plane metadata that wherein, " video identifier " is " 543210 ", and " focussing plane boundary coordinate " is " (3/5,0,4/5), (3/5,0,4/5), (3/5,0 ,-4/5) and (3/5,0 ,-4/5) ".At first, " video identifier " is that " 012345 " " focussing plane boundary coordinate " is " (0,3/5,4/5), (0 ,-3/5,4/5), (0 ,-3/5 ,-4/5), (1,3/5 ,-4/5) ".Therefore, described coordinate meets the zone of 0≤x≤1,0≤y≤1 and 0≤z≤1, and is grouped near the center.Next, " video identifier " is that " 543210 " " focussing plane boundary coordinate " is " (3/5,0,4/5), (3/5,0,4/5), (3/5,0 ,-4/5), (3/5,0 ,-4/5) ".Therefore, described coordinate meets the zone of 0≤x≤1,0≤y≤1 and 0≤z≤1, and, similarly, be grouped near the center.Thereby, described two video datas are judged to be belong to same group, and, add video identifier " 543210 " and near title " center " to focussing plane metadata that " video identifier " is " 012345 ", so that generate this data as the multi-angle metadata.Add video identifier " 012345 " and near title " center " to focussing plane metadata that " video identifier " is " 543210 ", so that generate this data as the multi-angle metadata.

Figure 18 is the figure that illustrates the data structure of the multi-angle metadata that is generated.Multi-angle information comprises: data ID, and it can be specified by catching the video data that same target obtains simultaneously; And video address, it can be appointed as each video address and the relative position of the video data that writes down and the information relevant with the title of presumptive area.In Figure 18, add the item " multi-angle information " and " title " of deriving to the video address shown in Figure 10 " 00:00:00:01 " in the above, " data ID:543210; video address 00:00:00:01 " is input in " multi-angle information ", and " near the center " is input in " title ".

Can be depending on all focussing plane boundary coordinates and whether be present in the presumptive area or whether at least one group of coordinate is present in this zone, and carry out judgement the existence in the presumptive area.

In this embodiment, can change rule of classification according to situation.The table of describing rule of classification can not be placed in the grouping identifying unit.Can adopt the configuration that this table is placed in the external data base and the grouping identifying unit is quoted external table.

This embodiment can be configured so that only when changing sensitive information, just to generate sensing metadata.In this configuration, reduced data volume to be processed, and, can improve processing speed.In fact, estimate that the adjacent image frame often has identical multi-angle information.Therefore, can be not be that each picture frame generates the multi-angle metadata, and, can generate and have the only multi-angle metadata of the data structure of the corresponding relation between the instruction video address and multi-angle information.In the case, reduced data volume to be processed, and, processing speed can be improved.In addition, can be grouped every group of generation multi-angle metadata that identifying unit is classified and can be not for each picture frame generates the multi-angle metadata.According to this configuration, reduced the processing of identical information duplicate record in the metadata of each video data, and, can improve processing speed.

This embodiment is configured so that to generate sensing metadata by imaging device.The invention is not restricted to this.For example, the outside from imaging device obtains sensing metadata.

[embodiment 3]

In embodiment 1 and 2, the example that is divided into groups by the image that a plurality of imaging devices are caught simultaneously to beginning has been described.In this embodiment, will the example that the image of being caught constantly in difference by single imaging device is divided into groups be described.In embodiment 1 and 2, judge whether obtain image by catching the same area at the N frame of all video datas.On the contrary, in this embodiment, the combination of all frames of video data is judged.

Figure 19 is the figure of configuration that the internal configurations of the interpolation information generation device in the embodiments of the invention is shown and comprises the interpolation information generating system of this interpolation information generation device.Interpolation information generating system shown in Figure 19 is made of following assembly: add information generation device 1010, it is to dividing into groups by catching the image that obtains by single imaging device; Imaging device 1020; Database 1030; And video searching apparatus 1040.Hereinafter, add the example that information generating system divides into groups to the video by a plurality of image constructions with describing.

Add information generation device 1010 and comprise sensing metadata acquiring unit 1101, focussing plane metadata lead-out unit 1102, grouping identifying unit 1103 and metadata record unit 1104.

Sensing metadata acquiring unit 1101 obtains the sensor information relevant with the capturing condition of imaging device 1020.Sensing metadata acquiring unit 1101 is via database 1030, and obtains the sensing metadata relevant with position, orientation, inclination angle, subtended angle and the focusing distance of each imaging device.In this embodiment, suppose by imaging device 1020 generation sensing metadata.The internal structure of imaging device 1020 and the details of sensing metadata will be described in the back.

Focussing plane metadata lead-out unit 1102 is based on resulting sensing metadata, and derivation comprises the focussing plane of the image of being caught by imaging device 1020, and, calculate rectangle based on sensing metadata, the seizure focussing plane in the real space of its indication imaging device 1020 as coordinate figure.To describe the focussing plane metadata in the back in detail.

Grouping identifying unit 1103 is based on the position of focussing plane relation and to image packets.When using the focussing plane of being derived by focussing plane metadata lead-out unit 1102, the grouping identifying unit is based on predetermined decision condition, and whether judgement has obtained described image by catching the same area.

Metadata record unit 1104 writes down the result of grouping as interpolation information, wherein, this information is relevant with image, and, as adding metadata, will be judged as the information relevant and export and record in the database 1030 with image by catching those images that the same area obtains.The interpolation metadata will be described in detail in the back.

Add information generation device 1010 and be connected to the database 1030 of storage from the video data of imaging device 1020, based on the sensing metadata that obtains from imaging device, and produce as with the interpolation metadata of relevant information by catching a plurality of video datas that same target obtains, and these data are outputed to data 1030.The video searching apparatus 1040 that is connected to database 1030 can be searched for video data based on adding metadata.

Next, imaging device will be described.Figure 20 is the figure of the internal configurations of the imaging device that uses in the interpolation information generating system that illustrates in an embodiment of the present invention.Imaging device 1020 comprises lens group 1201, CCD 1202, drive circuit 1203, timing signal generation unit 1204, sampling unit 1205, A/D converting unit 1206, video file generation unit 1207, video address generation unit 1208, video identifier generation unit 1209, machine information transducer 1210, sensing metadata generation unit 1211 and record cell 1212.

The timing signal synchronised ground driven CCD 1202 that is generated with the timing signal generation unit 1204 that is connected to drive circuit 1203, and, will be by lens group 1201 picture signal of the object images of incident output to sampling unit 1205.

Sampling unit 1205 comes picture signal is sampled with the sample rate that is exclusively used in CCD 1202.A/D converting unit 1206 will be converted to DID from the picture signal of CCD 1202 outputs, and these data are outputed to video file generation unit 1207.

Video address generation unit 1208 is in response to from the signal of timing signal generation unit 1204 and produce video address.Video identifier generation unit 1209 sends and adds the identifier that video is relevant with the sensing metadata of describing later (for example, filename or ID).

Utilize GPS (global positioning system) receiver, gyro sensor, aspect sensor, range sensor and subtended angle transducer to dispose machine information sensor 1210.

Gps receiver receives radio wave from satellite, to obtain the distance with the three or more artificial satellites of its position previously known, thus, can obtain the three-dimensional position (latitude, precision, highly) of gps receiver self.When using this function, might obtain imaging device absolute position on earth.

Gyro sensor is commonly called three acceleration sensors, and uses terrestrial gravitation to detect along the degree of the axial acceleration of looking from this transducer, that is, and and as the gradient vertically of numerical value.When using this function, might obtain the gradient (azimuth, inclination angle) of imaging device.

Aspect sensor is commonly called electronic compass, and uses the magnetic of the earth to detect tellurian north, south, east or western direction.When with gyro sensor and aspect sensor combination, might detect imaging device absolute direction on earth.

Range sensor is the transducer that is measured to the distance of object.This transducer sends infrared ray or ultrasonic wave from imaging device towards object, and can know the distance from the imaging device to the object from through time till imaging device receives reflection,, be used for the focusing distance that obtains to focus on that is.

The subtended angle transducer can obtain subtended angle from the height of focal length and CCD.Can obtain focal length by the distance of measuring between camera lens and the light receiving part, and the height of light receiving part is the value that is exclusively used in imaging device.

Based on output request from sensing metadata 1211, machine information transducer 1210 output and position, the orientation that will be used as benchmark, azimuth, inclination angle, subtended angle and the relevant sensitive information of focusing distance from the imaging device of GPS (global positioning system) receiver, gyro sensor, aspect sensor, range sensor and subtended angle transducer.Sensing metadata generation unit 1211 generates regularly according to the video address from video address generation unit 1208, and obtains sensitive information from machine information sensor 1210, produces sense data, and these data are outputed to record cell 1212.In response to the signal from timing signal generation unit 1204, machine information transducer 1210 and sensing metadata generation unit 1211 begin operation.

The generation of sensitive information and output and main purpose of the present invention are irrelevant, and, therefore, omit the detailed description of the operation of transducer.

Sampling rate (1/30 second) that can CCD is carried out obtaining of sensitive information, perhaps, can carry out obtaining of sensitive informations every several frames.

Indoor execution take or the idle situation of GPS transducer under, the positional information in input capture place manually.In the case, will be input to the machine information transducer by the positional information of not shown ground input unit input.

Hereinafter, the sensing metadata generating run that description is had the imaging device of above-mentioned configuration.Figure 21 is the flow chart of the operating process of the imaging device that uses in the interpolation information generating system that illustrates in an embodiment of the present invention.

At first, when pushing of the predetermined switch of the formant of carrying out imaging device waited, receive seizure commencing signal (step S1101).Subsequently, imaging device 1020 beginning videograph processes (step S1102), and imaging device 1020 begins to generate the process (step S1103) of sensing metadata.When timing signal generation unit 1204 received the seizure end signal, imaging device 1020 stopped videograph process and sensing metadata generative processes (step S1104).

To be described in videograph process that begins among the step S1102 and the sensing metadata generative process that in step S1103, begins by reference Figure 22 and 23.

Figure 22 is the flow chart that is illustrated in the process of the videograph operation among the step S102.When obtaining seizure commencing signal (step S1201), in response to operation directive command from timing signal generation unit 1204, and beginning capture operation (step S1202).In addition, in response to directive command, and generate video identifier (step S1203) by video identifier generation unit 1209 from timing signal generation unit 1204.

Obtain the video electrical signal (step S1204) from CCD 1202,1205 pairs of signals that obtained of sampling unit are carried out sampling (step S1205), and A/D converting unit 1206 is carried out the conversion (step S1206) of DID.

In response to directive command, and generate video address (step S1207) by video address generation unit 1208 from timing signal generation unit 1204, and, video file (step S1208) generated by video file generation unit 1207.In addition, add the video identifier (step S1209) that generates by video identifier generation unit 1209, and, final video file is recorded (step S1210) in the record cell 1212.

Figure 23 is the flow chart that is illustrated in the process of the sensing metadata generating run among the step S1103.When having obtained seizure commencing signal (step S1301), sensing metadata generation unit 1211 obtains the video address (step S1302) that is generated by video address generation unit 1208.Obtain the video identifier (step S1303) that generates by video identifier generation unit 1209.In addition, and obtain video address side by side, sensing metadata generation unit 1211 sends the request that is used to export sensitive information to machine information transducer 1210, with the information of the position, azimuth, inclination angle, subtended angle and the focusing distance that obtain camera.Can obtain the position of camera from gps receiver, can obtain azimuth and inclination angle, can obtain focusing distance from range sensor from gyro sensor, and, can obtain subtended angle (step S1304) from the subtended angle transducer.

Next, sensing metadata generation unit 1211 writes down position, azimuth, inclination angle, subtended angle and the focusing distance of camera with video identifier that is acquired and video address, generate and output sensing metadata (step S1305), and, with data record (step S1306) in record cell 1212.

Figure 24 is the figure that illustrates the data structure of the sensing metadata that is generated.Video identifier is added to a series of video datas of the frame configuration by arbitrary number.By this video identifier, allow video data uniquely corresponding to sensing metadata.For each video address, record camera coordinates, azimuth, inclination angle, subtended angle and focusing distance.The least unit of video address is the sampling rate of CCD 1202, that is, and and frame.For example, will be input to the video identifier of Figure 24 as " 12345 " of the information of obtaining from video identifier generation unit 1209.In addition, will be input to the video address as " 00:00:00:01 " of the information of obtaining from video address generation unit 1208.Input in video address " 00:00:00:01 ": as the camera position " 1; 0; 0 of the information of when obtaining video address, obtaining from machine information sensor 1210 ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".Represent camera position by " x, y, z ", wherein x represents latitude, and y represents longitude, and z represents height (more than the sea level).Latitude, precision and the height of the value of actual input for obtaining by the GPS transducer.Yet, in this embodiment, obtained latitude x=1, precision y=0, and height z=0 in order to describe for simplicity, to suppose.Input is as the information of obtaining from video address generation unit 1208 in next video address " 00:00:00:02 ".Input in next video address " 00:00:00:02 ": as the camera position " 1; 0; 0 of the information of obtaining from machine information sensor 1210 in the moment of obtaining video address ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".The information that input is obtained from video address generation unit 1208 in next video address " 00:00:00:03 ".Input in video address " 00:00:00:03 ": as the camera position " 1; 0; 0 of the information of obtaining from machine information sensor 1210 in the moment of obtaining video address ", orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and focusing distance " 1 meter ".

Next, the interpolation information generating run that description is had the interpolation information generation device of above-mentioned configuration.Figure 25 is the flow chart of process that the interpolation information generating run of the interpolation information generation device in the embodiments of the invention is shown.

At first, the sensing metadata acquiring unit 1101 of interpolation information generation device 1010 obtains all sensing metadata (step S1401) of one group of video being taken by imaging device 1020.Next, focussing plane metadata lead-out unit 1102 is derived focussing plane metadata (step S1402) based on the sensing metadata of being obtained.

Subsequently, focussing plane metadata lead-out unit 1102 determines whether to have finished for all sensing metadata the derivation of focussing plane metadata.If do not finish, the then operation of the derivation focussing plane metadata among the repeating step S1402.On the contrary, if finished the derivation of focussing plane metadata for all sensing metadata, then this process is transferred to subsequently and is generated the operation (step S1403) of adding metadata.Next, grouping identifying unit 1103 produces based on the focussing plane metadata of obtaining from focussing plane metadata lead-out unit 1102 and adds metadata (step S1404).

At last, the interpolation metadata of obtaining from grouping identifying unit 1103 (step S1405) is exported in metadata record unit 1104 to database 1030.

The operation of the derivation focussing plane metadata among the step S1402 will be described by reference Figure 26 and 27.Figure 26 is the figure of diagram focussing plane.Focussing plane is meant the rectangular surfaces that is shown as picture zone, wherein,, obtains focusing on or so-called focus carrying out when catching, and, can represent focussing plane by the coordinate figure (being called as boundary coordinate) at four angles of holding.As shown in the drawing, come to determine distance by focusing distance (that is, focal length) from imaging device (camera) to focussing plane, and, determine the size of rectangle by the subtended angle of camera.The center of this rectangle is a focus.

To describe the flow chart of Figure 27, it shows the focussing plane that adds information generation device and derives the process of operation.At first, focussing plane metadata lead-out unit 1102 obtains sensing metadata (step S1501).

In any camera as shown in figure 26 and the sensitive information of arbitrary timing be camera position (a, b, c), under the situation of orientation angles α degree, angle of inclination beta degree, subtended angle 2 γ degree and focusing distance L (rice), can obtain wherein camera position (a from orientation angles α degree and angle of inclination beta degree, b c) is set to the direction vector of the camera of reset condition.According to sensitive information, the direction vector of camera is (sin α cos β, cos α cos β, sin β).The direction vector of supposing resulting camera for (e, f, g).The camera direction vector be (e, f g) vertically pass focussing plane, and, be normal vector (step S1502) thus to focussing plane.

Next, can be from the camera direction vector (e, f, g) and camera position (a, b, c) derive by camera position (a, b, c) and the linear equation of focus.When using intermediate parameters z, linear equation can be expressed as (ez, fz, gz).According to this linear equation, can derive and be positioned at that (a, b c) separate the coordinate of distance L as focus on this straight line and with camera position.The expression formula that is used to obtain is $L=\sqrt{{(\mathrm{ez}-a)}^2+{(\mathrm{fz}-b)}^2+{(\mathrm{gz}-c)}^2}.$ From then on expression formula derives intermediate parameters z.When having found the solution $L=\sqrt{{(\mathrm{ez}-a)}^2+{(\mathrm{fz}-b)}^2+{(\mathrm{gz}-c)}^2}$ The time, obtain $z=\{(\mathrm{ae}+\mathrm{bf}+\mathrm{cg})\&PlusMinus;\sqrt{{(\mathrm{ae}+\mathrm{bf}+\mathrm{cg})}^2-(e+f+g)(a^2+{\mathrm{<figure-callout\; id="2"\; label="b"\; filenames="S2006800191746E00301.png"\; state="\{\{state\}\}">b</figure-callout>}}^2+c^2-L^2)}\}/(e+f+g),$ And, by replacing at (ez, fz, the z that obtains in gz) and obtain focus (step S1503).

Resulting focus is expressed as (h, i, j).Can from normal vector (e, f, g) and focus (h, i j) derive the focussing plane equation.The focussing plane equation is ex+fy+gz=eh+fi+gj (step S1504).

According to subtended angle 2 γ degree, (c) distance to the boundary coordinate of focussing plane is L/cos γ for a, b from camera position.Boundary coordinate can be: be at the center camera position (a, b, c) and have coordinate in the focussing plane that exists on the sphere of radius L/cos γ and obtain in the above.The center be in camera position (a, b, c) and the equation of sphere with radius L/cos γ for (x-a) ²+ (y-b) ²+ (z-c) ²=(L/cos γ) ²

The plane that use will be caught by camera (promptly, horizontal-shift does not appear (promptly, the height of the top on plane (z axle) is constant, and following height (z axle) is also constant) and focussing plane in length and wide ratio be those planes of fixing) feature, as the condition of finding the solution this equation.Because z constant (that is, the height of the top on plane (z axle) is constant, and following height (z axle) is also constant) is so z can be set to two value z1 and z2.By last, equation ex+fy+gz1=eh+fi+gj, ex+fy+gz2=eh+fi+gj, (x-a) have been obtained ²+ (y-b) ²+ (z1-c) ²=(L/cos γ) ², and (x-a) ²+ (y-b) ²+ (z2-c) ²=(L/cos γ) ²

When having found the solution described 4 equations, can derive the value x and the y that represent with z1 and z2 respectively.At first, consider that z is z1 or ex+fy+gz1=eh+fi+gj and (x-a) ²+ (y-b) ²+ (z1-c) ²=(L/cos γ) ²Situation.For simplicity, eh+fi+gj-gz1=A, (z1-c) are set ²=B and (L/cos γ) ²=C, and, subsequently, obtain x+fy+gz1=A and c ²+ (y-b) ²+ B=C.When from two equation cancellation x and be provided with A-ea=D, e ²(B-C)=E, e ²+ f ²=F ,-(2DF+2be ²)=G and e ²b ²During+E=H, obtain Fy ²+ Gy+H=0, and the value of y is $y=-G\&PlusMinus;\sqrt{G^2-4\mathrm{FH}}.$ Similarly, can obtain $x=(A-f(-G\&PlusMinus;\sqrt{G^2-4\mathrm{FH}})/2F).$ For simplicity, resulting x and y are set to X1, Y1, X2, Y2 respectively.

Next, be z2 or ex+fy+gz2=eh+fj+gj and (x-a) also at z ²+ (y-b) ²+ (z2-c) ²=(L/cos γ) ²Situation under obtain x and y.This method under the situation of the deriving method under the situation of z2 and z1 is identical, and therefore the descriptions thereof are omitted.Resulting x and y are set to X3, Y3, X4, Y4 respectively.Therefore, described 4 boundary coordinates be (X1, Y1, Z1), (X2, Y2, Z1), (X3, Y3, Z2) and (X4, Y4, Z2).

Since the length of focussing plane and wide ratio be fix (here, long: wide=P: Q), so, can derive the length of top: the length=P on the right: Q and following length: the length=P on the left side: Q.Schematically, will (X1, Y1, Z1), (X2, Y2, Z1), (X3, Y3, Z2) and (X4, Y4, Z2) be set to upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and the bottom right (X4, Y4, Z2).

And,

Therefore, obtain $\sqrt{{(X1-X2)}^2+{(Y1-Y2)}^2}:\sqrt{{(X2-X4)}^2+{(Y2-Y4)}^2+{(Z1-Z2)}^2}=P:Q$ With $\sqrt{{(X3-X4)}^2+{(Y3-Y4)}^2}:\sqrt{{(X1-X3)}^2+{(Y1-Y3)}^2+{(Z1-Z2)}^2}=P:Q,$ And, can obtain described two equations.Upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and (X4, Y4 Z2) are the value of representing by z1 and z2 in the bottom right.When the replacement that is used for abbreviation turns back to reset condition, therefore, can from $\sqrt{{(X1-X2)}^2+{(Y1-Y2)}^2}:\sqrt{{(X2-X4)}^2+{(Y2-Y4)}^2+{(Z1-Z2)}^2}=P:Q$ With $\sqrt{{(X3-X4)}^2+{(Y3-Y4)}^2}:\sqrt{{(X1-X3)}^2+{(Y1-Y3)}^2+{(Z1-Z2)}^2}=P:Q$ Obtain being used for the simultaneous equations of z1 and z2, and, z1 and z2 can be obtained.The expression formula of z1 and z2 is very complicated, and thus, the descriptions thereof are omitted.When upper left (X1, Y1, Z1), upper right (X2, Y2, Z1), the lower-left (X3, Y3, Z2) and (X4, Y4 when substituting resulting z1 and z2 in Z2), might obtain boundary coordinate in the bottom right.That resulting boundary coordinate is set to is upper left (k, l, m), upper right (n, o, p), the lower-left (q, r, s) and bottom right (t, u, v) (step S505).

At last, focussing plane metadata lead-out unit 1102 adds the boundary coordinate information of 4 points calculating to be used for each video address sensing metadata, to produce the data (step S1506) as the focussing plane metadata.

Hereinafter, the sensing metadata of using Figure 24 is actually described the method that derives focussing plane.The sensing metadata of the Fig. 6 that uses in this description is camera position (1,0,0), orientation and inclination angle " 90 degree, 0 degree ", subtended angle " 90 degree " and the focusing distance " 1 meter " that video address " 00:00:00:01 " is located.At first, orientation and inclination angle " 90 degree, 0 degree " are decomposed into x, y and z component with amplitude 1, and the vector of indication camera direction is (1,0,0) different with camera position (1,0,0).The vector of indication camera direction is the normal vector to focussing plane.

Next, according to normal vector (1,0,0) and camera position (1,0,0), might obtain such linear equation, wherein, normal vector is (1,0,0), and this straight-line pass camera position (1,0,0).This linear equation is y=0, z=0.According to linear equation and focusing distance=1 of y=0, z=0, on straight line and with the focusing distance of camera position (1,0,0) be that 1 coordinate (that is the coordinate of focus) is (0,0,0).

Next, according to the coordinate (0,0,0) and the normal vector (1,0,0) of focus, derive the equation of focussing plane.According to the coordinate (0,0,0) and the normal vector (1,0,0) of focus, the equation of focussing plane is x=0.

Because subtended angle is 90 degree, so, be 1/cos45 ° to the distance of the boundary coordinate of focussing plane, that is, and .Can think that boundary coordinate is present in and has on the sphere that radius  and center be in camera position (1,0,0), and in focussing plane.Having the equation that radius  and center be in the sphere of camera position (1,0,0) is (x-1) ²+ y ²+ z ²=2.Can be from spherical equation (x-1) ²+ y ²+ z ²=2 and the focussing plane EQUATION x=0 derive y ²+ z ²=1.When the supposition camera screen size of catching has 4: 3 length-width ratio, obtain z=4/3y.When finding the solution y ²+ z ²=1 and during z=4/3y, can derive y=± 3/5 and z=± 4/5.Therefore, boundary coordinate (0,3/5,4/5), (0 ,-3/5,4/5), (0 ,-3/5 ,-4/5) and (0,3/5 ,-4/5).

Figure 28 is the figure that illustrates the data structure of the focussing plane metadata that is generated.For each video address, the boundary coordinate of record focussing plane and the equation of focussing plane.In Figure 28, the item of Dao Chuing " focussing plane boundary coordinate " and " focussing plane equation " are added to the video address shown in Figure 24 " 00:00:00:01 " as mentioned above, and " (0; 3/5; 4/5), (0 ,-3/5,4/5), (0 ;-3/5 ;-4/5) and (0,3/5 ,-4/5) " is input in " focussing plane boundary coordinate ", and, " x=0 " is input in " focussing plane equation ".When adding the focussing plane metadata to image, allow the grouping of image described later.

Next, the operation of metadata is added in the generation that is described in by reference Figure 29 among the step S1404.Figure 29 is the flow chart that the process of the interpolation metadata generating run that adds information generation device is shown.At first, grouping identifying unit 1103 obtains information (equation) and the boundary coordinate (step S1601) of focussing plane metadata of all frames of all videos, and derivation is as N the pattern (step S1602) of the combination of all frames.

Figure 30 is the figure of image that the combination of all frames is shown.Figure 30 (b) shows the combination of all frames of the video B that the video A that is made up of the frame 1 to 3 shown in Figure 30 (a) and frame 1 to 3 form.For the frame 1 of video A, for example, have three patterns, perhaps, with the combination (first pattern) of the frame 1 of video B, with the combination (second pattern) of the frame 2 of video B, with the combination (the 3rd pattern) of the frame 3 of video B.Similarly, the combination of forming by the 7th to the 9th pattern of the frame 3 that has the combination of forming by the 4th to the 6th pattern of the frame 2 be directed to video A and be directed to video A.

Next, the pattern N that makes up is initialized as 1, and 1103 pairs of N patterns of grouping identifying unit are carried out the grouping decision, to produce interpolation metadata (step S1604).Next, grouping identifying unit 103 outputs to metadata record unit 104 (step S1605) with the interpolation metadata that is generated.Subsequently, constant N is increased progressively 1 (step S1606), and grouping identifying unit 1103 judges whether there is next combination pattern (N pattern) (step S1607).If there is next combination pattern, then this process turns back to step S1604, and repeats to add the metadata generating run.On the contrary, if there is no next combination pattern then finishes to add the metadata generating run.

To be described in grouping decision among the step S1604 by reference Figure 31 and 32.The grouping decision is based on operation predetermined decision condition, that from a plurality of video datas of catching simultaneously the video data that obtains by the seizure same target is divided into groups.In embodiment 3, the image classification that focussing plane is intersected each other is same group.That is, in embodiment 3, carry out as the decision condition that divides into groups " judgement of intersecting of focussing plane ".Figure 31 is the figure of the judgement of intersecting of diagram focussing plane.As shown in the drawing, the video data of the camera that focussing plane intersects each other (imaging device) is judged as the video data that obtains by the seizure same target, and the video data that focussing plane does not intersect each other is judged as by catching the video data that different objects obtains.

Figure 32 is the flow chart that the process of the grouping decision of adding information generation device is shown.At first, for the focussing plane metadata of being obtained to some extent, grouping identifying unit 1103 judges that the intersecting lens of plane equations is whether in boundary coordinate (step S1701).If the intersecting lens of plane equation is in boundary coordinate, then the video identifier information of the correspondence of indication n frame and video address being added to will be as adding the focussing plane metadata (step S1702) that metadata generates.

Hereinafter, will use the focussing plane metadata of Figure 28 to describe the grouping decision method by reality.In the focussing plane metadata of Figure 28, import as " video identifier " " 012345 ", import as " focussing plane boundary coordinate " " (0; 3/5; 4/5), (0 ,-3/5,4/5), (0 ;-3/5 ;-4/5) and (0,3/5 ,-4/5) ", and, import as " focussing plane equation " " x=0 ".Here, suppose to have another focussing plane metadata, wherein, " video identifier " is " 543210 ", and " focussing plane boundary coordinate " is " (3/5,0; 4/5), (3/5,0,4/5), (3/5; 0 ;-4/5) and (3/5,0 ,-4/5) ", and " focussing plane equation " is " y=0 ".Because the focussing plane equation is " x=0 " and " y=0 ", so the equation of intersecting lens is " x=0, y=0 ".

Next, judge that the intersecting lens of plane equation is whether in boundary coordinate.At boundary coordinate " (0; 3/5,4/5); (0 ,-3/5; 4/5); (0 ;-3/5 ,-4/5) by two planes " x=0 " and " y=0 ", (0,3/5,-4/5) "; and " (3/5,0,4/5), (3/5,0,4/5), (3/5,0 ,-4/5), (3/5,0,-4/5) " Biao Shi bounds-3/5≤x≤3/5 ;-3/5≤y≤3/5; in-4/5≤z≤4/5; the equation " x=0 of resulting intersecting lens, y=0 " be the x=0 between-4/5≤z≤4/5; and y=0, and, can be judged as in-3/5≤x≤3/5 ;-3/5≤y≤3/5; in the bounds of-4/5≤z≤4/5.Therefore, judge that two focussing planes intersect each other or obtain video data by catching same target.Subsequently, video identifier " 543210 " being added to will be as adding the focussing plane metadata that metadata generates, wherein " video identifier " is " 012345 ".Video identifier " 012345 " is added to will be as adding the focussing plane metadata that metadata generates, wherein " video identifier " is " 543210 ".

Figure 33 is the figure that illustrates the data structure of the interpolation metadata that is generated.Interpolation information comprises: data ID, and it can be specified by catching the video data that same target obtains simultaneously; And video address, it can be appointed as each video address and the relative position of the video data that writes down.In Figure 33, add the item " interpolation information " of deriving to the video address shown in Figure 28 " 00:00:00:01 " in the above, and " data ID:543210, video address 00:00:00:01 " is input to " interpolation information ".

As mentioned above, in relevant with corresponding video data, record adds metadata.Therefore, add metadata, add video searching apparatus 1040 and can search for and extract by catching the video data that same target obtains simultaneously by using.

In this embodiment, describe imaging device and added the configuration example that information generation device separates.Replacedly, imaging device can comprise sensing metadata acquiring unit and focussing plane metadata lead-out unit.

In this embodiment, by using video identifier that video data is relevant with various metadata.Replacedly, can then it be multiplexed to video data, so that do not use video identifier with various metadata conversion for stream.

In grouping is judged, can carry out grouping in such a way and judge.According to the depth of field as the front and back scope of the object that as if will obtain to focus on, and expansion or shrink focusing distance.Subsequently, calculate focussing plane for each focusing distance.

Therefore, can the video that have single camera to take constantly in difference be divided into groups.When in database, having registered the photo taken by conventional user or video, for example, according to the place that has object and automatically to its grouping.Can improve the work load under the situation of for example editing video significantly.

In the above, described by using focussing plane the example of image packets.When adding the focussing plane metadata to image, can apply the present invention to the purposes except the grouping of image.

Although described in detail the present invention, and with reference to specific embodiment, it is obvious to those skilled in the art that and can use various changes and modification, and can not deviate from the spirit and scope of the present invention.

The present invention is based on Japanese patent application of submitting to on May 30th, 2005 (2005-157179 number) and the Japanese patent application of submitting to on May 26th, 2006 (2006-146909 number), and, by reference it openly is herein incorporated.

Industrial applicability

According to the present invention, when by with the position of focussing plane as metadata add, flat based on focusing on During the grouping of the position relationship of face and carries out image, with traditional skill of carrying out grouping by graphical analysis Art is compared, and can reduce to process load. Therefore, the present invention has such effect: make it possible to low negative Carry and carry out in an easy manner search and the extraction of the image that obtains by seizure the same area. And the present invention can be used for metadata is added to by caught the image that obtains by imaging device Metadata adding set, metadata adding method etc.

Claims (8)

1. metadata adding set that is used for metadata is added to the image of being caught by imaging device comprises:

The sensitive information acquiring unit, it obtains the sensor information relevant with the capturing condition of imaging device;

The focussing plane lead-out unit, it is based on the sensor information of being obtained, and derivation is as the position of the focussing plane of the imaging surface of the image of being caught; And

The metadata adding device, its position with the focussing plane of being derived is added the image of being caught to as metadata.

2. metadata adding set as claimed in claim 1 also comprises:

Grouped element is used for based on the relation of the position between a plurality of focussing planes, and to a plurality of image packets; And

Add information recording unit, the result who is used to write down grouping is as interpolation information, and is simultaneously that interpolation information is relevant with image.

3. metadata adding set as claimed in claim 2, wherein, the image packets that grouped element will have the focussing plane that intersects each other is same group.

4. metadata adding set as claimed in claim 2, wherein, based on the table of the position relation between the storage focussing plane, it is same group that grouped element will have the image packets that is included in the focussing plane in the relation of position.

5. one kind is added metadata the metadata adding method of the image of being caught by imaging device to, comprising:

The sensitive information obtaining step obtains the sensor information relevant with the capturing condition of imaging device;

Focussing plane is derived step, and based on the sensor information of being obtained, and derivation is as the position of the focussing plane of the imaging surface of the image of being caught; And

Metadata is added step, adds the position of the focussing plane of being derived to caught image as metadata.

6. metadata adding method as claimed in claim 5 also comprises:

The grouping step, based on the relation of the position between a plurality of focussing planes, and to a plurality of image packets; And

Interpolation information recording step, the result who is used to write down grouping is as interpolation information, and is simultaneously that interpolation information is relevant with image.

7. metadata adding method as claimed in claim 6, wherein, the image packets that the grouping step will have the focussing plane that intersects each other is same group.

8. metadata adding method as claimed in claim 6, wherein, based on the table of the position relation between the storage focussing plane, it is same group that the grouping step will have the image packets that is included in the focussing plane in the relation of position.

Images