JP2019139310A - Information processing apparatus, method and program - Google Patents

Abstract

To provide an information processing apparatus capable of stabilizing a tracking result.SOLUTION: An information processing apparatus 10 comprises: a search unit 6 that searches for motion of an object in frames from all or a part between frames, for the plurality of frames arranged on a time axis satisfying a unit interval of a second frame rate lower than a first frame rate, obtained by performing imaging at the first frame rate; and an estimation unit 7 that obtains the motion at the second frame rate by combining the searched motion. It further comprises an output unit 8 that performs display at the second frame rate based on the combined motion. When performing the display at the second frame rate, the frames obtained by performing imaging at the first frame rate are displayed at the second frame rate.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, method, and program capable of stabilizing a tracking result.

  The technology for identifying and recognizing an object from an image can convert analog information described in a medium that can be easily distributed and presented as one application, and can be converted into digital information, improving user convenience. be able to. As this technique, the one of Non-Patent Document 1 is disclosed. In Non-Patent Document 1, a feature point is detected from an image, a feature amount is calculated from the periphery of the feature point, and then compared with a feature amount accumulated in advance. Is identified.

  On the other hand, as a technique for further stabilizing accuracy regarding the determination based on the feature points and feature amounts as described above, for example, a technique disclosed in Patent Document 1 is disclosed. Patent Document 1 discloses a method for dynamically updating a target tracking template. An image obtained by correcting an input image after detecting a target in Non-Patent Document 1 or the like is used as a tracking template, and feature points are obtained by tracking feature points using the tracking template by template matching, and the posture of the target is estimated. Since the tracking template is generated from the actual image of the object being tracked in subsequent images, it can enable effective and stable tracking.

JP 2006-28331 A

D. G. Lowe, `` Object recognition from local scale-invariant Features, '' Proc. Of IEEE International Conference on Computer Vision (ICCV), pp.1150-1157, 1999.

  However, the conventional techniques such as Non-Patent Document 1 and Patent Document 1 described above have a problem that the recognition result or the tracking result may become unstable.

  Specifically, Non-Patent Document 1 has a problem that when imaging is performed from an oblique direction, the feature amount changes depending on the projection distortion, and the determination process does not function at all if the feature amount cannot be matched with a pre-registered feature amount. . Even when the imaging target is imaged from a distance, the feature amount changes and is registered in advance because the region used for calculating the feature amount is relatively wide with respect to the region occupied by the target to be detected. There is a problem that the determination process does not function at all if it cannot be matched with the feature amount.

  Further, in Patent Document 1, since the object once detected is tracked, the posture can be estimated even if the object is moved obliquely as long as it can be detected in front, so that part of the problem can be solved. However, if there is an error in the posture information used for correction, there is a problem that not only the error occurs in the template, but also the error tracked by the template is accumulated, so that the accuracy is significantly lowered or the tracking fails. .

  In view of the above-described problems of the prior art, an object of the present invention is to provide an information processing apparatus, method, and program capable of stabilizing a tracking result.

  In order to achieve the above object, the present invention is an information processing apparatus that satisfies a unit interval of a second frame rate lower than the first frame rate obtained by performing imaging at the first frame rate on the time axis. For a plurality of frames arranged in a frame, a search unit that searches for a target motion in the frame from all or part of the frame, and an estimation unit that combines the searched motions to obtain a motion at the second frame rate And. Further, the present invention is characterized by a method and a program corresponding to the apparatus.

  According to the present invention, it is possible to stabilize the tracking result by performing a motion search on the premise of a high first frame rate and obtaining a motion at a low second frame rate by combining the results.

It is a functional block diagram of the information processor concerning one embodiment. It is a figure which shows typically the mutual relationship of the processing content of the search part in one Embodiment, an estimation part, and an output part. It is a flowchart of operation | movement of the information processing apparatus which concerns on one Embodiment as sequential processing. It is a flowchart of operation | movement of the information processing apparatus which concerns on one Embodiment as batch processing. It is a figure which shows the schematic example of the setting of the search range according to the space | interval between frames at the time of calculating | requiring a motion. It is a figure which shows the example of a model for demonstrating the advantage made into a search object even if it is a flame | frame away from an adjacent frame.

  FIG. 1 is a functional block diagram of an information processing apparatus according to an embodiment. As illustrated, the information processing apparatus 10 includes an imaging unit 1, a buffer unit 2, a switch unit 3, a detection unit 4, a storage unit 5, a search unit 6, an estimation unit 7, and an output unit 8. Here, as a hardware configuration for realizing the information processing apparatus 10, an arbitrary information terminal including the imaging unit 1 can be used. In addition to a mobile terminal, a tablet terminal, a desktop or laptop computer, and the like can be used. Can be used. Alternatively, some or all of the functional units other than the imaging unit 1 may be installed in a server, and information exchange between the functional units illustrated in FIG. 1 may be realized by communication via a network or the like. Conversely, the imaging unit 1 may exist on the network, and a captured image acquired from the imaging unit 1 may be processed in the information processing apparatus 10. The outline of the processing contents of each functional unit in FIG. 1 is as follows.

  The imaging unit 1 captures an object in response to a camera imaging operation by the user, and outputs the obtained captured image to the buffer unit 2. Imaging by the imaging unit 1 is performed at the first frame rate, and an image, that is, a captured image as a frame at each time in time series is output to the buffer unit 2. In FIG. 1, “frame rate” is abbreviated as “rate”. As hardware for realizing the imaging unit 1, in recent years, a digital camera that is often provided as a standard in a mobile terminal can be used.

  In the following description, as the notation of the variable name, the time at which the order is specified by the integer i is expressed as “time ti”, and the captured image obtained by imaging by the imaging unit 1 at the time ti is expressed as “frame Fi”. And That is, “frame Fi” is the same as “captured image Fi”, but when referring to processing on the time axis in the following description, expressions such as “frame Fi” are mainly used.

  The buffer unit 2 temporarily stores the frame Fi at each time ti obtained from the imaging unit 1, thereby referencing the frame Fi by the detection unit 4 and the search unit 6 (via the switch unit 3) and an output unit. 8 makes it possible to refer to the frame Fi. In one embodiment, the reference of the frame Fi from the detection unit 4 and the search unit 6 (via the switch unit 3) is made at the same first frame rate as the imaging rate in the imaging unit 1, and from the output unit 8 The reference of the frame Fi is made at a second frame rate lower than the first frame rate.

  Note that the period for temporarily storing the frame Fi at each time ti in the buffer unit 2 is at least a period in which processing of the detection unit 4, the search unit 6, and the output unit 8 described below is possible. Good. The buffer unit 2 may discard a frame Fi that has become unnecessary as a reference target after the period has elapsed.

  While the predetermined detection target in the frame Fi is not detected by the detection unit 4, the switch unit 3 supplies each frame Fi of the buffer unit 2 to the detection unit 4, and the detection unit 4 detects the target. Each frame Fi at a later time performs a switching process so as to be supplied to the search unit 6 so that the target is continuously tracked in the frame. When continuous tracking by the target search unit 6 becomes impossible, the switch unit 3 returns to the state in which each frame Fi is supplied to the detection unit 4 again.

  The detection unit 4 extracts the image feature amount from the frame Fi at each time ti supplied from the switch unit 3 when the predetermined target is not detected as described above, and stores the image feature amount of the predetermined target stored in the storage unit 5. Is attempted to detect the predetermined object. Here, assuming that the detection is successful at a certain time t0, the detection unit 4 outputs to the search unit 6 information on the region R0 occupied by the predetermined object in the frame F0 at the time t0 as the detection result.

  As the image feature amount for detection in the detection unit 4, any type of existing feature may be used, detection of any type of feature point (key point as a characteristic point such as a corner in the target) and the feature An image feature amount based on an arbitrary type of local feature amount extracted from a point may be used. With respect to the feature point detection and local feature extraction methods, existing methods such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded Up Robust Features) disclosed in Non-Patent Document 1 and the like can be used. Here, the image feature amount may be defined including the coordinate information of the feature points (particularly the coordinate information of the relative arrangement of the plurality of feature points). A feature amount may be defined. For example, after further quantizing individual local features (configured as real vectors) into visual words, the image features can be converted into bug-of-visual words that are histograms of the visual words of the entire target. You may define it.

  The storage unit 5 stores in advance the same type of image feature quantity that the detection unit 4 extracts as being extracted from the image for each of one or more predetermined detection targets, and the detection unit 4 When it is determined that the image feature value extracted from Fi and the image feature value stored in advance in the storage unit 5 are similar by the threshold determination, a predetermined target corresponding to the stored image feature value is detected from the frame Fi. Can be determined. In the similarity determination, the distance between the image feature amounts is calculated, and those determined to be small may be determined to be similar. When image feature values are defined including the coordinate information of feature points, matching between feature points is performed taking into account the correspondence of geometric matching between feature point coordinates evaluated by calculation of homography matrix etc. The distance between the image feature amounts may be calculated on the assumption that the degree of geometric matching is also reflected.

  The detection unit 4 may also perform detection based on evaluation of image features more generally, instead of detection based on the image feature amount extraction processing as described above. For example, target detection may be performed from the frame Fi by template matching. In this case, a template image may be stored in advance in the storage unit 5 for each of one or more detection targets, and the search within the frame Fi using the template image may be performed by the detection unit 4.

  In the search unit 6, regarding the frames F1, F2,... At the times t1, t2,... After the detection unit 4 has successfully detected the predetermined target at the time t0, the movement of the area occupied by the predetermined target in the frame is separated. The predetermined target is tracked by searching between frames of time. The motion search result as the tracking result is output to the estimation unit 7.

  In the search unit 6, when a predetermined target has already been detected or searched for as a region Ri in the frame Fi at a certain time ti, another time ti + k (k Can be positive or negative.) By searching the predetermined area Ri + k in the frame Fi + k of the frame Fi + k, the motion M (i, i + from the area Ri of the frame Fi to the area Ri + k of the frame Fi Get the information of k). Here, various embodiments are possible regarding how to set the search source frame Fi and the search destination frame Fi + k in the search and in what order the search is performed. The various embodiments will be described with reference to FIGS.

  In one embodiment, the search processing in the search unit 6 may use template matching which is an existing method as a region tracking method. As the template image to be used, when the detection unit 4 succeeds in detection, a predetermined template image that is stored in advance in the storage unit 5 with respect to the detection target may be read from the storage unit 5 and used. The area Ri of the frame Fi itself may be used as a template image updated as needed. At this time, only the detection region R0 by the detection unit 4 at time t0 may be fixedly used as a template image. In another embodiment, the search process in the search unit 6 may estimate the region Ri + k from the region Ri by using a particle filter that is an existing method as the region tracking method.

  The estimation unit 7 obtains the motion estimation result at the second frame rate by combining the motion search results obtained at the first frame rate from the search unit 6 on the time axis, and obtains the second frame rate. The motion estimation result at is output to the output unit 8.

  Here, when the information of the motion M (i, i + k) is given as a matrix M (i, i + k) of coordinate transformation between the regions Ri, Ri + k, the motion D of the second frame rate is It can be synthesized as a product of the matrix M (i, i + k) as in the following equation (1).

  When the information of the motion M (i, i + k) is given as the translation vector M (i, i + k) of the coordinates between the regions Ri, Ri + k, the following equation (2) Thus, the motion D can be obtained by synthesizing in a simplified form as the sum of the vectors, not the product.

  Note that the object to be multiplied or summed in the above two formulas (1) and (2) is within the period of the motion D of the second frame rate and is searched by the search unit 6 on the assumption of the first frame rate. All the motions M (i, i + k), and in the above two expressions, the object is represented by the index i. Since various embodiments of products and sums represented by the index i are determined according to a specific search by the search unit 6, the various embodiments are also shown in FIGS. 3 and 4 described later. A form is demonstrated. In addition, what is necessary is just to take a product in order of the time of the acquired motion, when taking a product.

  The output unit 8 obtains the motion estimation result of the target at the second frame rate obtained from the estimation unit 7 (that is, the estimation result is also detected by the detection unit 4 and the estimation result of the target intra-frame position at a later time) Is applied to each frame Fi at the second frame rate obtained from the buffer unit 2 to obtain and display output information at the second frame rate. That is, the update interval of the output information displayed by the output unit 8 is the reciprocal of the second frame rate.

  In the output unit 8, further, predetermined information (accompanying information according to the detection target) corresponding to the recognition result of what the predetermined target is identified when the predetermined target is detected in the detection unit 4, the information Can be read from the storage unit 5 that stores them in advance and the processing can be performed. In one embodiment, the output unit 8 displays the augmented reality display at the second frame rate with respect to the captured image Fi obtained at the first frame rate in the imaging unit 1 as output information at the second frame rate subjected to the processing. Can be obtained and displayed.

  In the output unit 8, by using the existing technology of augmented reality display, the captured image Fi is matched with the three-dimensional structure of the captured real space, and the detected and tracked target is expanded. It is also possible to obtain output information that has been displayed in reality. In this case, information on a spatial relative positional relationship between a camera as hardware constituting the imaging unit 1 and a predetermined target detected and tracked in the captured image Fi may be used.

  Here, the output unit 8 applies an existing method such as DLT or 8-point algorithm to the corresponding point group obtained from each output motion information (that is, each motion D on the time series obtained at the second frame rate). By applying, it is possible to estimate the relative positional relationship between the camera and the predetermined object or between the camera and the camera. That is, first, when detecting the predetermined target at the time t0 in the detection unit 4, by using what includes the coordinate information of the individual feature points as the image feature amount, between the camera and the predetermined target at the detection time t0 The relative positional relationship can be obtained in the form of a homography matrix H0. If the motions D1, D2, D3,... Are obtained at the second frame rate after detection, the relative positional relationship corresponding to the motions can be obtained as homography matrices H1, H2, H3,. . Therefore, the output unit 8 can obtain the relative positional relationship at the second frame rate after detection as H1, H0, H2, H1, H0, H3, H2, H1, H0,.

  The output unit 8 can be realized by a display as hardware. The display may be configured as a non-see-through type using a normal liquid crystal monitor or the like, or may be configured as a see-through type using a liquid crystal or an organic EL monitor. When the output unit 8 realizes augmented reality display when configured as a see-through type, the output unit 8 may omit receiving and displaying the captured image Fi from the buffer unit 2. Further, when the see-through type is configured in the form of a head-mounted display (HMD), for example, the detection position in the see-through type HMD with respect to the target detection position in the captured image Fi by the imaging unit 1 as a camera. When the augmented reality superimposed display is performed at the corresponding position, the position where the real detection target is visible in the user's view and the superimposed display position as the corresponding position of the detected position in the captured image Fi It is desirable to perform calibration between the camera and the see-through HMD so as to match.

  In the above, as an example suitable for applying the information processing apparatus 10, an example in which output information related to vision such as augmented reality display is obtained by the output unit 8 has been described, but instead of or in addition to this, The output information related to any other perception may be obtained as processed based on the motion D. For example, an audio output corresponding to the motion D may be obtained. Further, the motion D may be output as it is in the form of text information or the like. (In this case, the output unit 8 is omitted in the information processing apparatus 10, which corresponds to a configuration in which the output from the estimation unit 7 is the output from the information processing apparatus 10.) A control output for controlling a device or the like may be obtained. In this case, the output interval of output information such as voice, text information, and control corresponds to the second frame rate.

  In the storage unit 5, in order to enable detection in the detection unit 4 described above, feature information of a detection target configured as an image feature amount or the like is stored, and output information obtained by processing in the output unit 8 In order to enable the generation of the detection information, the accompanying information is stored for each type of the detection target, and the information is provided to the detection unit 4 and the output unit 8 for reference. The information stored in the storage unit 5 may be prepared in advance by an administrator or the like.

  FIG. 2 shows the interrelationship of the processing contents according to the embodiment by the search unit 6, the estimation unit 7 and the output unit 8 for each frame Fi after the detection of the target from the frame F0 at the time t0. It is a figure shown typically. In FIG. 2, [1] schematically shows search processing of the search unit 6, and [2] schematically shows estimation processing and output processing in the estimation unit 7 and the output unit 8 linked to the search processing.

  That is, as indicated by [1] in FIG. 2, the search unit 6 uses the adjacent times ti and ti + 1 (i = 0, 1, 2, 2) at the same first frame rate as the imaging frame rate of the imaging unit 1. ... 6) is detected between adjacent frames Fi and Fi + 1 (i = 0, 1, 2,... 6) (in the figure, they are schematically represented by white circles (O) in the frame). ) Motion M (i, i + 1) (i = 0, 1, 2,... 6) (represented schematically by curved arrows in the figure). Then, as shown in [2] in FIG. 2, in the estimation unit 7, the seven motions M (i, i + 1) (i = 0, 1, 2,... 6) searched by the search unit 6 As a combination of the above-described equations (1) or (2) to obtain a motion M (0,7) of the second frame rate defined between the frames F0 and F7, and the obtained motion M ( 0,7), the output unit 8 performs output such as augmented reality display at the second frame rate (represented schematically by stars (*) in the figure).

  2 is an example in which the first frame rate is seven times the second frame rate. For example, the first frame rate may be 210 fps (frames per second) and the second frame rate may be 30 fps.

  FIG. 3 is a flowchart of the operation of the information processing apparatus 10 according to an embodiment. The embodiment in FIG. 3 corresponds to an embodiment in which the search unit 6 performs a motion search by sequential processing on a captured image obtained at the first frame rate in the imaging unit 1, and realizes the operation of the schematic example in FIG. It corresponds to an example.

  3 corresponds to the state in which the captured image Fi is sent from the switch unit 3 to the detection unit 4 in the functional block expression of FIG. 1 in the loop composed of steps S1, S2, and S20. The state outside the loop corresponds to the state in which the captured image Fi is sent from the switch unit 3 to the search unit 6 in the functional block expression of FIG. That is, the switch unit 3 expresses the operation flow structure as an example shown in FIG. 3 in the functional block configuration of FIG. 1. Therefore, in the description of each step of FIG. A duplicate description of the operation will be omitted. Therefore, when in the loop, the detection unit 4 directly refers to the buffer unit 2 to acquire the captured image Fi, and when outside the loop, the search unit 6 directly refers to the buffer unit 2 to capture images. The operation of FIG. 3 will be described without particularly mentioning the switch unit 3 as acquiring the image Fi. (The same applies to FIG. 4 described later.) Hereinafter, each step of FIG. 3 will be described.

  The start time of the flow in FIG. 3 is assumed to be a state in which the target is not detected immediately after the start of imaging by the imaging unit 1 (for example, a state immediately after the camera of the imaging unit 1 is activated). Proceed to S1. In step S1, after the detection unit 4 tries to detect the target of the captured image Fi acquired by the imaging unit 1 at the current time ti, the process proceeds to step S2. In step S2, it is determined whether or not the detection by the detection unit 4 in the latest step S1 is successful. If successful, the process proceeds to step S3, and if unsuccessful, the process proceeds to step S20.

  In step S20, the time ti is updated to the next time ti + 1, and after the imaging unit 1 acquires the captured image Fi + 1 at the updated current time ti + 1, the process returns to step S1. The time update in step S20 may be such that the first frame rate is updated from time ti to time ti + 1 as described above, or the second frame rate (N is an integer greater than or equal to 2 It is also possible to update from time ti to time ti + N as the update of 1 / N of one frame rate. That is, the detection by the detection unit 4 in step S1 may be performed at either the first or second frame rate.

  In step S3, the detection unit 4 outputs information on the target area Ri detected from the frame Fi to the search unit 6 based on the detection result succeeded in the most recent step S1, and the detection is performed from the storage unit 5. Accompanying information according to the object (accompanying information for enabling the above-described augmented reality display or the like) is output to the output unit 8, and then the process proceeds to step S4.

  In step S4, the time ti is updated to the next time ti + 1 at the first frame rate. After the imaging unit 1 acquires the captured image Fi + 1 at the updated current time ti + 1, the process proceeds to step S5. Proceed with Note that the time ti is the current time in each step of FIG. 3 on the assumption that the latest time (current time) is always obtained by updating to the next time ti + 1 in steps S4 and S20. The time will be described as time ti. (The description of each step in FIG. 4 to be described later is also based on the same premise, and the current time is described as time ti.)

  In step S5, the search unit 6 refers to the buffer unit 2, and is the frame Fi-k of the latest past time ti-k (k ≧ 1) and has been successfully detected by the detection unit 4 or by the search unit 6. By using the target area Ri-k that has been successfully searched by searching, the target area Ri is searched from within the frame Fi at the current time ti, and then the process proceeds to step S6 after trying to track the area. In step S6, it is determined whether or not the tracking of the latest step S5 is successful. If successful, the process proceeds to step S7, and if unsuccessful, the process proceeds to step S70.

  In step S7, the search unit 6 acquires information on the area Ri in the frame Fi at the current time ti that has succeeded as information used for tracking in step S5 at the next time ti + 1, and then proceeds to step S8. . In step S70, the search unit 6 stores the fact that the tracking has failed for the time ti, and then proceeds to step S9. In addition, by storing in step S70, it is possible to obtain the latest past time ti-k (k ≧ 1) in step S5. In addition, when tracking fails for a certain time ti, step S7 is skipped in this way, and therefore the area Ri as the tracking / search source in step S5 is not acquired for the failed time ti.

  In step S8, the search unit 6 acquires the motion M (i, i-k) as a result of the successful tracking in the latest step S5, and then proceeds to step S9. In step S9, it is determined whether or not the current time ti (current time ti incremented and updated by 1 at the first frame rate in step S4) has reached the update timing of the second frame rate. If so, the process proceeds to step S10. If not reached, the process returns to step S4.

  In step S10, the current time ti corresponding to the update timing of the second frame rate and the time ti-N corresponding to the update timing of the most recent second frame rate (that is, N is an integer of 2 or more as described above, N times of the update of the first frame rate interval corresponds to one update of the second frame rate interval)), and the motion M (ik) acquired in the series of steps S8 in the period between After synthesizing and trying to obtain the motion D of the second frame rate (and outputting the output information by the output unit 8 based on the motion D), the process proceeds to step S11.

  In step S11, it is determined whether it is possible to obtain the motion D attempted in step S10 (and output of output information based on this), and if it is possible, the process is updated by returning to step S4. The search as described above by the search unit 6 is continued for the latest time ti. If it is determined in step S11 that it is impossible, the process returns to step S1, and the detection by the detection unit 4 is resumed for the updated latest time ti.

It should be noted that the determination of whether or not it was possible to obtain the motion D is the acquisition of the individual motion M (i, ik) for obtaining the motion D when all or any of the following conditions are satisfied. It may be carried out by determining that it is impossible to obtain the motion D as insufficient.
(Condition 1) At the update timing of the second frame rate, that is, when an affirmative determination is obtained in step S9, in the latest step S6 (that is, step S6 at the same time ti as the current time ti in step S9) Negative judgment is obtained.
(Condition 2) The number of times that a negative determination is obtained in step S6 regarding the first frame rate for N times constituting the second frame rate exceeds a predetermined threshold.

An example of motion acquisition and synthesis according to the flow of FIG. Here, assuming that N = 4 and the first frame rate is four times the second frame rate, 4 + 1 = 5 frames F0, F1, F2, F3, and F4 obtained at the first frame rate Then, an example of synthesizing the motion M (0,4) at the second frame rate is given.
(Example 1) When all the motions M (0,1), M (1,2), M (2,3), M (3,4) are successfully searched, these four motions are combined. A motion M (0,4) is obtained.
(Example 2) Of the above, the search for the motion M (1,3) succeeds after the search for the motion M (1,2) fails, and as a result, the motion M (0,1), M (1,3), When the search for M (3,4) is successful, the motion M (0,4) is obtained by combining these three motions.
(Example 3) Of the above, after the search for M (1,2) fails, the search for motion M (1,3) also fails, and then the search for motion M (1,4) succeeds. As a result, when the motion M (0,1) and M (1,4) are successfully searched, the motion M (0,4) is obtained by combining these two motions.

  Although the first frame rate is described as being N times the second frame rate, another one frame is added to N consecutive frames of the first frame rate because of so-called planting calculation. By setting the number to N + 1, a frame corresponding to one cycle of the second frame rate and including both end frames can be obtained.

  The above flow of FIG. 3 is an embodiment in the case where the search unit 6 performs a motion search by sequential processing on the captured image Fi of the first frame rate in the imaging unit 1. FIG. 4 is a flowchart of the operation of the information processing apparatus 10 according to another embodiment. The search unit 6 performs a search based on the captured image Fi given at the first frame rate as in FIG. Instead of searching as sequential processing, the captured image Fi of the first frame rate is accumulated in the buffer unit 2 by N + 1 sheets corresponding to one cycle (and both ends) of the second frame rate, and then It is the flow of an embodiment which searches by batch processing for N + 1 sheets or a part thereof. Hereinafter, each step of FIG. 4 will be described.

  Steps S21, S22, S23, and S220 after the flow of FIG. 4 is started are the same as steps S1, S2, S3, and S20 of FIG. In this way, after obtaining the detection success detection by the detection unit 4 in step S22 of FIG. 4 and obtaining the output corresponding to the detection result in step S23 as in steps S2 and S3 of FIG. 3, the process proceeds to step S31. move on.

  In step S31, the current time ti is updated to the next time ti + 1 at the first frame rate. After the imaging unit 1 acquires the frame Fi + 1 at the latest time ti + 1, the process proceeds to step S32. In step S32, it is determined whether or not the current time ti has reached the update timing of the second frame rate. If it has reached, the process proceeds to step S33, and if not, the process returns to step S31.

  In step S33, N + 1 frames Fi-N, Fi-N + 1 for one period (and both ends) of the second frame rate obtained at the current time ti through the loop of steps S31 and S32 described above. ,..., Fi-1, Fi, the search unit 6 tries to track the object by trying motion search between the end frames Fi and Fi-N, and then proceeds to step S34. Regarding the motion search in step S33, the search source frame Fi-N (that is, the frame successfully detected in the latest steps S21 and S22, or the search success in the latest steps “S33 and S34” or “S35 and S36”) The region Ri-N that has already been detected or searched in the frame) may be used.

  In step S34, it is determined whether or not the double-ended frame search in step S33 is successful. If successful, the process proceeds to step S37, and if unsuccessful, the process proceeds to step S35. In step S35, the search unit 6 receives a search failure in the both end frames Fi and Fi-N, and recursively follows the predetermined rule using internal frames other than both ends among the N + 1 frames existing. In particular, by performing a plurality of motion searches, an attempt is made to obtain a motion search result corresponding to both ends, and the process proceeds to step S36. Various embodiments are possible for step S35, which will be described later.

  In step S36, it is determined whether or not the search in step S35 has been successful. If successful, the process proceeds to step S37, and if unsuccessful, the process returns to step S21 to restart the process of the detection unit 4. The Rukoto. In step S37, the estimation unit 7 obtains the motion estimation result in the both end frames Fi and Fi-N and passes it to the output unit 8 to obtain output information, and the search unit 6 for motion estimation that is continued thereafter. After acquiring / holding the information of the area Ri at the current time ti, the process returns to step S31. In step S37, when the motion estimation of both end frames Fi and Fi-N in steps S33 and S34 is successful, the estimation unit 7 directly uses the one motion as the motion estimation result M (i, iN). When step S35 and S36 lead to step S37, a plurality of motions recursively tracked in step S35 are combined to obtain a motion estimation result M (i, iN).

  Hereinafter, each embodiment of step S35 will be described. In one embodiment, the sequential processing described in FIG. 3 (loop processing of S4, S5, S6, S7, S70, S8, and S9) can be employed as it is as a procedure for realizing step S35. In this case, the time update procedure in step S4 may be simply read as a procedure for setting the next adjacent frame Fi to be tracked, and the determination in step S9 is to determine whether the processing has been completed for all N + 1 frames. You can replace it.

In the embodiment using the sequential processing of FIG. 3 as it is, all N + 1 frames are used as motion search targets (unless there is no search failure). By following a predetermined procedure, only a part of the frames may be set as a motion search target. An example of a procedure for realizing the other embodiment in the case of searching for motion M (0, N) in N + 1 frames F0, F1,... FN corresponding to the second frame rate is as follows. is there.
(Procedure 1) The initial value i = 1 is set in the variable i, and the procedure proceeds to procedure 2.
(Procedure 2) The motion M (0, i) is searched, and if successful, the procedure proceeds to procedure 3, and if unsuccessful, the procedure proceeds to procedure 4.
(Procedure 3) The motion M (i, N) is searched for, and if successful, the process ends. If unsuccessful, the process proceeds to Procedure 4.
(Procedure 4) The value of variable i is incremented by 1, and the procedure returns to procedure 2.

  In step 2, if there is at least one motion M (0, ij) (j ≧ 1) that has already been successfully searched, M (ij, i) may be searched for, and the minimum of j If there is a successful search from those, it may be determined that step 2 has been successful. If the value of i reaches N in step 4, it is determined that the search according to the above procedure has failed.

  Further, the specific procedure is not limited to the above, and other procedures may be used. As a result (when successful), a plurality of movements M (0, i1), M (i1, i2), M (i2, i3 ), ..., M (im, N) (0 <i1 <i2 <i3 <... <im <N) may be used as an arbitrary procedure. The estimation unit 7 can obtain the motion M (0, N) by combining these.

  For example, the initial value of procedure 1 may be N-1, and the value of variable i may be subtracted by 1 in procedure 4. Then, the initial value of procedure 1 and the increment / subtraction value of procedure 4 may each be a predetermined value other than 1 (such as a predetermined value according to N). It is also possible to use a procedure that recursively divides N + 1 frames in half and explores the possibility of motion search at both ends. Similarly, N + 1 frames are recursively. Specifically, a procedure may be used in which the possibility of motion search is searched at both ends while dividing at the position of the ratio “a: (1-a)” (where 0 <a <1). When performing the recursive search, a section that has been successfully searched may be excluded from subsequent searches.

  As described above, according to the present invention, it is possible to estimate the motion with high accuracy at the update interval of the output unit 8 by combining a plurality of motions calculated from between the captured images Fi. If, for example, augmented reality display is realized using the motion estimated with high accuracy, the relative positional relationship between the camera of the imaging unit 1 and the object being imaged can be estimated with high accuracy. In addition, it is possible to realize highly accurate augmented reality display and the like. In the present invention, highly accurate motion estimation is possible from the following viewpoints.

(Viewpoint 1) Speeding up by reducing the amount of motion between captured images Fi accompanying shortening of the imaging interval. For example, if the imaging interval is 1/240 seconds, the amount of motion between the captured images Fi is reduced to 1/8 compared to the imaging interval of 1/30 seconds, so the range of motion search is controlled according to the imaging interval. To achieve high speed. In general, when the imaging interval is 1 / N times, the number of imaging information increases N times due to the increase in imaging information, but the motion amount itself is 1 / N, so the motion search range is (1 / N ) * (1 / N). Therefore, the number of searches for the entire process is 1 / N at the maximum, so shortening the imaging interval leads to higher speed.

(Viewpoint 2) Higher accuracy by linear approximation of motion between captured images Fi accompanying shortening of the imaging interval. When movement other than translation or a change in the appearance of the three-dimensional shape occurs, template matching based on translation is more likely to be approximated by translation as the imaging interval is shorter, leading to improved accuracy of motion search.

(Aspect 3) Higher accuracy due to a reduction in the amount of motion between captured images Fi accompanying a reduction in the imaging interval. In the case where there is a large movement, the shorter the imaging interval, the smaller the amount of movement, so the possibility that the search can be continued increases and the accuracy of the movement search is improved. On the other hand, when there is a small movement, the movement between the captured images Fi is suppressed, so that shortening the imaging interval leads to stabilization. For example, when the amount of motion is 1 pixel per 1/30 second, the average amount of motion at 1/240 second intervals is 1/8 pixel, but the amount of motion between pieces of imaging information can be estimated as 0. As a result, the output motion information is also estimated as 0.However, in augmented reality applications, the situation where the amount of motion is about 1 pixel is often a disturbance such as camera shake in a stationary state. It leads to stabilization.

(Aspect 4) Robust against disturbances such as fluorescent light flicker and flash. Since the result of the motion search is only used for the estimation of the output motion information in the estimation unit 7, it is only necessary to search for motion information that can estimate the output motion information even if the search fails for a part of the imaging motion information. This contributes to improving the stability of the entire system.

  Hereinafter, supplementary matters such as modifications in the present invention will be described.

(1) In relation to the above viewpoint 1, when the search unit 6 obtains the motion M (i, i + k), the search range by template matching or the like becomes wider as the interval k ≧ 1 for obtaining the motion is larger. It may be set. FIG. 5 shows a schematic example of the setting. When region R10 of frame F10 at time t10 is the search source region, when obtaining motion M (10,11) in frame F11 at time t11 where k = 1, region R10 (the same in frame F11) For example, when the region SR11 with the position expanded to 110% is set as the search range, and the motion M (10,12) is obtained in the frame F12 at t12 where k = 2, the region R10 (in the frame F12) A region SR12 in which the same position extends the region T12) to, for example, 120% may be set as the search range.

  In the example of FIG. 5, the regions T11 and T12 that are the search range have the center at the same position as the search source R10, but the center position has been moved as predicted from the motion estimated so far. You may make it set as a position.

(2) In order to perform an output such as display at the second frame rate, it is necessary to complete processing such as motion estimation according to the present invention, but at times t0, tN, t2N,. Assume that frames F0, FN, F2N,... Are obtained, and the time “t0” + Δ, “tN” corresponding to the second frame rate is actually obtained on the assumption that the motion estimation or the like is completed within a predetermined time Δ. Display and the like can be performed on + Δ, “t2N” + Δ,. In the above description of FIG. 3 and FIG. 4, it has been explained that calculation etc. is completed instantaneously when each frame Fi etc. is obtained at time ti etc., without particularly mentioning the time required for calculation etc. Assuming that the series of calculations are completed within a predetermined time Δ, it is possible to actually output a display or the like while maintaining the second frame rate.

(3) In relation to the above point of view 4, in the present invention, it is not necessarily limited to the motion M (i, i + 1) between the (most) adjacent frames Fi, Fi + 1. , The advantage that the motion M (i, i + k) between Fi + k (k ≧ 2) is a search target will be described with reference to FIG. Here, at the three adjacent times t20, t21, and t22, the one pixel as shown by hatching in the minute range (2 × 2 pixel range) of u, v = 0,1 of the image coordinates (u, v) An example is shown in which a pattern of approximately the same size as is moved in the (u, v) = (+ 1, + 1) direction. In this case, the pattern occupies only the position (0,0) at time t20, and the pattern occupies only the position (1,1) at time t21, but at the intermediate time t21, the 2 × 2 pixels The pattern is dispersed and arranged in the range. Under such circumstances (when the appearance of the distribution of edge boundaries etc. at the pixel boundary changes), the motion M (20,21) and motion M (21,22) related to the pattern is in principle detected accuracy. However, the motion M (20, 22) can be detected with high accuracy. The possibility of highly accurate detection is ensured by a motion M (i, i + k) search between distant frames Fi, Fi + k (k ≧ 2).

(4) The detection unit 4 may detect two or more objects. For example, if two objects A and B are detected, the motion estimation according to the present invention described above can be performed independently (simultaneously in parallel) for each of the objects A and B. The order in which the inter-frame motion M (i, i + k) is searched is independent for the objects A and B. Here, the time at which the objects A and B as the preprocessing for the search are detected may be different. Further, the detection unit 4 may detect the entire individual object as a rectangular region or the like, and the search unit 6 may perform a search for each part of the detected object, and the motion M ( The search order of i, i + k) is independent. The target portion may be configured as a predetermined neighborhood range (for example, a rectangular range of a predetermined size) of one or a plurality of feature points when detected by the detection unit 4 based on image features.

(5) The second frame rate accepts manual designation from the user, or manually or according to the analysis result such as the nature of the captured image Fi at the first frame rate (brightness, overall movement amount, etc.) It may be set as a variable that can be automatically changed.

(6) The information processing apparatus 10 can be realized as a computer having a general configuration. That is, a CPU (Central Processing Unit), a main storage device that provides a work area for the CPU, an auxiliary storage device that can be configured with a hard disk, SSD, etc., an input interface that receives input from the user such as a keyboard, mouse, touch panel, etc. The information processing apparatus 10 can be configured by a general computer including a communication interface for connecting to and communicating, a display for displaying, a camera, and a bus for connecting them. Further, the processing of each unit of the information processing apparatus 10 shown in FIG. 1 can be realized by a CPU that reads and executes a program for executing the processing, but any part of the processing is performed in a separate dedicated circuit or the like. It may be realized. The imaging unit 1 can be realized by a camera as the hardware.

  DESCRIPTION OF SYMBOLS 10 ... Information processing apparatus, 1 ... Imaging part, 2 ... Buffer part, 3 ... Switch part, 4 ... Detection part, 5 ... Memory | storage part, 6 ... Search part, 7 ... Estimation part, 8 ... Output part

Claims (11)

For a plurality of frames arranged on the time axis that satisfy the unit interval of the second frame rate lower than the first frame rate obtained by imaging at the first frame rate, all or part of the frames A search unit for searching for the movement of the object in the frame,
An information processing apparatus comprising: an estimation unit that combines the searched motions to obtain a motion at a second frame rate.   The information processing apparatus according to claim 1, further comprising an output unit configured to output at a second frame rate based on the synthesized motion.   The information processing apparatus according to claim 2, wherein the output unit performs output at the second frame rate by display.   The output unit further displays a frame obtained by performing imaging at the first frame rate at the second frame rate when performing display at the second frame rate. The information processing apparatus described.   5. The search unit according to claim 1, wherein when searching for a motion of a target in a frame between the frames, the search unit performs a search in a search range corresponding to an interval between the frames. Information processing device.   The information processing apparatus according to claim 1, wherein the search unit searches for motion of a target in the frame between the frames by region tracking. The search unit first searches for a target motion between frames at both ends of the plurality of frames,
When the initial search is successful, the estimation unit adopts the motion that succeeded in the search as the motion at the second frame rate,
When the first search fails, the search unit searches for a motion of a target in a frame between frames at both ends of the plurality of frames and frames other than the both ends, and the estimation unit 7. The information processing apparatus according to claim 1, wherein the searched motion is synthesized to obtain a motion at the second frame rate. Furthermore, a detection unit that detects a target based on feature information from a frame obtained by performing imaging at the first frame rate,
The search unit searches for a motion of a target in a frame between the frames based on the detection result with respect to a frame after the detected frame. The information processing apparatus according to any one of 7. Furthermore, a detection unit that detects a target based on feature information from a frame obtained by performing imaging at the first frame rate,
For the frame after the detected frame, the search unit searches for the movement of the target in the frame between the frames based on the detected result,
The information processing apparatus according to claim 2, wherein the output unit outputs incidental information corresponding to the detected object at the second frame rate. For a plurality of frames arranged on the time axis that satisfy the unit interval of the second frame rate lower than the first frame rate obtained by imaging at the first frame rate, all or part of the frames A search stage to search for the motion of the object in the frame, and
And an estimation step of synthesizing the searched motions to obtain motions at the second frame rate.   A program that causes a computer to function as the information processing apparatus according to claim 1.

Images