JP2021077283A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To continue tracking with less deviation when performing tracking on video images, while reducing an arithmetic load associated with matching processing.SOLUTION: An information processing apparatus has: a feature point extraction unit that extracts a feature point and a feature quantity from a whole image of a subject and photographed images obtained by continuously photographing the subject; a feature point comparison processing unit that performs feature point comparison processing between the whole image and the photographed images; a feature point tracking processing that performs feature point tracking processing for the feature point; a tracking processing unit that performs tracking processing on a predetermined area of the image by using a transformation matrix determined based on a processing result of the feature point comparison processing or the feature point tracking processing; and a tracking error calculation unit that, when an image on which the tracking processing is not performed occurs, calculates and stores an error in tracking that occurs. When the errors stored by the tracking error calculation unit exceed a threshold, the information processing apparatus performs the feature point comparison processing and resets the stored errors.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, an information processing method, and a program.

Camera-equipped portable terminal devices (mobile terminals) have become commonplace. In the past, users used scanners and the like to electronically capture paper documents, but now it is possible to easily capture paper documents electronically by using the camera of a mobile terminal. Patent Document 1 proposes a technique for recognizing and tracking the display area and coordinates of an electronic document captured by using a camera.

JP-A-2009-20890

Patent Document 1 describes a method of recognizing and tracking a display area and a place of an electronic document by using an invisible junction feature amount. In Patent Document 1, once the display area and the location are specified by the invisible junction feature amount, the feature points in the video shooting are tracked, and then the planar movement (projection conversion) is estimated between the video shootings. Are listed. However, Patent Document 1 does not describe processing for accumulating errors due to continuous image tracking. In order to suppress the occurrence of deviation due to the accumulation of tracking errors, it is conceivable to perform the matching process with high frequency, but the calculation load associated with the matching process becomes high, and high-speed processing becomes difficult. Therefore, an object of the present invention is to enable tracking with less deviation while reducing the calculation load due to the matching process when tracking the moving image.

The information processing apparatus according to the present invention includes an extraction means for extracting a feature point and a feature amount of the feature point from a whole image of the subject and a photographed image obtained by continuously photographing the subject, and a first extraction means extracted by the extraction means. A comparison processing means for comparing a feature amount in one image with a feature amount in a second image different from the first image to obtain a matching feature point, and a feature point extracted by the extraction means. A tracking processing means for performing tracking processing related to a predetermined region of an image by using a tracking processing means for obtaining a movement vector between images and a conversion matrix obtained based on the processing result by the comparison processing means or the tracking processing means. When an image that is not tracked by the tracking processing means is generated, an error calculating means that calculates and accumulates the generated tracking error, and an error that is accumulated by the error calculating means exceeds the threshold value. It is characterized by having a control means for executing a process by the comparison processing means and resetting an error accumulated by the error calculation means.

According to the present invention, it is possible to continue tracking with less deviation while reducing the calculation load due to the matching process.

It is a figure which shows an example of the appearance of a mobile terminal. It is a figure which shows the hardware configuration example of a mobile terminal. It is a figure which shows the software configuration example of a mobile terminal. It is a figure which shows an example of UI of a mobile application. It is a figure which shows an example of the whole image and the photographed image. It is a figure explaining the data input area. It is a figure explaining the tracking process. It is a flowchart which shows the example of the tracking process. It is a flowchart which shows the example of the creation / update process of the 1st transformation matrix and the 2nd transformation matrix. It is a figure explaining the tracking process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that the embodiments described below do not limit the present invention, and not all the configurations described in the embodiments are essential as means for solving the problems of the present invention. ..

Here, a configuration for acquiring and using the result information of character recognition processing (OCR) in a local region of an image of a paper document captured by a camera will be examined. When acquiring OCR result information, if the position coordinates of the area (data input area) in which the information to be acquired is written are known (for example, a form in a known format), the area to be OCR processed can be specified. The area may be subjected to OCR processing and the OCR result may be obtained.

When the input from the camera is a moving image, it is possible to obtain (match) a combination of feature points that match between the images by comparing the feature points between the images with the feature amount thereof. However, although the feature point comparison process is a highly accurate matching method, the arithmetic processing load is generally high and the processing speed is slow. Therefore, in the case of moving image processing having a certain high speed of about 30 FPS (frames / second), a high-speed arithmetic processing unit or the like is required to finish the matching processing at the moving image frame interval.

In the case of a low-speed mobile terminal, the matching process cannot be performed frequently. Therefore, the frequency of matching processing can be reduced by performing tracking processing for estimating the moving position of the feature point on the image to be compared. The tracking process estimates the movement vector of how much the image to be compared has moved by an optical flow or the like. This makes it possible to estimate to which position on the image to be compared the feature points on the original image have moved. Feature point tracking is relatively faster than feature point comparison.

However, if the position coordinates of the data input area deviate due to the accumulation of errors due to the continuation of image tracking, it becomes difficult to obtain accurate OCR result information. Therefore, in the present embodiment, when tracking is continued using the tracking information between each image of the moving image, the matching process is performed at an appropriate timing to reduce the calculation load due to the matching process and to reduce the deviation. To be able to continue.

Hereinafter, a mobile terminal will be described as an example of the information processing device according to the present embodiment. A mobile terminal is an example of a mobile terminal, and is a terminal that can be used in a free place by equipping it with a wireless communication function or the like.

FIG. 1 is a diagram showing an example of the appearance of a mobile terminal. The mobile terminal 100 includes various units 101 to 104 and the like. The front side of the mobile terminal 100 is the front side portion 101 of the mobile terminal. The touch panel 102 is an example of a display unit such as a display, and has two functions of output (display) and input. The back side of the mobile terminal 100 is the back side 103 of the mobile terminal. The mobile terminal back surface 103 has a camera 104 for capturing an image. In the present embodiment, the user of the mobile terminal 100 can start the process by taking an image of the subject 105 using the camera 104 with a mobile application (mobile application) described later. The subject 105 is, for example, an order form for an A4 size paper document. The subject 105 is not limited to a paper document, and may be a business card, a photograph, a card, or the like of various sizes. The mobile application described later can capture an image of the subject 105 using the camera 104 and output the image to the touch panel 102.

<Hardware configuration>
Next, the hardware configuration of the mobile terminal 100 will be described. FIG. 2 is a diagram showing a hardware configuration example of the mobile terminal 100. The mobile terminal 100 includes a CPU 201, a RAM 202, a ROM 203, an input / output interface 204, a NIC 205, and a camera unit 206. The CPU 201, RAM 202, ROM 203, input / output interface 204, NIC 205, and camera unit 206 are communicably connected via the bus 207 so that data can be transmitted and received.

The CPU (Central Processing Unit) 201 executes various programs and realizes various functions. The RAM (Random Access Memory) 202 stores various types of information. The RAM 202 is also used as a temporary working storage area for the CPU 201. The ROM (Read Only Memory) 203 stores various programs and the like.

For example, the CPU 201 loads the program stored in the ROM 203 into the RAM 202 and executes the program. Further, the CPU 201 executes processing based on a program stored in an external storage device such as a flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive). As a result, the software configuration constituting the mobile terminal 100 as shown in FIG. 3 and each process in the flowchart as shown in FIG. 8 and the like are realized. The functions of the mobile terminal 100 and all or part of the processing described later may be realized by using dedicated hardware.

The input / output interface 204 transmits / receives data to / from the touch panel 102. The NIC (Network Interface Card) 205 is an interface for connecting the mobile terminal 100 to a network (not shown). The camera unit 206 connects to the camera 104 and captures an image of the subject 105 into the mobile terminal 100.

<Software configuration>
Next, the software configuration in the mobile terminal 100 will be described. FIG. 3 is a diagram showing a software configuration example of the mobile terminal 100. The mobile terminal 100 has a data management unit 301 and a mobile application 302. A program that realizes each function in the software (application) shown in FIG. 3 is stored in ROM 203 or the like.

The data management unit 301 manages images and application data. The OS provides a control API (Application Programming Interface) for using the data management unit 301. The mobile application 302 acquires and saves images and application data managed by the data management unit 301 by using the control API.

The mobile application 302 is an application that can be executed by downloading and installing it on the mobile terminal 100, for example, by using the OS installation function of the mobile terminal 100. The mobile application 302 performs various data processing on the image of the subject 105 captured through, for example, the camera unit 206. The mobile application 302 has a main control unit 303, an information display unit 304, an operation information acquisition unit 305, a captured image acquisition unit 306, a storage unit 307, and a database (DB) unit 308. Further, the mobile application 302 has a feature point extraction unit 309, a feature point comparison processing unit 310, a feature point tracking processing unit 311, a coordinate conversion processing unit 312, a tracking processing unit 313, and a tracking error amount calculation unit 314.

The main control unit 303 controls the application (mobile application) 302 for the mobile terminal 100, and gives instructions and manages to the module units 303 to 314, which will be described later. The information display unit 304 provides the user with the user interface (UI) of the mobile application 302 according to the instruction from the main control unit 303. FIG. 4 is a diagram showing an example (mobile terminal screen 400) of a screen that provides a UI (UI for a mobile terminal) of the mobile application 302. The mobile terminal screen 400 is displayed on the touch panel 102 of the mobile terminal 100. Further, on the mobile terminal screen 400, an image captured via the camera 104 is displayed in the display / operation area 401, and an operation (user operation) by the user on the image or the like is accepted via the displayed UI. The UI form (position, size, range, arrangement, display content, etc.) of the mobile application 302 is not limited to that shown in FIG. 4, and the functions of the mobile terminal 100 can be realized. An appropriate configuration can be adopted.

The operation information acquisition unit 305 acquires information related to user operations via the UI of the mobile application 302 displayed by the information display unit 304, and notifies the main control unit 303 of the acquired information. For example, when the user touches the display / operation area 401 by hand, the operation information acquisition unit 305 acquires information indicating the touched position on the screen, and transmits the acquired position information to the main control unit 303. The captured image acquisition unit 306 acquires each captured image of the moving image captured via the camera unit 206 and transmits it to the storage unit 307.

The storage unit 307 stores, for example, a captured image acquired by the captured image acquisition unit 306. Further, the storage unit 307 can delete the captured image stored by the instruction of the main control unit 303. The DB unit 308 has a database function and manages the entire image 500, which will be described later, the data input area information table 601 for the rectangular area (data input area) including the data information to be extracted in the entire image 500, and the like. To do. The data of the DB unit 308 is transmitted to the storage unit 307 when the application 302 is started by the main control unit 303, and is acquired by the instruction of the control unit 303 when necessary.

The feature point extraction unit 309 extracts feature points and feature quantities from an image taken through the camera unit 206, an image previously held in the DB unit 308 and transmitted to the storage unit 307, and the like. The feature point extraction unit 309 calculates characteristic pixel points (feature points) such as a portion (edge) where the change in brightness on the image is large, and data (feature amount) representing the features of the feature points. As a method for obtaining a feature point or a feature amount, there are a method such as SIFT (Scale-Invariant Feature Transform) and SURF (Speeded-Up Robust Features). In the present embodiment, a method that is robust against changes such as rotation, enlargement / reduction, and image movement, and that uniquely determines matching feature points in the feature point comparison process described later is preferable.

The feature point comparison processing unit 310 performs feature point comparison processing on the feature points extracted from two images different from each other by the feature point extraction unit 309. In the feature point comparison process, a combination of feature points that match between images can be obtained (matched) by comparing the feature points between the images with the feature amount thereof. In addition, by using a method such as RANSAC (Random sample consensus) that eliminates outliers and estimates the law, it is possible to eliminate the combination of feature points that cause noise and perform more accurate matching. Become. The feature point comparison process is a highly accurate matching method, but the processing speed is generally slow. In the present embodiment, the feature point comparison processing unit 310 performs feature point comparison processing between a unique overall image and an arbitrary captured image. Hereinafter, the captured image used for the feature point comparison process is also referred to as a feature point comparison image.

The feature point tracking processing unit 311 estimates the moving position of the feature points on the image to be compared with respect to the feature points extracted by the feature point extraction unit 309 from the original original image (optical flow). I do. The feature point tracking processing unit 311 estimates the movement vector of how much each feature point on the original image has moved in the image to be compared. This makes it possible to estimate to which position on the image to be compared the feature points on the original image have moved. The feature point tracking process is faster than the feature point comparison process.

Here, the method of calculating the movement vector is generally shown below. Assuming that the brightness of points x and y on the image P at time t is P (x, y, t), the time advances by Δt, and if the coordinates move by Δx and Δy during that time, the brightness of the destination is P (x + Δx, y + Δy). , T + Δt). In order for P (x, y, t) = P (x + Δx, y + Δy, t + Δt), Pv _x + Pv _y + Pt = 0 _{using the movement amount vectors v x} , v _{y per unit time of x, y.} It becomes. Since _{there are two variables v x} and v _y , it is necessary to increase at least one constraint equation, and the Lucas-Kanade method assumes that the surrounding points behave in the same way. Therefore, in order to calculate the movement vector accurately, it is necessary that the movement amount of the object between the two images is less than a certain amount (the difference between the images is small).

In the case of the Lucas-Kanade method, when the difference between the images becomes large, the estimation error becomes large and the deviation from the actual position becomes large. Factors that increase the difference between images include high-speed movement of the camera between shooting frames, large changes in the brightness of the shooting target due to external factors, delays in movie shooting processing, dropped frames, etc. It is conceivable that the shooting interval (frame interval) becomes longer. Since high-speed movement and brightness change can be detected from the motion sensor, the brightness sensor, and the captured image, the deviation can be reduced by performing the matching process at the time of detection and then restarting the tracking process. In addition, as the calculation method of the movement vector, another method may be used.

The coordinate conversion processing unit 312 maps points between the entire image and the captured image (or between captured images different from the captured image). The coordinate transformation processing unit 312 performs point mapping by calculating a homography transformation matrix (hereinafter, transformation matrix) for performing homography transformation (homography transform) between images. The homography transformation can transform and transfer a point on one plane coordinate system onto a different plane coordinate system. Similar transformations include affine transformations that rotate, translate, and scale images, but in addition to these, homothety transformations can transform trapezoids by changing the scaling ratio according to the coordinate position. It becomes. The homography transformation can be expressed as follows using the coordinate points (x1, y1) on the current image, the coordinate points (x2, y2) on the transformed image, the transformation matrix H, and the constant s.

When a similar object exists between two images, the coordinate conversion processing unit 312 is based on the corresponding point coordinates between the images obtained by the matching process by the feature point comparison processing unit 310 or the feature point tracking processing unit 311. The parameters of the transformation matrix H are calculated. This makes it possible to obtain the transformation matrix between the entire image and the captured image and map the coordinates in the entire image in the captured image, or to obtain the inverse matrix of the transformation matrix and perform the reverse mapping. It becomes. However, when a similar object does not exist between the two images (when the difference between the images is large), the number of matching feature points becomes small, and the calculation of the transformation matrix H fails.

The tracking processing unit 313 tracks which part (area) of the entire image 500 is captured by the latest captured image by the camera 104 by the tracking process described later. Then, the tracking processing unit 313 maps and draws the data input area on the captured image based on the information of the data input area stored in the data input area information table 601 and displays it on the mobile terminal screen 400.

Therefore, in order to perform tracking, the coordinate conversion processing unit 312 converts between the captured image 500 and the latest captured image from the acquisition of the latest captured image to the display of the mapped image on the mobile terminal screen 400. It is necessary to wait for the process of finding the matrix. The calculation of the transformation matrix by the coordinate conversion processing unit 312 requires matching processing of feature points between two images, and the matching processing requires two calculation methods by the feature point comparison processing unit 310 and the feature point tracking processing unit 311. There is. Since the matching process by the feature point comparison processing unit 311 takes time, if the drawing is not performed until the feature point comparison process is completed, the shooting rate will be lowered. Further, although the feature point tracking process by the feature point tracking processing unit 312 is high-speed, when tracking is performed by multiplying the transformation matrices obtained between the captured images, the error of each matrix is accumulated, so that the error of each matrix is gradually accumulated. The tracking result will be out of sync. Therefore, in the tracking process in the present embodiment, the feature point comparison process and the feature point tracking process are used in combination to minimize the tracking deviation and reduce the shooting rate in drawing on the mobile terminal screen 400. I'm preventing it.

The tracking error amount calculation unit 314 calculates the error amount of the tracking process for each captured image. In the present embodiment, when the accumulation of tracking errors exceeds a predetermined threshold value, matching processing is performed and the tracking error amount is reset. When an unprocessed image is generated, the tracking error amount calculation unit 314 calculates the error amount of the tracking process from the error amount of the movement vector calculated by the feature point tracking processing unit 311. Specifically, it is determined whether or not the frame of the moving image can be correctly acquired from the captured image acquisition unit 306.

The determination of image acquisition from the captured image acquisition unit 306 determines whether or not frames can be acquired from the captured image acquisition unit 306 at predetermined intervals. The specified interval can be obtained from the camera unit 206. The specified interval may differ depending on the image resolution of the camera unit 206. For example, if the image generation from the camera unit 206 is 30 FPS, it is determined whether or not the captured image for one frame can be acquired within 1/30 second. Further, even if the captured image for one frame can be acquired within 1/30 second, if the tracking processing unit 313 cannot perform the tracking processing, the deviation becomes large, so that the image cannot be acquired. Similarly, the tracking error is calculated.

<Overall image and captured image>
Next, the whole image and the captured image will be described with reference to FIG. The whole image 500 is an example of the whole image. The whole image 500 is image data showing the whole image of the subject 105. It is assumed that the entire image 500 is stored in the DB unit 308 in advance. It should be noted that the image data including the subject 105 that has been photographed and acquired by the photographed image acquisition unit 306 is acquired by performing a paper surface detection process for excluding an area other than the subject and a distortion correction process for correcting the distorted portion and shaping the image data. It may be added to application 302.

Further, an image of a part (or the whole) of the subject 105 acquired by moving the camera 104 with respect to the subject 105 is referred to as a captured image. The captured images 501, 502, 503, and 504 are examples of captured images. The shooting areas 505, 506, 507, and 508 indicate the shooting areas of the shot image in the entire image. The captured images 501 to 504 are extracted continuous moving image shots acquired from the captured image acquisition unit 306, and show that the shooting areas 505 to 508 move with the movement of the camera 104.

<Data entry area information table>
Next, the data input area information table managed by the DB unit 308 will be described. FIG. 6 is a diagram for explaining the data structure of the data input area information table and the data input area information to be held in the present embodiment. As shown in FIG. 6A, the data input area information table 601 is composed of an id column, a key column, a point column, a width column, and a hight column.

The id column is a value that is incremented by 1 each time a record is added to the data input area information table 601 and is a primary key of the table. The key column stores information indicating what information each record is related to as data input area information. The point column stores the coordinates in the entire image 500 coordinate system corresponding to the position of the upper left corner of the data input area. The width column stores information indicating the width of the data input area in pixels. The height column stores information indicating the height of the data input area in pixels. For example, the display areas 608, 609, 610, 611, 612, and 613 shown on the entire image 500 shown in FIG. 6B are the data input area information 602, 603, and 604 of the data input area information table 601, respectively. , 605, 606, 607 are supported.

<Tracking process>
Next, the tracking process by the tracking process unit 313 will be described with reference to FIG. 7. In FIG. 7, the captured images 700 to 708 are captured images of continuous moving image acquisition acquired from the captured image acquisition unit 306, and are captured by bringing the camera 104 close to the entire image 500. Of the captured images 700 to 708, the captured image 700 is the first captured image acquired from the start of imaging.

The first transformation matrix 709 is obtained by the coordinate conversion processing unit 312 using the feature point comparison result obtained by the feature point comparison processing unit 310 with the entire image 500 and the captured image 700 as inputs. Since the feature point comparison process by the feature point comparison processing unit 310 takes time, captured images 701 and 702 are acquired by the time the first conversion matrix 716 is calculated, but the conversion matrix has not been generated and is not processed. The captured images 701 and 702 are displayed on the mobile terminal screen 400.

It is assumed that the calculation of the first transformation matrix 709 by the coordinate conversion processing unit 312 is completed when the captured image 703 is acquired, and the first transformation matrix 709 is obtained. Therefore, the second transformation matrix 710 is generated. The second transformation matrix 710 uses the captured image 700 used for the calculation of the first transformation matrix 709 and the latest captured image 703 as input images, and uses the feature point tracking processing result obtained by the feature point tracking processing unit 311. Therefore, it is obtained by the coordinate conversion processing unit 312.

By multiplying the first transformation matrix 709 and the second transformation matrix 710, a transformation matrix capable of converting coordinates between the entire image 500 and the captured image 703 is obtained. Then, the tracking processing unit 313 maps and draws each data input area on the captured image 703 based on the data input area information stored in the data input area information table 601 and displays it on the mobile terminal screen 400. ..

Next, using the latest captured image 704 and the previous captured image 703 as input images and the feature point tracking processing result obtained by the feature point tracking processing unit 311, the third transformation matrix 711 is the coordinate conversion processing unit. Obtained by 312. By multiplying the first transformation matrix 709, the second transformation matrix 710, and the third transformation matrix 711, a transformation matrix capable of converting the coordinates between the entire image 500 and the captured image 704 is obtained. Similarly, the third transformation matrix is obtained between the latest captured image and the previous captured image, and the unique first transformation matrix, the unique second transformation matrix, and a plurality of third transformation matrices are multiplied. Then, the transformation matrix for converting the coordinates between the entire image 500 and the latest captured image is obtained.

Here, the accuracy of the transformation matrix obtained by the coordinate conversion processing unit 312 is not 100% due to the influence of the estimation error of the feature point tracking processing by the feature point tracking processing unit 311 and the like, so a plurality of transformation matrices are multiplied. The error accumulates at. Therefore, when the tracking error amount calculation unit 314 determines that the accumulation of errors exceeds the threshold value, the first conversion matrix and the second transformation matrix are updated, and the accumulation of errors is reset.

The first transformation matrix 716 is a transformation matrix obtained by the coordinate conversion processing unit 312 using the feature point comparison result obtained by the feature point comparison processing unit 310 with the entire image 500 and the captured image 704 as inputs. Since it takes time to calculate the transformation matrix as in the case of obtaining the first transformation matrix 709, the captured images 705 and 706 are acquired by the time the first transformation matrix 716 is calculated. At this time, each time each captured image is acquired, the third transformation matrices 712 and 713 are calculated with the previous captured image. Meanwhile, the tracking processing unit 313 uses the generated first transformation matrix 709, second transformation matrix 710, and third transformation matrix 711, 712, 713 to coordinate between the latest captured image and the entire image 500. Find the transformation matrix that transforms.

Then, it is assumed that the calculation of the first transformation matrix 716 by the coordinate conversion processing unit 312 is completed when the captured image 707 is acquired, and the first transformation matrix 716 is obtained. Therefore, the second transformation matrix 717 is generated. The second transformation matrix 717 uses the captured image 704 used for the calculation of the first transformation matrix 716 and the latest captured image 707 as input images, and uses the feature point tracking processing result obtained by the feature point tracking processing unit 311. Therefore, it is obtained by the coordinate conversion processing unit 312.

When the second transformation matrix 717 is obtained, the update of the first transformation matrix and the second transformation matrix is completed. In the subsequent captured images, the tracking processing unit 313 uses the updated first transformation matrix 716 and the second transformation matrix 717, and the third transformation matrix between the captured images to perform the entire image 500 and the latest captured image. Find the transformation matrix that transforms the coordinates between them. This eliminates the need for the third transformation matrix 711 to 714 for tracking the latest captured image, so that the error caused by multiplying these transformation matrices is reset. In this way, by updating the first transformation matrix and the second transformation matrix when the accumulation of errors exceeds a predetermined threshold value, the error during tracking can be kept to a minimum.

Next, the basic tracking process executed by the mobile application 302 on the mobile terminal 100 will be described with reference to FIG. FIG. 8 is a flowchart showing an example of tracking processing in the present embodiment. The processing of the flowchart shown in FIG. 8 is triggered by the user invoking the mobile application 302 in the mobile terminal 100 and approaching the camera 104 with respect to the subject 105 to acquire an image.

In S801, the main control unit 303 transmits the entire image 500 stored in the DB unit 308 to the storage unit 307 so that it can be used.
Next, in S802, the main control unit 303 transmits the data input area information table 601 stored in the DB unit 308 to the storage unit 307 so that it can be used.

In S803, the main control unit 303 orders the captured image acquisition unit 306 to acquire one latest moving image captured image as a captured image.
Next, in S804, the main control unit 303 executes the creation / update processing of the first transformation matrix and the second transformation matrix, which will be described later, and performs the processing related to the creation / update of the first transformation matrix and the second transformation matrix.

In S805, the main control unit 303 determines whether or not tracking is possible. When the creation of the first transformation matrix and the second transformation matrix has been completed, the main control unit 303 determines that tracking is possible (Yes in S805) and transitions to S806. On the other hand, if either the first transformation matrix or the second transformation matrix has not been created, the main control unit 303 determines that tracking is impossible (No in S805) and transitions to S807.

In S806, the main control unit 303 instructs the coordinate conversion processing unit 312 to generate a third conversion matrix between the latest input captured image and the immediately input captured image.
Next, in S807, the main control unit 303 sends the entire image and the latest captured image to the coordinate conversion processing unit 312 by using the generated first conversion matrix, second transformation matrix, and third transformation matrix. Instruct to generate a transformation matrix that allows transformation of coordinates between.

In S808, the main control unit 303 determines whether or not the tracking process is completed within the specified time. When it is determined that the tracking process is completed within the specified time (Yes in S808), the main control unit 303 transitions to S812. On the other hand, when it is determined that the tracking process has not been completed within the specified time (No in S808), the main control unit 303 transitions to S809.

In S809, the main control unit 303 instructs the tracking error amount calculation unit 314 to calculate the tracking error amount from the acquisition result of the latest captured image. Further, the accumulation error is calculated based on the calculation result of the tracking error amount by the tracking error amount calculation unit 314.

In S810, the main control unit 303 acquires the accumulation error from the tracking error amount calculation unit 314, and compares the acquired tracking accumulation error with the error threshold value. The error threshold can be set in the user interface (UI) of the mobile application 302 or can be adaptively determined from the results of the tracking process and the matching process. When the main control unit 303 determines that the tracking accumulation error exceeds the error threshold value (Yes in S810), it transitions to S811, and when it determines that the tracking accumulation error does not exceed the error threshold value. (No of S810) transitions to S812.

In S811, the main control unit 303 sets a flag for creating the first transformation matrix, which will be described later. This first transformation matrix creation flag is used in the creation / update processing of the first transformation matrix and the second transformation matrix in S804.

In S812, the main control unit 303 maps the data input area on the captured image by using the transformation matrix generated in S807 and the information of the data input area information table 601 stored in the storage unit 307, and the mobile terminal. It is displayed on 100 mobile terminal screens 400. The main control unit 303 maps the data input area in the entire image coordinate system stored in the data input area information table 601 on the captured image based on the transformation matrix generated in S807. If the captured image is input and does not go through the tracking process by S807, the captured image without mapping the data input area is displayed as it is on the mobile terminal screen 400.

In S813, the main control unit 303 determines whether or not the input of the captured image by the captured image acquisition unit 306 is completed. When the main control unit 303 determines that the input of the captured image is completed (Yes in S813), the process shown in FIG. 8 is terminated, and when it is determined that the input of the captured image is continued (No in S813). Return to S803 and continue the process.

<Creation / update processing of the first transformation matrix and the second transformation matrix>
Next, the process of creating / updating the first transformation matrix and the second transformation matrix executed in S804 of FIG. 8 will be described with reference to FIG. FIG. 9 is a flowchart showing an example of creation / update processing of the first transformation matrix and the second transformation matrix.

In S901, the main control unit 303 determines whether or not the first transformation matrix has already been generated. When the main control unit 303 determines that the first transformation matrix has been generated (Yes in S901), it transitions to S902, and when it determines that the first transformation matrix has not been generated (No in S901). Transitions to S905.

In S902, the main control unit 303 determines whether or not to update the first transformation matrix. As for the update timing of the first transformation matrix, the first transformation matrix and the second transformation matrix have already been generated, and the main control unit 303 sets a flag in S811 to create the first transformation matrix for the latest captured image. And update. When the main control unit 303 determines that the first transformation matrix is not updated (No in S902), the transition is made to S903, and when it is determined that the first transformation matrix is not updated (Yes in S902), the transition is made to S908.

In S903, the main control unit 303 determines whether or not the second transformation matrix has already been generated. When the main control unit 303 determines that the second transformation matrix has been generated (Yes in S903), the main control unit 303 transitions to S904, and when it determines that the second transformation matrix has not been generated (No in S903). Transitions to S909.

In S904, the main control unit 303 determines whether or not the first conversion matrix has been updated. When the main control unit 303 determines that the first transformation matrix has been updated (Yes in S904), it transitions to S909, and when it determines that the first transformation matrix has not been updated (No in S904). ) Ends the creation / update process of the first transformation matrix and the second transformation matrix.

In S905, the main control unit 303 determines whether or not the feature point comparison processing by the feature point comparison processing unit 310 is being performed. When the main control unit 303 determines that the feature point comparison process is being executed (Yes in S905), the main control unit 303 transitions to S907, and when it is determined that the feature point comparison process is not being executed (No in S905), the main control unit 303 transitions to S907. Transition to S906.

In S906, the main control unit 303 instructs the feature point comparison processing unit 310 to start the feature point comparison process between the entire image and the latest captured image, and the first transformation matrix and the second transformation matrix End the creation / update process.

In S907, the main control unit 303 determines whether or not the feature point comparison process by the feature point comparison processing unit 310 is completed. The main control unit 303 transitions to S908 when it is determined that the feature point comparison process is completed (Yes in S907), and when it is determined that the feature point comparison process is not completed (No in S907), it is the first. The process of creating / updating the 1st transformation matrix and the 2nd transformation matrix is completed.

In S908, the main control unit 303 causes the coordinate conversion processing unit 312 to generate a first conversion matrix between the entire image and the latest captured image by using the feature point comparison processing result by the feature point comparison processing unit 310. Instruct. When the generation of the first transformation matrix is completed, the process proceeds to S901 and the processing is continued.

In S909, the main control unit 303 instructs the coordinate conversion processing unit 312 to obtain the second conversion matrix between the feature point comparison image used for creating the first conversion matrix and the latest captured image. When the generation of the second transformation matrix is completed, the creation / update processing of the first transformation matrix and the second transformation matrix is completed.

<Calculation of tracking error amount>
The error calculation process by the tracking error amount calculation unit 314 will be described with reference to FIG. The process already described in the tracking process described with reference to FIG. 7 will be omitted. The captured images 700 to 708 are captured images of continuous moving image acquisition acquired from the captured image acquisition unit 306, and are captured by bringing the camera 104 close to the entire image 500. Here, unused images 1001, 1002, 1003, 1004 indicate images whose images are unused or unacquired. The unused images 1001 to 1004 are unused or unused because the tracking processing unit 313 cannot perform the tracking process or the captured image acquisition unit 306 cannot acquire the image at a predetermined interval of continuous shooting that can be acquired from the camera unit 206. It is an image of acquisition.

The tracking processing unit 313 performs matching processing on the captured image 703 by using the captured image 700 and the latest captured image 703 as input images and using the feature point tracking processing result obtained by the feature point tracking processing unit 311. The captured image 700 is a captured image used for calculating the first transformation matrix 709 and the second transformation matrix 710. In the matching process, the transformation matrix obtained by multiplying the first transformation matrix 709, the second transformation matrix 710, and the third transformation matrix 711, which enables the transformation of coordinates between the entire image 500 and the captured image 704, is displayed on the image. The coordinates of are specified by the matching process. As a result of performing the matching process, the deviation of the coordinates is minimized, so the tracking error is reset.

Since the unused image 1003 exists, the tracking processing unit 313 determines the unused image 1003, generates a third transformation matrix 1006 between the captured image 704 and the captured image 706, and is a tracking error amount calculation unit. The tracking error amount is calculated from 314. The tracking error amount calculation unit 314 holds the calculated tracking error amount as an accumulation error in the DB unit 308.

The tracking error calculation states 1009 to 1016 indicate the processing states in each image. In the case of the tracking error calculation state 1014, the accumulation error from the tracking error calculation state 1013 is 0, and the occurrence error is 1 because the unused image 1003 generates an unused image for one frame. Further, as a result, it is shown that the accumulation error of the tracking error calculation state 1014 has become 1. Here, since the current threshold value of the accumulation error is 2, the accumulation error does not exceed the threshold value.

Similarly, since the unused image 1004 exists, the tracking processing unit 313 generates the third transformation matrix 1006 and calculates the tracking error amount from the tracking error amount calculation unit 314.

The accuracy of the transformation matrix obtained by the coordinate conversion processing unit 312 due to the generation of the unused images 1003, 1004, etc. accumulates an error by multiplying the transformation matrices when a plurality of unused images are generated. Therefore, when the tracking error amount calculation unit 314 determines that the accumulated error exceeds the threshold value, the accumulated error is reset by performing a matching process for updating the first transformation matrix and the second transformation matrix. ..

In the unused image 1004, the accumulation error is 2 in the tracking error calculation state 1016, and the occurrence error is 1 because the unused image 1004 generates an unused image for one frame, and the tracking error calculation state 1016 is accumulated. It shows that the error is 2. Here, since the current threshold value of the accumulation error is 2 and the accumulation error exceeds the threshold value 2, the main control unit 303 creates the first transformation matrix 1007 and the second transformation matrix 1008 to perform the matching process. Set a flag (S811 in FIG. 8).

According to the present embodiment, when an image is captured in close proximity to a subject such as a paper document, matching processing is performed at an appropriate timing to reduce the calculation load due to the matching processing and to perform tracking with less deviation. be able to.

In the above-described embodiment, when captured images that have not been tracked are continuously generated, it is conceivable that the tracking error becomes large. Therefore, weighting is performed for the tracking error, and when the captured images without tracking are continuously generated, the weight for the tracking error is increased as compared with the case where the captured images without tracking are generated alone. You may try to make it larger.

(Other Embodiments of the present invention)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

100: Portable terminal device (mobile terminal) 301: Data management unit 302: Mobile application 303: Main control unit 304: Information display unit 305: Operation information acquisition unit 306: Captured image acquisition unit 307: Storage unit 308: Database (DB) ) Part 309: Feature point extraction unit 310: Feature point comparison processing unit 311: Feature point tracking processing unit 312: Coordinate conversion processing unit 313: Tracking processing unit 314: Tracking error amount calculation unit

Claims (9)

An extraction means for extracting feature points and feature quantities of the feature points from the entire image of the subject and captured images obtained by continuously photographing the subject.
A comparison processing means that compares the feature amount in the first image extracted by the extraction means with the feature amount in the second image different from the first image and obtains a combination of matching feature points.
A tracking processing means for obtaining a movement vector between images of feature points extracted by the extraction means, and
A tracking processing means that performs tracking processing related to a predetermined region of an image by using the comparison processing means or a transformation matrix obtained based on the processing result by the tracking processing means.
An error calculation means for calculating and accumulating a tracking error that occurs when an image that is not tracked by the tracking processing means is generated.
An information processing apparatus including a control means for executing a process by the comparison processing means and resetting the error accumulated by the error calculating means when the error accumulated by the error calculating means exceeds a threshold value. .. The information processing apparatus according to claim 1, wherein the first image is the whole image, and the second image is the captured image. It has a transformation processing means for obtaining the transformation matrix used for the tracking processing, and has
The conversion processing means generates a first transformation matrix based on the first image and the second image each time the processing by the comparison processing means is executed, and generates the first transformation matrix. A second transformation matrix is generated based on the latest photographed image and the second image used for generating the first transformation matrix, and the first transformation matrix and the second transformation matrix are used. The information processing apparatus according to claim 1 or 2, wherein the transformation matrix used for the tracking process is obtained. The error calculation means according to claim 1 to 3, wherein the error calculation means calculates a tracking error when the captured image cannot be acquired at a predetermined interval or when the tracking processing means cannot perform the tracking process. The information processing apparatus according to any one item. The claim is characterized in that the error calculating means increases the weight for a tracking error that occurs when the captured image cannot be continuously acquired or when the tracking processing means cannot continuously perform the tracking process. Item 2. The information processing apparatus according to any one of Items 1 to 4. It has a holding means for holding information indicating the position of the predetermined region in the whole image.
The tracking processing means according to any one of claims 1 to 5, wherein the tracking processing means tracks the predetermined region in the captured image based on the transformation matrix and the information held by the holding means. Information processing device. The information processing apparatus according to any one of claims 1 to 6, wherein the predetermined area is a data input area including data information to be extracted in the whole image. It is an information processing method of an information processing device.
An extraction step of extracting feature points and feature quantities of the feature points from the entire image of the subject and the captured images of the subject continuously photographed.
A comparison processing step of comparing the feature amount in the first image extracted in the extraction step with the feature amount in the second image different from the first image and obtaining a combination of matching feature points. ,
A tracking process for obtaining a movement vector between images of feature points extracted in the extraction step, and a tracking process for obtaining a movement vector between images.
A tracking processing step of performing tracking processing related to a predetermined region of an image by using a transformation matrix obtained based on the processing result of the comparison processing step or the tracking processing step.
An error calculation process that calculates and accumulates the tracking error that occurs when an image that is not subjected to the tracking process occurs, and
Information characterized by having a control step of executing the process of the comparison processing step and resetting the error accumulated in the error calculation step when the error accumulated in the error calculation step exceeds the threshold value. Processing method. To the computer of the information processing device
An extraction step of extracting feature points and feature quantities of the feature points from the entire image of the subject and the captured images of the subject continuously captured.
A comparison processing step of comparing the feature amount in the first image extracted in the extraction step with the feature amount in the second image different from the first image and obtaining a combination of matching feature points. ,
A tracking process step for obtaining a movement vector between images of feature points extracted in the extraction step, and a tracking processing step.
A tracking processing step that performs tracking processing related to a predetermined region of an image by using a transformation matrix obtained based on the comparison processing step or the processing result of the tracking processing step.
An error calculation step that calculates and accumulates the tracking error that occurs when an image that is not subjected to the tracking process occurs, and
A program for executing the processing of the comparison processing step and the control step of resetting the error accumulated in the error calculation step when the error accumulated in the error calculation step exceeds the threshold value.

Images