JP2016025625A - Information processor, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To identify a subject of a photographed image accurately.SOLUTION: An information processor has: region extraction means extracting a plurality of content regions from a plurality of photographed images where a part of one paper medium is photographed; feature amount calculating means calculating feature amounts of the content regions; position identification means which identifies positional relations among the plurality of content regions; and electronic document identification means which refers to storage means that stores the feature amounts of the plurality of contents included in one electronic document in association with the electronic document corresponding to one paper medium and further stores positional information showing positional relations among the plurality of contents included in one electronic document and identifies an electronic document corresponding to the plurality of photographed images on the basis of the respective feature amounts of the respective plurality of content regions calculated by the feature amount calculating means and the positional relations among the plurality of content regions identified by the position identification means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

2. Description of the Related Art Conventionally, there has been known a technique for searching for information related to a subject photographed by a camera using a tablet terminal or a smartphone (hereinafter also referred to as a smart terminal) and displaying the searched related information on the screen of the smart terminal. Yes. For example, in augmented reality technology called Augmented Reality (hereinafter referred to as AR), information related to a subject photographed by a camera can be retrieved, and the retrieval result can be superimposed on a real-world object photographed by the camera. Furthermore, a service has been developed in which the subject in the technology is a paper document, and when the paper document is photographed, related information is superimposed and displayed on the paper document. Although content such as a moving image cannot be expressed on a paper medium, it is possible to search for a moving image as related information of a still image printed on a paper document and display the obtained moving image in a superimposed manner.
Further, as a technique for specifying a subject photographed by a camera, Patent Document 1 discloses a technique for reading a printed matter composed of a plurality of pages in units of pages and searching an original electronic document from the read information. ing.

JP 2004-349940 A

As described above, when searching for information related to a subject, it is necessary to accurately recognize the subject. For example, when a plurality of subject candidates are detected because the subject recognition accuracy is low, it is not possible to uniquely identify related information for the subject corresponding to this.
For example, if the subject to be photographed by a smart terminal is a large paper such as a street poster or a specification with repeated fine corrections, it may be difficult to recognize the subject, and appropriate related information may not be identified. is there. In this way, when related information cannot be uniquely identified, it may be possible to present a plurality of related information to the user. In this case, however, the user needs to select necessary related information, and the operation becomes complicated. was there.

  The present invention has been made in view of such problems, and an object thereof is to accurately specify a subject of a captured image.

  Therefore, the present invention is an information processing system, and includes an area extraction unit that extracts a plurality of content areas from a plurality of captured images obtained by capturing a part of one paper medium, and a feature amount of the content area. A feature amount calculating means for calculating, a position specifying means for specifying a positional relationship between the plurality of content areas, and a plurality of contents included in one electronic document in association with the electronic document corresponding to one paper medium And a storage unit that stores positional information indicating a positional relationship between a plurality of contents included in one electronic document, and stores each of the plurality of content areas calculated by the feature amount calculation unit. An electronic document feature that specifies an electronic document corresponding to the plurality of captured images based on the feature amount and the positional relationship between the plurality of content areas specified by the position specifying means. And having a means.

  According to the present invention, the subject of the captured image can be specified with high accuracy.

It is a figure which shows the whole structure of AR processing system. It is a figure which shows an example of a paper document. It is a figure which shows the hardware constitutions of a portable terminal. It is a figure which shows the software structure of a portable terminal. It is a figure which shows the software structure of a management server apparatus. It is a figure which shows an example of a data structure of a feature-value table. It is a sequence diagram which shows AR processing. It is a flowchart for an image analysis process. It is a figure which shows an example of the flame | frame of a process target. It is a figure which shows an example of the data structure of a picked-up image table. It is explanatory drawing of the process in which information is recorded on a picked-up image table. It is a flowchart which shows a feature-value extraction process. It is explanatory drawing of a reduced image generation process. It is a flowchart which shows an electronic document search process. It is a flowchart which shows a feature amount collation process. It is a flowchart which shows a support information transmission process. It is explanatory drawing of a support information transmission process. It is a figure which shows the example of a display of assistance information.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a diagram showing an overall configuration of an AR (Augmented Reality) processing system as an information processing system. The AR processing system includes a mobile terminal 100 and a management server device 110 as an information processing device. The mobile terminal 100 and the management server device 110 can communicate via the network 120. In the AR processing system, the mobile terminal 100 captures a paper document as a paper medium, displays the captured image on the display device 101, and transmits the captured image to the management server device 110. Here, it is assumed that the captured image is a moving image. When the management server device 110 receives the captured image, the management server device 110 identifies the electronic document associated with the captured image as related information and transmits it to the mobile terminal 100. The mobile terminal 100 displays the captured image on the display screen while capturing the captured image, and displays the related image superimposed on the captured image when the related information is received.
FIG. 2 is a diagram illustrating an example of a paper document 200 to be photographed by the mobile terminal 100. In the present embodiment, it is assumed that the paper document 200 to be photographed includes a plurality of contents as shown in FIG. In a paper document 200 shown in FIG. 2, character string content 201, photo content 202, graphic content 203, and character string content 204 are displayed. Thus, one paper document includes a plurality of contents, and the types of the contents may be the same or different.
In the present embodiment, the paper document 200 to be photographed is assumed to have a relatively large size, and therefore, each frame in the photographed image includes only a part of the paper document 200. To do. That is, each frame in the captured image includes only a part of the content included in the paper document 200.

FIG. 3 is a diagram illustrating a hardware configuration of the mobile terminal 100. The imaging unit 300 captures an image. The acceleration sensor 301 detects a change in the position or tilt of the mobile terminal 100. A CPU (Central Processing Unit) 302 performs calculations and logic determinations for various processes, and controls various elements connected to the bus 310. Note that the mobile terminal 100 may include an orientation sensor instead of the acceleration sensor 301.
The mobile terminal 100 is equipped with a memory including a program memory and a data memory. The program memory stores a program for control by the CPU 302 including various processing procedures to be described later with reference to flowcharts. The memory is realized by a ROM (Read-Only Memory) 303, a RAM (Random Access Memory) 304 loaded with a program from an external storage device or the like, or a combination thereof.

The storage device 305 is a device such as a hard disk for storing data and programs. The storage device 305 is also used for holding, for example, photographs and documents. Note that a data holding device (not shown) connected to an external connection or a network may be used as a device for holding photographs, documents, and the like. Here, the external storage device 306 is realized by, for example, a medium (recording medium) and an external storage drive for realizing access to the medium. Examples of such media include a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, and a flash memory.
Note that the functions and processes of the portable terminal 100 described later are realized by the CPU 302 reading a program stored in the ROM 303 or the like and executing the program.

The input device 307 is a device for inputting an instruction from the user. The user inputs an instruction for the AR processing system via the input device 307. The input device 307 can be realized by a keyboard or a pointing device, for example.
The display device 101 is a device that displays and outputs processing results and the like. The display device 101 can be realized by a display device such as a CRT (Cathode-Ray Tube) including a circuit for output and a liquid crystal display, for example. In the present embodiment, the display device 101 displays photos, documents, and the like.
An interface (hereinafter referred to as I / F) 309 is a device that mediates information, and the mobile terminal 100 exchanges data with an external device via the I / F 309.
The hardware configuration of the management server device 110 is the same as the hardware configuration of the mobile terminal 100 described with reference to FIG. However, the management server device 110 may not include the imaging unit 300, the acceleration sensor 301, and the like.

FIG. 4 is a diagram illustrating a software configuration of the mobile terminal 100. The communication unit 400 transmits / receives information to / from an external device via the network 120. For example, the communication unit 400 transmits a captured image to the management server device 110 and receives related information related to the captured image from the management server device 110. The accepting unit 401 accepts an instruction input by the user. The shooting control unit 402 performs shooting according to an instruction from the user to obtain a moving image.
The position related information generation unit 403 generates position related information of each frame constituting the moving image. In the present embodiment, the position-related information includes a detection result by the acceleration sensor 301 at the time when each frame is photographed and information indicating the zoom state of the photographing unit 300. The captured image and the position information are transmitted to the management server device 110 via the communication unit 400. The display unit 404 displays a moving image as a captured image obtained by the imaging control unit 402 on the display device 101. Furthermore, when the communication unit 400 receives related information from the management server device 110, the display unit 404 displays the related information superimposed on the captured image.

FIG. 5 is a diagram illustrating a software configuration of the management server device 110. The communication unit 500 transmits / receives information to / from an external device via the network 120. For example, the communication unit 500 receives a captured image from the mobile terminal 100 and transmits related information to the mobile terminal 100.
The analysis unit 501 analyzes the captured image and extracts the feature amount of each frame constituting the moving image as the captured image. Note that an existing document analysis technique can be used for the analysis processing. The analysis process will be described later.
The electronic document DB 502 stores electronic documents. Here, the electronic document is, for example, electronic data corresponding to the paper document 200 shown in FIG. The electronic document DB 502 stores an electronic document corresponding to a paper document to be photographed. When there are a plurality of paper documents to be photographed, the electronic document DB 502 stores an electronic document corresponding to each of the plurality of paper documents. Note that the electronic document DB 502 is stored in the external storage device 306 or the like.

The electronic document DB 502 further stores a feature amount table. FIG. 6A is a diagram illustrating an example of a data configuration of the feature amount table 600. The feature amount table 600 stores a content ID, an attribute, a feature amount, position information, and related information in association with the document ID. Note that the electronic document with the document ID “001” shown in FIG. 6A corresponds to the paper document 200.
The document ID is identification information of the electronic document. The content ID is identification information of each content included in the electronic document. The content is information corresponding to the content 201 to 204 shown in FIG. The attribute indicates the type of content. In FIG. 6A, the attribute “1” means a character string, “2” means a photograph, and “3” means a figure type. The position information includes the position (X, Y) of each content in the document, the content width W, and the content height H.
The related information is information set by an administrator or the like as information related to the content. In the present embodiment, the related information is an electronic document, but the type of related information is not limited to this. As another example, the related information may be a UI screen for the associated paper document. Further, it is not necessary that the related information is associated with all the contents registered in the feature amount table.

Note that the position information only needs to be information indicating the positional relationship between a plurality of contents included in the same document, and the specific information content is not limited to the embodiment. FIG. 6B is a diagram illustrating another example of the feature amount table 610. In the example shown in FIG. 6B, the position information is information indicating content adjacent to the target content. That is, the position information is information indicating the content ID of content that is adjacent to each content in each of the vertical and horizontal directions.
In addition, when there is only one document for which related information is presented, the related information table may not record the document ID.

  Returning to FIG. 5, the document search unit 503 refers to the electronic document DB 502 and performs a search process of the electronic document corresponding to the photographed image based on the content feature amount obtained by the analysis unit 501. The related information extraction unit 504 refers to the electronic document DB 502 and extracts related information corresponding to the content displayed on the display unit 404 of the mobile terminal 100 at the time of processing from among the specified electronic documents. The extracted related information is transmitted to the mobile terminal 100 via the communication unit 500.

FIG. 7 is a sequence diagram showing AR processing by the AR processing system. In step S700, the shooting control unit 402 of the mobile terminal 100 starts shooting a paper document in accordance with an instruction from the user. Note that the shooting control unit 402 continues shooting until a start instruction is input from the user. In step S <b> 701, the display unit 404 starts displaying a moving image as a captured image captured by the capturing control unit 402 on the display device 101. Note that shooting and display of the shot image continue until an end instruction is input from the user.
Further, in S702, the position related information generation unit 403 generates position related information for each frame included in the captured image. In step S <b> 703, the communication unit 400 transmits the captured image and the position related information to the management server device 110.

On the other hand, in the management server device 110, when the communication unit 500 receives the captured image and the position information in S703, the analysis unit 501 performs analysis processing on the captured image and extracts the feature amount of each frame in S704. . Note that the image analysis processing in S703 will be described later with reference to FIG. In step S <b> 705, the document search unit 503 searches the electronic document corresponding to the paper document as the subject of the photographed image with reference to the feature amount table 600 based on the feature amount extracted in step S <b> 704. The document search unit 503 further specifies the angle (rotation angle) of the captured image with respect to the electronic document.
In step S <b> 706, the document search unit 503 confirms whether an electronic document corresponding to the paper document has been identified. If the electronic document can be identified (S706: Yes), the document search unit 503 can identify the electronic document. The process proceeds to S707. If the electronic document cannot be identified (No in S706), the document search unit 503 again waits for reception of a subsequent frame in S703.
In step S <b> 707, the related information extraction unit 504 refers to the feature amount table 600 and extracts related information associated with the content displayed in the last frame from the captured image (related information extraction processing). Further, the related information extraction unit 504 identifies the position where the related information is to be displayed based on the position of the content area (referred to as the target content area) where the content corresponding to the related information is displayed in the last frame. To do. Next, in S708, the communication unit 500 transmits the related information, the display position of the related information, and the rotation angle of the target content area specified in S705 to the mobile terminal 100.

In portable terminal 100, when communication unit 400 receives related information or the like in S708, display unit 404 displays data to be superimposed based on the related information, the display position of the related information, and the rotation angle in S709. Is generated. The display unit 404 superimposes the generated data on the captured image being displayed (superimposition display process).
Note that paper images may be taken at various angles. Therefore, in the present embodiment, the mobile terminal 100 uses the rotation angle to match the vertical direction of the paper document in the captured image being displayed on the display device 101 with the vertical direction of the related information to be superimposed. . Thereby, the portable terminal 100 can display related information in a state in which the related information is easier to see. Through the above processing, the user can browse related information related to the photographed paper document on the portable terminal 100.

FIG. 8 is a flowchart showing detailed processing in the image analysis processing (S704) described with reference to FIG. In S800, the analysis unit 501 recognizes meaningful blocks (regions) from each frame of the captured image as a block, and determines the attribute of the content displayed in each block.
FIG. 9 is a diagram illustrating an example of a frame to be processed. The analysis unit 501 extracts three content areas 901 to 903 from the frame 900 shown in FIG. 9A (area extraction processing). Here, the content area 901 is an area for displaying the content 201. The content area 902 is an area for displaying the content 202. The content area 903 is an area for displaying the content 203. The analysis unit 501 further specifies the attribute of each content area. For the content areas 901 to 903 shown in FIG. 9, the attributes of text, photograph, and figure are specified, respectively.
Also, from the frame 910 shown in FIG. 9B, the analysis unit 501 extracts only one content area 911 and identifies the attribute of the content area 911 as text.

Hereinafter, the process of extracting the content area and specifying the attribute will be described more specifically. Upon receiving the frame, the analysis unit 501 binarizes the frame into a black and white image. Then, the analysis unit 501 performs contour tracking and extracts a block of pixels surrounded by a black pixel contour. When black pixels having a certain area or more are extracted, the analysis unit 501 performs contour tracking for white pixels inside to extract a block of white pixels. The analysis unit 501 further extracts a black pixel block if the extracted white pixel block is larger than a certain area. The analysis unit 501 performs this extraction process recursively if the extracted cluster is larger than a certain area.
Then, the analysis unit 501 classifies the block of black pixels obtained by the extraction process as blocks having various attributes depending on the size and shape. For example, the analysis unit 501 determines a block having an aspect ratio close to 1 as a block of pixels corresponding to a character, and sets a block of pixels corresponding to a character as a text block when the blocks of pixels corresponding to adjacent characters are aligned. Further, the analysis unit 501 classifies a block of irregular pixels into a photographic block, and classifies a block other than that as a graphic block.

In S800, the analysis unit 501 further specifies the positional relationship between the frames based on the position related information of each of the other frames already processed in the captured image and the frame to be processed. And the analysis part 501 specifies the positional relationship between the some content area | region obtained from the some flame | frame based on the positional relationship between frames (position specifying process).
When a plurality of content areas are included in one frame as in a frame 900 shown in FIG. 9, the positional relationship between the plurality of content areas in the frame is specified. In the frame 900 shown in FIG. 9, the positional relationship that the content area 902 exists to the right of the content area 901 and the content area 903 exists below the content area 902 is specified.

The analysis unit 501 further assigns an area ID to the content area obtained as a block. Here, the area ID is identification information of the content area. The analysis unit 501 records the area ID, the attribute specified for the area, and position information indicating the positional relationship in the captured image table. Here, it is assumed that the captured image table is stored in the RAM 304 or the external storage device 306, for example.
FIG. 10 is a diagram illustrating an example of a data configuration of the photographed image table 1000. The captured image table 1000 stores a region ID, an attribute, position information, and a feature amount in association with each other. Each row shown in FIG. 10 corresponds to one record. In S800, the analysis unit 501 records the area ID, the attribute, and the position information in the captured image table 1000 as one record in association with each other. Note that the feature amount is additionally recorded in association with the region ID in S802 to be described later.

  Returning to FIG. 8, after the process of S800, in S801, the analysis unit 501 calculates the feature amount of each content region obtained in S800 (feature amount calculation process). In step S <b> 802, the analysis unit 501 records the feature amount in the captured image table 1000 in association with the content area.

Here, a process for recording information in the captured image table 1000 will be specifically described. For example, as illustrated in FIG. 11A, it is assumed that photographing of a paper document 200 is started on the mobile terminal 100 and the image of the content 201 is displayed on the display device 101 as a photographed image.
Thereafter, when the user moves the photographing direction to the right of the paper document 200 (the direction of the arrow A) by changing the tilt of the mobile terminal 100, the display device 101 has a display as shown in FIG. An image of the content 202 is displayed as a captured image. Further, when the shooting direction moves downward (in the direction of arrow B), the image of the content 203 is displayed on the display device 101 as a shot image, as shown in FIG.

When a captured image with a change in shooting direction as described above is to be processed, a frame 1111 shown in FIG. 11A is first input to the analysis unit 501, and an area analysis process (S 704) for the frame 1111 is executed. . In the area analysis process (S800), the analysis unit 501 extracts the image area of the content 201 as the content area 1101, and issues a content ID “a001” to the content area 1101. The analysis unit 501 further identifies the attribute of the content area 1101 as text.
At this time, the analysis of other frames has not been performed, and the positional relationship with other content areas is not known only from the frame 1111. Therefore, at this time, the analysis unit 501 records the region ID “a001” in the captured image table 1000 and records the attribute “1” in association with the region ID “a001”. Thereafter, the analysis unit 501 calculates the feature amount of the content area 1101 in S801, and records it in association with the area ID “a001” in S802.

Subsequently, when the frame 1112 is input to the analysis unit 501, an image analysis process (S 704) is performed on the frame 1112. That is, in the region analysis process (S800), the analysis unit 501 extracts the image region of the content 202 as the content region 1102, and issues a content ID “a002” to the content region 1101. The analysis unit 501 further identifies the attribute of the content area 1102 as a photograph.
Further, the analysis unit 501 specifies that the shooting direction has moved to the right of the paper document 200 between the processed frame 1111 and the frame 1112 from the position related information and the pixel change in each of the frames 1111 and 1111. . Then, the analysis unit 501 specifies that the content area 1101 exists on the left side of the content area 1102 from the change in the shooting direction.

Then, the analysis unit 501 records the region ID “a002” in the captured image table 1000, records the attribute “2” in association with the region ID “a002”, and further identifies the content region identified by “a001” in the left direction. Position information indicating that 1101 exists is recorded. At this time, the analysis unit 501 also records position information indicating that the content area 1102 identified by “a002” exists in the right direction in association with the area ID “a001”. Thereafter, in S801, the feature amount of the content area 1102 is calculated, and in S802, this is recorded in association with the area ID “a002”.
Further, a content area 1103 is registered in the captured image table 1000 shown in FIG. From the position information of the captured image table 1000 shown in FIG. 10, it can be seen that the content area 1102 with the area ID “a002” exists to the left of the content area 1101 with the area ID “a001”. Furthermore, it can be seen from this position information that a content area 1103 with area ID “a003” exists below the content area 1102 with area ID “a002”.

If the area ID associated with the feature amount extracted in S802 is already registered in the captured image table 1000, no new content area is registered in the captured image table 1000. Note that the case where the area ID corresponding to the extracted feature amount is already registered means that the same content is displayed in another processed frame, and the content area has already been displayed in the processing of this other frame. This is a case where it is registered in the photographed image table 1000.
When a plurality of content areas are included in one frame, the analysis unit 501 identifies the positional relationship between the content areas and registers this in the captured image table 1000 as position information.

FIG. 12 is a flowchart showing detailed processing in the feature amount extraction processing (S801) described with reference to FIG. The feature amount extraction processing is processing for extracting a local feature amount (local feature amount) of an image. A local feature has a property of rotation invariance and enlargement / reduction invariance. With this feature, even when the image is rotated, enlarged or reduced, a search process using the feature amount is possible. The local feature quantity extraction processing can be performed using an existing local feature quantity extraction processing technique.
In step S1200, the analysis unit 501 reads a processing target frame. Next, in S1201, the analysis unit 501 extracts a luminance component from the input frame and creates a luminance component image. Next, in S1202, the analysis unit 501 generates a reduced image from the luminance component image. Specifically, the analysis unit 501 sequentially reduces the latitude component image according to the magnification p, and generates n reduced images. Here, it is assumed that the magnification p and the number n of reduced images are defined in advance and stored in the ROM 303 or the like.

FIG. 13 is a diagram for explaining the reduced image generation processing (S1202). FIG. 13 shows an example where the magnification p is set to 2 to the power of-(1/4) and the number n of reduced images is set to 9. However, in this example, the magnification p is not the area ratio but the side length ratio.
In FIG. 13, reference numeral 1301 denotes the luminance component image created in S1201. Reference numeral 1302 denotes a reduced image obtained by reducing the luminance component image 1001 four times according to the magnification p, and corresponds to an image obtained by reducing the luminance component image 1301 to ½. Reference numeral 1303 denotes a reduced image obtained by reducing the luminance component image 1301 eight times in accordance with the magnification p, and corresponds to an image obtained by reducing the luminance component image 1301 to ¼. Note that the scale number 1310 is a number assigned in descending order of the size of the reduced image.
Other examples of the method of reducing the image include a method of simply thinning out pixels, a method of using linear interpolation, a method of sampling after applying a low-pass filter, and the like.

Returning to FIG. 12, after S1202, in S1203, the analysis unit 501 extracts local feature points (local feature points) from each of the n reduced images obtained in S1202. The local feature points extracted here are robust local feature points that are stably extracted from the same place even if image processing such as rotation or reduction is performed on the image. As a method for extracting such local feature points, a Harris operator or the like can be cited.
Specifically, the analysis unit 501 examines the pixel values of the pixel of interest and the pixels in the vicinity of the periphery 8 (total of 9 pixels) for each pixel of the image obtained by applying the Harris operator. Then, the analysis unit 501 extracts a point where the pixel of interest is located as a local feature point when the pixel value of the pixel of interest is equal to or greater than the threshold value and has a local maximum (the pixel value of nine pixels is maximum). . In addition, the process which the analysis part 501 extracts a robust local feature point is not limited to embodiment.

Next, in S1204, the analysis unit 501 calculates, for each local feature point obtained in S1203, a feature quantity (local feature quantity) that is defined so as to remain unchanged even when the image is rotated. The analysis unit 501 uses a local jet of the following literature and a combination of derivatives thereof as a method for calculating the local feature amount.
J. et al. J. et al. Koenderink and A.K. J. et al. van Doorn, "Representation of local geometry in the visual system," Riologic Cybernetics, vol. 55, pp. 367-375, 1987

ただし、式（１）の右辺で用いている記号は、以下に示す式（２）〜式（７）で定義される。ここで、式（２）右辺のＧ（ｘ，ｙ）はガウス関数、Ｉ（ｘ，ｙ）は画像の座標（ｘ，ｙ）における画素値であり、"＊"は畳み込み演算を表す記号である。また、式（３）は式（２）で定義された変数Ｌのｘに関する偏導関数、式（４）は当該変数Ｌのｙに関する偏導関数である。式（５）は式（３）で定義された変数Ｌｘのｙに関する偏導関数、式（６）は式（３）で定義された変数Ｌｘのｘに関する偏導関数、式（７）は式（４）で定義されたＬｙのｙに関する偏導関数である。

なお、局所特徴量の算出方法は、上述の方法に限定されるものではなく、他の局所特徴量の算出方法も適用可能である。 The local feature amount calculated by this method can have characteristics that have a relatively high resistance to scaling and rotation. Specifically, the analysis unit 501 calculates a local feature amount v shown in Expression (1).

However, the symbols used on the right side of the equation (1) are defined by the following equations (2) to (7). Here, G (x, y) on the right side of Expression (2) is a Gaussian function, I (x, y) is a pixel value at image coordinates (x, y), and “*” is a symbol representing a convolution operation. is there. Equation (3) is a partial derivative of variable L defined by equation (2) with respect to x, and equation (4) is a partial derivative of variable L with respect to y. Equation (5) is the partial derivative of variable Lx defined in equation (3) with respect to y, equation (6) is the partial derivative of variable Lx defined in equation (3) with respect to x, and equation (7) is the equation. It is a partial derivative with respect to y of Ly defined in (4).

Note that the local feature amount calculation method is not limited to the above-described method, and other local feature amount calculation methods are also applicable.

FIG. 14 is a flowchart showing detailed processing in the electronic document search processing (S705) described with reference to FIG. In the electronic document search process, as described above, the electronic document corresponding to the captured image is searched based on the feature amount, and the rotation angle of the related information with respect to the captured image is specified. In step S <b> 1400, the document search unit 503 selects one unprocessed content area that has not been subjected to a feature amount matching process (S <b> 1401) described below from the content areas registered in the captured image table 1000.
In order to specify an unprocessed content area, a flag indicating whether or not the process has been performed may be stored in association with each area ID of the captured image table 1000. Further, as a process of sequentially selecting unprocessed content areas, a search may be performed in order from the top of the captured image table 1000, or as another example, a search may be performed randomly.

In step S <b> 1401, the document search unit 503 collates the feature amount of the content area selected in step S <b> 1400 with the feature amount of the electronic document content registered in the feature amount table 600. Furthermore, when it is determined that the two match, the rotation angle of the related information to be displayed superimposed on the content area is calculated using the value calculated at the determination home. The feature amount matching process will be described in detail later with reference to FIG.
In step S <b> 1402, the document search unit 503 confirms whether or not the processing in step S <b> 1401 has been completed for all content areas that are registered in the captured image table 1000. If there is an unprocessed content area (Yes in S1402), the document search unit 503 advances the process to S1400, selects an unprocessed content area, and continues the process. If there is no unprocessed content area (No in S1402), the document search unit 503 advances the process to S1403.
Note that the processing of S1400 to S1402 is an example of content specifying processing for specifying content corresponding to each content area in the feature amount table 600.

In step S1403, the document search unit 503 specifies an electronic document including content corresponding to all content areas registered in the captured image table 1000, which is obtained in the repetition processing in steps S1400 and S1401. Then, the document search unit 503 sets this as a candidate for the electronic document corresponding to the captured image.
In step S <b> 1404, the document search unit 503 determines whether the positional relationship between all the content areas registered in the captured image table 1000 satisfies the positional relationship indicated in the positional information registered in the feature amount table 600. Judge whether or not. If the position relationship indicated by the position information is not satisfied (No in S1404), the document search unit 503 advances the process to S1405. In step S <b> 1405, the document search unit 503 determines that the electronic document candidate is not the electronic document corresponding to the photographed image, and the electronic document cannot be specified, and ends the electronic document search process.

On the other hand, when the positional relationship indicated by the positional information is satisfied (Yes in S1404), the document search unit 503 advances the process to S1406. In step S1406, the document search unit 503 determines whether there are a plurality of electronic document candidates that satisfy the positional relationship indicated by the positional information. If there are a plurality of documents (Yes in S1406), the document search unit 503 advances the process to S1405, determines that the electronic document cannot be specified, and ends the electronic document search process.
On the other hand, when there is only one electronic document candidate that satisfies the positional relationship indicated by the positional information (No in S1406), the document search unit 503 advances the process to S1407. In step S1407, the document search unit 503 specifies the obtained electronic document candidate as an electronic document corresponding to the captured image (electronic document specifying process).
Note that the document search unit 503 may specify an electronic document corresponding to the captured image based on the extracted feature amount of the content area and the positional relationship between the content areas, and specific processing for that is performed. The form is not limited.

FIG. 15 is a flowchart showing detailed processing in the feature amount matching processing (S1401) shown in FIG. In the feature amount matching process, the similarity between the image displayed in the content area obtained in the captured image and the content of the electronic document is calculated, and the content corresponding to the content area is specified based on the calculated similarity. . Specifically, the document search unit 503 specifies content that is equal to or greater than the threshold and has the maximum similarity as content corresponding to the image in the content area.
Prior to the description of the feature amount matching process, symbols used in the feature amount matching process will be described. The local feature point extracted from the captured image is Q, the coordinate is Q (x ′, y ′), and the local feature amount of the local feature point is Vq. Further, a local feature point on one content of the electronic document to be collated is S, a coordinate is S (x, y), and a local feature amount of the local feature point is Vs.

In step S1500, the document search unit 503 calculates the distance between the local feature amounts of Vq and Vs for all combinations, and creates a minimum distance corresponding point list. Specifically, the document search unit 503 first calculates the distance between feature amounts for all combinations of Vq and Vs. Next, the document search unit 503 extracts a combination (corresponding point) of Vq and Vs such that the calculated distance between the feature amounts is equal to or less than the threshold value Tv and the minimum distance. Then, the document search unit 503 creates the minimum distance corresponding point list by registering the extracted corresponding points in the list. Here, the k-th minimum distance corresponding point is represented as Qk and Sk, respectively, and these coordinates are described as Qk (x ′ _k , y ′ _k ), Sk (x _k , y _k ), etc., with subscripts. I decided to. The local feature values of Qk and Sk are described as Vq (k) and Vs (k), respectively.
Note that there may be two or more local feature quantities associated with one local feature point, but here, for simplicity, a case where only one local feature quantity is associated with one local feature point will be described. To do. Also, let m be the number of pairs of corresponding points registered in the minimum distance corresponding point list.

After the processing of S1500, in S1501, the document search unit 503 checks whether m is 3 or more. If m is less than 3 (No in S1501), the document search unit 503 determines that the similarity cannot be calculated and ends the feature amount matching process. If m is 3 or more (Yes in S1501), the document search unit 503 advances the process to S1503. In step S1503, the document search unit 503 initializes a variable VoteMax indicating the final number of votes to 0. In step S <b> 1504, the document search unit 503 initializes a variable “Count” indicating the number of repetition counts of the similarity calculation process to 0.
In step S <b> 1505, the document search unit 503 compares the iteration count number Count with a predetermined maximum iteration number Rn. If the iteration count Count is equal to or greater than the maximum iteration count Rn (No in S1505), the document search unit 503 advances the process to S1506. In step S1506, the document search unit 503 outputs the final vote number VoteMax and the rotation angle, and ends the feature amount matching process.

文書検索部５０３は、具体的には、式（８）中の変数ａ，ｂ，ｅ，ｆを求める。文書検索部５０３は、Ｑ１（ｘ'₁，ｙ'₁）とＳ１（ｘ₁，ｙ₁）、Ｑ２（ｘ'₂，ｙ'₂）とＳ２（ｘ₂，ｙ₂）を用いて、式（９）〜式（１２）により、変数ａ，ｂ，ｅ，ｆを求める。

If the iteration count number Count is less than the maximum iteration count Rn (Yes in S1505), the document search unit 503 advances the process to S1507. In step S1507, the document search unit 503 initializes a variable Vote representing the number of votes to 0. In step S <b> 1508, the document search unit 503 randomly extracts two sets of coordinates of corresponding points from the minimum distance corresponding point list. Here, the extracted two sets of coordinates are Q1 (x ′ ₁ , y ′ ₁ ), S1 (x ₁ , y ₁ ), Q2 (x ′ ₂ , y ′ ₂ ), and S2 (x ₂ , y ₂ ). To do.
In step S1509, the document search unit 503 calculates a conversion matrix M. Here, the transformation matrix M is represented by Expression (8). In the present embodiment, only similarity conversion is considered for the sake of simplicity.

Specifically, the document search unit 503 obtains variables a, b, e, and f in Expression (8). The document search unit 503 uses Q1 (x ′ ₁ , y ′ ₁ ), S1 (x ₁ , y ₁ ), Q2 (x ′ ₂ , y ′ ₂ ), and S2 (x ₂ , y ₂ ) to The variables a, b, e, and f are obtained from (9) to (12).

In step S1510, the document search unit 503 initializes the corresponding point selection variable k to 3 in order to select a point other than the two sets of points selected in step S1508. In step S <b> 1511, the document search unit 503 checks whether the corresponding point selection variable k is equal to or less than the number m of pairs of corresponding points registered in the shortest distance corresponding point list. If k is equal to or smaller than m (Yes in S1511), the document search unit 503 advances the process to S1512. If k is larger than m (No in S1511), the document search unit 503 advances the process to S1517.
In step S1512, the document search unit 503 extracts one new set of corresponding points from the minimum distance corresponding point list. The extracted coordinates are represented by Sk (x _k , y _k ) and Qk (x ′ _k , y ′ _k ). In step S1513, the document search unit 503 obtains the coordinates Sk (x ′ _k , y ′ _k ) to which the coordinates Sk (x _k , y _k ) extracted in step S1512 are transferred according to the equation (8).

In step S1514, the document search unit 503 calculates the Euclidean distance D as a geometric distance between the coordinates Sk (x ′ _k , y ′ _k ) and the coordinates Qk (x ′ _k , y ′ _k ). The Euclidean distance D is compared with the threshold value Td. If the Euclidean distance D is equal to or smaller than the threshold Td (Yes in S1514), the document search unit 503 advances the process to S1515.
In step S1515, the document search unit 503 increments the vote number Vote and advances the process to step S1516. If the Euclidean distance D is greater than the threshold Td (No in S1514), the document search unit 503 advances the process to S1516 without performing the process of S1515. In step S1516, the document search unit 503 increments the corresponding point selection variable k. Thereafter, the process proceeds to S1511. The above processing is repeated until the corresponding point selection variable k exceeds the number m of corresponding points registered in the shortest distance corresponding point list.

Next, a process when the corresponding point selection variable k exceeds the number m of corresponding points registered in the corresponding point list in S1511 will be described. In step S1517, the document search unit 503 compares the value of the vote number Vote with the value of the final vote number VoteMax. If the value of the vote number Vote is larger than the value of the final vote number VoteMax (Yes in S1517), the document search unit 503 advances the process to S1518. If the value of the vote number Vote is equal to or smaller than the value of the final vote number VoteMax (No in S1517), the document search unit 503 advances the process to S1519.
In step S1518, the document search unit 503 replaces the value of the final vote number VoteMax with the value of the vote number Vote, and stores the conversion matrix M at that time in the variable Mmax. Thereafter, in S1519, the document search unit 503 increments the iteration count number Count, and advances the process to S1505 described above.

In S1509, it is possible to deal with other geometric transformations such as affine transformation by using a transformation matrix other than the transformation matrix shown in Expression (8). In the case of affine transformation, the number of coordinates of the pair of corresponding points selected at random in S1508 is set to 3. In S1509, equation (13) is used instead of equation (8), and the variables a to f may be obtained using the three corresponding points (total of 6 points) selected in S1508.

Through the above processing, the document search unit 503 can calculate the final vote number VoteMax as the similarity (similarity calculation processing). Then, the document search unit 503 can specify the content having the calculated similarity higher than a predetermined threshold and the highest similarity as the content corresponding to the content area to be processed.
In the present embodiment, the final vote number VoteMax is output as the similarity in S1506, but the method of calculating the similarity is not limited to the embodiment. As another example, the document search unit 503 may output the number m of corresponding points as the similarity without performing the processing from S1503 onward.
In the above description, a method using RANSAC as an image matching method based on the comparison of local feature points / local feature amounts has been described. However, the feature amount matching process may be another method as long as there is a method that can calculate the similarity between two images and can estimate the rotation angle by a method with a low processing load in the process of calculating the similarity.

As described above, the AR processing system according to the present embodiment considers the positional relationship between the content obtained from the photographed image and the content registered in the feature amount table 600, not the simple content. To identify the content. Therefore, the AR processing system can specify the content with higher accuracy, and thereby can present the related information associated with the content being browsed by the user with high accuracy.
Further, the AR processing system specifies the positional relationship between the frames based on the change in the shooting direction at the time of shooting, and specifies the positional relationship between the contents based on the positional relationship. Therefore, the AR processing system can accurately specify the positional relationship of each content even when a plurality of contents arranged in one paper document cannot be accommodated in one frame.

As a first modification of the AR processing system according to the first embodiment, it is not necessary to set all frames in a captured image as a processing target. For example, a frame at a predetermined interval is set as a processing target, and position related information generation (S702), transmission of a photographed image, etc. (S703) and image analysis (S704) shown in FIG. Based on the result, the processing after S705 may be performed. As a result, the number of processes can be reduced, and the processing load can be reduced.
Further, the captured image may be a still image. In this case, the AR processing system may process continuously captured images captured at different timings. Specifically, in step S702, the AR processing system generates position related information and further specifies the timing at which the captured image is obtained. And you may perform the process after S703 with respect to the still image as a picked-up image.
As a second modification, the processing of S704 to S706 illustrated in FIG. 7 may be performed by the mobile terminal 100 instead of the management server device 110. As another example, the management server device 110 is configured by a plurality of devices, and the processing of S704 to S706 may be shared and executed by a plurality of devices.

(Second Embodiment)
Next, an AR processing system according to the second embodiment will be described. When the AR processing system according to the first embodiment cannot identify the corresponding electronic document from the captured image, the AR processing system waits for the subsequent frame to be input, and based on the information obtained from the subsequent frame. Try to identify the electronic document again.
On the other hand, the AR processing system according to the second embodiment, when the corresponding electronic document cannot be identified from the captured image, provides support information for capturing an image necessary for specifying the mobile terminal 100. Displayed on the display device 101. Hereinafter, of the specific processing of the AR processing system according to the second embodiment, a part different from the processing of the AR processing system according to the first embodiment will be described.

FIG. 16 is a flowchart showing support information transmission processing by the management server device 110. When it is determined No in S706 of the AR process described with reference to FIG. 7, the management server apparatus 110 executes the support information transmission process instead of proceeding to S703. In step S1601, the document search unit 503 of the management server apparatus 110 checks whether or not a plurality of electronic document candidates that satisfy the position information are obtained in the electronic document search process (S705). If a plurality of electronic document candidates are obtained (Yes in S1601), the document search unit 503 advances the process to S1603. If the electronic document candidate satisfying the position information is not obtained (No in S1601), the document search unit 503 advances the process to S1602.
In S1602, the communication unit 500 transmits presentation information indicating that there is no related information to the mobile terminal 100. In response to this, the communication unit 400 of the mobile terminal 100 receives the presentation information. Then, the display unit 404 of the portable terminal 100 displays the presentation information on the display device 101. Thereby, the user of the portable terminal 100 can grasp that there is no related information for the paper document being browsed.

In step S <b> 1603, the document search unit 503 refers to the feature amount table 600 and specifies content necessary for narrowing down to one electronic document corresponding to the photographed image from the content associated with the plurality of candidates. (Content identification process). Here, the plurality of candidates are obtained in the electronic document search process (S705).
In step S1604, the document search unit 503 generates support information based on the content specified in step S1603 (support information generation process). Specifically, the document search unit 503 creates, as support information, an instruction for changing the shooting direction by the shooting unit 300 of the mobile terminal 100 so that the specified content is shot.
Next, in S1605, the communication unit 500 transmits the support information to the mobile terminal 100, and then the process proceeds to S704 (FIG. 7). In response to this, the communication unit 400 of the mobile terminal 100 receives support information. Then, the display unit 404 of the portable terminal 100 displays support information on the display device 101 (support information display process). Thereby, the user of the portable terminal 100 can grasp that the related information can be viewed by changing the shooting direction.

The support information transmission process will be specifically described with reference to FIG. As a premise, as shown in FIG. 17A, it is assumed that electronic documents having document IDs “011” and “012” are registered in the feature amount table 1700. In the AR process, it is assumed that three content areas are extracted, and records corresponding to the three content areas are recorded in the captured image table 1710 as shown in FIG.
As illustrated in FIG. 17, it is assumed that the feature amount of the content area with the region ID “a011” matches the feature amount of the content with the content ID “A” of the document ID “011”. Also, as shown in FIG. 17, the feature amount of the content area with the region ID “a012” is assumed to match the feature amount of the content with the content ID “C” of the document ID “012”. In this case, the document search unit 503 cannot identify the two electronic documents identified by the document IDs “011” and “012” as candidates, and which candidate the captured image corresponds to.

  Therefore, in step S1603, the document search unit 503 searches the content registered in the positional relationship with the content that matches the content area recorded in the captured image table 1000 among the content associated with the candidate electronic document. Identify. That is, the content specified at this time is content other than content that matches the content area. In the example of FIG. 17, the document search unit 503 specifies the content with the content ID “D” with the document ID “011” and the content with the content ID “E” with the document ID “012”.

  In step S <b> 1604, the document search unit 503 determines a shooting direction for shooting each of the content IDs “D” and “E” specified in step S <b> 1603. For example, it is assumed that the content area with the area ID “a013” is displayed in the frame displayed on the display device 101 of the mobile terminal 100 at the time of the processing of S1604. In this case, the document search unit 503 creates support information for instructing to turn the shooting direction downward or upper left. As a result, if the content area displaying the content with the content ID “E” is photographed, it can be understood that the electronic document corresponding to the photographed image is the document with the document ID “012”. In addition, if the content area displaying the content with the content ID “D” is captured, it can be understood that the electronic document corresponding to the captured image is the document with the document ID “011”.

FIG. 18 is a diagram illustrating a display example of support information. As described above, the display unit 404 of the mobile terminal 100 creates the arrow image 1800 indicating the shooting direction from the support information, and displays this on the display device 101. The output form of support information is not limited to the embodiment. As another example, the display device 101 may display support information in a text format. As another example, when the mobile terminal 100 includes a speaker (not shown), the support information may be output as sound.
The remaining configuration and processing of the AR processing system according to the second embodiment are the same as the configuration and processing of the AR processing system according to the first embodiment.
As described above, in the AR processing system according to the second embodiment, when the electronic document corresponding to the photographed image cannot be uniquely specified, the user is requested to perform a necessary operation, thereby more quickly and reliably. An electronic document can be identified and appropriate relevant information can be presented.

<Other embodiments>
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code (computer program) of software that implements the functions of the above-described embodiments is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium.
Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Moreover, the program code of software may be supplied to the system or apparatus via a network.

  An operating system (OS) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

Furthermore, it is assumed that the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing, and the processing of the above-described embodiment is realized by the processing. Needless to say.
When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

  As mentioned above, according to each embodiment mentioned above, the photographic subject can be specified accurately.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

DESCRIPTION OF SYMBOLS 100 Mobile terminal 101 Display apparatus 110 Management server apparatus 300 Image pick-up part 301 Acceleration sensor 302 CPU

Claims (11)

Area extracting means for extracting a plurality of content areas from a plurality of photographed images obtained by photographing a part of one paper medium;
A feature amount calculating means for calculating a feature amount of the content area;
Position specifying means for specifying a positional relationship between the plurality of content areas;
A position that stores feature quantities of a plurality of contents included in one electronic document in association with an electronic document corresponding to one paper medium, and further indicates a positional relationship between the plurality of contents included in one electronic document With reference to the storage means for storing information, based on the feature amounts of each of the plurality of content areas calculated by the feature amount calculation means and the positional relationship between the plurality of content areas specified by the position specifying means, An information processing system comprising electronic document specifying means for specifying an electronic document corresponding to the plurality of photographed images.   The information processing system according to claim 1, wherein the region extraction unit extracts a plurality of content regions from a plurality of frames included in a moving image as the plurality of captured images. Similarity calculation means for calculating the similarity based on the feature quantities of the plurality of content regions extracted by the area extraction means and the feature quantities of the content;
First content specifying means for specifying content corresponding to each of the plurality of content areas extracted by the area extracting means based on the similarity;
The electronic document specifying means indicates the positional relationship specified by the position specifying means in the position information stored in the storage means for the plurality of contents specified by the first content specifying means. 3. The information processing system according to claim 1, wherein an electronic document corresponding to the position information is specified as an electronic document corresponding to the photographed image when the positional relationship is satisfied.   In the electronic document specifying means, all the contents associated with the electronic document are specified by the first content specifying means, and the positional relationship specified by the position specifying means is indicated in the position information. 4. The information processing system according to claim 3, wherein an electronic document corresponding to the position information is specified as an electronic document corresponding to the captured image when the positional relationship is satisfied. The storage means further stores related information in association with the content,
Related information extraction for extracting the related information stored in the storage means in association with the content included in the electronic document specified by the electronic document specifying means, corresponding to the content area extracted by the area extracting means The information processing system according to claim 3 or 4, further comprising means.   The apparatus further includes a superimposed display unit that displays the captured image by superimposing the related information at a position corresponding to the content area corresponding to the related information extracted by the related information extracting unit of the captured image. The information processing system according to claim 5, wherein the system is an information processing system. Second content specification that specifies content necessary for narrowing down based on the positional information when the positional relationship specified by the position specifying means satisfies the positional relationship indicated by each of the positional information of a plurality of electronic documents. Means,
Support information creating means for creating support information that prompts the user to change the shooting direction based on the second content;
The information processing system according to claim 3, further comprising support information display means for displaying the support information. Area extracting means for extracting a plurality of content areas from a plurality of photographed images obtained by photographing a part of one paper medium;
A feature amount calculating means for calculating a feature amount of the content area;
Position specifying means for specifying a positional relationship between the plurality of content areas;
A position that stores feature quantities of a plurality of contents included in one electronic document in association with an electronic document corresponding to one paper medium, and further indicates a positional relationship between the plurality of contents included in one electronic document With reference to the storage means for storing information, based on the feature amounts of each of the plurality of content areas calculated by the feature amount calculation means and the positional relationship between the plurality of content areas specified by the position specifying means, An information processing apparatus comprising: an electronic document specifying unit that specifies an electronic document corresponding to the plurality of captured images. An information processing method executed by an information processing system,
A region extracting step of extracting a plurality of content regions from a plurality of captured images obtained by capturing a part of one paper medium;
A feature amount calculating step for calculating a feature amount of the content area;
A position specifying step for specifying a positional relationship between the plurality of content areas;
A position that stores feature quantities of a plurality of contents included in one electronic document in association with an electronic document corresponding to one paper medium, and further indicates a positional relationship between the plurality of contents included in one electronic document With reference to storage means for storing information, based on the feature amounts of each of the plurality of content areas calculated in the feature amount calculation step and the positional relationship between the plurality of content areas specified in the position specifying step, And an electronic document specifying step of specifying an electronic document corresponding to the plurality of photographed images. An information processing method executed by an information processing apparatus,
A region extracting step of extracting a plurality of content regions from a plurality of captured images obtained by capturing a part of one paper medium;
A feature amount calculating step for calculating a feature amount of the content area;
A position specifying step for specifying a positional relationship between the plurality of content areas;
A position that stores feature quantities of a plurality of contents included in one electronic document in association with an electronic document corresponding to one paper medium, and further indicates a positional relationship between the plurality of contents included in one electronic document With reference to storage means for storing information, based on the feature amounts of each of the plurality of content areas calculated in the feature amount calculation step and the positional relationship between the plurality of content areas specified in the position specifying step, And an electronic document specifying step of specifying an electronic document corresponding to the plurality of photographed images. Computer
Area extracting means for extracting a plurality of content areas from a plurality of photographed images obtained by photographing a part of one paper medium;
A feature amount calculating means for calculating a feature amount of the content area;
Position specifying means for specifying a positional relationship between the plurality of content areas;
A position that stores feature quantities of a plurality of contents included in one electronic document in association with an electronic document corresponding to one paper medium, and further indicates a positional relationship between the plurality of contents included in one electronic document With reference to the storage means for storing information, based on the feature amounts of each of the plurality of content areas calculated by the feature amount calculation means and the positional relationship between the plurality of content areas specified by the position specifying means, A program for functioning as electronic document specifying means for specifying an electronic document corresponding to the plurality of captured images.

Images