JP6168872B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program for outputting extended information related to a captured image.

  Search for extended information related to the subject imaged using a camera of a tablet terminal or a smartphone (hereinafter also referred to as “smart terminal” together with the tablet terminal and the smartphone), and the searched extended information is displayed on the screen of the smart terminal. The technique of displaying on is known. For example, in augmented reality technology called Augmented Reality (hereinafter also referred to as “AR”), the augmented information related to the subject imaged by the camera is identified, and the identified augmented information is displayed on the subject imaged by the camera. Displayed on the screen.

  Furthermore, a service is also developed in which, when a paper document is captured as a subject, extended information (hereinafter also referred to as “AR content”) is superimposed and displayed on a screen on which the captured paper document is displayed. Although content such as moving images cannot be expressed (reproduced) on paper, a moving image is specified as AR content of a still image printed on a paper document, and the moving image is superimposed on the still image on the screen of the smart terminal. It becomes possible. For example, Patent Document 1 discloses a technique for detecting a noticed word from an imaged paper document and superimposing and displaying extended information related to the noticed word.

JP2012-22442

J. et al. J. et al. Koenderink and A.K. J. et al. van Doorn, “Representation of local geometry in the visual system,” Riologic Cybernetics, vol. 55, pp. 367-375, 1987 Yuji Kikuta Sozo Inoue Mizuho Iwaihara, “About the presentation method of large amount of information in consideration of importance and attractiveness and evaluation using display achievement”, IEICE Technical Report, May 2000, Vol. 100, no. 31, pp. 41-48

  In the technology described above, there is a possibility that the viewer does not pay attention to information displayed as AR content of a paper document. For example, when browsing a paper document, if there is content with a very conspicuous color in the paper document, the viewer may focus on the content with a conspicuous color and may not pay attention to the AR content. For the AR content provider, in the case of important AR content that the viewer wants to be noticed, it becomes a problem that the AR content is not noticed by the viewer. For example, important information associated with a paper document is associated as AR content, and the provider of the AR content expects the viewer to attract attention. However, in reality, since the viewer does not pay attention to the AR content, it is considered that a situation occurs in which important AR content information is not understood by the viewer.

  Regardless of the content to be imaged, it may be possible to prepare AR content that is more noticeable than that, but if it is too conspicuous, the beauty will be impaired. In addition, there are various environments for browsing information on paper documents. It is difficult to prepare AR content that is likely to attract attention in any environment. For example, when the AR content is projected from a projection device onto a plane on which a paper document is mounted and viewed, the surrounding scenery, other documents / objects, the position of the light source, and the like vary depending on the situation. It is difficult to create AR content that is common to these situations and that attracts attention.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an information processing apparatus in which extended information is easily noticed by changing extended information displayed together with a captured image according to imaging information. And Moreover, it aims at providing the method and program.

  An image processing apparatus according to the present invention has the following configuration. In other words, the acquisition means for acquiring extended information to be displayed together with the captured image based on the feature information of the captured image, and the attention level of each of the plurality of region images obtained by dividing the captured image are derived. Deriving means, comparing means for comparing the respective attention degrees of the plurality of area images with the attention degree of the extended information, and changing means for changing the display condition of the extended information based on the comparison result.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the extended information which attracts attention easily corresponding to various imaging environments by dynamically producing | generating the extended information which attracts attention based on imaging information.

A conceptual diagram of AR processing in an embodiment of the present invention is shown. The block diagram which shows an example of the hardware constitutions in embodiment of this invention is shown. The block diagram which shows the structure of the AR processing system in embodiment of this invention is shown. 3 is a flowchart showing the overall flow of AR processing in the embodiment of the present invention. 5 is a flowchart showing a flow of feature amount extraction processing in the embodiment of the present invention. An example of the reduced image in embodiment of this invention is shown. An example of the AR content management table in the embodiment of the present invention is shown. An example of the AR content table in the embodiment of the present invention is shown. The flowchart which shows the flow of the attention degree calculation process in embodiment of this invention is shown. The conceptual diagram of the imaging information attention degree calculation process in embodiment of this invention is shown. The flowchart which shows the flow of the AR content change process in embodiment of this invention is shown. An example of the AR content change result in the embodiment of the present invention is shown. 1 shows an example of the appearance and configuration of an information processing apparatus according to an embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Embodiment 1>
First, an outline of processing in the present embodiment will be described. In this system, AR content (extended information) related to feature information in a paper document is superimposed and displayed on a paper document displayed on a screen of an information display device such as a smart terminal. Hereinafter, this system is simply referred to as an AR processing system.

  In the AR processing system, when a paper document is captured by a camera of a smart terminal, AR content related to feature information extracted from the captured image is specified, and the AR content (extended information) is displayed on a screen of an information display device such as a smart terminal. Is superimposed on the paper document displayed on the screen.

  FIG. 1 is a conceptual diagram of the AR processing system. When the paper document 101 is captured by the smart terminal 102, AR content (extended information) related to the feature information extracted from the captured image of the paper document is specified from the AR content DB 103. Then, the specified AR content (extended information) is displayed on the screen of the smart terminal together with the imaging information of the paper document. In the display result example A in FIG. 1, a view including a paper document and the AR content 105 is displayed as the display result 104. In the display result example B, a view including the AR content 107 is displayed as the display result 106. In the display result example B, when the display result is generated, the AR content (extended information) is subjected to display processing so that a viewer of the AR content (extended information) is likely to receive attention. After the AR content (extended information) identified from the AR content DB 103 is converted into a format that is easy for the viewer to watch, the converted AR content (extended information) is displayed and output. In the display result example B of FIG. 1, the display position and size of the AR content (extended information) are changed so as to hide a graph that tends to attract attention. In addition, the colors may be converted so that the AR content (extended information) itself is likely to attract attention. This process makes it easy for the AR content to attract attention, so that it is possible to dynamically generate a display output that makes it difficult for the viewer to miss the AR content.

  Hereinafter, a detailed description will be given with reference to FIGS.

  FIG. 2 is an example of each hardware configuration included in the information processing system (hereinafter also referred to as “AR processing system”) in the present embodiment.

  A CPU (Central Processing Unit) 202 performs calculations and logic determinations for various processes, and controls various elements connected to the bus 201.

  The AR processing system is equipped with a memory including a program memory and a data memory. The program memory stores a program for control by the CPU including various processing procedures described later with reference to flowcharts. This memory may be a ROM (Read-Only Memory) 203 or a RAM (Random Access Memory) 204 into which a program is loaded from an external storage device or the like. Or you may implement | achieve with these combination.

  The storage device 205 is a device such as a hard disk for storing data and programs according to the present embodiment. In the present embodiment, it is also used for holding photographs, documents, and the like. Note that a data holding device (not shown) connected to an external connection or a network may be used as a device for holding photographs, documents, and the like. Here, the external storage device 206 can be realized by, for example, a medium (recording medium) and an external storage drive for realizing access to the medium. As such media, for example, a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, a flash memory, and the like are known.

  The input device 207 is a device for inputting an instruction from the user. The user gives an instruction to the AR processing system via the input device 207. The input device 207 can be realized by, for example, a keyboard or a pointing device.

  The output device 208 is a device as display means for displaying and outputting the processing result and the like. The output device 208 can be realized, for example, by a display device such as a CRT (Cathode-Ray Tube) including a circuit for output or a liquid crystal display. In the present embodiment, photographs, documents, etc. are displayed on the output device 208.

  An interface (hereinafter also referred to as “I / F”) 209 is a device that mediates information, and exchanges data with an external device via this I / F 209. Network interface and input / output interface.

  FIG. 3 is a block diagram showing the configuration of the AR processing system in the present embodiment. The system includes an AR display device 301 and an AR processing device 302. Each device has the hardware configuration described in FIG.

  The AR display device 301 includes an imaging unit 303 that performs imaging processing by a camera, and a display unit 304 that displays AR content generated by the AR processing device. The imaging unit 303 images a paper document or the like with a camera. The AR content is displayed on the display unit 304 as extended information related to the captured paper document.

  The AR processing device 302 is a device that generates AR content to be output to the display unit 304 based on image information input by the imaging unit 303 via the I / F 209. The AR processing device 302 includes a feature amount extraction unit 305, a content specification unit 306, an attention level calculation unit 307, and an AR content change unit 308. The feature amount extraction unit 305 extracts a feature amount from the image information input by the imaging unit 303. Details of the feature amount extraction method will be described later. The content identifying unit 306 identifies the corresponding AR content using the feature amount extracted by the feature amount extracting unit 305. The content specifying process is performed with reference to the AR content management table held in the AR content DB of FIG. The AR content DB exists in the external storage device 206 that can be accessed via the storage device 205 or the I / F 209 of the AR processing device 302. Details of the AR content management table will be described later with reference to FIG. The attention level calculation unit 307 performs attention level calculation processing on the image information input by the imaging unit 303 and the AR content specified by the AR content specification unit 306. Details of the attention level calculation processing will be described later with reference to FIG. The AR content changing unit 308 changes the AR content based on the attention level for the image information and the attention level for the AR content calculated by the attention level calculation unit 307. Details of the AR content change process will be described later with reference to FIG.

  Each process is controlled by the CPU 202 of FIG. Details of the processing group will be described later.

  The photographing unit 301, the display unit 302, the feature amount extracting unit 305, the content specifying unit 306, the attention level calculating unit 307, and the AR content changing unit 308 can be provided as a single device.

  The CPU can function as various means by executing a program.

  Note that a control circuit such as an ASIC that operates in cooperation with the CPU may function as these means. These means may be realized by cooperation between the CPU and a control circuit that controls the operation of the image processing apparatus. Further, the CPU need not be a single one, and may be a plurality. In this case, a plurality of CPUs can perform processing in a distributed manner. The plurality of CPUs may be arranged in a single computer, or may be arranged in a plurality of physically different computers. Note that the means realized by the CPU executing the program may be realized by a dedicated circuit.

  FIG. 4 is a flowchart for explaining the overall flow of the AR processing system according to this embodiment. Hereinafter, the flowchart is realized by the CPU executing the control program.

  Step S4000 is document imaging processing, in which a paper document is imaged by the imaging unit 303 shown in FIG.

  Step S4010 is a feature quantity extraction process, which is executed by the feature quantity extraction unit 305 shown in FIG. In the feature amount extraction process, the image feature amount of the feature information is extracted from the paper document imaged in step S4000. The feature information may be printed on paper or may be feature information that can be recognized by the imaging unit. Other feature information includes invisible watermark information and paper text. Details of the feature amount extraction processing will be described later.

  Step S4020 is an AR content specifying process, which is realized by the content specifying unit 306 shown in FIG. In the AR content specifying process, the AR content is specified using the feature amount extracted in step S4010 and an AR content management table described later with reference to FIG.

  Step S4030 is attention level calculation processing, which is realized by the attention level calculation unit 307 shown in FIG. In the attention level calculation process, the attention level is calculated for the image information of the paper document imaged in step S4000 and the AR content specified in step S4020. Information on the specified AR content can be acquired by referring to an AR content table described later with reference to FIG. The degree of attention of the AR content can be calculated in advance, and can be derived using the degree of attention as long as it is managed in the AR content table. Details of the attention level calculation processing will be described later.

  Step S4040 is attention level determination processing. In the attention level determination process, a comparison process between the attention level of the captured paper document calculated in step S4030 and the attention level of the AR content is performed. As a result of the comparison, when the attention level of the AR content is higher than that of the captured paper document, the process ends. On the other hand, in the comparison result, when the attention level of the AR content is lower than that of the captured paper document, the process proceeds to step S4050.

  Step S4050 is an AR content change process, which is realized by the AR content change unit 308 shown in FIG. In the AR content change process, the AR content is changed so that the attention level of the AR content is higher than that of the captured paper document.

  Next, details of the feature amount extraction processing in step S4010 of FIG. 4 will be described. The feature amount extraction is a process of extracting a local feature amount of an image (hereinafter also referred to as a local feature amount). A local feature has a property of rotation invariance and enlargement / reduction invariance. With this feature, even when the image is rotated, enlarged, or reduced, it is possible to specify the image using the feature amount.

  The local feature quantity extraction process can be performed using an existing local feature quantity extraction technique. Hereinafter, an example of a local feature extraction technique will be simply shown. In the following example, the local feature amount of the entire image data obtained from the imaging information of the paper document is obtained, but the local feature amount of the partial image may be obtained. Further, a QR code (registered trademark) or the like may be recognized without using the local feature amount of the image.

  FIG. 5 is a flowchart for explaining the flow of local feature extraction processing from an image.

  In the input image reading step S5000, the designated image is read.

  In the luminance component image creation step S5010, a luminance component is extracted from the input image read in step S5000 to generate a luminance component image.

  In a reduced image generation step S5020, the luminance component image created in step S5010 is sequentially reduced according to the magnification p, and n reduced images are generated. Here, the magnification p and the number n of reduced images are defined in advance.

  FIG. 6 is a diagram illustrating an example of the reduced image generation process in step S5020. FIG. 6 shows an example in which the magnification p is 2 to the power of-(1/4) and the number n of reduced images generated by reduction is 8. However, in this example, the magnification p is not the area ratio but the side length ratio. In FIG. 6, reference numeral 601 denotes the luminance component image created in step S5010. Reference numeral 602 denotes a reduced image obtained by reducing the luminance component image 601 four times in accordance with the magnification p, and corresponds to an image obtained by reducing the luminance component image 601 by half. Reference numeral 603 denotes a reduced image obtained by reducing the luminance component image 601 eight times according to the magnification p, and corresponds to an image obtained by reducing the luminance component image 601 to ¼. Note that there are various methods for reducing an image, such as a method for thinning out pixels, a method using linear interpolation, a method for sampling after applying a low-pass filter, and the generation method of a reduced image in this embodiment may be any method. . Reference numeral 604 denotes a scale number, which is assigned in order from the larger size of the reduced image.

  In step S5030, local feature points (local feature points) are extracted from each of the luminance component image created in step S5010 and the n reduced images obtained in step S5020. The local feature points extracted here are robust local feature points that can be stably extracted from the same place even if image processing such as rotation or reduction is performed on the image. As a method for extracting such local feature points, there is a Harris operator or the like.

  Specifically, for each pixel of the image obtained by applying the Harris operator, the pixel values of the pixel of interest and pixels in the vicinity of the surrounding 8 (a total of 9 pixels) are examined. When the pixel value of the pixel of interest is greater than or equal to a predetermined value and has a local maximum (the pixel value of the pixel of interest is the maximum among the nine pixels), the point where the pixel of interest is located is defined as a local feature point Extract. Note that any feature point extraction method is applicable as long as it is a method capable of extracting a robust local feature point, not limited to the above-described feature point extraction method using the Harris operator.

  Next, in step S5040, for each local feature point obtained in step S5030, a feature quantity (local feature quantity) defined so as to remain unchanged even when the image is rotated is calculated. As a method for calculating the local feature amount, in this embodiment, a local jet of Non-Patent Document 1 and a combination of derivatives thereof are used.

  By this method, the calculated local feature amount can have characteristics that have a relatively high resistance to enlargement, reduction, or rotation. Specifically, the local feature amount shown in Expression (1) is calculated.

  However, the symbols used on the right side of the equation (1) are defined by the following equations (2) to (7). Here, G (x, y) on the right side of Expression (2) is a Gaussian function, I (x, y) is a pixel value at image coordinates (x, y), and “*” is a symbol representing a convolution operation. is there. Equation (3) is a partial derivative of variable L defined by equation (2) with respect to x, and equation (4) is a partial derivative of variable L with respect to y. Equation (5) is the partial derivative of y of the variable Lx defined in Equation (3), Equation (6) is the partial derivative of Lx defined in Equation (3), and Equation (7) is the equation ( This is a partial derivative of Ly defined by 4) with respect to y.

  In addition, the calculation method of a local feature-value is not restricted to the above-mentioned method, The calculation method of another local feature-value is applicable. Examples of other local feature values include SIFT feature values and SURF feature values.

  Next, the AR content management table referred to in step S4020 of FIG. 4 will be described in detail. The AR content management table is a table that manages AR content in association with image feature amounts that are conditions for displaying the AR content. In step S4020 of FIG. 4, the AR content related to the captured paper document is specified by referring to the AR content management table. In the AR content management table, when a paper document is created, the AR content related to the paper document and the image feature amount of the paper document are registered in association with each other.

  FIG. 7 is an example of an AR content management table. The AR content management table includes an AR content ID 701 for uniquely specifying the AR content, and an image feature amount 702 serving as a key for specifying the AR content. “ID 701” is identification information of the AR content. Using the AR content management table, it is determined whether the image feature amount 702 of the AR content management table includes the same image feature amount as the image feature amount extracted in step S4010 of FIG. When the image feature amount of the imaging information of the paper document is included in the AR content management table, the AR content associated with the matched image feature amount is used as the AR content associated with the captured paper document. Can be identified.

  In the present embodiment, the AR content is registered at the same time as the paper document is created, but it is not necessary to be at the same time as the paper document. For example, after a paper document is created, it may be registered in the AR content management table as related AR content. Also, information contained in the paper document is extracted from the captured paper document using a technique such as a character recognition technique. Then, using the extracted information as a search query, information may be searched from the Internet or the like, and AR content may be dynamically created and registered from the searched information.

  Next, the details of the AR content table referred to in step S4030 will be described. The AR content table holds entity data and display conditions (display position and display size) of the AR content indicated by the AR content ID 701 included in the AR content management table described with reference to FIG.

  FIG. 8 is an example of an AR content table. The AR content table includes an AR content ID 801 for specifying the AR content, a data path 802 indicating the data storage location, a width 803 indicating the display size, a height 804, and a display position 805 indicating the display position. In the AR content ID 801, the same information as the information described in the AR content ID 701 described with reference to FIG. 7 is described.

  By referring to the AR content table using the AR content ID specified in step S4020 of FIG. 4, the entity data and display conditions of the AR content can be acquired.

  A data path 802 is a data path to a file indicating the actual data of the AR content, and is described as a relative path or an absolute path supported by the file system. In the example of FIG. 8, the data path is expressed as a relative path, and a path 806 to a file indicating image data, a path 807 to a file indicating moving image data, and a path 808 to a file indicating HTML data are described. ing. In the present embodiment, the path information to the file is described. However, any format may be used as long as it can represent the entity data. For example, image binary data indicating entity data of AR content may be stored. It is also possible to store data describing HTML in text format.

  The display size (width) 803 is data indicating the width when displaying the AR content. The unit of data indicating the width is a pixel, and when displaying the AR content, the data is displayed so as to have a value specified by the display size (width) 803. The display size (height) 804 is data indicating the height when the AR content is displayed. Similar to the display size (width), the unit of this data is expressed in pixels. A display position 805 is data indicating a position when the AR content is displayed. The unit of the position data is expressed in pixels, and corresponds to the pixel of the device having the upper left on the display device as the origin (0, 0). In this embodiment, the units of the display size (width) 803, the display size (height) 804, and the display position 805 are pixels, but different units may be used. For example, a unit such as mm can be used. For the display position 805, a coordinate system having the origin at the upper left of the display device is used, but a different method may be used. Specifically, instead of using the absolute coordinates of the display device, it is also possible to use relative coordinates with the upper left corner of the paper document being imaged as the origin. By using the relative coordinates, it becomes possible to define information on AR content that does not depend on the display device.

  Next, details of the attention level calculation processing in step S4030 will be described. In the attention level calculation processing, attention level calculation processing is performed on the captured image of the paper document captured in step S4010 of FIG. 4 and the AR content specified in step S4020. Details of the processing will be described with reference to FIG.

FIG. 9 is a flowchart for explaining the flow of attention level calculation processing.
Step S9000 is an AR content attention level calculation process. In the AR content attention level calculation processing, attention level calculation processing is performed for the AR content specified in step S4020 of FIG. The AR content data that is subject to attention level calculation can be acquired using the AR content table described above. The attention level is calculated for the acquired AR content with reference to the data path 802 of the AR content table. In the present embodiment, an existing attention level calculation technique is used for calculating the attention level. As an existing attention level calculation technique, for example, there is the following Non-Patent Document 2. Non-Patent Document 2 discloses a method of calculating the degree of attraction by expressing the ease of attracting attention for the displayed object as the degree of attraction. Specifically, it is possible to calculate the degree of attraction using the hue of the object.

  In the present embodiment, using the calculation formula for calculating the degree of attraction described in Non-Patent Document 2 above, the degree of attraction is obtained, and the obtained degree of attraction is used as the degree of attention in the AR content attention degree DB (not shown). Store. As an example of the calculation formula, there is formula (8). Formula (8) is a formula for calculating the degree of attention using the hue. In Equation (8), H indicates the hue of the displayed object, and Hd indicates the degree of attention (attraction) calculated using the hue.

  Using the above calculation formula (8), when the AR content is image data, the attention level can be calculated from the hue of the image data. Further, even when the AR content is not image data such as HTML data, it is possible to calculate the degree of attention by generating image data rendered as a display result and calculating the degree of attraction for the image data. is there.

  Step S9010 is imaging information attention level calculation processing. In the imaging information attention level calculation processing, attention level calculation is performed on the imaging information of the paper document captured in step S4010 of FIG. The imaging information may include surrounding information in addition to the paper document. If there is information that attracts attention around the paper document, the AR content is changed accordingly.

  In the calculation of the attention level of the imaging information, the imaging information is divided into a prescribed number, and the attention level is calculated for each of the divided area images. The method of calculating the attention level is the same as the method described in step S9000. For example, it is possible to calculate the degree of attention of imaging information by using the above equation (8). An imaged paper document can be configured as image information, and Expression (8) can be used for the image information. By configuring as image information, attention level information can be appropriately calculated even in situations where the position of the light source is different.

  FIG. 10 is a conceptual diagram of the attention level calculation process of the imaging information. This is information on a paper document in which the paper document 1001 is captured. This paper document 1001 is divided into 24 areas having a predetermined fixed size. The area division result 1002 is a result of dividing into 24 areas. Each region of the division result 1002 is regarded as one image, and the degree of attention for each image is calculated. The calculated attention level is associated with each area and is stored as attention level information 1003.

  In the present embodiment, the paper document of the imaging information is divided into a prescribed number of area images, and the attention level is calculated for each area image, but a different method may be used. For example, a method of calculating the degree of attention for an area or object extracted using an existing document area analysis technique or object extraction technique may be used. With the existing document area analysis technology, it is possible to extract a plurality of types of areas such as a character area in which a character string in a captured paper document is described, an image area in which an image exists, and the like. The attention level can be calculated for the image area using the above-described method.

  Next, the details of the AR content change process described as step S4040 and step S4050 of FIG. 4 will be described. In the AR content change process, when the attention level of the AR content is lower than the captured information, the display condition of the AR content is changed so that the attention level of the AR content is higher than the captured information. Details of the processing will be described with reference to FIG.

  FIG. 11 is a flowchart for explaining the flow of the AR content change process.

  Step S11000 is AR content attention level acquisition processing. In the AR content attention level acquisition process, the AR content attention level calculated in step S9000 of FIG. 9 is acquired. The AR content attention level acquisition processing refers to an AR content attention level DB (not shown) to acquire the AR content attention level.

  Step S11010 is an imaging information attention level acquisition process. In the attention level acquisition process of the imaging information, the attention level of the imaging information is acquired by referring to the attention level information described as the attention level information 1003 in FIG. The degree of attention is the degree of attention for each of the plurality of regions as described above.

  Step S11020 is attention level comparison processing. In the attention level comparison process, the attention level of the AR content acquired in step S11000 is compared with the attention level associated with each area of the imaging information acquired in step S11010. As a result of the comparison, if there is a region of imaging information having a higher attention level than the attention level of the AR content, information on the region is stored in a not-shown attention region DB. After the comparison process is performed on all areas of the imaging information, if no data is stored in the attention area DB, it is determined that the attention level of the imaging information is lower than the attention level of the AR content, and the process is performed. finish. When data is stored in the attention level DB described above, it is determined that the attention level of a part or all of the imaging information is higher than the attention level of the AR content, and the process proceeds to step S11030.

  Step S11030 is a process of changing display conditions such as the display position and display size of the AR content. In the display position / display size change process, the attention area DB created in step S11020 is referred to, and the display position / display size of the AR content is set so that the attention area having a higher attention degree than the attention degree of the AR content is hidden. change. The AR content whose display conditions are changed is transmitted to the AR image display device 301 via the I / F 209.

  This will be described in detail with reference to FIG. The attention area result 1201 in FIG. 12 is information obtained by superimposing the information of the attention level DB on the captured paper document. The areas with higher attention level than the AR content 1210 stored in the attention level DB are attention areas 1202 to 1207. The region where only the attention regions 1202 to 1207 are cut out is the high attention region 1209, and in the AR content change processing, the display position and display size of the AR content 1210 are determined so that the high attention region 1209 is hidden. A display result 1208 indicates the AR content display result after the AR content is changed. As can be seen from the display result 1208, an area having a higher degree of attention than the AR content (high attention area 1209) is hidden by the display of the AR content 1210, and as a result, the AR content is displayed with the highest degree of attention. It becomes possible to do.

  In the example of the present embodiment, the example in which the attention regions are adjacent to each other has been described, but the attention regions may not be adjacent. When the attention area is not adjacent, the AR content can be changed so as to include all the attention areas including the area that is not the attention area.

  In addition, when compared with AR content, there is a possibility that the attention level of a meaningful area is high. The area is specified for each AR content in advance so that the meaningful area is not hidden, and the area is excluded from the area to be hidden.

  For the displayed AR content, a button for non-display is presented together with the AR content. When the user desires to hide the AR content, the AR content can be hidden by instructing this button.

  As described above in detail, by dynamically changing the AR content based on the information of the captured paper document, it is possible to display the AR content easily in various browsing environments and browsing targets. It becomes possible.

  In this embodiment, a paper document is used as a target for calculating the degree of attention of imaging information, but different information may be included. For example, it is possible to calculate the attention level including information such as a table on which a paper document is placed and a landscape. Also in this case, attention level calculation can be realized by using the method described in the present embodiment for image information including information such as a table and a landscape.

  Moreover, although the method of extracting the feature-value with respect to an image was shown as a feature-value extraction method, you may use a different method. For example, characters and images can be recognized and the recognized result can be handled as a feature amount.

  In the present embodiment, a single feature amount is used, but a plurality of feature amounts may be used. For example, the type of feature quantity such as a feature quantity for a natural image such as a landscape photograph and a feature quantity for a character string image may be changed according to the content. When there is content that allows easy extraction of feature amounts for each type of feature amount, the accuracy of AR content specifying processing can be improved by using a plurality of feature amounts.

<Embodiment 2>
In the first embodiment, the method of Non-Patent Document 2 described above has been described as a method of calculating the attention level of a captured paper document. In the present embodiment, a method of using the user's line-of-sight information will be described as a method of calculating the attention level of a captured paper document.

  Specifically, the attention level calculation method described in step S9010 of FIG. 9 will be described as a method using the user's line-of-sight information. The other processes are the same as those in the first embodiment, and will be omitted. The hardware configuration of the AR processing system is the same as that of the first embodiment, and the same functional units as those of the first embodiment are included.

  In the attention level calculation method using the user's line-of-sight information, the user's line-of-sight information is acquired by the imaging unit 303 in FIG. 3, and an area where the acquired line-of-sight information is concentrated is specified as a high-attention area. As described in the first embodiment, the imaging information area is divided into predetermined areas, and the line-of-sight information of each area is acquired. When the line-of-sight information in each area is collected for a specified number of times or for a specified time or more, it is specified as the attention area.

  As described above, the AR content can be changed according to the degree of attention of the user by specifying the attention area using the user's line-of-sight information. By changing the AR content in accordance with the degree of attention of the user, there is no need to change the AR content that the user has already noticed, and a display that does not give the viewer a sense of incongruity is realized. Calculation load can be reduced.

<Embodiment 3>
In the first embodiment and the second embodiment, as a method for changing the AR content, a method for determining the display position / display size of the AR content so that an area having a higher degree of attention than the AR content cannot be seen has been described. In the present embodiment, a method for changing the AR content by changing the display condition of the color of the AR content and improving the degree of attention of the AR content will be described.

  Specifically, a method for realizing the AR content changing method described in step S11030 of FIG. 11 by changing the color of the AR content will be described. The other processes are the same as those in the first embodiment, and will be omitted. The hardware configuration of the AR processing system is the same as that of the first embodiment, and the same functional units as those of the first embodiment are included.

  In the process of changing the color of the AR content, the color of the AR content is changed so that the attention level of the AR content is higher than the attention level associated with each area of the attention area DB described above. As described in Non-Patent Document 2, the degree of attention can be calculated based on color. Using the calculation formula, it is possible to change the AR content so that the degree of attention is high. For example, if redness is strong, the degree of attention calculated by Expression (8) increases, so the redness of the AR content is increased.

  As described above in detail, by changing the color of the AR content, it is possible to increase the degree of attention of the AR content as compared with a relatively captured paper document. By increasing the attention level of the AR content by changing the color of the AR content, the AR content can be focused without hiding other imaging information with a high level of attention.

  In this embodiment, the color of the AR content is changed, but other parameters used in the existing attention level calculation technique may be used. Specifically, it is possible to improve the degree of attention by changing the size of the AR content without hiding other imaging information with a high degree of attention. As an example of calculating the degree of attention using the size, the following formula (9) described in Non-Patent Document 2 described above can be used. In Equation (9), pe is the degree of attention, m is the area of the object or the captured image (or each region obtained by dividing the captured image into a plurality of regions), and ar is the area of the entire captured image.

  In addition to the size, it is possible to improve the degree of attention by giving a display effect such as blinking AR content.

<Embodiment 4>
In the present embodiment, the AR content is projected onto a flat work table on which a paper document is mounted.

  FIG. 13A is a hardware configuration diagram of the information processing apparatus 1300 in the present embodiment. The CPU 1301 is a CPU (Central Processing Unit), performs calculations and logic determinations for various processes, and controls each component connected to the system bus 1307. A ROM (Read-Only Memory) 1302 stores program data for control by the CPU including various processing procedures shown in flowcharts to be described later. A RAM 1303 (Random Access Memory) 1303 is a data memory, and includes a work area for the program of the CPU 1301, a data save area during error processing, a load area for the control program, and the like. The storage device 1304 includes a hard disk, a storage device connected to the outside, and the like, and stores various data and programs such as electronic data according to the present embodiment. The CPU 1301 reads out and executes processing steps and programs stored in the ROM 1302 or the storage device 1304. These processing steps and programs can be read from a storage medium such as a disk device or acquired by network communication and stored in the ROM 1302 or the storage device 1304. At that time, the CPU 1301 functions as a controller. In the present embodiment, the CPU 1301 operates as each functional unit of the information processing apparatus 1300 by executing the steps shown in the flowchart. The imaging device 1305 images a work space in which a user operates, and provides the system as an input image. The projection device 106 projects an image including electronic data and user interface components onto a workbench 1321 described later. In this embodiment, the imaging device 1305 and the projection device 1306 are configured in the information processing device 1300, but may be external devices connected by a wired or wireless interface.

  Here, FIG. 13B is a diagram illustrating an example of an appearance of the information processing apparatus 1300 according to the present embodiment. In the present embodiment, the imaging device 1305 is installed on the upper side of the workbench 1321, takes a bird's-eye image of physical data 1322 such as paper placed on the workbench 1321, and reads the reading target surface facing upward. Recognize information. Similarly, the projection apparatus 1306 projects a projection image including an image constituting the user interface, electronic data 1323 generated by the information processing apparatus 1300, and the like on the upper surface of the work table 1321 or the physical data 1322. In the present embodiment, the imaging device 1305 and the projection device 1306 are configured with the same casing. However, as long as the information processing apparatus grasps in advance the information on the installation position and imaging angle of view of the imaging apparatus 1305 and the position and projection angle of view of the projection apparatus 1306, they may be installed in different places.

  The imaging device 1305 corresponds to the imaging unit 301 of FIG. The projection device 1306 projects and displays the AR content on the plane of the work table 1321. Each functional configuration of the AR processing apparatus may be provided in the information processing apparatus 1300, or may be provided in an AR processing apparatus different from the information processing apparatus 1301 accessible from the information processing apparatus 1300 via the network interface. Also good.

  The information processing apparatus 1300 performs processing of the entire AR processing system shown in FIG. 4, local feature amount extraction processing shown in FIG. 5, attention degree calculation processing shown in FIG. 9, and AR content change processing shown in FIG. Run.

  Although the embodiment has been described in detail above, the present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist of the present invention. It is also possible to combine the processes of the embodiments.

  The present invention can take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 301 AR display apparatus 302 AR processing apparatus 303 Imaging part 304 Display part 305 Feature-value extraction part 306 Content specific | specification part 307 Attention degree calculation part 308 AR content change part

Claims (21)

Acquiring means for acquiring extended information to be displayed together with the captured image based on feature information of the captured image;
Derivation means for deriving the degree of attention of each of the plurality of region images obtained by dividing the captured image;
Comparison means for comparing the attention level of each of the plurality of region images with the attention level of the extended information;
An image processing apparatus comprising: changing means for changing a display condition of the extended information based on the result of the comparison. The changing means is any one of the If there is a little attention of the extension information than attention area images, the image processing apparatus according to claim 1, wherein changing the display condition of the extension information. Said comparing means includes a respective saliency of said plurality of area image excluding the specific area image, the image processing apparatus according to claim 1 or 2, wherein the comparing the saliency of the extension information. It said acquisition means, according to claim 1 to 3 or 1 and acquires the extended information corresponding to the characteristic information by referring to a table in which the feature information and the identification information of the extended information associated with The image processing apparatus according to item. It said changing means according to any one of claims 1 to 4 any one and changes the display condition and the extension information and display condition of the extension information is obtained by referring to the table associated with Image processing apparatus. The feature information, the image processing apparatus according to any one of claims 1 to 4, characterized in that an image feature amount. The changing means, according to claim 1 to 6 any one and changes the display position of the extended information to a high degree of attention area image than attention of the extension information is the extended information hiding An image processing apparatus according to 1. The changing means, according to claim 1 to 6 any one and changes the display size of the high prominence the expansion information to the region image corresponding extension information hide than attention of the extension information An image processing apparatus according to 1. The changing means, according to claim 1, any of the If there is a little attention of the extension information than attention area images, and changing the hue of the extension information such attention increases 7. The image processing device according to any one of 6 to 6 . The image processing apparatus according to claim 6 , wherein the image feature amount is a feature point obtained by applying a Harris operator and a feature amount corresponding to the feature point. The image processing apparatus according to claim 6 , wherein the image feature amount is a SIFT feature amount or a SURF feature amount. It said deriving means, the attention of the region image, the image processing apparatus according to claim 1 to 6 any one, characterized in that derived using the hue. Said deriving means, the prominence of the area image, the image processing apparatus according to any one of claims 1 to 6, characterized in that derived using the number of times or time the line of sight is directed to each area image . The area image is an image processing apparatus according to claim 1 to 6 any one characterized in that it is obtained by dividing the captured image to a certain size. The area image is an image processing apparatus according to any one of claims 1 to 6, characterized in that it is obtained by dividing into a plurality of kinds of regions including the said captured image character area and an image area . The changed extended information display conditions, the information processing apparatus according to claim 1 to 6 any one characterized by further comprising transmission means for transmitting to the image display apparatus including a display unit. The information processing apparatus according to claim 1 to 6 any one characterized by further comprising a projection means for projecting the extended information that has changed the display condition. 4. The image processing according to claim 3, wherein the specific area image is specified and stored for each extension information in a table in which the extension information and a display condition of the extension information are associated with each other. apparatus. An image processing method for processing a captured image,
An acquisition step of acquiring extended information associated with feature information extracted from the captured image;
A comparison step of comparing each of the attention levels of the region images obtained by dividing the captured image with the attention level of the extended information;
An image processing method comprising: a changing step of changing a display condition of the extended information based on a result of the comparison. Computer
Acquisition means for acquiring extended information associated with feature information extracted from the captured image;
Comparison means for comparing each of the attention levels of the region images obtained by dividing the captured image with the attention level of the extended information;
Based on the result of the comparison, changing means for changing the display condition of the extended information;
Program to function as. An image processing system comprising an image processing device and an image display device,
The image processing apparatus includes:
Acquisition means for acquiring extended information associated with feature information extracted from the captured image;
Comparison means for comparing each of the attention levels of the region images obtained by dividing the captured image with the attention level of the extended information;
Based on the result of the comparison, changing means for changing the display condition of the extended information;
Transmitting means for transmitting the extended information whose display condition has been changed to the image display device;
With
The image display device includes:
Receiving means for receiving the transmitted extension information;
Display means for displaying the extended information superimposed on the captured image under the changed display condition;
An image processing system comprising:

Images