JP2013084313A - Information display system, information display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve convenience of service using AR.SOLUTION: A collation processing unit 113 controls a key information acquisition unit 130, acquires key information such as positional information, and gives notification to an external device by a notification unit 120. The collation processing unit 113 stores the acquired key information in association with identification information acquired from the external device into a storage unit 160. If the acquired identification information is not in a table for recording AR data acquired from the external device, the collation processing unit 113 gives notification of the identification information to the external device and acquires AR data. If there is no empty record in a table for recording key information, key information with an earlier use date becomes a deletion object. However, if the key information shows vicinity of a current position, this key information is excluded from the deletion object. If there is no empty record in the table for recording AR data, AR data with an earlier use date becomes a deletion object. However, if corresponding key information is excluded from the deletion object, this AR data is also excluded from the deletion object.

Description

  The present invention relates to an information display system, an information display device, and a program, and more particularly, to an information display system, an information display device, and a program suitable for information display using AR technology.

  High-performance mobile communication terminals equipped with an imaging function, such as so-called smartphones, are becoming widespread, and services (hereinafter referred to as “AR services”) in which AR (Augmented Reality) technology is applied to such terminal devices. Is realized). In the AR service, the additional information is superimposed in real time on the real image captured by the terminal, so that various information at the location where the image is captured can be easily obtained (for example, Patent Document 1).

  When superimposing information on a real image, if there are a plurality of subjects in the image, it is necessary to align each subject to accurately correspond to the information to be superimposed. As an alignment method, it may be possible to identify the subject by performing image recognition on the photographed photographed image and comparing it with various photographed images stored in a server or database on the network. (For example, Patent Document 2).

  In the case of this method, as the AR service is enriched, various objects become targets of the AR service, and the number of inquiries to the server increases. In addition, an enormous amount of collation images are accumulated in the database, and the amount of information provided in response to an inquiry increases, increasing the traffic load on the network.

  In addition, since the amount of information to be acquired increases in a terminal that uses the provided information, the amount of memory used increases, and the number of objects to be collated also increases, so that collation takes time. As a result, the performance when using the AR service is lowered, and convenience is impaired.

Special table 2008-510254 gazette JP 2009-230255 A

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an information display system, an information display device, and a program capable of improving convenience in AR service.

In order to achieve the above object, an information display system according to a first aspect of the present invention includes:
An information display device that includes an imaging unit and superimposes and displays information provided from outside on a captured image captured by the imaging unit;
An information display system for performing information display (AR display) based on augmented reality (AR), comprising: an information providing device that provides information superimposed on a captured image by the information display device to the information display device;
The information providing apparatus includes:
AR data for identifying AR data including a collation image for collating with a subject in the captured image and superimposition information to be superimposed and displayed on the captured image in which the subject collated with the collation image is captured. AR data storage means for storing identification information in association with key information that can be notified from the information display device;
When key information is provided from the information display device, identification information corresponding to the key information is transmitted to the information display device. When identification information is provided from the information display device, the identification information is provided. Information providing means for transmitting AR data corresponding to the information display device,
The information display device includes:
Key information acquisition means for acquiring key information for specifying a subject in the captured image;
Key information notification means for notifying the information providing apparatus every time the key information is acquired;
Identification information acquisition means for acquiring identification information provided from the information providing device in response to the notification of the key information;
Each time the identification information is acquired, new key information and the identification information are associated with each other and added to the first table in the first table that records a plurality of the key information and the identification information. Key information recording means for recording
Identification information notification means for notifying the information providing apparatus of any identification information recorded in the first table;
AR data obtaining means for obtaining AR data used for the AR display provided from the information providing apparatus in response to the notification of the identification information;
AR display means for performing the AR display using the AR data acquired by the AR data acquisition means,
It is characterized by that.
In the above information display system,
AR data recording means for recording the acquired AR data in association with the first table in a second table in which the number of records is defined,
Preferably, the AR display means performs the AR display using the AR data recorded by the AR data recording means.

In order to achieve the above object, an information display device according to the second aspect of the present invention provides:
Augmented reality (AR) information display comprising: an imaging unit; a communication unit that communicates with an external device; and a display unit that superimposes and displays information provided from the external device on an image captured by the imaging unit. An information display device for performing (AR display),
Key information acquisition means for acquiring key information used to specify information provided by the external device to the information display device;
Key information notifying means for notifying the external device of the key information acquired by the key information acquiring means via a communication network;
The verification image that is transmitted by the external device in response to the key information notification from the key information notification unit and is collated with the subject in the subject imaged by the imaging unit, and the subject that is collated with the verification image Identification information acquisition means for acquiring identification information indicating AR data including superimposition information to be superimposed on a captured image, via the communication network;
The key information acquired by the key information acquisition unit is recorded in the first table in which the number of records is defined in association with the use date and time of the key information and the identification information acquired by the identification information acquisition unit. Key information recording means;
Identification information notification means for notifying the external device of the identification information recorded in the first table via the communication network;
AR data acquisition means for acquiring the AR data transmitted by the external device in response to notification of identification information by the identification information notification means;
AR data recording means for recording the AR data acquired by the AR data acquisition means in a second table in which the number of records is defined in association with the notified identification information and the use date and time of the AR data; ,
AR display means for performing AR display using the AR data recorded in the second table,
The identification information notifying means notifies the external device of the identification information when the identification information recorded in the first table is not recorded in the second table;
The key information recording means performs deletion based on the use date and time of already recorded key information when there is no empty record in the first table when the key information acquiring means acquires key information.
The AR data recording means performs deletion based on the use date and time of already recorded AR data when there is no empty record in the second table when the identification information notification means notifies the identification information.
It is characterized by that.

In the above information display device,
The key information recording means records the date and time when the key information is acquired as a use date and is updated with the date and time when the AR display using the AR data corresponding to the key information is performed,
The AR data recording means records the date and time when the AR data was acquired as a usage date and updated with the date and time when AR display using the AR data was performed,
It is desirable that the key information recording unit and the AR data recording unit specify a record in which the oldest date and time is recorded as the usage date and time as a deletion target.

In the above information display device,
Preferably, the key information includes position information indicating a position of the information display device at the time of imaging by the imaging unit.
The key information recording means calculates the distance between the position indicated by the position information and the current position of the information display device when the key information recorded in the record specified as the deletion target is position information. When the distance is smaller than a predetermined threshold, it is desirable to exclude the record from the deletion target.

In the above information display device,
In the AR data recording means, the identification information recorded in the record specified as the deletion target is recorded in the first table, and the record in which the identification information is recorded in the first table When the key information recording unit excludes the specified record from the deletion target, it is desirable to exclude the specified record from the deletion target.

In order to achieve the above object, a program according to the third aspect of the present invention is:
Augmented reality (AR) information display comprising: an imaging unit; a communication unit that communicates with an external device; and a display unit that superimposes and displays information provided from the external device on an image captured by the imaging unit. To the computer that controls the information display device that performs (AR display)
A function of acquiring key information used to specify information provided by the external device to the information display device;
A function of notifying the acquired key information to the external device via a communication network;
A collation image that is transmitted by the external device in response to the key information notification and is collated with the subject in the subject imaged by the imaging means, and is superimposed on the captured image in which the subject collated with the collation image is shown. A function of acquiring identification information indicating AR data including superimposition information to be transmitted via the communication network;
A function of recording the acquired key information in a first table in which the number of records is defined in association with the use date and time of the key information and the acquired identification information;
When the identification information recorded in the first table is not recorded in the second table that records the identification information acquired from the external device in association with the AR data, the identification information is A function of notifying the external device via a communication network;
A function of acquiring the AR data transmitted by the external device in response to the notification of the identification information;
A function of recording the acquired AR data in the second table in which the number of records is defined in association with the notified identification information and the use date and time of the AR data;
A function of performing AR display using AR data recorded in the second table;
A function for performing deletion based on the use date and time of already recorded key information when there is no empty record in the first table when the key information is acquired;
A function of performing deletion based on the use date and time of AR data already recorded when there is no empty record in the second table when the identification information is notified to the external device;
It is characterized by realizing.

  According to the present invention, it is possible to improve convenience when using a service using AR technology.

It is a mimetic diagram showing an information display system concerning an embodiment of the present invention. 2A and 2B are schematic diagrams illustrating an external configuration of the information display device illustrated in FIG. 1, in which FIG. 1A illustrates an external configuration on the main surface side, and FIG. It is a block diagram which shows the structure of the information display apparatus shown in FIG. It is a functional block diagram which shows the function implement | achieved by the control part shown in FIG. It is a block diagram which shows the structure of the information provision apparatus shown in FIG. It is a functional block diagram which shows the function implement | achieved by the control part shown in FIG. It is a figure which shows the example of the "collation image storage table" stored in the memory | storage part shown in FIG. It is a flowchart for demonstrating "AR display process (1)" concerning Embodiment 1 of this invention. It is a flowchart for demonstrating "AR information provision process (1)" concerning Embodiment 1 of this invention. It is a figure for demonstrating the operation | movement concerning Embodiment 1 of this invention, (a) shows the example of the assumed scene, (b) shows the example of the collation image used. FIG. 9 is a diagram illustrating a screen display example according to “AR display processing (1)” illustrated in FIG. 8, where (a) illustrates an example of distribution display, and (b) illustrates an example in which a guide is displayed on the distribution display. FIG. 9 is a diagram illustrating a screen display example according to “AR display processing (1)” illustrated in FIG. 8, where (a) illustrates a display example when the guide moves, and (b) illustrates an example in which superimposition information is displayed. Show. It is a figure which shows the other example of a guide display, (a) shows the example of the guide display which urges approach to a to-be-photographed object, (b) shows the example of the guide display to urge away from a to-be-photographed object, (c) is An example of a guide display showing a required angle of view is shown. It is a flowchart for demonstrating "AR display process (2)" concerning Embodiment 2 of this invention. It is a figure which shows the example of the "collation image storage table" concerning Embodiment 2 of this invention. 15 is a flowchart for explaining “key information management processing” executed in “AR display processing (2)” shown in FIG. 14; The flowchart for demonstrating "AR data management process" performed by "AR display process (2)" shown in FIG. FIG. 15 is a diagram for describing a table used in “AR display processing (2)” illustrated in FIG. 14, in which (a) illustrates an example of a “key information management table”, and (b) illustrates an “AR data management table”; An example is given. It is a flowchart for demonstrating "AR display process (3)" concerning Embodiment 3 of this invention. FIG. 20 is a diagram for explaining a collation image and 3D modeling data in the “AR display process (3)” illustrated in FIG. 19, in which (a) schematically illustrates a line-of-sight direction with respect to an object, and (b) illustrates a collation image. (C) shows an example of generated 3D modeling data, and (d) shows an example of 3D modeling data in which the line of sight is changed.

  Embodiments according to the present invention will be described below with reference to the drawings.

(Embodiment 1)
An information display system 1 according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of an information display system 1 according to the embodiment.

  As shown in the figure, the information display system 1 according to the present embodiment includes an information display device 100, an information providing device 200, and the like connected by a communication network NW.

  The communication network NW according to the present embodiment is, for example, a WAN (Wide Area Network) including an IP network such as the Internet and a cellular network for mobile communication.

  The information display device 100 according to the present embodiment is a portable terminal device that includes at least a communication function, an imaging function, and a display function, and a so-called smartphone is a preferred form. In this embodiment, it is assumed that the smartphone is as shown in FIG. In this embodiment, an information display system 1 including such an information display device 100 realizes a service (hereinafter referred to as “AR service”) to which AR (Augmented Reality) technology is applied. Shall. That is, information on the subject in the image is superimposed and displayed in real time on a real image captured by the imaging function of the information display device 100.

  The configuration of the information display device 100 will be described with reference to FIGS. 2 and 3. 2A and 2B are schematic views showing the external configuration of the information display device 100. FIG. 2A shows the main surface side of the information display device 100, and FIG. 2B shows the back side of the information display device 100. Is shown. FIG. 3 is a block diagram showing a configuration of the information display device 100.

  As shown in the respective drawings, the information display device 100 according to the present embodiment includes a control unit 110, a communication unit 120, a key information acquisition unit 130, an imaging unit 140, an input / output unit 150, a storage unit 160, and the like. ing.

  The control unit 110 includes, for example, a CPU (Central Processing Unit), a RAM (Random Access Memory), and the like. The control unit 110 controls each unit of the information display device 100 and executes programs as described below. Perform processing.

  The communication unit 120 is configured to perform communication via the communication network NW, and can directly access the WAN such as a cellular communication module such as CDMA2000 or a communication module for IEEE 802.16e (so-called mobile WiMAX). Consists of a communication module and / or an IEEE 802.11 series (wireless LAN) communication module for communicating with, for example, an external communication device (such as a mobile router) that can access the WAN, a public wireless LAN access point, or the like Is done.

  The key information acquisition unit 130 is configured to acquire key information necessary for receiving the AR service. For example, a GPS (Global Positioning System) module, an RFID (Radio Frequency IDentification), an IrDA (Infrared Data Association), a short-range wireless communication module such as Bluetooth (trademark), a visible light communication module, and the like, and obtains key information provided from the outside.

  The “key information” here is, for example, a store where information superimposed on the AR service is registered in the information providing apparatus 200 in addition to position information (latitude / longitude information) indicating the current position of the information display apparatus 100. ID information assigned to various facilities, etc., and the key information acquired by the information display device 100 is transmitted to the information providing device 200, so that it is provided by a collation image or AR service required for alignment. Information selection is made.

  Therefore, a GPS module or the like is used as a configuration for acquiring current position information of the information display device 100. In addition, as a configuration for acquiring ID information that can be acquired at various facilities, a short-range wireless communication module, a visible light communication module, or the like is used. For example, as shown in FIG. 2B, these configurations are appropriately configured on the outer surface of the information display device 100 or the inside thereof. Although not all of these configurations may be provided, in the present embodiment, it is assumed that at least a configuration (such as a GPS module) for acquiring position information is provided.

  In the present embodiment, it is assumed that at least a configuration for acquiring the current position (such as a GPS module) is included in the key information acquisition unit 130, and a configuration for acquiring ID information is arbitrarily provided. And The current position information may be acquired by a method other than GPS, for example, the position information may be acquired from a base station accessed by cellular communication or an access point accessed by a wireless LAN. In this case, the key information acquisition unit 130 includes the communication unit 120 for performing these communications.

  The imaging unit 140 is a configuration for realizing the imaging function of the information display device 100, and includes, for example, a digital camera module using a solid-state imaging device. In the present embodiment, information provided by the AR service is superimposed on a moving image (actual image) as a live view image obtained by the imaging unit 140.

  Note that when the key information described above is provided as a visual object such as a two-dimensional code such as a QR code (trademark), the key information is acquired by the imaging operation of the imaging unit 140. In this case, the key information acquisition unit 130 includes the imaging unit 140.

  The imaging unit 140 is preferably a twin-lens stereo camera as shown in FIG. In other words, the imaging apparatus has two lenses arranged in parallel in the horizontal direction at intervals corresponding to parallax. By using such a stereo camera, distance measurement to a subject and generation of 3D modeling data can be performed based on the principle of triangulation.

  The input / output unit 150 includes at least a display device that displays and outputs a captured image by the imaging unit 140, information provided by the AR service, and an input device that receives an input operation by a user of the information display device 100. In the embodiment, a touch panel (touch screen) that integrally realizes a display function and an input function is configured as the input / output unit 150. The input / output unit 150 also includes a speaker, a microphone, and the like that are used for voice input / output such as a call function in a smartphone.

  The storage unit 160 includes, for example, a semiconductor storage device such as a flash memory. The storage unit 160 includes a program executed by the control unit 110, data used in various processes, data generated by various processes, the communication unit 120, and key information. Data acquired by the operation of the acquisition unit 130 is stored.

  Functions realized when the control unit 110 executes the program stored in the storage unit 160 will be described with reference to FIG. FIG. 4 is a functional block diagram showing functions realized by the control unit 110.

  In the present embodiment, functions such as the AR processing unit 111, the imaging processing unit 112, and the matching processing unit 113 are realized by the control unit 110, as illustrated.

  The AR processing unit 111 executes an application related to the AR service (hereinafter referred to as “AR application”) stored in the storage unit 160 based on an input signal from the input / output unit 150 based on a user operation. The operation is controlled, and image processing necessary for a display operation related to the AR service (hereinafter referred to as “AR display”) is performed and displayed on the input / output unit 150. In this case, AR display is performed by performing image processing using image data or the like stored in the storage unit 160.

  The imaging processing unit 112 controls an operation related to the imaging function of the information display apparatus 100 by controlling the imaging unit 140. In the present embodiment, the imaging operation necessary for the AR application executed by the AR processing unit 111 is controlled.

  The verification processing unit 113 controls the communication unit 120 and the key information acquisition unit 130 to acquire information from the information providing device 200 and various external devices, thereby performing a verification operation necessary for alignment in AR display. Do it. Here, in accordance with execution of the AR application, the key information acquisition unit 130 is controlled to acquire key information, the acquired key information is stored in the storage unit 160, and the acquired key information is stored in the information providing apparatus 200 or the like. The communication unit 120 is controlled to acquire the collation image and the superimposition information that are transmitted and required for executing the AR application from the information providing apparatus 200. Then, the acquired collation image or superimposed image is stored in the storage unit 160, and the captured image acquired by the imaging processing unit 112 is collated with the collation image.

  In the present embodiment, these functions are logically realized by the control unit 110. However, some or all of these functions are hardware such as an ASIC (Application Specific Integrated Circuit). It may be realized by wear.

  Each configuration of the information display device 100 described above is a configuration necessary for realizing the present invention, and other configurations necessary for functioning as a smartphone are appropriately provided.

  Next, the information providing apparatus 200 will be described. The information providing apparatus 200 is an information processing apparatus such as an ASP (Application Service Provider) server, for example. In this embodiment, the information providing apparatus 200 provides an AR service to the information display apparatus 100 connected via the communication network NW. The configuration of such an information providing apparatus 200 will be described with reference to FIG. FIG. 5 is a block diagram illustrating a configuration of the information providing apparatus 200.

  As illustrated, the information providing apparatus 200 includes a control unit 210, a communication unit 220, an input / output unit 230, a storage unit 240, and the like.

  The control unit 210 includes, for example, a CPU, a RAM, and the like, and controls each unit of the information providing apparatus 200 and executes each process described later by executing a program.

  The communication unit 220 is configured to perform communication via the communication network NW. For example, the communication unit 220 includes communication devices such as a NIC (Network Interface Card), a router, and a modem, and transmits and receives data using a suitable communication protocol. To do.

  The input / output unit 230 includes, for example, an input device that performs input in accordance with an operation by the operator of the information providing device 200 such as a keyboard and a mouse, and an output device that outputs processing results, such as a display device. Is done.

  The storage unit 240 is configured by a storage device such as a hard disk device, for example, and stores a program executed by the control unit 210, data necessary for each process performed by the execution of the program, and data generated by the process. Is stored.

  Functions realized when the control unit 210 executes the program stored in the storage unit 240 will be described with reference to FIG. FIG. 6 is a functional block diagram illustrating functions realized by the control unit 210.

  In the present embodiment, functions such as a communication processing unit 211, an information registration unit 212, and an information search unit 213 are realized by the control unit 210 as illustrated.

  The communication processing unit 211 controls the communication unit 220 and performs communication with the information display device 100 via the communication network NW. In this embodiment, in addition to receiving the key information transmitted from the information display device 100, the information retrieved by the information retrieval unit 213 described later is transmitted to the information display device 100.

  The information registration unit 212 performs processing for registering information necessary for providing the AR service in the storage unit 240. An example of information registered in the storage unit 240 is shown in FIG. In the present embodiment, a “collation image storage table” as shown in the figure is created in the storage unit 240, and various information necessary for providing the AR service is stored.

  In this collation image storage table, information serving as an index of each collation image is added to a plurality of collation images prepared for collating subjects captured in a real image captured by the information display device 100 using the AR service. (Hereinafter referred to as “index information”), information to be superimposed and displayed on the subject to be collated by the collation image (hereinafter referred to as “superimposition information”), and the like are stored in association with each other.

  Here, the “collation image” in the present embodiment is an image in which an object on which information is to be superimposed by the AR service is shown as a subject. For example, the appearance of a building such as a store is the subject. Or an image in which an object such as a product is a subject is registered as a collation image.

  The index information associated with such a collation image is, for example, position information (latitude / longitude information) indicating a place where the subject is present. These pieces of information can be obtained as additional information of Exif (Exchangeable Image File Format), for example, by using a captured image obtained by an imaging device having a GPS function as a collation image, and also when registering in the storage unit 240 Input is performed by the operator of the providing device 200 operating the input / output unit 230.

  In addition to such index information, “scale information” and “unit scale” as shown in FIG. 7 are recorded in the collation image storage table.

  The scale information is information used to specify the scale of the subject among the shooting data related to the collation image. Here, the image size (number of pixels) of the image, the size of the image sensor (sensor size), The distance to the subject, the focal length of the lens, etc. are recorded as scale information. These pieces of information are acquired from additional information such as Exif data relating to the collation image, and are recorded by an input by an operator operation.

  The scale information includes information indicating the angle of view. The angle of view is calculated from the scale information and recorded. In this case, for example, the angle of view [degree] is calculated by calculating Formula 1.

(Equation 1)
2 x tan <sup> -1 </ sup> {sensor size [mm] / (2 x focal length [mm])}

  Here, the size of the image sensor is either a vertical size or a horizontal size, and the focal length is, for example, a value converted into a lens focal length in a 35 mm format in a still camera.

  The unit scale is information indicating the scale of the subject calculated based on the scale information. In the present embodiment, for example, an actual size corresponding to one pixel (pixel) is recorded as the unit scale. Such calculation of the unit scale is performed by the information registration unit 212 when the collation image is registered. Alternatively, a value calculated in advance may be input by the operator.

  The unit scale is calculated in the following procedure, for example. First, the size of the subject in the collation image is calculated by calculating Formula 2 from the distance to the subject, the size of the image sensor, and the focal length of the lens.

(Equation 2)
(Distance to subject [mm] x sensor size [mm]) / focal length [mm]

  Here, the size of the image sensor is either vertical or horizontal, and the focal length is, for example, a value converted into a lens focal length in a 35 mm format in a still camera. By the above calculation, the actual size [mm] of the subject in the direction (vertical or horizontal) of the substituted image sensor size is specified.

  By dividing the subject size calculated in this way by the image size (number of pixels) in the direction of the imaging size (vertical or horizontal) used for the calculation, an actual size corresponding to one pixel is calculated.

  The information search unit 213 searches the collation image storage table based on the key information from the information display device 100 received by the communication processing unit 211, and acquires a corresponding record. Here, position information indicating the current position of the information display device 100 is acquired as key information, and a record in which the acquired key information is index information is specified. Among the information recorded in the specified record, at least image data of the collation image, superimposition information, scale information, unit scale, etc. (hereinafter referred to as “AR data”) is displayed by the communication processing unit 211 as an information display device. 100.

  In the present embodiment, each of the above functions is logically realized by the control unit 210, but some or all of these functions may be realized by hardware such as an ASIC.

  An example of operations performed by the information display device 100 and the information providing device 200 configured as described above will be described below.

  First, the operation of the information display apparatus 100 will be described with reference to the flowchart shown in FIG. FIG. 8 is a flowchart for explaining the “AR display process (1)” executed in the information display apparatus 100 according to the present embodiment.

  When the AR application is started by the AR processing unit 111, the AR processing unit 111 notifies the imaging processing unit 112 and the matching processing unit 113 to that effect. Upon receiving the notification, the imaging processing unit 112 controls the imaging unit 140 to start an imaging operation (step S1101).

  When the imaging operation is started, the collation processing unit 113 controls the key information acquisition unit 130 to perform the key information acquisition operation. Here, it is assumed that at least acquisition of position information is performed by a GPS module or the like, and acquisition of other key information is attempted when acquisition of position information is successful. If the location information cannot be acquired, the AR service cannot be received, and error processing such as waiting until it can be acquired or terminating this processing is performed.

  When key information including at least position information is acquired, the collation processing unit 113 records the acquired key information in the storage unit 160 (step S1102). The key information recorded here is deleted at the end of this process.

  Here, the collation processing unit 113 determines whether or not there is a change in the recorded key information (step S1103). When key information is acquired immediately after the start of processing, since there is no previous recording, it is determined that there is “variation” (step S1103: Yes).

  In this case, the collation processing unit 113 controls the communication unit 120 to transmit the acquired key information to the information providing apparatus 200 via the communication network NW (step S1104).

  When the information providing apparatus 200 receives the key information transmitted from the information display apparatus 100 in this manner, the information providing apparatus 200 performs “AR information provision processing (1)” for transmitting the AR data to the information display apparatus 100. Executed. The “AR information providing process (1)” executed by the information providing apparatus 200 according to the present embodiment will be described with reference to the flowchart shown in FIG.

  When the communication processing unit 211 receives the key information transmitted by the information display device 100, the information search unit 213 acquires information corresponding to the index information in the collation image storage table from the received key information (step S1201). Here, position information indicating a shooting location on the information display device 100 is acquired from the key information.

  The information search unit 213 searches the collation image storage table of the storage unit 240 using the acquired index information (position information) as a search target (step S1202), and whether there is a record in which position information corresponding to the shooting location is recorded. Determination is made (step S1203). In the search based on the position information, a record in which position information indicating an approximate value in a predetermined range is recorded so that the vicinity of the shooting location is included instead of the complete match of the position information is searched.

  When there is a record in which position information corresponding to the shooting location by the information display device 100 is recorded (step S1203: Yes), the information search unit 213 acquires AR data from the record (step S1204), and the communication processing unit 211. To hand over.

  The communication processing unit 211 controls the communication unit 220, transmits the AR data acquired by the information search unit 213 to the information display device 100 via the communication network NW (step S1205), and ends the process.

  On the other hand, when the record in which the position information corresponding to the shooting location is recorded does not exist in the collation image storage table (step S1203: No), the information providing apparatus 200 ends the process without transmitting the AR data to the information display apparatus 100. . In this case, the information providing apparatus 200 may transmit notification information indicating that the AR service cannot be provided at the current position to the information display apparatus 100.

  The transmission / reception operation with the information display apparatus 100 as described above is realized by using a known transaction technique using, for example, a CGI (Common Gateway Interface) technique.

  Returning to the flowchart of FIG. 8, the operation of the information display apparatus 100 to which the AR data has been returned will be described.

  When the AR data is transmitted from the information providing apparatus 200, the communication unit 120 of the information display apparatus 100 receives the communication data via the communication network NW. When such reception is made (step S1105: Yes), the collation processing unit 113 stores the received AR data in the storage unit 160 and notifies the AR processing unit 111 accordingly.

  The AR processing unit 111 controls the input / output unit 150 to display and output the live view image currently captured by the imaging unit 140, and based on the unit scale of the received AR data, An image showing the distribution is generated and displayed on the input / output unit 150 (step S1106).

  Here, processing contents will be described with reference to the drawings based on examples of possible scenes. In the present embodiment, it is assumed that the AR service is used in a city where stores A to E are located as shown in FIG. Here, for the store A, a collation image or the like is registered in the information providing apparatus 200, and the AR service can be provided. However, for other stores B to E, information is registered in the information providing apparatus 200. Therefore, it is assumed that the service is not covered by the AR service.

  And about the shop A, the collation image as shown in FIG.10 (b) shall be registered into the information provision apparatus 200. FIG. As shown in the figure, here, in addition to the collation image AA in which the appearance of the store A is photographed, collation images AB to AZ in which products sold in the store A are photographed are registered in the information providing apparatus 200. To do. The image sizes of these verification images are all the same.

  In the collation image, it is assumed that only the subject subject is photographed so as to fit within the image size. When a plurality of collation images for a plurality of subjects are prepared in this way, the subject sizes are not uniform, and the subject scale for each image is also different. In other words, the collation image AA is taken with a full view of the exterior of the building, while the collation image AZ is taken with a full view of the pen, which is a product, so the actual size corresponding to one pixel is to differ greatly.

  In this case, if the appearance of the store A is captured to the full angle of view of the live-action image, collation using the collation image AA can be performed, but the scale is too small even if the product pen is reflected there. Therefore, the collation with the collation image AZ cannot be performed. That is, information provided by the AR service cannot be obtained properly unless shooting is performed at an appropriate angle of view according to the subject to be processed.

  In order to eliminate such an inconvenience, in this embodiment, the unit scale registered together with the collation image in the collation image storage table of the information providing apparatus 200 is used to guide the user to shoot at an appropriate angle of view. .

  Therefore, in this embodiment, in step S1106, distribution display based on the unit scale of the collation image received from the information providing apparatus 200 is performed. A display example in this case is shown in FIG. As shown in the drawing, a “captured image display area” and a “guide display area” are set in the input / output unit 150, and a live view image from the image capturing unit 140 is displayed in the captured image display area. In the guide display area, a distribution display as shown in FIG.

  In this distribution display, for example, a double arrow in the vertical direction is displayed, and the distribution of the collation image based on the unit scale is displayed in the form of a graph on the right side. In the example of FIG. 11A, the upper side of the double arrow indicates that the distance from the information display device 100 (= camera) to the subject is far, and the lower side is the distance from the information display device 100 (= camera) to the subject. It is shown that.

  That is, since the unit scale is an actual size corresponding to one pixel, the value is relatively large in an image obtained by photographing the appearance of a building such as the collation image AA (FIG. 10B), and the collation image AZ. In an image obtained by photographing a pen like this, the value is relatively small. Therefore, the larger the unit scale value, the farther away from the subject, and the smaller the unit scale value, the closer to the subject.

  For this reason, in the distribution display shown in FIG. 11A, the larger the unit scale value, the more distant the distribution is displayed above the double-headed arrow, and the smaller the value, the closer the value is. The distribution is displayed on the lower side. The horizontal direction of the distribution display represents the number of images of the unit scale. In the example of FIG. 11A, the collation image corresponding to the distribution on the uppermost side is a collation image AA (FIG. 10A) obtained by photographing the appearance of the store, and the plurality of distributions below it are as follows. A plurality of collation images AB to AZ (FIG. 11B) in which the product is photographed.

  Here, returning to the flowchart of FIG. 8, the description of the processing is continued. As described above, when the distribution display is performed in step S1106, the AR processing unit 111 calculates a unit scale for the image currently captured by the imaging unit 140 (step S1107). Here, the performance information (sensor size, etc.) of the imaging unit 140, the shooting settings at that time (image size, lens focal length, etc.), the distance to the subject obtained by the ranging operation, and the like are used. The unit scale is calculated by the same method as the information registration unit 212 of the information providing apparatus 200. Here, if the imaging unit 140 is a stereo camera, distance measurement using the principle of triangulation can be performed, so that the distance to the subject can be obtained more accurately.

  When the unit scale for the captured image is calculated, the AR processing unit 111 displays a guide as shown in FIG. 11B in the guide display area (step S1108). This guide reflects the calculated unit scale of the captured image and is displayed, for example, on the left side of the double arrow in the distribution display.

  Here, in the live view image displayed in the captured image display area of FIG. 11B, not only the store A that is the target of the AR service but also stores B to E that are not the target of the service are reflected. Wide angle of view. In such a case, even if the store A is included in the angle of view, since the size of the entire image is small, it may differ from the scale of the verification image AA (FIG. 10B) and may not be appropriately verified. .

  In this case, as shown in FIG. 11B, the guide reflecting the unit scale of the captured image is displayed so as to be positioned above the distribution display representing the collation image AA. In the example of FIG. 11B, since the image distribution does not exist at the position of the guide, the user must perform collation for performing the AR display unless the user approaches the subject or zooms to the telescopic side angle of view. You can recognize that you can't.

  As described above, when the current captured image cannot be collated using the collation image acquired from the information providing apparatus 200 (step S1109: No), the process returns to step S1107, and the unit scale of the captured image is calculated again. That is, when the user who recognizes that the collation is not possible by the guide display approaches the subject or zooms to the telephoto side and narrows the angle of view as shown in FIG. A unit scale is calculated (step S1107).

  In this case, guide display is performed according to the calculated unit scale (step S1108), but the guide is displayed so as to move downward as shown in FIG. As shown in the drawing, when the position is moved to a position corresponding to the distribution of the collation image AA, collation with the collation image AA is possible at the current angle of view (step S1109: Yes).

  In this case, the collation processing unit 113 collates the corresponding unit scale collation image (in this case, the collation image AA) with the captured image by performing known image processing in AR display (step S1110). The position of store A in the image is recognized. When collation is performed in this way, the AR processing unit 111 displays the captured image on the full screen as shown in FIG. 12B, for example, and associates it with the collation image AA to provide the information providing apparatus 200. Is superimposed and displayed at the position of the store A in the captured image (step S1111).

  For example, if a predetermined end event such as an AR application end instruction or power-off of the information display apparatus 100 does not occur (step S1112: No), the processing from step S1102 onward is repeated. Here, the key information acquisition operation is performed in step S1102, but if the information display device 100 does not move significantly, it does not vary from the previously acquired key information (step S1103: No). In such a case, the process proceeds to step S1107, and guide display based on the current unit scale (update of the guide position according to the change in the angle of view) is performed.

  On the other hand, when there is a change in the newly acquired key information (step S1103: Yes), the key information is transmitted again to the information providing apparatus 200 (step S1104), and the AR data provided based on the key information is updated. After that, the subsequent processing is performed (step S1105).

  If the information providing apparatus 200 does not have a collation image corresponding to the key information transmitted from the information display apparatus 100, the AR data is not transmitted from the information providing apparatus 200 or the AR service cannot be provided as described above. Notification information to that effect is sent. In such a case (step S1105: No), the process proceeds to step S1112, and an operation according to whether or not an end event has occurred is performed. That is, if the end event does not occur (step S1112: No), the process is repeated from the key information acquisition operation again (step S1102).

  If an end event has occurred (step S1112: Yes), the key information recorded in the storage unit 160 is erased, and the process ends.

  As described above, according to the present embodiment, when AR display is performed on the terminal device, the distribution based on the subject scale of the collation image prepared at the shooting location is displayed, and the subject in the captured image is displayed. Since the scale is displayed as a guide on the distribution display, the angle of view and the shooting position suitable for collation using the provided collation image can be easily obtained, which is convenient when using the AR display. Can be improved.

  In this embodiment, the guide display as shown in FIGS. 11 and 12 is exemplified, but the method and form of the guide display are arbitrary and are not limited thereto. For example, when the subject image in the captured image is small because the subject is away from the subject, an upward arrow image as shown in FIG. 13A is guided to approach the subject or to change the angle of view by zooming in. May be displayed.

  Conversely, when the subject image is not within the angle of view because it is too close to the subject, the subject is moved away or the angle of view is changed by zooming out, as shown in FIG. A downward arrow image may be displayed as a guide.

  Alternatively, as shown in FIG. 13C, a rectangular frame indicating the angle of view necessary for collation may be displayed as a guide, and the angle of view change may be urged so that the subject image fits in the rectangular frame.

  These guide displays are effective, for example, when the unit scales of the captured image and the collation image are close to each other to bring the angle of view closer to an appropriate angle. A corner change can be prompted. Therefore, first, the guide display as illustrated in FIG. 11 may be performed, and the display may be switched to the guide display as illustrated in FIG. 13 when the unit scale is approximated to some extent. Alternatively, when the number of collation images is small, a guide display as shown in FIG. 13 may be used from the beginning.

  In this embodiment, more accurate distance information is obtained by obtaining the distance to the subject necessary for calculating the unit scale by ranging with a stereo camera, but the distance to the subject can be obtained. If so, the method is arbitrary. For example, the focusing distance of the lens may be used. In this case, the imaging unit 140 provided in the information display apparatus 100 may not be a stereo camera as illustrated, and the imaging unit 140 may be configured by a general monocular camera module.

(Embodiment 2)
In the first embodiment, when the information display device 100 acquires key information such as position information by GPS or the like, the information providing device 200 searches the AR data based on the key information and provides it to the information display device 100. However, as the AR service target expands, the AR service target near the current location increases, so the number of inquiries to the information providing apparatus 200 increases and the amount of AR data provided at one time also increases. The traffic load and the amount of memory used in the information display device 100 may increase, leading to a decrease in performance.

  In addition, as described above, since there are a plurality of types of key information acquired by the information display device 100, it may be necessary to alternately collate a large amount of collation images corresponding to the plurality of key information. Similarly, the performance of the information display device 100 is reduced.

  Even when the same type of key information is used, for example, when position information by GPS is used, the user may take an action of returning the route before reaching the target location. In such a case, if the information providing apparatus 200 is accessed according to the change of the position information, the performance of the information display apparatus 100 is lowered, which is not preferable for the convenience of the user.

  In this case, if as many AR data as possible are stored in the information display device 100, the number of accesses to the information providing device 200 can be reduced. However, as long as the verification is performed in the information display device 100, it is stored. If there is a lot of AR data, it will take time for the verification process, so performance degradation is inevitable.

  In the present embodiment, by managing the acquired key information and AR data, the number of inquiries to the information providing apparatus 200 is reduced, and the use of memory in the information display apparatus 100 is improved.

  For this purpose, the “AR display process (2)” executed by the information display apparatus 100 will be described with reference to the flowchart of FIG. This process is executed in place of the “AR display process (1)” of the first embodiment.

  Here, in the first embodiment, position information acquired by GPS or the like is essential as key information. However, in this embodiment, other types of key information are acquired without acquiring position information. It is assumed that an inquiry is made to the information providing apparatus 200 even in the case of the case.

  As described above, the key information may be transmitted from a facility such as a store. For example, if a transmitter using short-distance wireless communication or visible light communication is installed at a storefront of a store and the key information related to AR data about the store is transmitted, the key information is acquired when the store is approached can do. The key information transmitted in this case can be, for example, position information corresponding to GPS, and indicates the facility when the facility registers AR data or the like in the information providing apparatus 200. The identification information may be transmitted as key information.

  In such a case, it is assumed that a “group ID” as shown in FIG. 15 is added to the collation image storage table of the information providing apparatus 200 as index information. By associating a plurality of AR data with one group ID, the search efficiency in the information providing apparatus 200 can be improved.

  In this embodiment, as shown in FIG. 15, it is assumed that unique identification information (AR data identification information) is assigned to each AR data. In this case, when key information is transmitted from the information display device 100, the information providing device 200 according to the present embodiment returns identification information (AR data identification information) corresponding to the key information to the information display device 100. When the identification information (AR data identification information) is transmitted from the information display device 100, the AR data corresponding to the identification information (AR data identification information) is returned to the information display device 100.

  The key information acquired by the information display device 100 is, for example, transmitted from an RFID chip attached to a price card or the like attached to a product sold in a store, or displayed. The key information includes identification information associated with the AR data for the product, including those represented by a two-dimensional code. In this embodiment, such identification information is the same as the AR data identification information described above.

  The operation by the “AR display process (2)” executed under the above assumption will be described. Note that, similarly to the first embodiment, it is assumed that processing is started in response to an instruction to execute an AR application by a user.

  When the processing is started, the imaging processing unit 112 controls the imaging unit 140 to start the imaging operation (step S2101), and the collation processing unit 113 controls the key information acquisition unit 130 to thereby obtain key information. Is acquired (step S2102). The acquired key information is temporarily held in a storage area such as a RAM, for example.

  As in the example of the first embodiment, the acquired key information is recorded in the storage unit 160. In this embodiment, a “key information management table” as shown in FIG. The acquired key information and additional information are stored in this table.

  As shown in FIG. 18A, this key information management table includes a record No. It is composed of records with key as. Here, the number of records constituting the table is defined by, for example, the capacity allocated to the AR application among the capacity of the storage unit 160, the processing capability of the information display apparatus 100, and the like. That is, the number of records is defined such that the key information to be recorded has a data amount that does not deteriorate the performance for executing the AR application. In the present embodiment, for example, it is assumed that the table is composed of n records.

  Further, the number of key information acquired in step S2102 is N (pieces). As described above, since the key information is acquired by a plurality of methods, when the key information acquisition operation is performed at the same place, for example, the position information is acquired by GPS and the group ID is acquired by short-range wireless communication. Since a plurality of pieces of key information, such as being acquired, may be acquired at the same time, the number N of key information is N ≧ 1.

  In such a key information management table, as shown in FIG. 18A, the acquired key information (for example, position information, group ID, identification information, etc.), and the type of method for acquiring the key information are indicated. Information (for example, GPS, visible light communication, RFID, IrDA, Bluetooth (trademark), two-dimensional code, etc.), use date and time, identification information (AR) transmitted from the information providing apparatus 200 for the key information Data identification information), a flag indicating that the record should be excluded from the deletion target (“exclusion flag”), and the like are recorded.

  Here, the date and time when the acquired key information is recorded is recorded in the “use date and time”. In addition, when the key information already recorded in the key information management table is acquired again, it is updated at the date and time at that time.

  As a process for storing key information in such a key information management table, a “key information management process” is executed (step S2200). This “key information management process” will be described with reference to the flowchart shown in FIG.

  When the processing is started, the collation processing unit 113 accesses the key information management table of the storage unit 160, and when the same key information as the key information held in the RAM is stored in the key information management table, “Use date and time” for the key information is updated to the current date and time (step S2201). Here, among the key information acquired this time that is stored in the RAM, the key information already recorded in the key information management table is deleted from the RAM after the use date is updated.

  The key information remaining in the RAM after deleting the already recorded key information from the RAM is the key information not recorded in the key information management table. If the key information acquired this time is not recorded in the key information management table (step S2202: No), the process is terminated simply by updating the use date and time, and the “AR display process (2)” ( Returning to the flow of FIG.

  On the other hand, when there is unrecorded key information in the key information management table (step S2202: Yes), the collation processing unit 113 determines whether or not such unrecorded key information can be added to the key information management table. Is discriminated (step S2203). As described above, the number of records in the key information management table according to the present embodiment is limited to n based on the capacity of the storage unit 160 and the like. If there are one or more empty records among such n records, it is determined that the key information can be added (step S2203: Yes).

  In this case, for one of the key information remaining in the RAM, the verification processing unit 113 sets the key information, information indicating the acquisition method, and information indicating the current date and time as the acquisition date and time as one of the empty records. (Step S2209). In this case, the collation processing unit 113 deletes the key information on the RAM.

  When one of the key information acquired this time is recorded in the key information management table, the verification processing unit 113 controls the communication unit 120 to transmit the key information to the information providing apparatus 200 via the communication network NW. Corresponding identification information (AR data identification information) is acquired from the information providing apparatus 200 via the communication network NW (step S2210).

  In this case, in the information providing apparatus 200, when the communication unit 220 receives the key information transmitted from the information display apparatus 100, the information search unit 213 accesses the storage unit 240, and from the collation image storage table shown in FIG. Then, the identification information (AR data identification information) associated with the received key information is searched, and the communication unit 220 transmits it to the information display device 100.

  When the information providing apparatus 200 transmits identification information (AR data identification information) in this manner, the communication unit 120 of the information display apparatus 100 receives the identification information. The collation processing unit 113 records the received identification information (AR data identification information) in the key information record recorded in the key information management table in step S2209 (step S2211).

  When the recording of the key information in one record is completed, the collation processing unit 113 has not recorded in the key management information table by determining whether or not the key information acquired this time remains on the RAM. It is determined whether or not all key information has been recorded (step S2212).

  If the key information acquired this time still remains in the RAM (step S2212: No), the process returns to step S2203. Here, when information is recorded in all n records, the collation processing unit 113 determines that the key information cannot be added (step S2203: No), and deletes the contents of one of the records to make room. Prepare a record.

  In this case, the collation processing unit 113 scans the usage date and time recorded in the key information management table, and identifies the record in which the oldest usage date and time is recorded (step S2204). And it is discriminate | determined whether the key information currently recorded on the record is things other than position information (group ID, identification information, etc.) (step S2205).

  When the key information other than the position information is recorded (step S2205: Yes), the collation processing unit 113 deletes the content of the record (step S2208). As a result, one vacant record is prepared, and information corresponding to the vacant record is recorded by the processing in step S2209 and subsequent steps.

  On the other hand, when the key information of the record in which the oldest usage date and time is recorded is position information (step S2205: No), the collation processing unit 113 is recorded in the record from the position information and the current position. It is determined whether or not the position information is close to the current position (step S2206).

  Here, if the position information is included in the key information acquired this time, the position indicated by the position information is set as the current position. If the position information is not included, the current position information is controlled by controlling the key information acquisition unit 130. Acquire. Then, the distance between the two points is calculated from the latitude and longitude indicating each position, and the calculated distance is compared with a predetermined threshold value to determine whether or not it is close to the current position. The threshold in this case is arbitrary, and may be set arbitrarily by the user in addition to being set by the manufacturer of the information display device 100.

  By such an operation, if the position information recorded in the record with the oldest usage date and time indicates a position close to the current position (step S2206: Yes), the collation processing unit 113 deletes the record. (Step S2207), and the process returns to step S2204. In this case, for example, a flag (exclusion flag) indicating that the record should be excluded from the deletion target is set in the record.

  On the other hand, if it is a record in which position information indicating a position far from the current position is recorded (step S2206: No), the process proceeds to step S2208, and the contents of the record are deleted. And various information is recorded on the said record by performing the process after step S2209.

  In this way, when all the key information acquired this time is recorded in the key information management table (step S2212: Yes), this process is terminated, and the flow of “AR display process (2)” (FIG. 14). Return to.

  Here, in the present embodiment, it is assumed that AR data acquired from the information providing apparatus 200 is recorded in an “AR data management table” as illustrated in FIG. 18B created in the storage unit 160.

  As shown in the figure, in the “AR data management table”, the record No. Is used as a key. In each record, identification information (AR data identification information) corresponding to the key information is recorded, and AR data (that is, AR data identification information) corresponding to the identification information (AR data identification information) is recorded. Collation image, superimposition information, scale information, unit scale information, etc.), information indicating the use date and time of the AR data, and the like are recorded. Here, in “use date and time”, the date and time when the AR data is acquired from the information providing apparatus 200 is recorded.

  Note that the number of records constituting the AR data management table is also defined by the capacity of the storage unit 160 allocated to the AR application, the processing capability of the information display device 100, and the like. That is, the number of records is defined so that the amount of information that does not decrease the performance for executing the AR application is recorded. In the present embodiment, it is assumed that the table is composed of m records.

  After executing the “key information management process” (step S2200) described above, in the “AR display process (2)”, an “AR data management process” for recording AR data in such an AR data management table is executed. (Step S2300). This AR data management processing will be described with reference to the flowchart shown in FIG.

  When the processing is started, the collation processing unit 113 first accesses the key information management table (FIG. 18A) of the storage unit 160, and the date and time corresponding to the current key information acquisition operation is recorded as the use date and time. And the identification information (AR data identification information) recorded in the record is specified. That is, identification information (AR data identification information) associated with the key information acquired this time is specified. In this case, the collation processing unit 113 acquires the identified identification information (AR data identification information) from the key information management table (step S2301), and temporarily holds it on the RAM.

  Next, the matching processing unit 113 accesses the AR data management table (FIG. 18B) in the storage unit 160, and specifies a record in which the identification information (AR data identification information) acquired in step S2301 is already recorded. The “use date / time” of the record is updated to the current date / time (step S2302). That is, if the AR data associated with the key information acquired this time has already been recorded in the AR data management table, it is assumed that the AR data has been acquired from the information providing apparatus 200 at this time, and “use date / time” is set. Update.

  When the “use date / time” is updated for the identification information (AR data identification information) indicating the AR data recorded in the AR data management table, the identification information is deleted from the RAM. If the identification information remains in the RAM after such deletion, the AR data indicated by the identification information is not recorded in the AR data management table.

  If no identification information remains on the RAM, all AR data corresponding to the key information acquired this time is already recorded in the AR data management table. In this case (step S2303: No), the process is ended only by updating the “use date / time” for the AR data corresponding to the key information acquired this time, and the “AR display process (2)” (FIG. 14) is completed. Return to flow.

  On the other hand, when there is AR data not recorded in the AR data management table (step S2303: Yes), the collation processing unit 113 determines whether it can be added to the AR data management table (step S2304). As described above, the AR data management table according to the present embodiment is limited to m records based on the capacity of the storage unit 160 and the like. If there are one or more empty records among these m records, it is determined that AR data can be added (step S2304: Yes).

  In this case, the matching processing unit 113 selects one of identification information (AR data identification information) held in the RAM. Then, by controlling the communication unit 120 and transmitting the selected identification information (AR data identification information) to the information providing apparatus 200 via the communication network NW, it is associated with the identification information (AR data identification information). AR data is acquired from the information providing apparatus 200 (step S2310).

  At this time, in the information providing apparatus 200, when the communication unit 220 receives the identification information (AR data identification information) transmitted from the information display apparatus 100, the information search unit 213 accesses the storage unit 240, as shown in FIG. The AR data associated with the received identification information (AR data identification information) is searched from the collation image storage table, and the communication unit 220 transmits the AR data to the information display device 100.

  When the information providing apparatus 200 transmits AR data in this way, the communication unit 120 of the information display apparatus 100 receives the AR data. The collation processing unit 113 records the received AR data in one of the empty records (step S2311). The collation processing unit 113 also records information indicating the current date and time at this time as the use date and time of the AR data. At this time, the collation processing unit 113 deletes the identification information (AR data identification information) for the AR data recorded in the AR data management table from the RAM among the identification information (AR data identification information) held in the RAM. .

  The collation processing unit 113 determines whether or not the identification information (AR data identification information) remains in the RAM, whereby all of the AR data not recorded in the AR data management table is stored in the AR data management table. It is determined whether or not it has been recorded (step S2312).

  If identification information (AR data identification information) remains on the RAM (step S2312: No), the process returns to step S2304, and AR data is acquired and recorded for the identification information (AR data identification information). . Here, when information is recorded in all m records of the AR data management table, the collation processing unit 113 determines that AR data cannot be added (step S2304: No), and determines the contents of any record. Prepare empty records by deleting them.

  In this case, the collation processing unit 113 scans the use date and time recorded in the AR data management table, and specifies the record in which the oldest use date and time is recorded (step S2305). Then, identification information (AR data identification information) recorded in the record is specified.

  Next, the verification processing unit 113 accesses the key information management table and determines whether there is a record in which the identified identification information (AR data identification information) is recorded. That is, it is determined whether or not the key information corresponding to the AR data with the oldest usage date remains in the current key information management table (step S2306).

  When the corresponding key information does not remain in the key information management table (step S2306: No), the collation processing unit 113 deletes the content of the record specified in step S2305 from the AR data management table (step S2307). . As a result, one empty record has been prepared, and information corresponding to the record is recorded by the processing from step S2310.

  On the other hand, if the key information corresponding to the AR data with the oldest usage date remains in the current key information management table (step S2306: Yes), the collation processing unit 113 displays “exclusion flag” in the key information record. Is determined, it is determined whether or not the key information has been excluded from the deletion target in step S2207 of the “key information management process” (FIG. 16) ( Step S2308).

  Here, when the key information corresponding to the AR data identified in step S2305 has been excluded from the deletion target in the key information management table (step S2308: Yes), the collation processing unit 113 determines the record identified in step S2305. Are excluded from the deletion targets (step S2309), and the process returns to step S2305. In this case, for example, it can be indicated by setting a flag indicating that the record is excluded, or may be excluded by updating the use date and time of the record to the current date and time.

  By the above-mentioned “key information management process” (FIG. 16), the key information management table remains so that key information acquired relatively recently, or key information indicating a position close to the current position but old in use date and time remains. Is managed. Therefore, if the key information corresponding to the AR data specified in step S2305 remains in the key information management table and has been excluded from the deletion target, it is acquired this time even if the use date of the AR data is old. There is a high possibility that it will be used for AR display using key information or AR display using key information that can be acquired in the near future. Therefore, such AR data is left in the AR management table by being excluded from the deletion target from the AR data management table.

  After such an exclusion process, the process returns to step S2305, a record that can be deleted is searched, and the contents are deleted (steps S2305 to S2307). Then, newly acquired AR data is recorded in the empty record prepared in this way (steps S2310 to S2311).

  On the other hand, even if the corresponding key information remains in the key information management table, it is not excluded from the deletion target (step S2308: No), the key information remains in the key information management table. Among them, the use date is relatively old and not close to the current position, and the corresponding AR data has the oldest use date. For this reason, it is considered that the AR data recorded in the record identified in step S2305 is unlikely to be used this time or in the near future, and the contents of the record are deleted to prepare an empty record (step S2307).

  The above operation is repeated until all target AR data is recorded (step S2312: No), and when all AR data is recorded (step S2312: Yes), this process is performed. Then, the process returns to the flow of “AR display process (2)” (FIG. 14).

  In the “AR display process (2)”, AR display using AR data corresponding to the key information acquired in step S2102 is performed. Here, the AR display using the acquired AR data is performed by performing the same operations as in Steps S1106 to S1111 of the “AR display process (1)” (FIG. 8) described in the first embodiment ( Step S2103).

  If a predetermined end event does not occur (step S2104: NO), the AR display is performed by repeatedly executing the processing after step S2102. Then, when an end event occurs (step S2104: Yes), this process ends.

  As described above, according to the present embodiment, the key information and AR data held in the information display device 100 are displayed in an efficient AR display in accordance with the memory resources and processing capabilities available in the information display device 100. In addition to being appropriately managed so as to contribute, it is possible to effectively reduce data transfer for providing AR data from the information providing apparatus 200 to the information display apparatus 100. Therefore, in addition to reducing the traffic load, the information display apparatus 100 performance degradation can be prevented, and the convenience of using AR services can be improved.

  That is, when key information is acquired by the information display device 100, only identification information corresponding to the key information is acquired from the information providing device 200. Therefore, AR data having a relatively large data size such as a collation image or the like. Without transfer. Then, based on the acquired identification information, the AR data is received from the information providing apparatus 200 only when the AR data corresponding to the AR data held by the information display apparatus 100 is not included. Thereby, compared with the case where AR data is acquired from the information providing apparatus 200 every time key information is acquired, the data transfer amount on the communication network NW can be significantly reduced.

  Also, in the information display device 100, key information acquired relatively recently and key information indicating position information close to the current position are retained preferentially, and AR data corresponding to these key information is stored in the information display device 100. Since it is managed so as not to be deleted, AR data that is likely to be used now or in the near future will be preferentially held in the information display device 100, and information for acquiring AR data The number of transactions with the providing device 200 can be further reduced.

  In this embodiment, when the AR data corresponding to the key information acquired at that time has already been recorded in the AR data management table, the “use date / time” for the AR data is updated (“ The “use date and time” may be updated when the AR data recorded in the AR data management table is used for the AR display in step S2302 of the “AR data management process” (FIG. 17). Thereby, the AR data actually used for the AR display can be preferentially held in the information display device 100.

  When the key information is deleted from the key information management table (step S2208 of the “key information management process” (FIG. 16)), the AR data associated with the key information is deleted from the AR data management table. In other words, when the AR data is deleted from the AR data management table (step S2307 in the “AR data management process” (FIG. 17)), the key associated with the identification information (AR data identification information) of the AR data. Information may be deleted from the key information management table. Thereby, when the key data or the AR data is to be deleted, the AR data or the key information corresponding to each other is also deleted. Therefore, in both the key information management table and the AR data management table, The possibility that an empty record is prepared in advance can be increased.

(Embodiment 3)
In the second embodiment, the group ID is exemplified as the key information transmitted from the facility such as the store. However, not only the key information but also AR data (collation image) can be acquired on the spot by such a method. You may do it. In this case, for example, even when position information cannot be obtained by GPS, AR data necessary for using the AR service can be obtained, and AR data can be obtained without accessing the information providing apparatus 200 or the like. This can also contribute to the reduction of traffic load in the communication network NW.

  In this mode, a short-range wireless communication device or a visible light communication device installed in a facility such as a store functions as the information providing device 200, but is different from the information providing device 200 exemplified in the above embodiments. Since only the information on the installation location needs to be stored and transmitted, it can be realized in a small and simple manner, and a large amount of AR data can be distributed.

  As the collation image provided in this way, in each of the above embodiments, a so-called 2D image such as a photographic image is used. In this case, the collation may not be performed successfully unless the image is taken from a line-of-sight direction that matches the line-of-sight direction from the camera that has photographed the collation image. In particular, when AR display is performed for each of a large number of products in a store or the like, if the interior of the store is narrow or an object is close, the line-of-sight direction of the verification image is not necessarily the same. It is not always possible to shoot. In addition, since the collation operation is performed in the background, the user does not recognize what collation image is used, and therefore, it is usually not possible to actively perform shooting according to the line-of-sight direction of the collation image. .

  Here, in the case of the information display apparatus 100 in which the imaging unit 140 is configured with a stereo camera, 3D modeling data can be generated from the captured image. Therefore, by collating such 3D modeling data with the verification image, Such problems can be solved. In this embodiment, an operation example in this case will be described below.

  The “AR display process (3)” when the information display apparatus 100 performs collation using the 3D modeling data will be described with reference to the flowchart shown in FIG. In this embodiment, it is assumed that AR data is provided to the information display device 100 by a short-range wireless communication device or a visible light communication device installed in a facility such as a store.

  When the processing is started, the imaging processing unit 112 controls the imaging unit 140, which is a stereo camera, and the imaging operation is started (step S3101). In the present embodiment, it is assumed that the storage unit 160 includes a storage device in which image data is expanded for image processing, such as a VRAM (Video RAM). In this case, the collation processing unit 113 extracts the subject from the captured image and performs depth estimation on the image data developed in the storage unit 160 using a known technique in 3D modeling. 3D modeling data is generated (step S3102).

  When the 3D modeling data is generated, the matching processing unit 113 uses, for example, a SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust) for a subject for which 3D modeling data is generated from a captured image using a known technique in image processing. The feature points are extracted and the feature amount is calculated by “Features” (step S3103).

  In addition, the verification processing unit 113 controls the key information acquisition unit 130 to acquire AR data transmitted from a short-range wireless communication device or a visible light communication device installed in a facility such as a store. Is recorded in the storage unit 160 (step S3104). When recording the AR data, the AR data is recorded in an AR data management table as exemplified in the second embodiment.

  When a plurality of collation images included in the acquired AR data are included, the collation processing unit 113 sequentially collates the generated 3D modeling data with these collation images, so that the index number i of the collation image to be processed is set. After setting the initial value “0” (step S3105), the value of i is incremented by 1 (step S3106), and the first collation image is designated.

  The matching processing unit 113 performs the same processing as the processing in step S3103 on the matching image (i-th matching image) stored in the storage unit 160, thereby extracting feature points by SIFT or SURF. And a feature amount are calculated (step S3107).

  Here, in the case where the currently imaged subject is a three-dimensional object as shown in FIG. 19A, the prepared collation image is from the direction indicated by “line of sight of collation image” in FIG. It is assumed that the photographed image (2D image) as shown in FIG. Here, assuming that the current imaging direction by the information display device 100 is the direction indicated by “current camera line of sight” in FIG. 19A, the 3D modeling data generated in step S3102 is as shown in FIG. It will be as shown in

  In this case, since the line-of-sight of the verification image is different from the line-of-sight in the current imaging, the generated 3D modeling data cannot be verified using the verification image. Therefore, the matching processing unit 113 compares the feature points / features of the captured image with the feature points / features of the matching image, and changes the line-of-sight direction of the 3D modeling data so that the line-of-sight of the matching image is obtained ( Step S3108). Since the 3D modeling data whose line of sight is changed in this manner is similar to the collation image as shown in FIG. 19D, collation using the collation image can be performed.

  Note that if the comparison image currently being compared is not for the subject that generated the 3D modeling data, the verification will not succeed even if the line of sight of the 3D modeling data is changed (step S3109: No). In this case, the process returns to step S3106, and the above-described processing is performed on the next collation image by incrementing the index number i of the collation image to be processed by +1.

  If the collation is successful (step S3109: YES), the same processing as step S1106 to step S1111 of the “AR display process (1)” (FIG. 8) illustrated in the first embodiment is performed. Display is performed (step S3110).

  If the predetermined end event as exemplified in each of the above embodiments does not occur (step S3111: No), the process returns to step S3104, and the subsequent processing is repeated, and the occurrence of the end event (step S3111: Yes). This process is terminated.

  According to the present embodiment as described above, the line-of-sight direction of the 3D modeling data generated from the captured image is changed based on the feature points and feature amounts extracted from the captured image and the collation image. Even when shooting from a different direction, it is possible to appropriately perform collation, and it is possible to accurately display the AR even when the shooting is about. That is, even when everything becomes an AR service target due to the expansion of the AR service, accurate AR display can be performed, and the convenience of the user can be improved.

  In this embodiment, a collation image as a photographic image is acquired from a short-range wireless communication device or a visible light communication device installed in a facility such as a store. The image itself may be 3D modeling data. This can also be set to 3D modeling data when the information providing apparatus 200 exemplified in the first embodiment or the second embodiment provides a collation image. If collation is performed between 3D modeling data, a more accurate collation operation based on numerical computation can be performed without performing image processing.

  As described above, according to each of the above embodiments of the present invention, the convenience of the user can be improved even when the target is enormous when the AR service is used.

  The said embodiment is an example and the application range of this invention is not restricted to this. That is, various applications are possible, and all embodiments are included in the scope of the present invention.

  For example, the standards and formats shown in the above embodiments are examples, and any standards and formats that can be adopted are applicable as long as the same effects can be obtained.

  Moreover, in each said embodiment, although the smart phone was illustrated as the information display apparatus 100, if it has a structure which can acquire an imaging function and key information and / or AR data at least, it will not be restricted to a smart phone, Any device can be used as the information display device 100.

  As shown in FIG. 4, by applying the program to any device that can be employed as the information display device 100, even if the device does not have the function realized by the control unit 110 described above, If a simple functional configuration can be realized, an existing device can be made to function as the information display device according to the present invention.

  Similarly, if a functional configuration as shown in FIG. 6 can be realized by applying a program to a general-purpose server device or the like, an existing information processing device or the like is caused to function as the information providing device according to the present invention. Can do.

  The application method of such a program is arbitrary. For example, the program can be applied by being stored in a storage medium such as a CD-ROM or a memory card, or can be applied via a communication medium such as the Internet.

DESCRIPTION OF SYMBOLS 1 ... Information display system, NW ... Communication network, 100 ... Information display apparatus, 110 ... Control part, 111 ... AR process part, 112 ... Imaging process part, 113 ... Collation process part, 120 ... Communication part, 130 ... Key information acquisition , 140 ... Imaging unit, 150 ... Input / output unit, 160 ... Storage unit, 200 ... Report providing device, 210 ... Control unit, 211 ... Communication processing unit, 212 ... Information registration unit, 213 ... Information search unit, 220 ... Communication unit, 230 ... input / output unit, 240 ... storage unit

Claims (7)

An information display device that includes an imaging unit and superimposes and displays information provided from outside on a captured image captured by the imaging unit;
An information display system for performing information display (AR display) based on augmented reality (AR), comprising: an information providing device that provides information superimposed on a captured image by the information display device to the information display device;
The information providing apparatus includes:
AR data for identifying AR data including a collation image for collating with a subject in the captured image and superimposition information to be superimposed and displayed on the captured image in which the subject collated with the collation image is captured. AR data storage means for storing identification information in association with key information that can be notified from the information display device;
When key information is provided from the information display device, identification information corresponding to the key information is transmitted to the information display device. When identification information is provided from the information display device, the identification information is provided. Information providing means for transmitting AR data corresponding to the information display device,
The information display device includes:
Key information acquisition means for acquiring key information for specifying a subject in the captured image;
Key information notification means for notifying the information providing apparatus every time the key information is acquired;
Identification information acquisition means for acquiring identification information provided from the information providing device in response to the notification of the key information;
Each time the identification information is acquired, new key information and the identification information are associated with each other and added to the first table in the first table that records a plurality of the key information and the identification information. Key information recording means for recording
Identification information notification means for notifying the information providing apparatus of any identification information recorded in the first table;
AR data obtaining means for obtaining AR data used for the AR display provided from the information providing apparatus in response to the notification of the identification information;
AR display means for performing the AR display using the AR data acquired by the AR data acquisition means,
An information display system characterized by that. AR data recording means for recording the acquired AR data in association with the first table in a second table in which the number of records is defined,
The AR display means performs the AR display using the AR data recorded by the AR data recording means.
The information display system according to claim 1. Augmented reality (AR) information display comprising: an imaging unit; a communication unit that communicates with an external device; and a display unit that superimposes and displays information provided from the external device on an image captured by the imaging unit. An information display device for performing (AR display),
Key information acquisition means for acquiring key information used to specify information provided by the external device to the information display device;
Key information notifying means for notifying the external device of the key information acquired by the key information acquiring means via a communication network;
The verification image that is transmitted by the external device in response to the key information notification from the key information notification unit and is collated with the subject in the subject imaged by the imaging unit, and the subject that is collated with the verification image Identification information acquisition means for acquiring identification information indicating AR data including superimposition information to be superimposed on a captured image, via the communication network;
The key information acquired by the key information acquisition unit is recorded in the first table in which the number of records is defined in association with the use date and time of the key information and the identification information acquired by the identification information acquisition unit. Key information recording means;
Identification information notification means for notifying the external device of the identification information recorded in the first table via the communication network;
AR data acquisition means for acquiring the AR data transmitted by the external device in response to notification of identification information by the identification information notification means;
AR data recording means for recording the AR data acquired by the AR data acquisition means in a second table in which the number of records is defined in association with the notified identification information and the use date and time of the AR data; ,
AR display means for performing AR display using the AR data recorded in the second table,
The identification information notifying means notifies the external device of the identification information when the identification information recorded in the first table is not recorded in the second table;
The key information recording means performs deletion based on the use date and time of already recorded key information when there is no empty record in the first table when the key information acquiring means acquires key information.
The AR data recording means performs deletion based on the use date and time of already recorded AR data when there is no empty record in the second table when the identification information notification means notifies the identification information.
An information display device characterized by that. The key information recording means records the date and time when the key information is acquired as a use date and is updated with the date and time when the AR display using the AR data corresponding to the key information is performed,
The AR data recording means records the date and time when the AR data is acquired as a use date and time,
Update the date and time when AR display using the AR data was performed,
The key information recording unit and the AR data recording unit specify a record in which the oldest date and time is recorded as a usage date and time as a deletion target.
The information display device according to claim 3. The key information includes position information indicating a position of the information display device at the time of imaging by the imaging unit,
The key information recording means calculates the distance between the position indicated by the position information and the current position of the information display device when the key information recorded in the record specified as the deletion target is position information. If the distance is smaller than a predetermined threshold, exclude the record from the deletion target.
The information display device according to claim 4. In the AR data recording means, the identification information recorded in the record specified as the deletion target is recorded in the first table, and the record in which the identification information is recorded in the first table If the key information recording means is excluded from the deletion target, the specified record is excluded from the deletion target;
The information display device according to claim 5. Augmented reality (AR) information display comprising: an imaging unit; a communication unit that communicates with an external device; and a display unit that superimposes and displays information provided from the external device on an image captured by the imaging unit. To the computer that controls the information display device that performs (AR display)
A function of acquiring key information used to specify information provided by the external device to the information display device;
A function of notifying the acquired key information to the external device via a communication network;
A collation image that is transmitted by the external device in response to the key information notification and is collated with the subject in the subject imaged by the imaging means, and is superimposed on the captured image in which the subject collated with the collation image is shown. A function of acquiring identification information indicating AR data including superimposition information to be transmitted via the communication network;
A function of recording the acquired key information in a first table in which the number of records is defined in association with the use date and time of the key information and the acquired identification information;
When the identification information recorded in the first table is not recorded in the second table that records the identification information acquired from the external device in association with the AR data, the identification information is A function of notifying the external device via a communication network;
A function of acquiring the AR data transmitted by the external device in response to the notification of the identification information;
A function of recording the acquired AR data in the second table in which the number of records is defined in association with the notified identification information and the use date and time of the AR data;
A function of performing AR display using AR data recorded in the second table;
A function for performing deletion based on the use date and time of already recorded key information when there is no empty record in the first table when the key information is acquired;
A function of performing deletion based on the use date and time of AR data already recorded when there is no empty record in the second table when the identification information is notified to the external device;
A program characterized by realizing.

Images