JP2011227875A - Information processor, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to easily acquire stock information and price information of a commodity which the user will purchase.SOLUTION: A mobile terminal 200 acquires, by photographing, image data relating to a commodity posted in a flier 300 and transmits the image data relating to the commodity to a shopping management server 100. On reception of the image data relating to the commodity from the mobile terminal 200, the shopping management server 100 analyzes the image data relating to the commodity to specify this commodity, acquires at least one of stock information and sale information of the commodity, and transmits at least one of the stock information and sale information of the commodity to the mobile terminal 200. The mobile terminal 200 presents computer graphic image data showing at least one of the stock information and sale information of the commodity, which is transmitted from the shopping management server 100, in combination with an image of the flier in a real space.

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program that use augmented reality technology to synthesize and superimpose a real-space video and computer graphics image data.

Navigation technology is used in various fields such as route guidance to a destination and presentation of work procedures during work. Since navigation technology is a technology that assists in facilitating certain actions, a mechanism that can be understood as intuitively as possible has been demanded. As one of such mechanisms, navigation using a technology called augmented reality that superimposes data managed by computers such as characters and images on the real environment and gives supplementary information to the real environment. Technology exists.

  In the theme parks and shopping malls, a technology for three-dimensional navigation using augmented reality (hereinafter referred to as AR) is disclosed for the route to the desired attraction and the display information of the product recommended by the store side (for example, AR) (for example, , See Patent Document 1). Specifically, the position / orientation of the portable information terminal at the time of shooting is calculated, and a marker on the image shot by the portable information terminal is used, and an image obtained by shooting predetermined information previously associated with the marker is displayed. This is a technique for superimposing and displaying on the screen. Note that AR is a technique for providing supplementary information by superimposing and displaying computer-managed data (hereinafter referred to as computer graphics image data) such as characters, graphics, still images, and moving images in an actual environment.

  In addition, a technique is known in which merchandise to be purchased is registered in a list of mobile terminals in advance, and product information of the merchandise is displayed on the mobile terminal when walking near the merchandise registered in the list (for example, , See Patent Document 2).

JP 2004-48674 A JP-A-10-198731

  However, in the technique disclosed in Patent Document 1, the navigation information may include information that is recommended for the store side but is unnecessary for the user. On the other hand, the technology disclosed in Patent Document 2 can display the product information of the products scheduled for shopping registered in the list in advance on the mobile terminal. However, there is a possibility that the product to be purchased is out of stock or the price of the product to be purchased fluctuates when going to shopping.

  Furthermore, in order to be able to understand intuitively in an environment where the situation is constantly changing, such as the price and availability of goods, the timing and type of information that can be obtained is almost as realistic as possible, and immediately It was required to present it in a state where it could be perceived by. SUMMARY OF THE INVENTION An object of the present invention is to provide support for easily acquiring product information and guide information of shopping-scheduled products at an arbitrary timing.

An information processing apparatus according to the present invention includes an augmented reality presentation apparatus capable of combining and displaying real space video and computer graphics image data, and inventory information and sales information of each product posted on the flyer. An information processing system having a management device that manages at least one of them, wherein the augmented reality presentation device includes imaging means for acquiring image data relating to a product posted in the leaflet by imaging, First transmission means for transmitting image data relating to the product acquired by the imaging means to the management device, wherein the management device receives image data relating to the product from the augmented reality presentation device. When received, the product is identified by analyzing image data related to the product, and at least one of the inventory information and the sales information of the product A second acquisition unit that transmits at least one of inventory information and sales information of the product acquired by the first acquisition unit to the augmented reality presentation device. The augmented reality presentation device includes computer graphics image data indicating at least one of inventory information and sales information of the product transmitted from the management device, and the flyer in the real space. It further has presentation means for synthesizing and presenting the video.
The information processing method according to the present invention includes an augmented reality presentation device capable of combining and displaying real-space video and computer graphics image data, and inventory information and sales information of each product posted on the flyer. An information processing method executed by a management device that manages at least one of them, wherein the augmented reality presentation device captures image data relating to a product posted in the leaflet by photographing, and A first transmission step of transmitting image data related to the product acquired in the imaging step to the management device, wherein the management device receives an image related to the product from the augmented reality presentation device. When the data is received, the product is identified by analyzing the image data related to the product, and the inventory information and the sales information of the product are An acquisition step for acquiring at least one of the information, and a second transmission for transmitting at least one of the inventory information and the sales information of the product acquired by the acquisition step to the augmented reality presentation device The augmented reality presentation device includes computer graphic image data indicating at least one of the inventory information and the sales information of the product transmitted from the management device, and the image of the flyer in the real space. And a presentation step of combining and presenting.
The program according to the present invention includes an imaging step in which image data relating to a product posted on a flyer is acquired by photographing on a computer, and image data relating to the product acquired in the imaging step is displayed on each flyer. A transmission step of transmitting to at least one of inventory information and sales information of the product, and at least one of inventory information and sales information of the product transmitted from the management device A presentation step of combining the presented computer graphic image data with the flyer image in the real space and presenting it.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to assist so that the merchandise information and guidance information of shopping planned goods can be acquired easily at arbitrary timings.

It is a figure showing roughly the composition of the shopping navigation system concerning the embodiment of the present invention. It is a figure which shows the hardware constitutions of the shopping management server which concerns on embodiment of this invention. It is a figure which shows the hardware constitutions of the portable terminal which concerns on embodiment of this invention. It is a flowchart which shows the flow of registration processing of the goods planned for purchase in the shopping navigation system concerning the embodiment of the present invention. It is a flowchart which shows the presentation process of the merchandise information of the goods planned for purchase, and the navigation process from the present location to the store in the shopping navigation system according to the embodiment of the present invention. It is a figure which shows the state which the user pointed to the purchase plan goods posted on the flyer, and the state was caught in the angle of view of the imaging part of the portable terminal. It is a figure which shows the example of a display which synthesize | combined the computer graphics image data of merchandise information with the image | video of real space. It is a figure which shows the example of AR display when the product information of the apple which is a user's purchase plan goods can be acquired for 3 stores. It is a figure which shows typically the example of the database structure of flyer information DB. It is a figure which shows typically the example of the database structure of user information DB. It is a figure which shows typically the example of the database structure of merchandise information DB. It is a figure which shows schematically the structure of the shopping navigation system which concerns on other embodiment of this invention. It is a flowchart which shows the flow of the registration process of the goods planned for purchase in the shopping navigation system concerning other embodiments of the present invention. It is a flowchart which shows the presentation process of the merchandise information of a goods planned for purchase, and the navigation process from a present location to a store in the shopping navigation system concerning other embodiments of the present invention. It is a figure which shows the example of a shopping list.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram schematically showing a configuration of a shopping navigation system according to an embodiment of the present invention. As shown in FIG. 1, the shopping navigation system according to the present embodiment includes a shopping management server 100 and a mobile terminal 200, which are connected to each other via a wireless communication line.

  The user browses the flyer 300 and selects a product to be purchased from the products posted on the flyer 300. Next, the user points to the selected purchase planned product, and captures the state in the angle of view of the imaging unit of the mobile terminal 200, whereby image data in a state in which the purchase planned product is pointed is generated by the imaging unit. The portable terminal 200 detects image data of a product scheduled for purchase from the image data generated by the imaging unit. As a method for detecting image data of a product to be purchased, for example, several finger shape patterns in the state of pointing are prepared, and image data matching or similar to those finger shape patterns is detected in the image data. In such a case, a method of recognizing the image data as a finger and detecting an object that exists within a certain distance from the finger in the direction of the finger may be considered.

  When the mobile terminal 200 detects the image data of the purchase planned product, the mobile terminal 200 transmits the image data of the purchase planned product to the shopping management server 100. When the shopping management server 100 receives the image data of the product to be purchased from the mobile terminal 200, the product matching unit 106 stores the image data of each product listed in the flyer 300 stored in the flyer information DB 102, and the mobile terminal. The consistency with the image data of the product scheduled for purchase received from 200 is determined, and it is determined whether or not the product planned for purchase is a product listed on the flyer 300. If the purchase planned product is a product listed on the flyer 300, the registration / inquiry unit 105 registers the purchase planned product and the user in association with each other in the user information DB 101. As a result, the user's planned purchase product is registered in the shopping management server 100.

  After the product to be purchased is registered in the shopping management server 100, when the user wants to know the transition of price information or stock information of the product to be purchased, the mobile terminal 200 is again held over the flyer 300, and the mobile terminal 200 The product to be purchased is captured at the angle of view of the imaging unit. Thereby, the image data of the product to be purchased is transmitted from the mobile terminal 200 to the shopping management server 100. The product matching unit 106 of the shopping management server 100 analyzes the image data received from the mobile terminal 200 to detect that the image data is image data of a product posted on the flyer 300. Then, the registration / inquiry unit 105 determines that the product is a user's planned purchase product. Then, the product information search unit 107 acquires product information such as the latest (current) sales price and inventory information of the product to be purchased and transmits it to the mobile terminal 200. The mobile terminal 200 synthesizes computer graphics image data indicating the latest product information of a product scheduled for purchase with a video in the real space and displays it on the display. At this time, the computer graphics image data indicating the product information is displayed in the vicinity of the real space image so that the product information is related to the product to be purchased. Is synthesized.

  The map information search unit 108 acquires the current location information of the mobile terminal 200 and the address information of the store selling the product to be purchased, and includes a location indicated by the current location information and a location indicated by the address information. Information is acquired from the map information DB 104. Then, the map information search unit 108 generates guide information for guiding the route from the current location of the mobile terminal 200 to the store selling the product to be purchased using the acquired map information, and stores the information in the mobile terminal 200. Send to. Using the guidance information, the mobile terminal 200 displays computer graphics image data that guides the route from the current location to the store that sells the product to be purchased with the video in the real space. Accordingly, by capturing the street at the angle of view of the imaging unit of the mobile terminal 200, the user confirms on the display of the mobile terminal 200 an image in which the route guidance based on the computer graphics image data is combined with the image of the real space on the street. While moving to the store.

  In the present embodiment, an expanded sense of reality is presented by displaying computer graphics image data at a position that matches the image in the real space on the display of the mobile terminal 200. Hereinafter, such display of computer graphics image data that gives an expanded sense of reality may be referred to as AR (Augmented Reality) display. There are various types of HMD such as non-transmission type, video transmission type, and optical transmission type. The composite display (AR display) is to superimpose and display computer graphics image data at a matching position on the real space image displayed on the lens portion of the HMD by any method. In the case of the type method, the image is superimposed and displayed on the image of the real space seen through, and in the case of the video transmission type, the image is displayed on the image of the real space captured by the video camera.

  In addition, the mobile terminal 200 includes a mobile terminal type AR display device that displays a real-space image captured by the imaging unit on a display and displays computer graphics image data in combination with the displayed real-space image. Assuming, the AR display device applicable to the present embodiment is not limited to this. For example, a glasses-type AR display device worn on the user's head, which displays computer graphics image data at a position that matches optically transmitted and projected images in a real space. An AR display device (HMD) may be used, or a head that is installed in the direction of the line of sight of the driver's seat of a moving vehicle and displays computer graphics image data in combination with a real space image that is optically transmitted and projected. An AR display device of an up display type may be used.

  In addition, when computer graphics image data is displayed in alignment with an image in the real space, coordinates between the object in the real space image and the computer graphics image data are performed. As a method of coordinate alignment, based on the position (latitude, longitude) and posture (direction, elevation angle) of the mobile terminal 200, it is estimated where the user holding the mobile terminal 200 is looking in the real space through the imaging unit. Then, the computer graphics image data may be synthesized so as to match the estimated position, or a marker is attached to the object in the real space, the marker is photographed by the imaging unit, and the image data is captured from the photographed image data. You may detect the position of a marker and synthesize | combine computer graphics image data so that it may correspond with the detected position. Furthermore, the position of the object in the real space may be detected by analyzing the image data in the real space captured by the imaging unit, and the computer graphics image data may be synthesized so as to match the detected position.

  FIG. 2A is a diagram illustrating a hardware configuration of the shopping management server 100 according to the present embodiment. The CPU 1001 comprehensively controls each device and controller connected to the system bus. The ROM 1003 or the HD (hard disk) 1009 stores a basic input / output system (BIOS) that is a control program for the CPU 1001 and an operating system program, for example, the shopping management server 100 in the processes shown in FIGS. A program of processing to be executed is stored.

  In the example of FIG. 2A, the HD 1009 is configured to be arranged inside the shopping management server 100. However, as another embodiment, a configuration corresponding to the HD 1009 is arranged outside the shopping management server 100. Also good. Further, for example, the program for performing the processing shown in FIG. 3A and FIG. 3B according to the present embodiment is recorded on a computer-readable recording medium such as a flexible disk (FD) or a CD-ROM. It is good also as a structure supplied from a recording medium, and good also as a structure supplied via communication media, such as the internet.

  A RAM 1002 functions as a main memory, work area, and the like for the CPU 1001. The CPU 1001 implements various operations by loading a program necessary for execution of processing into the RAM 1002 and executing the program.

  The HD 1009 and the FD 1008 function as an external memory. The CPU 1001 implements various operations by loading a program necessary for execution of processing into the RAM 1002 and executing the program.

  The disk controller 1007 controls access to an external memory such as the HD 1007 and the FD 1008. The communication I / F controller 1006 is connected to the Internet or a LAN, and controls communication with the outside by, for example, TCP / IP.

  A display controller 1010 controls image display on the display 1011.

  The KB controller 1004 receives an operation input from the KB (keyboard) 1005 and transmits it to the CPU 1001. Although not shown, in addition to the KB 1005, a pointing device such as a mouse can also be applied to the shopping management server 100 according to the present embodiment as a user operation means.

  The registration / inquiry unit 105, the product matching unit 106, the product information search unit 107, and the map information search unit 108 illustrated in FIG. 1 are stored in, for example, the HD 1009 and loaded into the RAM 1002 as necessary, and execute the program. This is a configuration realized by the CPU 1001. The user information DB 101, flyer information DB 102, product information DB 103, and map information DB 104 have a configuration corresponding to a partial storage area in the HD 1009.

  FIG. 2-2 is a diagram illustrating a hardware configuration of the mobile terminal 200 according to the present embodiment. The CPU 2001 comprehensively controls each device and controller connected to the system bus. The ROM 2003 stores, for example, a program for processing executed by the mobile terminal 200 among the processing illustrated in FIGS. 3A and 3B. Note that the program for performing the processes shown in FIGS. 3A and 3B may be supplied via a communication medium such as the Internet.

  A RAM 2002 functions as a main memory, work memory, and the like for the CPU 2001. The CPU 2001 implements various operations by loading a program necessary for execution of processing into the RAM 2002 and executing the program. A communication I / F controller 2005 is connected to the Internet or a LAN, and controls communication with the outside by TCP / IP, for example.

  The imaging unit 2004 converts a subject image incident through an optical lens and formed on the imaging element into an electrical signal, and outputs moving image data or still image data. A display controller 2006 controls image display on the display 2007. In the present embodiment, it is assumed that the display 2007 employs a liquid crystal display.

  FIG. 3A is a flowchart illustrating a flow of registration processing of a product scheduled for purchase in the shopping navigation system according to the present embodiment. First, the user points to a purchase planned product posted on the flyer 300 and captures the state in the angle of view of the imaging unit 2007 of the mobile terminal 200. As a result, in step S101, the imaging unit 2007 generates image data captured at the angle of view, and the mobile terminal 200 analyzes the image data to detect image data of a product to be purchased designated by pointing. . In step S <b> 102, the mobile terminal 200 transmits the image data of the product scheduled for purchase detected in step S <b> 101 to the shopping management server 100.

  FIG. 4 is a diagram illustrating a state in which the user points to a purchase planned product posted on the flyer 300 and the state is captured by the angle of view of the imaging unit 2007 of the mobile terminal 200. The user selects and points to the apple to be purchased among the products posted on the flyer 300. In the example of FIG. 4, the photographing unit of the mobile terminal 200 keeps this state in the angle of view, and the flyer 300 The display 2007 shows a video in which the user points to the product to be purchased (apple) above. Therefore, in the example of FIG. 4, the image data of the apple is detected as the image data of the product scheduled for purchase. In addition to the method described above, for example, the user may specify a product to be purchased by moving his / her finger so as to surround the apple on the flyer and capturing it in the angle of view of the imaging unit 2007. In this case, the mobile terminal 200 detects the trajectory of the user's finger and determines a product that falls within the trajectory as a purchase-scheduled product. By adopting the method for designating a purchase planned product as described above, it is possible to easily designate a product planned for purchase.

  In step S <b> 201, the shopping management server 100 receives image data of a product scheduled for purchase from the mobile terminal 200. In step S <b> 202, the product matching unit 106 acquires the image data of each product posted in the flyer 300 from the flyer information DB 102 and compares it with the image data of the planned purchase product received from the mobile terminal 200. In step S <b> 203, the product matching unit 106 determines whether the image data of the product scheduled for purchase and the image data of the product listed on the flyer 300 match, whereby the product scheduled for purchase is the product listed on the flyer 300. It is determined whether or not. This matching is determined by performing pattern matching between the image data of the product to be purchased and the image data of the product listed in the leaflet 300, and when it is determined that both are the same or similar, It is determined that the product is on the flyer 300. When it is determined that the product to be purchased is a product posted on the flyer 300, the process proceeds to step S204. On the other hand, if it is determined that the product to be purchased is not a product listed in the flyer 300, the process returns to step S201 and waits for reception of image data of a new product to be purchased.

  FIG. 7 is a diagram schematically illustrating an example of the database structure of the flyer information DB 102. In FIG. 7, the flyer information DB 102 manages the ID (product ID) of each product listed in the flyer 300 and image data in association with each other. In step S201, when the shopping management server 100 receives, for example, apple image data from the portable terminal 200, in step S203, the product matching unit 106 displays the image data of the product to be purchased and each product listed in the flyer 300. As a result of the comparison with the image data, it is determined that the image data of the planned purchase product is identical or similar to the image data of the apple 701, and the purchase planned product is determined to be an apple.

  In step S <b> 204, the registration / inquiry unit 105 acquires the product ID associated with the image data of the product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. In addition, the registration / inquiry unit 105 acquires the transmission source user ID from the header information of the image data of the product to be purchased received in step S201. Then, the registration / inquiry unit 105 registers the user ID and the product ID of the product scheduled for purchase in the user information DB 101 in association with each other.

  FIG. 8 is a diagram schematically illustrating an example of the database structure of the user information DB 101. In FIG. 8, the user information DB 101 manages a user ID and a purchase planned product ID in association with each other. For example, when the product ID of the product to be purchased is “1011” and the user ID “aaa” can be acquired from the header information of the image data of the product to be purchased, as shown in the record 801, the user ID “ “aaa” and purchase planned product ID “1011” are managed in association with each other.

  FIG. 3-2 is a flowchart illustrating a process for presenting product information of a product to be purchased and a navigation process from the current location to the store in the shopping navigation system according to the present embodiment. For example, when the user wants to grasp the latest price information, inventory information, etc. of the product scheduled to be purchased that is already registered in the shopping management server 100, the mobile terminal 200 is held over the product scheduled to be posted on the flyer 300 again. The product to be purchased is captured at the angle of view of the imaging unit 2007. As a result, in step S103, the imaging unit 2007 of the mobile terminal 200 generates image data of a product to be purchased. In step S <b> 104, the mobile terminal 200 transmits the image data of the product scheduled for purchase to the shopping management server 100.

  In step S <b> 205, the shopping management server 100 receives image data of a product scheduled for purchase from the mobile terminal 200. In step S <b> 206, the product matching unit 106 acquires the image data of each product listed in the flyer 300 from the flyer information DB 102 and compares it with the image data of the planned purchase product received from the mobile terminal 200. In step S <b> 207, the product matching unit 106 determines a match between the image data of the product planned for purchase and the image data of the product listed in the flyer 300, so that the product planned for purchase is the product listed in the flyer 300. It is determined whether or not. When it is determined that the product to be purchased is a product posted on the flyer 300, the process proceeds to step S213. On the other hand, when it is not determined that the product to be purchased is a product listed in the flyer 300, the process returns to step S205.

  In step S 213, the registration / inquiry unit 105 acquires a product ID associated with the image data of the product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. In addition, the registration / inquiry unit 105 acquires the transmission source user ID from the header information of the image data received in step S205. In step S208, the registration / query unit 105 determines whether the product ID acquired in step S213 is managed in the user information DB 101 in association with the user ID. If the product ID is associated with the user ID, the process proceeds to step S209. On the other hand, if the product ID is not associated with the user ID, the process returns to step S205.

  In step S209, the product information search unit 107 searches the product information DB 103 for product information corresponding to the product ID acquired in step S213, and transmits the product information to the mobile terminal 200.

  FIG. 9 is a diagram schematically illustrating an example of the database structure of the product information DB 103. In FIG. 9, the product information DB 103 manages the product ID, product name, price (price 1, price 2,...), Store, inventory quantity, and product attributes in association with each other. As the product attribute, if the product is food, the production location, date of production, and the like are managed. If the product is clothing, the color, size, and the like are managed. The data managed in the product information DB 103 may be updated at an arbitrary timing in cooperation with a POS system (Point Of Sales system) not shown in FIG. For example, if the product ID acquired in step S213 is “1011”, in step S209, a record 901 is retrieved from the product information DB 103 shown in FIG. 9 and the product information registered in the record 901 is stored in the mobile terminal 200. Sent. Note that the latest information regarding the price and the stock amount included in the product information is transmitted to the mobile terminal 200. In the product information DB 103, the price of the product posted in the flyer 300 is registered as the initial price (price 1), and the price is additionally registered every time the price is updated (price 2, price 3,...・). In step S <b> 209, the last registered price is transmitted to the mobile terminal 200 from the history of prices registered in this way. As a result, the latest (current) price of the product scheduled for purchase is displayed on the mobile terminal 200. It should be noted that the product information DB 103 may manage the price history, overwrite the price itself, and manage only the latest price. In the product information DB 103, the stock quantity is overwritten at a predetermined timing, and only the latest stock quantity is managed. Accordingly, in step S209, the latest inventory amount is transmitted to the mobile terminal 200. As a result, the latest stock quantity of the product scheduled for purchase is displayed on the mobile terminal 200.

  In step S <b> 105, the mobile terminal 200 receives product information from the shopping management server 100. In step S <b> 106, the mobile terminal 200 displays the received product information computer graphics image data in combination with a real-space video. FIG. 5 is a diagram showing a display example in which computer graphics image data of product information is combined with a real space video. In the example of FIG. 5, computer graphics image data indicating product information 501 and computer graphics image data indicating a tag 502 that is an operation switch for requesting route guidance are displayed in the vicinity of the product to be purchased. .

  In step S110, the mobile terminal 200 determines whether or not there is a request for route guidance (navigation). That is, the mobile terminal 200 determines whether or not the tag 502 has been operated. If the tag 502 is operated, the process proceeds to step S107. On the other hand, if the tag 502 has not been operated, the process ends. In step S <b> 107, the mobile terminal 200 issues a route guidance request to the shopping management server 100.

  In step S <b> 210, the shopping management server 100 receives a route guidance request from the mobile terminal 200. In step S <b> 211, the map information search unit 108 acquires the current location information of the mobile terminal 200, and acquires store information selling the products scheduled for purchase from the product search unit 107. Note that the current location of the mobile terminal 200 can be acquired by using a GPS (Global Positioning System). Although not shown in FIG. 9, the product information DB 103 includes address information of stores that sell products, and the map information search unit 108 plans to purchase from the product information DB 103 via the product information search unit 107. The address information of the store that sells the product shall be acquired. In step S <b> 211, the map information search unit 108 searches for map information including the position indicated by the current location information of the mobile terminal 200 and the position indicated by the address information of the store selling the product to be purchased. In step S212, the map information search unit 108 uses the searched map information to generate guide information for guiding a route from the current location of the mobile terminal 200 to a store selling the product to be purchased, and the mobile terminal 200 is transmitted.

  In step S <b> 108, the mobile terminal 200 receives the guide information from the shopping management server 100. In step S <b> 109, the mobile terminal 200 synthesizes computer graphics image data that guides a route from the current location of the mobile terminal 200 to the store that sells the product to be purchased, using the received guidance information, into a real-space image. And displayed on the display 2007.

  In the embodiment described above, when the user wants to know the product information of the registered shopping planned product, the user performs a predetermined designation operation on the shopping planned product on the flyer, and this is performed on the imaging unit 2004. The product information such as the latest stock quantity and price of the shopping-scheduled product can be easily obtained. In addition, when the association between the user ID and the product ID of the planned purchase product is registered in the user information DB 801, the product information of the planned purchase product can be notified to the mobile terminal 200. Step S209 in FIG. 3-2 may be executed after step S204 in 3-1. That is, when the shopping management server 100 registers the association between the user ID and the product ID of the product to be purchased in the user information DB 801 in step S204, the product information corresponding to the product ID is stored in the product information DB 103 in the subsequent step S209. Are transmitted to the portable terminal 200. By doing in this way, it becomes possible to notify the portable terminal 200 of the product information of the purchase planned product at the time of registering the association between the user ID and the product ID of the planned purchase product.

  In the above-described embodiment, the user needs to hold the mobile terminal 200 over the flyer 300 again in the process of presenting the product information of the product to be purchased. However, the present invention is not limited to this. Another embodiment that does not require the portable terminal 200 to be held over the flyer 300 will be described with a focus on differences from the above-described embodiment.

  FIG. 10 is a diagram schematically illustrating a configuration of a shopping navigation system according to another embodiment. As shown in FIG. 10, the system according to the present embodiment does not include the user information DB 101 in the shopping management server 100 as compared with the system according to the above-described embodiment. Instead, the shopping list 201 is stored in the mobile terminal 200. In preparation. Note that the shopping list 201 is a file in which the product ID of a product to be purchased is recorded, and is managed in association with a product image, a product name, or the like as shown in FIG.

  FIG. 11A is a flowchart illustrating a flow of registration processing of a product planned for purchase in the shopping navigation system according to the present embodiment. Compared to FIG. 3A, step S1101 is added, and step S1201 is added instead of step S204 being deleted.

  In step S <b> 1201, the registration / inquiry unit 105 acquires a product ID associated with image data of a product determined to be a purchase planned product from the flyer information DB 102 via the product matching unit 106. Then, the acquired product ID is transmitted to the mobile terminal 200.

  In step S1101, the product ID is received from the portable terminal 200. The received product ID is stored in the ROM 2003.

  Next, FIG. 11-2 is a flowchart illustrating a process for presenting product information of a product scheduled for purchase and a navigation process from the current location to the store in the shopping navigation system according to the present embodiment. Compared to FIG. 3-2, step S <b> 1102 in FIG. 3-2 is deleted instead of step S <b> 103 and step S <b> 104. Also, step S1202 is added instead of step S208 being deleted from step S205 in FIG. Hereinafter, the present embodiment will be described focusing on differences from the above-described embodiment.

  For example, when the user wants to grasp the latest price information, inventory information, etc. of the product to be purchased that has already been registered in the mobile terminal 200 in step S1101 in FIG. 11A, in step S1102 in FIG. The product ID of the product is transmitted to the shopping management server 100. As a method for specifying the product ID of the product, for example, a shopping list is displayed on the mobile terminal 200 as shown in FIG. 12, and when the product name displayed on the shopping list is tapped, the mobile terminal corresponding to the product name is displayed. The product ID registered in 200 is selected and transmitted to the shopping management server 100.

  In step S1202, the shopping management server 100 acquires a product ID. Thereafter, the process proceeds to step S209 similar to FIG. According to the configuration of the present embodiment, the user does not need the flyer 30 when acquiring the inventory information and price information of the product to be purchased, and thus can be acquired more easily.

  In the above-described embodiment, when the user selects a purchase planned product from the products listed on the flyer 300, the product information of the purchase planned product in the store selling the purchase planned product is displayed as an AR. However, it is not limited to this. For example, the user's planned purchase product may be sold at a plurality of stores. In such a case, in step S209 described above, product information at a plurality of stores selling the product is searched, and the product information is obtained. AR display can also be performed. That is, in the flyer information DB 102, the association between the image data of the flyer products of a plurality of stores and the product ID is managed, and there are a plurality of image data of products that match or are similar to the image data of the product to be purchased. When a combination of any one of the product IDs associated with all the image data and the user ID recorded in the header of the image data of the product to be purchased exists in the user information DB 101, each of the products All product information corresponding to the ID may be acquired from the product information DB 103. Alternatively, a method may be used in which all the product IDs corresponding to image data that matches or is similar to the image data of the product to be purchased are specified from the flyer information DB 102. Alternatively, although not shown in FIG. 7, a column for managing product names is added in the flyer information DB 102, and a product name that is the same as or similar to the product name of a product that matches or is similar to the image data of the product to be purchased. A method of specifying all the product IDs of the products managed in the column “product name” may be used. Alternatively, instead of the image data of the product to be purchased, a method may be used in which all product IDs of products that match or are similar to the product name or product image data managed in association with the product ID received from the mobile terminal 200 may be used. . And all the acquired merchandise information is transmitted with respect to the portable terminal 200 from the shopping management server 100, and the portable terminal 200 can carry out AR display of the merchandise information of several stores regarding a purchase plan commodity.

  FIG. 6 is a diagram illustrating an example of an AR display when product information of apples that are products to be purchased by the user can be acquired for three stores. In FIG. 6, 601, 602, and 603 are product information about apples at A store, B store, and C store, respectively, and indicate current apple prices, stock quantities, and the like at each store. Thereby, the user can refer to product information of apples other than the A store that issued the flyer. For example, if an apple is sold at a cheaper price than the A store in a store other than the A store, or if the stock is out of the A store and the stock remains in the other store, buy the apple in the other store You may change your shopping destination. In this case, in step S110, the mobile terminal 200 determines whether the tag 604 relating to the product of which store is operated among the plurality of stores in which the product information is AR-displayed, and then in step S107, the mobile terminal 200 200 issues a request for route guidance to the store to the shopping management server 100. In FIG. 6, a store selection form of the tag 604 that is an operation switch for requesting route guidance is displayed in a pull-down menu, but other configurations are possible. For example, a tag for each of a plurality of stores may be provided, or voice recognition for recognizing a store name uttered by the user may be used.

  In addition, if you originally planned to purchase apples at store A, but you are selling apples at a cheaper price than store A at other stores, or if you are out of stock at store A and have stock at other stores, The management server 100 may automatically detect it, generate guidance information for navigating to the multi-store with respect to the mobile terminal 200, and transmit it to the mobile terminal 200. At this time, in step S211 and step S212, the shopping physical server 100 compares the product information in the plurality of stores searched in the above-described step S209 at an arbitrary timing, and generates guide information based on a predetermined rule. The rule is stored in the ROM 1003 or the HD (hard disk) 1009. For example, the route to the nearest store among the plurality of stores searched in the current location and in step S209 is guided or searched in step S209. The route that guides the route to the store with the most stock among the plurality of stores that have been acquired, or separately acquires information indicating the congestion status of people in each route of the plurality of stores searched in step S209, and the route that is most easily passed It may be possible to guide the user. One rule may not be fixedly applied, but a plurality of rules may be applied in combination, or the combination may be dynamically changed and applied. For example, by first applying the rule “Guide the route to the store with the shortest distance”, guide information to the store A is generated, and the user goes to the store A based on the information displayed on the mobile terminal 200 as AR. On the way, when the shopping management server 100 refers to the product information DB 103 and grasps that the product is not in stock, the purchase management server 100 purchases based on another rule “Guide the route to the store with the most stock”. The management server 100 generates new guidance information and displays the AR on the mobile terminal 200. Further, not only the route from the current location to the store but also the location where the target product is displayed in the store may be guided. Furthermore, when the user has reached the vicinity of the target product but cannot find the product, the posture may be detected by the posture detection function of the mobile terminal 200 to guide the user to see the direction.

  Further, as another example of information displayed on the mobile terminal 200 as an AR, there is an alarm that a product registered in the past is posted on the flyer currently being viewed by the user, a discount price or a discount period of the product, etc. is there. That is, once consumables such as powdered milk and tissue that are regularly required by the user are registered once by the method described with reference to FIG. 3A, the user holds the portable terminal 200 over the leaflet issued thereafter, and the leaflet If the registered product is posted, it is detected by the method described in FIG. 3-2 (steps S205 to S208), and the alarm and the discount price or discount period of the product are processed by the same processing as step S209. Or the like may be transmitted to the mobile terminal 200. As a result, information such as an alarm is displayed in an AR display while the user looks at the flyer over the mobile terminal 200.

  As another embodiment of the present invention, the shopping management server 100 automatically selects a product according to the user's preference from the flyer, and guides the user to the selected product and the store that handles the product. May be. Specifically, the user master information is registered in the shopping management server 100. Information that is registered as master information includes personal preferences (whether or not quality is important, whether or not seasonal things are important, whether or not there are allergies, the target food cost for one week, the target calorie for the day, Whether you want to increase the number of items with the same ingredients and the same cooking procedure halfway, whether you want to finish shopping in the neighborhood, family preferences (Father: oil mode, children: meat center, adults: dry curry, children: Sweet curry), cooking history (cooking success or failure, leftover history, family reaction). Moreover, the shopping management server 100 acquires the flyer information of each store. As a method of acquiring flyer information, the image of the product, the price, etc. are acquired from the electronic flyer data of each store site, or the image of the product is obtained from the image data obtained by taking a photograph of the folded flyer of the newspaper, The method of acquiring a price etc. is mentioned. Note that the flyer information may be obtained from a plurality of stores. For example, it is conceivable to obtain leaflet information about curry ingredients from a store A which is a supermarket, and to obtain leaflet information about seasonings from a store B which is a cheap liquor store. On the other hand, when the user views a cooking program, a restaurant introduction program, a cooking researcher's blog, and a recommended cooking recipe on the mobile terminal 200 and operates a button such as “Looks good” on the screen, the mobile terminal 200 displays an image. Recognition, voice recognition, character recognition, and the like are performed, and the analysis result is combined with master information and flyer information to select a food and a store.

  The shopping management server 100 generates guide information for guiding the route from the current location of the mobile terminal 200 to the sales store based on the food and the sales store thus selected, and transmits the guidance information to the mobile terminal 200. To do. In the portable terminal 200, using the guidance information, computer graphics image data for guiding from the current location to the sales store is combined with the video in the real space and displayed. Thereby, the route to the store where the food that matches the user's preference can be purchased is automatically guided. In addition, when a user moves into a sales store and holds the mobile terminal 200 in the store, a product such as the price of the food, the amount of inventory, the production area, etc. is displayed on the image of the real space in which the shelf on which the food is placed is reflected. The computer graphics image data indicating the information may be combined and displayed, or the computer graphics image data displayed in an arrow indicating the position of the shelf may be combined and displayed on the image in the real space.

  Also, the history of ingredients actually purchased by the user may be registered in the master information and used when selecting the next ingredients. Furthermore, you may make it output the log | history of the purchased foodstuff as some information of a household account book.

  In the above-described embodiment, at the time of registration of a product planned for purchase or acquisition of product information of a product planned for purchase, image data of the product planned for purchase is captured by the imaging unit 2004 and the image data is captured from the mobile terminal 200 to the shopping management server 100. However, the present invention is not limited to this. That is, when a marker corresponding to a product is posted on the leaflet, the marker may be photographed by the imaging unit 2004 and image data of the marker may be transmitted from the mobile terminal 200 to the shopping management server 100. The shopping management server 100 manages the association between the marker and the product, and identifies the planned purchase product corresponding to the marker by analyzing the received image data of the marker. Then, the shopping management server 100 compares the ID of each product listed in the leaflet 300 stored in the leaflet information DB 102 with the ID of the planned purchase product, and if there is a matching product, the purchase planned product is the flyer. And the user information DB 101 is managed by associating the user with the product to be purchased.

  In addition, when the user wants to obtain product information of the product to be purchased, the user also captures a marker corresponding to the product to be purchased with the imaging unit 2004, and the image data of the marker is transferred from the portable terminal 200 to the shopping management server 100. Send to. The shopping management server 100 identifies the product to be purchased corresponding to the marker by analyzing the received image data of the marker. Then, the shopping management server 100 compares the ID of each product listed in the leaflet 300 stored in the leaflet information DB 102 with the ID of the product to be purchased. When there is a matching product, the shopping management server 100 determines whether or not the association between the ID of the planned purchase product and the user is registered in the user information DB 101. When the association between the ID of the planned purchase product and the user is registered in the user information DB 101, it is determined that the planned purchase product has been specified by the user in the past. Therefore, the shopping management server 100 transmits the product information of the planned purchase product to the mobile terminal 200 of the user. As described above, it is also possible to easily obtain the product information of the product scheduled for purchase at an arbitrary timing by using the marker corresponding to each product posted on the flyer.

  In the above-described embodiment, the flyer has been described as an example. Needless to say, the present invention can be similarly applied to any advertising medium such as a poster or a TV commercial attached to the city.

  100: Shopping management server, 200: Mobile terminal, 300: Flyer, 101: User information DB, 102: Flyer information DB, 103: Product information DB, 104: Map information DB, 105: Registration / inquiry unit, 106: Product matching Part 107: Product information search part 108: Map information search part

Claims (7)

Augmented reality presentation device capable of combining and displaying real space video and computer graphics image data, and managing at least one of inventory information and sales information of each product posted on the flyer An information processing system having a management device for performing
The augmented reality presentation device includes:
Imaging means for acquiring image data relating to the product posted in the leaflet by photographing;
First transmission means for transmitting image data relating to the product acquired by the imaging means to the management device;
The management device
When image data related to the product is received from the augmented reality presentation device, the product is identified by analyzing the image data related to the product, and at least one of inventory information and sales information of the product First acquisition means for acquiring
Second transmission means for transmitting at least one of inventory information and sales information of the product acquired by the first acquisition means to the augmented reality presentation device;
The augmented reality presentation device includes:
It further has a presentation means for presenting computer graphic image data indicating at least one of the inventory information and sales information of the product transmitted from the management device in combination with the flyer image in the real space. A characteristic information processing apparatus.   The management device acquires at least one of inventory information and sales information registered last by the first acquisition unit, and inventory information registered last by the second transmission unit. The information processing apparatus according to claim 1, wherein at least one of the sales information and the sales information is transmitted to the augmented reality presentation apparatus. The augmented reality presentation device transmits a guide information request by a user for the product indicated by the computer graphic image presented by the presentation means to the management device,
The management device further includes third transmission means for transmitting information related to the store where the product is sold to the augmented reality presentation device in response to the acquired guidance information request, and presenting the augmented reality presentation device The information processing apparatus according to claim 1, wherein the means presents computer graphic image data indicating information about the sales store acquired from the management apparatus in combination with an image of a real space. The augmented reality presentation device includes:
In response to an operation for designating the product by the user, the image data related to the product is transmitted to the management device by the first transmission unit,
The management device
Based on the image data relating to the product received from the augmented reality presentation device, further comprising a registration means for registering the product as a purchase planned product,
When the product corresponding to the image data is registered in the registration unit as a result of analyzing image data relating to the product received later from the augmented reality presentation device, the product is obtained by the first acquisition unit. The information processing apparatus according to claim 1, wherein at least one of inventory information and sales information is acquired. The augmented reality presentation device includes:
Detecting means for detecting a designation operation of the product by the user with respect to the flyer captured at the angle of view of the imaging device;
A second acquisition unit that acquires image data relating to the product that is a target of the designated operation detected by the detection unit;
The information processing apparatus according to claim 4, wherein the first operation unit transmits image data relating to the product acquired by the second acquisition unit to the management apparatus. Augmented reality presentation device capable of combining and displaying real space video and computer graphics image data, and managing at least one of inventory information and sales information of each product posted on the flyer An information processing method executed by a management device that performs
The augmented reality presentation device includes:
An imaging step of acquiring image data relating to the product posted in the leaflet by shooting;
A first transmission step of transmitting image data relating to the product acquired in the imaging step to the management device;
The management device
When image data related to the product is received from the augmented reality presentation device, the product is identified by analyzing the image data related to the product, and at least one of inventory information and sales information of the product An acquisition step to acquire,
A second transmission step of transmitting at least one of inventory information and sales information of the product acquired by the acquisition step to the augmented reality presentation device;
The augmented reality presentation device includes:
The method further includes a presentation step of combining computer graphic image data indicating at least one of inventory information and sales information of the product transmitted from the management device with the flyer image in the real space. A characteristic information processing method. On the computer,
An imaging step of acquiring image data relating to the product posted on the flyer by photographing;
A transmission step of transmitting image data relating to the product acquired in the imaging step to a management device that manages at least one of inventory information and sales information of each product posted on the flyer;
Presenting the computer graphic image data indicating at least one of the inventory information and the sales information of the product transmitted from the management device, combined with the flyer image of the real space,
A program for running

Images