JP2011253324A - Merchandise information providing terminal and merchandise information providing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a merchandise information providing terminal capable of efficiently suppressing power consumption to provide merchandise information on merchandise in a shop in real time by utilizing an AR technology.SOLUTION: The merchandise information providing terminal includes: an imaging means for picking up a real space; a merchandise information acquisition part which acquires the merchandise information regarding the merchandise included in a picked-up image picked up by the imaging means; an image generation part which generates an extended real image for superposing the merchandise information acquired by the merchandise information acquisition part on the real space to be displayed based on a position of the merchandise included in the picked-up image picked up by the imaging means; a display part which can display the extended real image generated by the image generation part; an acceleration detection means for detecting acceleration; an operating state determination part which determines an operating state of a user based on acceleration information to be detected by the acceleration detection means; and an operation control part which activates the imaging means and stops an operation of the imaging means based on the operating state determined by the operating state determination part.

Description

  The present invention relates to a merchandise information provision terminal device and a merchandise information provision system for providing merchandise information about merchandise displayed in a store in real time to a visitor using augmented reality technology.

In recent years, research on mixed reality technology has been active, and in particular, images of virtual objects such as computer graphics (CG) created by a computer are as if the virtual objects exist in real space. The so-called augmented reality (AR :) displayed on the display unit
Augmented Reality) technology is drawing attention.

  As a display unit for realizing the AR technology, an optical see-through method and a video see-through method are known. In the optical see-through method, a display unit that can transmit visible light, for example, a display unit composed of a half mirror, is arranged in the direction in which the observer directly views the real space, and the real image of the real space reflected on the display unit is displayed. In this method, the image of the virtual object is reflected and superimposed on the display unit. In this method, the scene in the real space and the virtual object are fused on the retina of the observer. The video see-through method is a method in which a combined image obtained by superimposing and drawing an image of a virtual object on a captured image captured by an imaging unit such as a video camera is displayed on a display unit.

  Such AR technology is expected to be applied to various fields. As one of them, in stores where various products are displayed such as supermarkets, a portable terminal device equipped with a display unit adopting an optical see-through method or a video see-through method is lent to a store visitor, and an optical see-through method is provided. In the case of the product, the visitor knows about the product, such as the actual image of the product shown on the display unit, the captured image including the product captured by the imaging means in the case of the video see-through method, and the product's characteristics, sales and evaluation by the purchaser By displaying an augmented reality image for superimposing and displaying desired information (hereinafter referred to as “product information”) on the display unit of the terminal device, a service for providing product information to a store visitor in real time is provided. Applications are being considered.

  For example, in Patent Document 1, a consumer uses E.C. attached to a product by a mobile communication terminal reader. A technique is disclosed in which information indicating an evaluation related to the production environment of a product is acquired from a server by reading a bar code called an S (Environment protection and Safety) code and displayed on the screen of the terminal.

JP 2002-63389 A

  The portable terminal device for realizing the service as described above includes at least an imaging means for imaging a real space and an augmented reality image for displaying product information superimposed on the real space, that is, an optical see-through. In the case of the system, the product information image for displaying the product information superimposed on the real image of the product displayed on the display unit, and in the case of the video see-through method, the product information is superimposed on the captured image including the product imaged by the imaging means. An image generation unit that generates a composite image, a display unit that can display an augmented reality image generated by the image generation unit, and a power supply unit that supplies power to the terminal device.

  By using such a portable terminal device, the imaging means is operated to image a product, and based on the captured image including the product imaged by the imaging unit, the product information is in an appropriate position and an appropriate size. Then, an augmented reality image is generated so as to be displayed superimposed on a real image of the product displayed on the display unit or a captured image captured by the imaging unit, and the generated augmented reality image is displayed on the display unit. Product information can be provided in real time.

  By the way, a large amount of power is required to operate each means for realizing the AR, such as an image pickup means, and much power is consumed even in a short time. In a portable terminal device, since the power supply is realized by a battery, if each means for realizing the AR is operated during the period from the start to the end of the shopping by the customer, There is a problem in that the capacity is reduced, the necessary power cannot be secured, and the product information cannot be provided.

  Further, in order to realize the service as described above, it is necessary to recognize what the product displayed on the display unit or the product imaged by the imaging means is to display the appropriate product information in a superimposed manner. is there. As a product recognition method, it is conceivable to employ either a marker method or a markerless method. The marker method is a method for recognizing a product by reading a product-specific marker displayed on each product. The markerless method is a method for recognizing a product based on a captured image obtained by capturing a part or the whole of the appearance of the product.

  When the marker method is employed, it is cheap and easy to implement a bar code assigned in advance for each product as a product-specific marker. However, in most cases, the barcode is displayed at a position that cannot be easily confirmed in the normal display state such as the back or side of the product. In this case, in order to read the barcode as a marker, the customer side Therefore, it is necessary to pick up the product and find the bar code display, which requires troublesome work for the customer. On the other hand, the markerless method can recognize a product based on a captured image including a part or the whole of the front surface of the product, that is, based on a captured image when a product in a normal display state is captured by the imaging unit. Therefore, the work which becomes troublesome as described above can be omitted.

  However, in the markerless method described above, product information is superimposed on the image-recognized product. Therefore, if a plurality of products are included in the captured image captured by the imaging unit, the image-recognized plurality Product information is acquired from the product information database storing the product information of each product. In this case, if all the acquired product information is superimposed and displayed, the product information is displayed in a complicated manner, so that the store visitor cannot easily obtain only the desired product information. .

  An object of the present invention is to provide a product information providing terminal device capable of efficiently suppressing power consumption and providing product information about a product displayed in a store using AR technology in real time. That is.

  Another object of the present invention is to provide a product information providing terminal device and a product information providing system that can easily provide only product information that a customer wants to know.

The present invention is a product information providing terminal device that includes a power supply unit, operates with power from the power supply unit, and is carried by a user,
An imaging means for imaging the real space;
A product information acquisition unit that acquires product information related to a product included in the captured image captured by the imaging unit;
Image generation for generating an augmented reality image for displaying the product information acquired by the product information acquisition unit superimposed on a real space based on the position of the product included in the captured image captured by the imaging unit And
A display unit capable of displaying the augmented reality image generated by the image generation unit;
Acceleration detecting means for detecting acceleration;
An operation state determination unit that determines an operation state of the user based on acceleration information detected by the acceleration detection unit;
A product information providing terminal device comprising: an operation control unit that activates and stops the operation of the imaging unit based on the operation state determined by the operation state determination unit.

In the present invention, the operation state determination unit determines whether the user is moving or not moving based on the acceleration information detected by the acceleration detection unit. And
The operation control unit activates the imaging unit when it is determined that the imaging unit is in a non-moving state when the imaging unit is not operating, and the operation control unit determines that the imaging unit is in a moving state when the imaging unit is operating. The operation of the imaging means is stopped.

The present invention also includes a head mounting portion for mounting on the user's head,
The acceleration detection means is held by the head mounting portion,
The operation state determination unit determines whether the user is moving or not moving based on the acceleration information detected by the acceleration detection unit. Determine whether or not the head is moving,
The operation control unit activates the imaging unit when it is determined that the imaging unit is in a non-moving state in a non-operating state and it is determined that the position of the head is not moved. The operation of the imaging unit is stopped when it is determined in the operating state that it is in a moving state or when it is determined that the position of the head is moving.

Further, the present invention provides an instruction information determination unit that determines whether or not the captured image captured by the imaging unit includes predetermined instruction information for specifying a product for which product information is to be displayed;
A captured image storage unit that stores a captured image to be determined by the instruction information determination unit,
The product information acquisition unit acquires product information related to a product included in a captured image based on a captured image when the instruction information determination unit determines that predetermined instruction information is included. To do.

  Further, the present invention is characterized in that the predetermined instruction information is a human finger included in a captured image.

Further, according to the present invention, the captured image storage unit determines that a first storage area in which a captured image is stored when it is determined that predetermined instruction information is included, and does not include predetermined instruction information. And a second storage area for storing a captured image when
When the instruction information determination unit determines that the predetermined instruction information is not included, the instruction information determination unit performs a determination on a captured image captured after the determination target captured image,
In the second storage area, a captured image when it is determined that predetermined instruction information is not included is updated and stored,
The product information acquisition unit acquires product information related to a product included in the captured image based on the captured image stored in the first storage area and the captured image stored in the second storage area. .

  Further, the present invention is characterized in that the operation control unit stops the operation of the imaging means when it is not determined that the predetermined instruction information is included within a predetermined period or within a predetermined number of determinations.

  According to the present invention, the product information acquisition unit is based on a captured image stored in the second storage area when it is not determined that the predetermined index is included within a predetermined period or within a predetermined number of determinations. Product information relating to the product included in the captured image is acquired.

  In the invention, it is preferable that the image generation unit has a function of tracking a product for which product information is to be displayed with respect to a captured image captured by the imaging unit.

In the present invention, the display unit is configured to transmit visible light.
The image generation unit generates an augmented reality image for superimposing and displaying the product information of the product on the real image of the product shown on the display unit at a position and size corresponding to the position and size of the real image of the product It is characterized by doing.

  According to the present invention, the image generation unit superimposes and draws the product information of the product on the captured image captured by the imaging unit at a position and a size corresponding to the position and size of the product included in the captured image. And generating an augmented reality image.

Further, the present invention is a product information providing system including a product information providing terminal device and a server device that includes a power supply unit, operates with power from the power supply unit, and is carried by a user,
The product information provision terminal device
An imaging means for imaging the real space;
A product information acquisition unit that acquires product information related to a product included in the captured image captured by the imaging unit;
Image generation for generating an augmented reality image for displaying the product information acquired by the product information acquisition unit superimposed on a real space based on the position of the product included in the captured image captured by the imaging unit And
A display unit capable of displaying the augmented reality image generated by the image generation unit;
Acceleration detecting means for detecting acceleration;
An operation state determination unit that determines an operation state of the user based on acceleration information detected by the acceleration detection unit;
An operation control unit that activates and deactivates the imaging unit based on the operation state determined by the operation state determination unit;
An instruction information determination unit that determines whether or not the captured image captured by the imaging unit includes predetermined instruction information for specifying a product for which product information is to be displayed;
A captured image storage unit that stores a captured image to be determined by the instruction information determination unit;
Including a terminal-side wireless transceiver that wirelessly communicates information with the server device,
Server device
A server-side wireless transmission / reception unit that wirelessly communicates information with the product information providing terminal device;
A product information database that stores product information for each product;
A product information providing system that includes a product information extraction unit that identifies a product for which product information is to be displayed based on predetermined instruction information and extracts product information related to the identified product from a product information database. .

  According to the present invention, the product information provision terminal device includes an imaging unit, an acceleration detection unit, an operation state determination unit, and an operation control unit, and is determined based on the acceleration of the product information provision terminal device. The imaging means can be activated and deactivated based on the operating state of the store visitor carrying the information providing terminal device.

  For example, when a visitor is moving in a store, the visitor is usually in a state of looking at multiple products and may want to know product information about a specific product. Is low. On the other hand, when stopping in the store, there is a strong interest in a specific product compared to other products, and there is a high possibility of wanting to know product information related to the specific product. Thus, according to the operation state of the store visitor, the imaging means is started and stopped, and the product information is efficiently provided, thereby efficiently suppressing power consumption in the product information providing terminal device. Can do.

  According to the present invention, the product information providing terminal device includes an instruction information determination unit, a captured image storage unit, and a product information acquisition unit, and predetermined instruction information for specifying a product for which product information is to be displayed. Based on the captured image when it is determined that the product is included, product information relating to the product included in the captured image is acquired. Therefore, even when a plurality of products are included in the captured image, it is possible to display only the product information of the product specified according to the predetermined instruction information. Thereby, it is possible to easily provide only product information that the customer wants to know.

  According to the present invention, the product information providing system includes a product information providing terminal device and a server device, and the product information providing terminal device includes an imaging unit, an acceleration detecting unit, an operation state determination unit, an operation The imaging means can be activated and deactivated based on the operating state of the store visitor carrying the merchandise information provision terminal device determined based on the acceleration of the merchandise information provision terminal device. Therefore, the power consumption in the product information providing terminal device can be efficiently suppressed by starting and stopping the imaging means according to the operating state of the store visitor and efficiently providing the product information. .

  The product information provision terminal device includes an instruction information determination unit, a captured image storage unit, a product information acquisition unit, and a terminal side wireless transmission / reception unit, and the server device includes a server side wireless transmission / reception unit, a product information database, and And a product information extraction unit. Thereby, the captured image when it is determined that the predetermined instruction information for specifying the product for which the product information is to be displayed is included is transmitted from the product information providing terminal device to the server device. Identify products for which product information is to be displayed based on predetermined instruction information, extract product information about the identified products from the product information database, send the extracted product information from the server device to the product information providing terminal device, and receive The augmented reality image generated based on the merchandise information is displayed on the display unit. Therefore, even when a plurality of products are included in the captured image, it is possible to display only the product information of the product specified according to the predetermined instruction information. Thereby, it is possible to easily provide only product information that the customer wants to know.

It is a perspective view showing typically HMD100 concerning one embodiment of the present invention. It is a block diagram which shows the structure of HMD100 which concerns on one Embodiment of this invention. 3 is a block diagram showing a functional configuration of a control device 30. FIG. It is a block diagram which shows the structure of the server apparatus 110 used in cooperation with HMD100 which concerns on one Embodiment of this invention. 3 is a block diagram showing a functional configuration of a control device 80. FIG. It is a flowchart for demonstrating operation | movement of HMD100 which concerns on one Embodiment of this invention. It is a flowchart for demonstrating operation | movement of HMD100 which concerns on one Embodiment of this invention. It is a figure which shows the state in which the wearer of HMD100 stood still in front of the goods shelf S in which goods AF were displayed. 2 is a diagram illustrating an example of a captured image captured by a camera. FIG. It is a figure which shows an example of the captured image in which a human finger is contained. 4 is a diagram illustrating an example of a captured image transmitted to the server apparatus 110. FIG. It is a figure which shows the display part 21 on which product information image Q1 was displayed. It is a figure which shows an example of the change of the merchandise information image Q1 accompanying the movement of the real image of the merchandise reflected on the display part 21. FIG. 5 is a flowchart for explaining the operation of a server device 110. It is a flowchart for demonstrating operation | movement when an interruption request | requirement generate | occur | produces. It is a flowchart for demonstrating operation | movement of HMD100 which concerns on other embodiment of this invention. It is a flowchart for demonstrating operation | movement of HMD100 which concerns on other embodiment of this invention.

Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the drawings.
FIG. 1 is a perspective view schematically showing a head-mounted display device (HMD: Head Mounted Display) 100 which is a product information providing terminal device according to an embodiment of the present invention. FIG. 2 is a block diagram showing a configuration of the HMD 100 according to the embodiment of the present invention.

  The HMD 100 is used in stores where various products are displayed. By using augmented reality technology, the merchandise information in the real space and product information drawn in CG are displayed in an integrated manner. The apparatus is configured to provide product information in real time. In the present embodiment, the HMD 100 is configured to provide product information in cooperation with a server device 110 described later.

  HMD100 which concerns on this embodiment is comprised so that it may be used in the state with which it attached to the visitor's head. HMD100 should just be formed in the shape which can be detachably mounted | worn with a human head. Thereby, since the visitor does not need to hold the product information providing terminal device by hand, he / she can shop with both hands as before. As schematically shown in FIG. 1, the HMD 100 according to the present embodiment includes a vine portion 51 that is a head-mounted portion that is worn on the ear of a visitor, and is formed in a glasses-like shape.

  The HMD 100 includes a camera 10, a display device 20, a control device 30, an acceleration detection unit 40, a wireless transmission / reception unit 50, and a power supply unit 60.

The camera 10 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (
This is a video camera that uses a Complementary Metal Oxide Semiconductor) sensor as an image sensor and can image a subject with an imaging cycle per image of, for example, 1/30 seconds (that is, a frame rate of 30 fps). The camera 10 is attached at a position close to the display unit 21 so that its imaging direction substantially matches the line-of-sight direction of the wearer of the HMD 100, images the real space, and sends image data of the captured image to the processing unit 31. Output. In the present embodiment, the camera 10 has an autofocus function.

  The display device 20 includes a pair of display units 21 for the right eye and the left eye that can display an augmented reality image for displaying product information in the virtual space superimposed on the product in the real space. The pair of display units 21 are respectively attached so as to be arranged in front of the eyes of the wearer of the HMD 100.

  In the present embodiment, the display unit 21 employs an optical see-through method. That is, the display unit 21 is configured to be able to transmit visible light, and the wearer of the HMD 100 can directly see the scene in the real space that has passed through the display unit 21 disposed in front of the eyes, and the display unit 21. The augmented reality image displayed on the screen can also be viewed. The augmented reality image in this case is a product information image indicating product information drawn by CG or the like.

  Therefore, the wearer of the HMD 100 can see the scene of the real space seen through the display unit 21, that is, the real image of the real space reflected on the display unit 21 and the product information image. Such a display unit 21 may be realized by, for example, a half mirror. Although not shown, the display device 20 includes a projection device for projecting a product information image generated by an image generation unit 106 described later on the display unit 21.

The control device 30 includes a processing unit 31, a ROM (Read Only Memory) 32, and a RAM (Random Access Memory) 33. The processing unit 31 is a CPU (Central
A program stored in the ROM 32 is executed to control each part of the HMD 100 and perform data processing. The ROM 32 stores programs and data executed by the processing unit 31. The RAM 33 is used as a program execution processing area, a temporary storage area for image data, and various work areas.

  The acceleration detection unit 40 is configured to detect acceleration acting on the HMD 100. In the present embodiment, the acceleration detection unit 40 uses a three-axis acceleration sensor that can detect accelerations in three directions orthogonal to each other (X-axis direction, Y-axis direction, and Z-axis direction). The acceleration detection unit 40 outputs a signal indicating a change in acceleration in three directions detected by the triaxial acceleration sensor to the processing unit 31.

  The wireless transmission / reception unit 50 includes a wireless communication module using a wireless communication method such as ZigBee and IEEE 802.15.4, and is configured to wirelessly communicate information with the server device 110 described later.

  The power supply unit 60 supplies necessary power to the battery 61, which is a power supply source for operating the HMD 100, and the camera 10, the display device 20, the control device 30, the acceleration detection unit 40, and the wireless transmission / reception unit 50. And a power supply circuit unit 62.

  The acceleration detection unit 40, and an operation state determination unit 101 and an operation control unit 102, which will be described later, are configured to always operate by power supply from the power supply unit 60 to detect acceleration and to execute predetermined calculation processing. Yes. That is, the acceleration detection unit 40 constantly outputs a signal indicating a change in acceleration to the operation state determination unit 101.

  The control device 30, the acceleration detection unit 40, and the wireless transmission / reception unit 50 are built in a device main body 52 attached to a vine 51 that is a head-mounted unit. That is, the acceleration detection unit 40 is held by the vine unit 51 and outputs a signal indicating a change in acceleration according to the movement of the head of the wearer of the HMD 100.

  FIG. 3 is a block diagram illustrating a functional configuration of the control device 30. The control device 30 includes an operation state determination unit 101, an operation control unit 102, an instruction information determination unit 103, a captured image storage unit 104, a product information acquisition unit 105, and an image generation unit 106. .

  The operation state determination unit 101 continuously receives signals indicating changes in acceleration in three directions detected by the acceleration detection unit 40, and determines the operation state of the wearer of the HMD 100 based on the signals.

  Specifically, it is determined whether the wearer is moving or not moving based on a continuous change in acceleration in the vertical direction accompanying the wearer's walk. . Further, in the present embodiment, since the acceleration detection unit 40 is mounted on the HMD 100, the position of the head is determined by an operation such as a wearer of the HMD 100 looking around based on a change in horizontal acceleration perpendicular to the vertical direction. Whether or not is moving.

  The operation control unit 102 activates and deactivates the camera 10 based on the operation state of the wearer of the HMD 100 determined by the operation state determination unit 101. In the present embodiment, the operation control unit 102 is configured to start and stop the display device 20 as well as the camera 10.

  When the camera 10 is activated by the operation control unit 102, the instruction information determination unit 103 includes predetermined instruction information for specifying a product for which product information is to be displayed in a captured image captured by the camera 10. It is determined whether or not. Then, based on the determination result, the image data of the captured image that is the determination target is stored in a predetermined storage area in the captured image storage unit 104. In the present embodiment, the instruction information determination unit 103 determines whether or not a human finger is included in the captured image as the predetermined instruction information.

  The captured image storage unit 104 stores image data of a captured image that is a determination target by the instruction information determination unit 103. The captured image storage unit 104 is determined to include a first storage area for storing image data of a captured image when it is determined that the predetermined instruction information is included, and the predetermined instruction information is not included. And a second storage area in which image data of the captured image is stored. In the second storage area, image data of a captured image when it is determined that predetermined instruction information is not included is sequentially updated and stored.

  When the image data is stored in the first storage area, the product information acquisition unit 105 instructs the wireless transmission / reception unit 50 to store the image data stored in the first storage area and the image data stored in the second storage area. Is transmitted to the server apparatus 110. If image data is not stored in the second storage area, only the image data stored in the first storage area is transmitted to the server device 110. When the wireless transmission / reception unit 50 receives information from the server device 110, the product information acquisition unit 105 acquires the information.

  Based on the information acquired by the product information acquisition unit 105, the image generation unit 106 recognizes a partial area in the captured image transmitted to the server device 110 as a product for which product information is to be displayed. Then, the image of the area recognized as the product for which the product information is to be displayed and the captured image captured by the camera 10 are collated (matched), and the image of the product for which the product information is to be displayed is included in the captured image. It is determined whether or not. When an image of a product is included, image data of a product information image is generated at a position and a size corresponding to the position and size of the product image. The image data generated by the image generation unit 106 is output to the display device 20 and displayed on the display unit 21.

  Further, the image generation unit 106 has a tracking function for tracking a product for which product information is to be displayed with respect to a captured image captured by the camera 10. As a result, the image generation unit 106 follows the movement of the real image of the product even if the real image of the product displayed on the display unit 21 is slightly moved due to the slight movement of the head of the wearer of the HMD 100. A product information image can be generated.

  FIG. 4 is a block diagram showing a configuration of the server device 110 used in cooperation with the HMD 100 according to an embodiment of the present invention. The server device 110 includes a product information database 71, an image recognition database 72, a control device 80, and a wireless transmission / reception unit 90.

  The merchandise information database 71 is merchandise information about each merchandise displayed in the store, such as merchandise ID, merchandise ID, merchandise name, manufacturer name, price, raw material, competing merchandise, characteristics, sales and evaluation by the purchaser. Is stored.

  The image recognition database 72 stores a product ID of each product displayed in the store and image data of a reference image captured in advance for each product in association with each other. The reference image is an image obtained by capturing the front surface of the product in the display state.

  The control device 80 includes a processing unit 81, a ROM 82, and a RAM 83. The processing unit 81 includes a CPU, executes a program stored in the ROM 82, controls each unit of the server device 110, and performs data processing. The ROM 82 stores programs executed by the processing unit 81 and data. The RAM 83 is used as a program execution processing area, a temporary storage area for image data, and various work areas.

  The wireless transmission / reception unit 90 includes a wireless communication module using a wireless communication method such as ZigBee and IEEE802.15.4, and is configured to wirelessly communicate information with the HMD 100.

  FIG. 5 is a block diagram showing a functional configuration of the control device 80. The control device 80 includes a product information extraction unit 111 and a transmission / reception unit control unit 112.

  When the wireless transmission / reception unit 90 receives the image data of the captured image from the HMD 100, the product information extraction unit 111 extracts the product information in the received captured image based on the image data of the reference image stored in the image recognition database 72. Identify products to be displayed. Then, product information relating to the identified product is extracted from the product information database 71. Further, the product information extraction unit 111 calculates the position (coordinates) and size of the specified product in the captured image transmitted from the HMD 100.

  When the product information extraction unit 111 extracts the product information and calculates the position and size of the product in the captured image, the transmission / reception unit control unit 112 calculates the extracted product information and the calculated product information to the wireless transmission / reception unit 90. A command for transmitting the position and size information of the product to the HMD 100 that has transmitted the captured image is output.

  6A and 6B are flowcharts for explaining the operation of the HMD 100 according to the embodiment of the present invention. In step s0, the store visitor wears the HMD 100 to start the operation.

  In step s1, the operation state determination unit 101 determines whether the wearer is in a moving state or a non-moving state based on a signal output from the acceleration detection unit 40. If it is determined that it is in the moving state, the process proceeds to step s1.

  In step s2, the operation state determination unit 101 determines whether the wearer is moving the position of the head based on the signal output from the acceleration detection unit 40. If it is determined that the position of the head is moved, the process proceeds to step s1, and if it is determined that the position of the head is not moved, the process proceeds to step s3.

  In step s3, the operation control unit 102 permits the acceptance of the interrupt request from the operation state determination unit 101. In step s4, the operation control unit 102 activates the camera 10. FIG. 7 is a diagram illustrating a state in which the wearer of the HMD 100 is stationary in front of the product shelf S on which the products A to F are displayed. As illustrated in FIG. 7, when the operation state determination unit 101 determines that the wearer of the HMD 100 is stationary, the operation control unit 102 activates the camera 10 mounted on the HMD 100 and captures an image in the imaging range V. Is started. FIG. 8 is a diagram illustrating an example of a captured image captured by the camera 10, FIG. 8A illustrates a captured image P1 including one product A, and FIG. 8B illustrates two products. A captured image P2 including A and B is shown. In step s 5, image data of the captured image captured by the camera 10 is input to the instruction information determination unit 103.

  In step s6, the instruction information determination unit 103 determines whether a captured image captured by the camera 10 includes a human finger based on the input image data. FIG. 9 is a diagram illustrating an example of a captured image including human fingers. FIG. 9A illustrates a state in which the wearer points to the product A for which the product information is desired to be known with his / her finger f. A captured image P3 is shown, and FIG. 9 (b) shows a captured image P4 in which the wearer is holding the product B that the wearer wants to know about the product information with his / her hand h. When it is determined that the human fingers f and h are included in the captured image as in the captured images P3 and P4 illustrated in FIG. 9, the process proceeds to step s7, and the captured images P1 and P2 illustrated in FIG. If it is determined that the fingers f and h are not included, the process proceeds to step s9.

  In step s7, the instruction information determination unit 103 stores the image data of the captured image determined to include a finger in the first storage area of the captured image storage unit 104. In step s8, the product information acquisition unit 105 causes the wireless transmission / reception unit 50 to transmit the image data stored in the first storage area to the server device 110. When the image data is transmitted, the process proceeds to step s21.

  In step s9, the operation control unit 102 starts measuring a determination continuation period that is a determination criterion for determining whether to continue the determination. The determination continuation period is set in advance and stored in the RAM 33. In step s <b> 10, the instruction information determination unit 103 stores the image data of the captured image determined not to include a finger in the second storage area of the captured image storage unit 104. If image data is already stored in the second storage area, it is updated and stored.

  In step s11, the operation control unit 102 determines whether or not the elapsed time from the measurement start time of the determination continuation period exceeds the set determination continuation period. When it is determined that the set determination duration has not been exceeded, the process proceeds to step s12, and the image data of the captured image captured by the camera 10 is input to the instruction information determination unit 103. If it is determined that the set determination duration has been exceeded, the process proceeds to step s16.

  In step s 12, the image data of the captured image captured by the camera 10 is input to the instruction information determination unit 103. In step s13, the instruction information determination unit 103 determines whether a captured image captured by the camera 10 includes a human finger based on the input image data. If it is determined that the finger is included, the process proceeds to step s14. If it is determined that the finger is not included, the process proceeds to step s10.

  In step s <b> 14, the instruction information determination unit 103 stores the image data of the captured image determined to include a finger in the first storage area of the captured image storage unit 104. In step s15, the product information acquisition unit 105 causes the wireless transmission / reception unit 50 to transmit the image data stored in the first storage area and the image data stored in the second storage area to the server device 110. FIG. 10 is a diagram illustrating an example of a captured image transmitted to the server apparatus 110 in step s15. FIG. 10A illustrates an image of a state where the wearer points to the product A with his / her finger f. FIG. 10B is an image P5 that is an image P5 obtained by capturing the image immediately before pointing to the product A with the finger f. When the image data is transmitted, the process proceeds to step s21.

  In step s <b> 16, the operation control unit 102 stops measuring the determination continuation period because it cannot capture an image including a finger within the determination continuation period. In step s17, the operation control unit 102 stops the imaging operation of the camera 10. In step s18, the operation control unit 102 prohibits acceptance of an interrupt request from the operation state determination unit 101.

  In step s19, the motion state determination unit 101 determines whether the wearer is in a moving state or a non-moving state based on a signal output from the acceleration detection unit 40. If it is determined that it is in the moving state, the process proceeds to step s1.

  In step s20, the operation state determination unit 101 determines whether the wearer is moving the position of the head based on the signal output from the acceleration detection unit 40. If it is determined that the position of the head is moved, the process proceeds to step s1, and if it is determined that the position of the head is not moved, the process proceeds to step s19.

  As described above, after the operation of the camera 10 is stopped in step s17, it is confirmed in steps s19 and s20 that the wearer has shifted to a moving state or a state of moving the head position. The process proceeds to step s1.

  In step s <b> 21, the product information acquisition unit 105 determines whether the wireless transmission / reception unit 50 has received data from the server device 110. If it is determined that the data has been received, the data is acquired, and the process proceeds to step s22. If it is determined that the data has not been received, the process proceeds to step s21.

  In step s22, the image generation unit 106 determines whether the product information is included in the data acquired by the product information acquisition unit 105. If it is determined that the product information is included, the process proceeds to step s23. If it is determined that the product information is not included, the process proceeds to step s27.

  In step s23, the image generation unit 106, based on the product position and size information acquired by the product information acquisition unit 105, and the image data of the captured image transmitted to the server device 110 in step s8 or step s15, A part of the region in the captured image is recognized as a product for which product information is to be displayed, and image data of the product information image is generated based on the product information acquired by the product information acquisition unit 105. In addition, the operation control unit 102 activates the display device 20.

  Then, the image generation unit 106 outputs the image data of the generated product information image to the display device 20. A product information image is displayed on the display unit 21 by the image data output to the display device 20. FIG. 11 is a diagram showing the display unit 21 on which the product information image Q1 is displayed. As shown in FIG. 11, a product information image Q1 is displayed on the display unit 21 adopting the optical see-through method so that the product information q is superimposed on the real image of the product A to be displayed. The wearer of the HMD 100 can acquire product information in real time by browsing the product information image Q1.

  In step s24, image data of a captured image captured by the camera 10 is input to the image generation unit 106. In step s25, the image generation unit 106 collates (matches) the image of a part of the region recognized as the product for which the product information is to be displayed in step s23 with the captured image input from the camera 10, and determines the captured image. It is determined whether or not an image of a product for which product information is to be displayed is included. If it is determined that the product image is included, the process proceeds to step s26. If it is determined that the product image is not included, the process proceeds to step s28.

  In step s26, in order to continuously display the product information image on the display unit 21, the image generation unit 106 tracks the product for which the product information is to be displayed, and the imaging input from the camera 10 based on the tracking result. Image data of a new product information image is generated for the image. Then, the image data of the generated product information image is output to the display device 20. Accordingly, the product information image is displayed on the display unit 21 following the movement of the real image of the product shown on the display unit 21.

  FIG. 12 is a diagram illustrating an example of a change in the product information image Q1 accompanying the movement of the real image of the product displayed on the display unit 21, and FIG. 12 (a) is a diagram before the real image of the product displayed on the display unit 21 is moved. FIG. 12B shows a state after the real image of the product shown on the display unit 21 has moved. As shown in FIG. 12, when the real image of the product shown on the display unit 21 moves slightly in the vertical direction of the display unit 21, for example, the product in which the position of the balloon Q2 is changed in accordance with the movement of the real image of the product. An information image is generated and displayed on the display unit 21. When the newly generated product information image is displayed on the display unit 21, the process proceeds to step s24.

  In step s27, the image generation unit 106 generates an image for indicating that product information has not been found. In addition, the operation control unit 102 activates the display device 20. Then, the image generation unit 106 outputs the image data of the generated image to the display device 20. Based on the image data output to the display device 20, the display unit 21 displays that product information has not been found.

  In step s28, the image generation unit 106 discovers the product information image or the product information for a preset display period in order to secure time for the wearer to confirm the display content displayed on the display unit 21. An image indicating that it has not been displayed is displayed on the display unit 21. When the display period elapses in step s29, the operation control unit 102 stops the display operation of the display device 20. In step s30, the operation control unit 102 stops the imaging operation of the camera 10. In step s31, the operation control unit 102 prohibits acceptance of an interrupt request from the operation state determination unit 101, and proceeds to step s1.

  In the present embodiment, the instruction information determination unit 103 is configured to repeat the determination on the captured image until a predetermined determination duration elapses when the instruction information is not included in the determination target captured image. ing. In another embodiment, the determination on the captured image may be repeated until a predetermined number of determinations is exceeded.

  FIG. 13 is a flowchart for explaining the operation of the server apparatus 110. In step a0, the operation is started. In step a <b> 1, the transmission / reception unit control unit 112 determines whether the wireless transmission / reception unit 90 has received image data from the HMD 100. If it is determined that the image data has been received, the process proceeds to step a2, and if it is determined that the image data has not been received, the process proceeds to step a1.

  In step a2, the product information extraction unit 111 determines whether the received image data is only image data of a captured image that is determined to include a finger. If it is determined that only the image data of the captured image determined to include the finger, the process proceeds to step a3, and it is determined that the image data is not only the image data of the captured image determined to include the finger. The process proceeds to step a4.

  In step a <b> 3, the product information extraction unit 111 recognizes an image based on the image data of the captured image determined to include a finger and the image data of the reference image stored in the image recognition database 72, and captures the image. Identify the products included in the image. The product information extraction unit 111 calculates the position and size of the specified product in the captured image.

  In step a4, the product information extraction unit 111 recognizes an image based on the image data of the captured image determined not to include a finger and the image data of the reference image stored in the image recognition database 72, and captures an image. Identify the products included in the image. The product information extraction unit 111 calculates the position and size of the specified product in the captured image.

  In step a5, the product information extraction unit 111 determines whether there is product information corresponding to the product information database 71 for the product included in the captured image. If it is determined that there is product information, the process proceeds to step a6. If it is determined that there is no product information, the process proceeds to step a11.

  In step a6, the product information extraction unit 111 determines whether there are a plurality of products specified as products included in the captured image. If it is determined that the number is not plural, the process proceeds to step a7. If it is determined that the number is plural, the process proceeds to step a8.

  In step a7, the transmission / reception unit control unit 112 causes the wireless transmission / reception unit 90 to transmit the product information acquired by the product information extraction unit 111 and the calculated product position and size information to the HMD 100. If each information is transmitted to HMD100, it will progress to Step a1.

  In step a8, the product information extraction unit 111 determines whether the received image data is only image data of a captured image that is determined not to include a finger. If it is determined that only the image data of the captured image determined not to include the finger, the process proceeds to step a9, and it is determined that the image data is not only the image data of the captured image determined not to include the finger. The process proceeds to step a10.

  In step a <b> 9, the product information extraction unit 111 selects a product closest to the camera 10 or a product closest to the center of the captured image among a plurality of products included in the captured image.

  As a method of selecting the product having the closest distance from the camera 10, the actual product size ratio obtained from each product information and the product size ratio calculated from the proportion of the product in the captured image are used. You may choose. As another method, the camera 10 may be set so as to be in focus with respect to the closest subject, and the one in focus may be selected.

  In step a10, the merchandise information extraction unit 111 selects a merchandise on which fingers are overlapped from among a plurality of merchandise included in the captured image. When a product is selected in step a9 or step a10, in step a7, product information about the selected product, and the position and size information of the product are transmitted to the HMD 100.

  In step a11, the transmission / reception unit control unit 112 causes the wireless transmission / reception unit 90 to transmit information indicating that product information corresponding to the product information database 71 has not been found to the HMD 100. When the information is transmitted to the HMD 100, the process proceeds to step a1.

  FIG. 14 is a flowchart for explaining the operation when an interrupt request is generated. The interrupt request is generated during a period in which the camera 10 is operating, that is, a period from when the interrupt request is permitted by the operation control unit 102 to when it is prohibited.

  In step b0, the operation is started by allowing the operation control unit 102 to accept the interrupt request from the operation state determining unit 101.

  In step b1, the operation state determination unit 101 determines whether the wearer is in a moving state or a non-moving state based on a signal output from the acceleration detection unit 40. If it is determined that it is in the moving state, the process proceeds to step b3. If it is determined that it is in the non-moving state, the process proceeds to step b2.

  In step b <b> 2, the motion state determination unit 101 determines whether the wearer is moving the position of the head based on the signal output from the acceleration detection unit 40. If it is determined that the position of the head is moved, the process proceeds to step b3. If it is determined that the position of the head is not moved, the process proceeds to step b1.

  In step b3, the operation state determination unit 101 makes an interrupt request to the operation control unit 102. In step b4, the operation control unit 102 prohibits acceptance of an interrupt request from the operation state determination unit 101 so as not to generate multiple interrupts, and performs the processing executed by the instruction information determination unit 103 or the image generation unit 106. Interrupt.

  In step b4, the operation control unit 102 stops the imaging operation of the camera 10, and ends the operation in step b5.

  As described above, the HMD 100 includes the acceleration detection unit 40, the operation state determination unit 101, and the operation control unit 102, and based on the operation state of the wearer of the HMD 100 determined based on the acceleration of the HMD 100, The camera 10 can be activated and stopped. That is, since the camera 10 is started and stopped according to the operating state of the wearer, power consumption in the HMD 100 can be efficiently suppressed by providing product information efficiently.

  The HMD 100 also includes an instruction information determination unit 103 and a captured image storage unit 104, and the captured image when it is determined that predetermined instruction information for specifying a product for which product information is to be displayed is included. Based on the information, the product information related to the product included in the captured image is acquired. Therefore, even when a plurality of products are included in the captured image, it is possible to display only the product information of the product specified according to the predetermined instruction information. Thereby, only the merchandise information which the wearer of HMD100 wants to know can be provided easily.

  15A and 15B are flowcharts for explaining the operation of the HMD 100 according to another embodiment of the present invention. The flowchart shown in FIG. 15A is similar to the flowchart shown in FIG. 6A. Specifically, the remaining steps are the same except that step s35 is adopted instead of steps s16 to s20 in FIG. 6A. is there. 15B is the same as the flowchart shown in FIG. 6B. Therefore, overlapping description of the same process is omitted.

  In the present embodiment, if it is determined in step s11 that the elapsed time from the measurement start time of the determination continuation period has exceeded a predetermined determination continuation period, the process proceeds to step s35. In step s35, the product information acquisition unit 105 causes the wireless transmission / reception unit 50 to transmit the image data stored in the second storage area to the server device 110.

  That is, in this embodiment, even when a captured image including a finger is not acquired, only the image data of the captured image not including the finger is transmitted to the server apparatus 110. As described above, a process for specifying a product based on a captured image that does not include a finger may be executed.

  In the embodiment described above, the case where the HMD 100 including the display unit 21 adopting the optical see-through method is used has been described. However, the present invention is not limited thereto, and product information is provided by a handy terminal configured to be carried by hand. A terminal device may be realized. In the case of a handy terminal, the display unit may be an optical see-through method or a video see-through method. When the video see-through method is adopted, a composite image in which product information is superimposed and drawn on a captured image including a product is generated as an augmented reality image.

DESCRIPTION OF SYMBOLS 10 Camera 20 Display apparatus 21 Display part 30 Control apparatus 40 Acceleration detection part 50 Wireless transmission / reception part 60 Power supply part 61 Battery 71 Product information database 72 Image recognition database 80 Control apparatus 90 Wireless transmission / reception part 100 Head-mounted display apparatus 101 Operation state determination Unit 102 Operation Control Unit 103 Instruction Information Determination Unit 104 Captured Image Storage Unit 105 Product Information Acquisition Unit 106 Image Generation Unit 110 Server Device 111 Product Information Extraction Unit 112 Transmission / Reception Unit Control Unit

Claims (12)

A product information providing terminal device that includes a power supply unit, operates with power from the power supply unit, and is carried by a user,
An imaging means for imaging the real space;
A product information acquisition unit that acquires product information related to a product included in the captured image captured by the imaging unit;
Image generation for generating an augmented reality image for displaying the product information acquired by the product information acquisition unit superimposed on a real space based on the position of the product included in the captured image captured by the imaging unit And
A display unit capable of displaying the augmented reality image generated by the image generation unit;
Acceleration detecting means for detecting acceleration;
An operation state determination unit that determines an operation state of the user based on acceleration information detected by the acceleration detection unit;
A product information provision terminal device comprising: an operation control unit that activates and deactivates the imaging unit based on the operation state determined by the operation state determination unit. The operation state determination unit determines whether the user is moving or not moving based on the acceleration information detected by the acceleration detection unit,
The operation control unit activates the imaging unit when it is determined that the imaging unit is in a non-moving state when the imaging unit is not operating, and the operation control unit determines that the imaging unit is in a moving state when the imaging unit is operating. The product information providing terminal device according to claim 1, wherein the operation of the imaging unit is stopped. It has a head mounting part for mounting on the user's head,
The acceleration detection means is held by the head mounting portion,
The operation state determination unit determines whether the user is moving or not moving based on the acceleration information detected by the acceleration detection unit. Determine whether or not the head is moving,
The operation control unit activates the imaging unit when it is determined that the imaging unit is in a non-moving state in a non-operating state and it is determined that the position of the head is not moved. 2. The product information provision according to claim 1, wherein the operation of the imaging unit is stopped when it is determined that the moving state is determined in the operating state or the position of the head is moved. Terminal device. An instruction information determination unit that determines whether or not the captured image captured by the imaging unit includes predetermined instruction information for specifying a product for which product information is to be displayed;
A captured image storage unit that stores a captured image to be determined by the instruction information determination unit,
The product information acquisition unit acquires product information related to a product included in a captured image based on a captured image when the instruction information determination unit determines that predetermined instruction information is included. The merchandise information provision terminal device according to any one of claims 1 to 3.   The product information providing terminal device according to claim 4, wherein the predetermined instruction information is a human finger included in a captured image. The captured image storage unit stores a first storage area for storing a captured image when it is determined that predetermined instruction information is included, and an imaging when it is determined that predetermined instruction information is not included A second storage area for storing images,
When the instruction information determination unit determines that the predetermined instruction information is not included, the instruction information determination unit performs a determination on a captured image captured after the determination target captured image,
In the second storage area, a captured image when it is determined that predetermined instruction information is not included is updated and stored,
The product information acquisition unit acquires product information related to a product included in the captured image based on the captured image stored in the first storage area and the captured image stored in the second storage area. The product information provision terminal device according to claim 4 or 5.   7. The operation control unit according to claim 6, wherein the operation control unit stops the operation of the imaging unit when it is not determined that the predetermined instruction information is included within a predetermined period or within a predetermined number of determinations. Product information providing terminal device.   When the product information acquisition unit does not determine that a predetermined index is included within a predetermined period or within a predetermined number of determinations, the product information acquisition unit generates a captured image based on the captured image stored in the second storage area. The product information providing terminal device according to claim 6, wherein the product information on the included product is acquired.   The product according to claim 1, wherein the image generation unit has a function of tracking a product for which product information is to be displayed with respect to a captured image captured by the imaging unit. Information providing terminal device. The display unit is configured to transmit visible light,
The image generation unit generates an augmented reality image for superimposing and displaying the product information of the product on the real image of the product shown on the display unit at a position and size corresponding to the position and size of the real image of the product The merchandise information provision terminal device according to any one of claims 1 to 9, wherein   The image generation unit superimposes and renders the product information of the product on the captured image captured by the imaging unit at a position and a size corresponding to the position and size of the product included in the captured image, thereby generating an augmented reality image. The product information provision terminal device according to claim 1, wherein the product information provision terminal device is generated. A product information providing system that includes a power supply unit, operates with power from the power supply unit, and includes a product information providing terminal device and a server device carried by a user,
The product information provision terminal device
An imaging means for imaging the real space;
A product information acquisition unit that acquires product information related to a product included in the captured image captured by the imaging unit;
Image generation for generating an augmented reality image for displaying the product information acquired by the product information acquisition unit superimposed on a real space based on the position of the product included in the captured image captured by the imaging unit And
A display unit capable of displaying the augmented reality image generated by the image generation unit;
Acceleration detecting means for detecting acceleration;
An operation state determination unit that determines an operation state of the user based on acceleration information detected by the acceleration detection unit;
An operation control unit that activates and deactivates the imaging unit based on the operation state determined by the operation state determination unit;
An instruction information determination unit that determines whether or not the captured image captured by the imaging unit includes predetermined instruction information for specifying a product for which product information is to be displayed;
A captured image storage unit that stores a captured image to be determined by the instruction information determination unit;
Including a terminal-side wireless transceiver that wirelessly communicates information with the server device,
Server device
A server-side wireless transmission / reception unit that wirelessly communicates information with the product information providing terminal device;
A product information database that stores product information for each product;
A product information providing system comprising: a product information extraction unit that identifies a product for which product information is to be displayed based on predetermined instruction information and extracts product information relating to the identified product from a product information database.

Images