WO2024013777A1

WO2024013777A1 - Shopping assistance system, shopping assistance method, and recording medium

Info

Publication number: WO2024013777A1
Application number: PCT/JP2022/027181
Authority: WO
Inventors: 佑樹鶴岡; 明彦大仁田; 祐史丹羽; 峰三宅
Original assignee: 日本電気株式会社
Priority date: 2022-07-11
Filing date: 2022-07-11
Publication date: 2024-01-18

Abstract

This shopping assistance system comprises a first output control unit, a commodity detection unit, and a second output control unit. The first output control unit causes a first display to show information relating to sales floors of a store where commodities are displayed. The commodity detection unit detects, on the basis of the action of a customer, a commodity that the customer wants to acquire. The second output control unit causes a second display to show information relating to the detected commodity.

Description

Shopping support system, shopping support method, and recording medium

The present disclosure relates to a shopping support system and the like.

Dome-shaped displays are known as means for displaying virtual spaces and images (for example, Patent Documents 1 and 2). Additionally, a wearable information processing terminal is known as a means for displaying virtual spaces and images (for example, Patent Document 3).

International Publication No. 2018/101279 International Publication No. 2017/187821 Japanese Patent Application Publication No. 2020-129356

For example, when using a large display such as a dome-shaped display as a display means, the user may perform operations using a controller. When operations are performed using a controller, there is a problem in that it is difficult to operate. For example, when shopping at a store in a virtual space or at a remote store, it may be troublesome for some users to use a controller to perform operations such as viewing products displayed at the store.

An example of the purpose of the present disclosure is to provide a shopping support system that can improve convenience when using a large display.

A shopping support system according to an aspect of the present disclosure includes a first output control means for displaying information regarding a sales floor of a store where products are displayed on a first display; The apparatus includes a product detection means for detecting a product, and a second output control means for displaying information regarding the detected product on a second display.

A shopping support method according to an aspect of the present disclosure includes displaying information about the sales floor of a store where products are displayed on a first display, detecting a product that the customer wants to acquire based on the customer's behavior, and displaying information on a second display. information regarding the detected product is displayed.

A program according to an aspect of the present disclosure causes a computer to display information regarding a sales floor of a store where products are displayed on a first display, detects a product that the customer wants to acquire based on the customer's behavior, and displays a second display. A display is caused to display information regarding the detected product and a process is executed.

The program may be stored in a computer-readable non-transitory recording medium.

According to the present disclosure, it is possible to improve convenience when using a large display.

1 is a block diagram showing an example of a configuration of a shopping support system according to a first embodiment; FIG. 3 is a flowchart illustrating an example of the operation of the shopping support system according to the first embodiment. FIG. 2 is an explanatory diagram showing an example of a dome-shaped display. It is an explanatory view showing an example of installation of an aerial display and an example of installation of devices around a customer. It is an explanatory view showing an example of a sales floor of a store. It is an explanatory view showing an aerial display simply. FIG. 2 is a block diagram showing an example of a configuration of a shopping support system according to a second embodiment. FIG. 2 is an explanatory diagram showing an example in which a store's sales floor is displayed on a dome-shaped display. FIG. 2 is an explanatory diagram showing an example of detecting a product from a customer's hand movement and an example of displaying information regarding the product on an aerial display. FIG. 6 is an explanatory diagram showing an example of detecting a product from the movement of a customer's line of sight and an example of displaying information regarding the product on an aerial display. FIG. 6 is an explanatory diagram showing an example of detecting a product from a conversation and an example of displaying information regarding the product on an aerial display. FIG. 6 is an explanatory diagram showing an example of changing the orientation of a displayed product by changing the position and orientation of an imaging device that images the product. FIG. 3 is an explanatory diagram showing an example of changing the orientation of a displayed product by switching the imaging device. It is an explanatory view showing an example of changing the orientation of a displayed product by rotating the stand. 7 is a flowchart illustrating an example of the operation of the shopping support system according to the second embodiment. FIG. 2 is an explanatory diagram showing an example of implementation of a shopping support system. FIG. 2 is an explanatory diagram showing an example of the hardware configuration of a computer.

Embodiments of a shopping support system, a shopping support method, a program, and a non-temporary recording medium for recording the program according to the present disclosure will be described in detail below with reference to the drawings. This embodiment does not limit the disclosed technology. Here, each embodiment may be explained using an aerial display as an example of a non-contact display.

In addition, in each embodiment, shopping is taken as an example, the user of the first display is called a customer, and the description is given as a shopping support system, but there are use cases in which the first display is combined with a second display such as an aerial display. is not limited to shopping.

(Embodiment 1)
First, in Embodiment 1, basic functions of a shopping support system will be explained. FIG. 1 is a block diagram showing an example of a configuration of a shopping support system according to a first embodiment. The shopping support system 10 includes a first output control section 101, a product detection section 102, and a second output control section 103.

The first output control unit 101 controls the display on the first display. The first display is, for example, a large display. Here, large size means at least larger than the second display. For example, the first display may be a display device structured to cover the customer's field of view. This allows customers to shop in an immersive manner. Here, a dome-shaped display will be described as an example of the first display. For example, the first output control unit 101 causes a dome-shaped display to display information regarding the department of a store where products are displayed. Here, the information regarding the sales floor is a video of the sales floor, an image of the sales floor, or the like. Note that the video here refers to a moving image, and the image refers to a still image. Note that the first output control unit 101 may cause the dome-shaped display to display information regarding other products, such as the name of the sales floor, as additional information, along with the video of the sales floor and the image of the sales floor. Here, stores include various retail stores such as supermarkets, super centers, convenience stores, mass retailers, home centers, drug stores, stores that handle apparel products, department stores, and individual stores such as bakeries and delicatessen stores. may be used, and is not particularly limited.

The product detection unit 102 detects products that the customer wants to acquire based on the customer's behavior. For example, the product that the customer wants to acquire is the product that the customer wants to check the details of. For example, a product that a customer wants to acquire is a product that attracts attention of the customer. Here, the customer's behavior is, for example, the customer's movement. Specifically, for example, the customer's behavior may be a specific movement of the customer's body, such as a movement of the customer's hand or a movement of the customer's line of sight. For example, the product detection unit 102 detects a customer's movement from an image captured by an imaging device. For example, if the customer's behavior is a movement of the customer's hand, the product detection unit 102 specifies the position where the customer extends his/her hand based on the movement of the customer's hand, and detects the product according to the specified position. do. The product corresponding to the specified position is, for example, a product displayed on the dome-shaped display that is on an extension of the specified position, a product that is within a predetermined range from the specified position, etc.

Also, for example, the customer's behavior may be a conversation between the customer and a store clerk. Therefore, the product detection unit 102 may detect the customer's behavior by performing voice recognition on the conversation.

The second output control unit 103 controls the display on the second display. The second display is a display for the customer's hand. The second display is, for example, a display that is closer to the customer than the first display. For example, the second display is between the customer and the first display. Further, for example, the second display is a display installed at a position where the customer can easily view the display. For example, the display area of the second display is smaller than the display area of the first display. For example, the first display displays information about the sales floor, such as images and videos of the sales floor, while the second display displays images and videos of products that have been noticed, such as products selected by customers. Display information about products. For this reason, for example, it is desirable that the second display be installed closer to the customer than the dome-shaped display and display information about the product in a way that is easy to see.

Furthermore, the second display may be a fixed display or a portable display. Here, as the second display, a non-contact type display will be described as an example, but a contact type display may be used.

Further, for example, the display target is not particularly limited, such as images, videos, characters and numbers, and special codes that can be read by any device.

For example, the second output control unit 103 displays information regarding the detected product on a non-contact display. For example, the information regarding the product includes an image of the product, a video of the product, information related to the product, and the like. For example, the information related to the product includes characters, numbers, codes, etc. related to the product, and is not particularly limited to the price of the product, the name of the product, the place of production, and the manufacturer of the product. The product code may be a code representing characters and numbers related to the product. The image of the product may be a real-time image or may be an image captured in advance. The image of the product may be a new image taken in real time, or may be an image taken in advance. Products are also shown in videos and images of the sales floor, but videos and images of products are videos and images that focus on the product.

(flowchart)
FIG. 2 is a flowchart showing an example of the operation of the shopping support system 10 according to the first embodiment. The first output control unit 101 causes the dome-shaped display to display information regarding the sales floor of the store (step S101).

The product detection unit 102 determines whether the product that the customer wants to acquire is detected (step S102). In step S102, the product detection unit 102 detects a product that the customer wants to acquire based on the customer's behavior. If the product that the customer wants to acquire is not detected (step S102: No), the product detection unit 102 returns to step S102.

If the product that the customer wants to acquire is detected (step S102: No), the second output control unit 103 displays information regarding the detected product on the non-contact display (step S103). The shopping support system 10 then ends the process.

As described above, in the first embodiment, the shopping support system 10 displays the store on the first display, which is a large display, and displays the products detected based on the customer's behavior on the second display. This makes it possible to improve convenience when using a large display such as a dome-shaped display. Therefore, a more realistic shopping experience can be provided. For example, the shopping support system 10 displays information about the store's sales floor on a dome-shaped display, and when a displayed product is selected by a customer, a large image or video of the selected product is displayed on a display at hand. You can also display additional information about the selected product. In addition, it is easier for customers to check information about each product or product if they can see it closer using a handheld display or the like, compared to checking information about each product or product on a dome-shaped display.

Also, for example, the second display is installed between the customer and the first display. When an aerial display is used as the second display, there is no physical space between the customer and the first display when the aerial display is not displaying information about the product. Therefore, when an aerial display is used as the second display, it does not interfere with the customer's hand movements and does not interfere with the customer's view when viewing the first display. Therefore, it is possible to provide a shopping experience that is closer to actual shopping, such as reaching for products and having an unobstructed conversation with store staff. Furthermore, by making the aerial display portion invisible to surrounding people, information such as the customer's personal information that the customer does not want other people to see can be presented to the customer via the aerial display. Note that since the first display is large, there is a risk that other people may see the information displayed. In this way, information that the user does not want others to see is content that is difficult to display on the first display. Furthermore, when an aerial display is used as the second display, the customer may be able to view the product three-dimensionally.

Further, for example, when a wearable information processing terminal such as an HMD (Head Mounted Display) is used as a display means, the user may not be able to see his/her hand while the virtual space is displayed. On the other hand, for example, when a dome-shaped display is used as the display means, the user has the advantage of being able to see his or her hand even when the virtual space is displayed. Further, when a dome-shaped display is used as a display means, since there is a distance between the dome-shaped display and the user, a second display such as a spatial display can be installed.

Additionally, when a dome-shaped display is used, it can be used for general purposes. For example, it is assumed that a dome-shaped display is installed in a place where everyone can access it, and that various users use the dome-shaped display. In such cases, some users may be unfamiliar with the controller. According to the first embodiment, it is possible to improve the operability during shopping. By improving convenience when using a dome-shaped display as in the first embodiment, it is possible to further enhance versatility. In addition, since videos and images of the sales floor are displayed on the dome-shaped display 21 that covers the customer's field of view, the customer can shop with an immersive feeling. Therefore, it is possible to provide a shopping experience closer to actual shopping.

However, the operation by the controller and the shopping support system 10 may be used in combination as appropriate.

(Embodiment 2)
Next, Embodiment 2 will be described in detail with reference to the drawings. In Embodiment 2, an example of a dome-shaped display as a first display, an installation example of an aerial display as a second display, etc. will be described. Furthermore, in Embodiment 2, an example will be described in which the display of the dome-shaped display is controlled in order to make the displayed product easier to see while displaying information about the product on the aerial display. Hereinafter, a description of contents that overlap with the above description will be omitted to the extent that the description of the second embodiment is not unclear.

FIG. 3 is an explanatory diagram showing an example of a dome-shaped display. FIG. 4 is an explanatory diagram showing an example of installing an aerial display and an example of installing devices around a customer. In FIG. 3, the dome-shaped display 21 includes, for example, a screen 2101, a projection device 2102, and a table 2103.

When the dome-shaped display 21 displays a video or image, it means that the projection device 2102 projects the video or image onto the screen 2101.

For example, the aerial display 22 is installed on the table 2103. Further, in FIG. 4, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are omitted. In FIG. 4, a customer is sitting on a chair and looking at apples displayed on the dome-shaped display and the aerial display 22. Although FIG. 4 shows an example in which the table 2103 is viewed from the side, in reality, the aerial display 22 may not have a thickness for aerial imaging, which will be described later. In addition, in such a case, if the aerial image is viewed from the side in the thickness direction, the aerial image cannot be seen, but in the following explanation, for ease of understanding, This will be explained using an example in which an apple or the like is displayed. Furthermore, in FIG. 4, an imaging device 23 and a recording device 24 may be installed at the location where the dome-shaped display 21 is installed. For example, the imaging device 23 and the recording device 24 are used to detect customer behavior. Although not shown, an audio output device such as a speaker for outputting the voice of the store clerk may be installed at the location where the dome-shaped display 21 is installed.

The number of imaging devices 23, recording devices 24, and aerial displays 22 is not particularly limited. For example, the imaging device 23 may transmit the captured video or image to a shopping support system or the like. Furthermore, the recording device 24 may transmit the audio to a shopping support system or the like.

The dome-shaped display 21 is, for example, a device in which at least a portion of the screen 2101 is curved to cover the customer's field of view. Note that the dome-shaped display 21 may not have a complete dome shape such as 360 degrees, but may have a partially broken shape such as 180 degrees. Furthermore, in the dome-shaped display 21, a portion of the screen 2101 does not need to be curved. Furthermore, the dome-shaped display 21 is sized to create a gap between it and the user that is large enough to allow the installation of the table 2103 on which the aerial display 22 is installed. In this way, the size and type of the dome-shaped display 21 are not particularly limited. For example, the dome-shaped display 21 may be a 180-degree dome-shaped display 21 or a 360-degree dome-shaped display 21. Further, for example, the size of the dome-shaped display 21 may be, for example, approximately 1 meter to 2 meters in length, 1 meter to 2 meters in width, and 1 meter to 2 meters in height.

Further, the installation location of the dome-shaped display 21 is not particularly limited. For example, the dome-shaped display 21 may be installed in a customer's home, office, etc., or may be installed in a place where anyone can use it.

FIG. 5 is an explanatory diagram showing an example of a store's sales floor. For example, in the sales floor of a store, for example, product shelves are installed. Products are displayed on the installed shelves. The store may be a store in a virtual space or an actual store. The store in the virtual space may be a virtual store that does not actually exist, or a virtual store that imitates an actual store. If there is an actual store and an image of the actual store is displayed on the dome-shaped display 21, for example, an imaging device 25 and a recording device 26 are installed in the sales floor of the actual store. Further, although not shown, an audio output device such as a speaker for outputting customer voices may be installed in the sales floor of the actual store.

In FIG. 5, two imaging devices 25-1 and 25-2 are installed, but the number of imaging devices 25 is not particularly limited. Note that if the imaging device 25 is not particularly limited, it will be referred to as an imaging device 25. The imaging device 25 images the sales floor of the store. Further, the imaging device 25 may take an image of a store clerk. Further, the imaging device 25 may take images of each product in the store. Here, the imaging device 25-1 may take an image of the sales floor of the store including the store staff, and the imaging device 25-2 may take an image of a specified product. For example, the imaging device 25 may transmit the captured video to a shopping support system or the like.

Additionally, the recording device 26 records or collects the voices of the sales floor. Then, for example, the recording device 26 may transmit the audio to a shopping support system or the like.

Although FIG. 5 shows an example of an actual store, a store in a virtual space may also be used. For example, a store in virtual space may be used when there is no actual store.

FIG. 6 is an explanatory diagram that simply shows an aerial display. The aerial display 22 includes, for example, an optical element, a display, and a sensor.

In the aerial display 22, optical elements pass the light rays emitted by the display's display and form the same image on the side opposite the display. In FIG. 6, this image is an aerial image. Sensors are used to manipulate images in the air. The sensor is, for example, a motion sensor. In the following description, for example, an operation that can be detected by a sensor will also be referred to as an operation on the aerial display. The type of aerial display is not particularly limited.

FIG. 7 is a block diagram showing an example of the configuration of a shopping support system according to the second embodiment. The shopping support system 20 includes a first output control section 201, a product detection section 202, a second output control section 203, an acquisition section 204, a video analysis section 205, an audio analysis section 206, and an imaging device control section 207. , a platform control section 208 , a superimposition position specifying section 209 , a visual position specifying section 210 , and an operation receiving section 211 .

The shopping support system 20 includes an acquisition unit 204, a video analysis unit 205, an audio analysis unit 206, an imaging device control unit 207, a stand control unit 208, in addition to the shopping support system 10 according to the first embodiment. A superimposition position specifying section 209, a visible position specifying section 210, and an operation receiving section 211 are added. The first output control section 201 has the functions of the first output control section 101 according to the first embodiment as a basic function. The product detection unit 202 has the function of the product detection unit 102 according to the first embodiment as a basic function. The second output control section 203 has the functions of the second output control section 103 according to the first embodiment as a basic function.

For example, the shopping support system 20 includes a customer DB (DataBase) 2001, a store DB 2002, and a product DB 2003. Note that each functional unit of the shopping support system 20 can refer to and update various databases as appropriate.

For example, the customer DB 2001 stores customer video and audio, customer behavior history, and the like. The behavior history may be a history of the customer's movements detected from video, a history of conversation behavior detected from audio, or a history of the customer's operations on the aerial display 22.

Also, for example, the store DB 2002 stores store videos, store images, store sounds, and the like.

The product DB 2003 stores product identification information and product information in association with each other for each product. The product information includes, for example, a video of the product, an image of the product, a price of the product, a name of the product, a display position of the product on the sales floor, characteristics of the product, and the like. The characteristics of the product may be the color of the product, the shape of the product, the production area of the product, the raw materials of the product, the manufacturing company of the product, and the like. For example, the characteristics of a product may be used to identify the product based on a conversation between a customer and a store employee, so there is no particular limitation on the characteristics as long as the information allows the product to be identified.

Next, each functional section will be explained.

For example, the acquisition unit 204 acquires a captured image of the customer from the imaging device 23. For example, the acquired video data is stored in the customer DB 2001, for example. Further, for example, the acquisition unit 204 acquires the customer's voice from the recording device 24. For example, the acquired voice data is stored in the customer DB 2001, for example.

For example, if there is an actual store, the acquisition unit 204 acquires a captured image of the store from the imaging device 25. For example, the acquired video data is stored in the store DB 2002, for example. Note that the acquisition unit 204 may acquire an image of a store created in advance. For example, when there is no actual store, an image of the store in virtual space may be created. For example, the acquired video data is stored in the store DB 2002, for example.

If there is an actual store, the acquisition unit 204 acquires the store's audio from the recording device 26. The acquired voice data is stored in the store DB 2002, for example.

Additionally, the acquisition unit 204 may acquire audio in the virtual space.

The first output control unit 201 causes the dome-shaped display 21 to display information regarding the department of the store where the products are displayed. For example, information regarding the sales floor of a store includes a video of the sales floor, an image of the sales floor, and the like. Specifically, if there is an actual store, for example, the first output control unit 201 causes the dome-shaped display 21 to display an image of the sales floor acquired from the imaging device 25. This allows customers to view video of the sales floor captured in real time, giving them the feeling of shopping in an actual store.

FIG. 8 is an explanatory diagram showing an example in which the sales floor of a store is displayed on the dome-shaped display 21. In FIG. 8, the dome-shaped display 21 displays a sales floor as shown in FIG. For example, the customer can sit on a chair and view the sales floor.

Returning to the explanation of FIG. 7. The first output control unit 201 further causes the dome-shaped display 21 to display information regarding the store clerk. Information regarding the store clerk includes a video of the store clerk, an image of the store clerk, the name of the store clerk, the employee's affiliation, characteristics of the store employee, and the like. The employee's affiliation is the employee's affiliation at the store, such as "in charge of vegetable section," for example. The characteristics of the store clerk may be, for example, qualifications held. For example, if a store employee is present in an actual store, the imaging device 25 captures an image of the sales floor including the store employee. Then, the first output control unit 201 causes the dome-shaped display 21 to display the captured video. In the example of FIG. 8, an image captured including the store clerk at the sales floor of FIG. 5 is displayed. Further, when displaying information regarding a store clerk, the first output control section 201 may display an avatar of the store clerk. For example, when displaying a store clerk's avatar, the imaging device 25 images the sales floor. The first output control unit 201 may then combine the captured video with the store clerk's avatar and display the combined video. Alternatively, for example, the first output control unit 201 may combine the store clerk's avatar with the video of the sales floor in the virtual space, and display the combined video.

Note that if the voice of the store clerk can be acquired from the recording device 24, the acquisition unit 204 may acquire the voice of the customer and the store clerk from the recording device 24.

The product detection unit 202 detects products that the customer wants to acquire based on the customer's behavior. First, in order for the product detection unit 202 to detect the product, for example, the video analysis unit 205 analyzes the customer's behavior from the video. Then, the product detection unit 202 detects the product based on the analyzed customer behavior. Alternatively, in order for the product detection unit 202 to detect the product, the voice analysis unit 206 analyzes the customer's behavior from the voice. Then, the product detection unit 202 detects the product based on the analyzed customer behavior. Note that examples of product detection will be described later for each customer's behavior.

Then, the second output control unit 203 causes the aerial display 22 to display information regarding the detected product. The information regarding the product may be a video of the product, an image of the product, or information related to the product such as the name of the product. The image of the product may be, for example, a captured image or an image that reproduces the product in a virtual space. The captured video may be a video of the product captured in real time by the imaging device 25, or may be a video of the product captured in advance. Further, the image of the product may be a captured image of the product, an image of the product such as an illustration, or an image of the product reproduced in virtual space. The captured image may be an image of the product captured in real time by the imaging device 25, or may be an image of the product captured in advance.

<Example of product detection>
Next, examples of product detection will be explained based on customer behavior.

<Example of product detection based on hand movements>
First, we will explain the case where the customer's behavior is the customer's hand movement as an example. For example, if the customer's behavior is a movement of the customer's hand, the video analysis unit 205 may detect the movement of the customer's hand from the video captured by the imaging device 23. For example, the product detection unit 202 detects a product that the customer wants to acquire based on the customer's hand movements. As a more specific method of detecting the product, for example, the product detection unit 202 detects the product based on the position where the customer extends his/her hand. For example, the product detection unit 202 detects a product located at a position where a customer extends his or her hand or a product located near the position. A product near the position where the customer extends his/her hand may be a product located on an extension of the position where the customer extends his or her hand, or a product that is closest to the position where the customer extends his or her hand. You can.

Here, an example of displaying information regarding a product on the aerial display 22 based on detection of a customer's hand movement will be described using FIG. 9.

FIG. 9 is an explanatory diagram showing an example of detecting a product from a customer's hand movement and an example of displaying information regarding the product on the aerial display 22. In FIG. 9, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are not shown. For example, the imaging device 23 images a customer. The acquisition unit 204 then acquires the captured image of the customer in real time. The product detection unit 202 then detects the customer's behavior from the video.

In FIG. 9, the video analysis unit 205 detects that the customer stands up from a sitting position and extends his hand toward the dome-shaped display 21 based on the video. Then, in FIG. 9, the product detection unit 202 detects the product at the end of the customer's outstretched hand from among the products on the sales floor displayed on the dome-shaped display 21. Then, the second output control unit 203 causes the aerial display 22 to output information regarding the detected product. Accordingly, in FIG. 9, the aerial display 22 displays information regarding the product, such as an image of the product or a video of the product. In the example of FIG. 9, an apple is displayed.

Returning to the explanation of FIG. 7. Hand movements are not limited to those detected by a sensor such as the imaging device 23, but may also be detected by a dedicated glove or the like. Note that the sensor is not limited to the imaging device 23, and can be modified in various ways.

<Example of product detection based on eye movement>
Next, an example will be explained in which the customer's behavior is the movement of the customer's line of sight. For example, if the customer's behavior is the movement of the customer's line of sight, the video analysis unit 205 may detect the movement of the customer's line of sight from the image captured by the imaging device 23.

For example, the product that the customer is watching may be the product that the customer wants to check. Therefore, the product detection unit 202 detects the product that the customer is watching as a product that the customer wants to acquire. Note that the products that customers are looking at include products that the customer's gaze is directed toward, products that the customer's gaze is directed toward for more than a predetermined amount of time, and products that the customer's gaze is directed toward for more than a predetermined number of times. . The method of identifying the product that the customer is watching is not particularly limited.

Here, an example of displaying information regarding products on the aerial display 22 from detection of a customer's line of sight will be described using FIG. 10.

FIG. 10 is an explanatory diagram showing an example of detecting a product from the movement of the customer's line of sight and an example of displaying information regarding the product on the aerial display 22. In FIG. 10, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are not shown. For example, the imaging device 23 images a customer. The acquisition unit 204 then acquires the captured image of the customer in real time. Then, the video analysis unit 205 detects the movement of the customer's line of sight from the video.

In FIG. 10, the product detection unit 202 detects the product on which the customer has been looking for a predetermined period of time or more from among the products on the sales floor displayed on the dome-shaped display 21 based on the video. Then, the second output control unit 203 causes the aerial display 22 to display information regarding the detected product. Accordingly, in FIG. 10, the aerial display 22 displays information regarding the product, such as an image of the product or a video of the product. In the example of FIG. 10, an apple is displayed.

Returning to the explanation of FIG. 7. The movement of the line of sight is not limited to the example detected by the imaging device 23, but may be detected by another sensor. Furthermore, the direction of the face may be used regardless of the movement of the line of sight.

<Example of product detection through conversation>
Next, an example will be explained in which the customer's action is a conversation with a store clerk. For example, if the customer's behavior is a conversation between a store clerk and the customer, the voice analysis unit 206 recognizes the conversation between the customer and the store clerk. Then, the voice analysis unit 206 detects keywords that can identify the product, such as the name of the product, the display position of the product on the sales floor, the price of the product, and the characteristics of the product, from the conversation. The characteristics of the product may be the color of the product, the shape of the product, the production area of the product, the raw materials of the product, the manufacturing company of the product, and the like. The position of the product on the sales floor may be the right end, the left end, the upper right, the position on the shelf, etc., and is not particularly limited. For example, a keyword that includes the location of a product is, for example, "apple on the far right." For example, the product detection unit 202 detects a product matching the detected keyword from the product DB 2003. Note that the keyword may be a keyword issued by a customer or a keyword issued by a store clerk.

Here, an example of detecting a product from a conversation and displaying information regarding the product on the aerial display 22 will be described using FIG. 11.

FIG. 11 is an explanatory diagram showing an example of detecting a product from a conversation and an example of displaying information regarding the product on the aerial display 22. In FIG. 11, the screen 2101 of the dome-shaped display 21 and the projection device 2102 are not shown. For example, the recording device 24 records audio or collects audio. The acquisition unit 204 then acquires audio from the recording device 24 in real time. Then, the voice analysis unit 206 detects information that can identify the product from the conversation by performing voice recognition on the conversation.

In FIG. 11, for example, the product detection unit 202 detects the keyword "apple on the far right" from the conversation. Then, the second output control unit 203 causes the aerial display 22 to display information regarding the detected product. Accordingly, in FIG. 11, the aerial display 22 displays information regarding the product, such as an image of the product or a video of the product. In the example of FIG. 11, an apple is displayed.

Returning to the explanation of FIG. 7. The customer's behavior obtained from the voice is not limited to the conversation between the customer and the store clerk, and the customer's behavior may be, for example, the customer's utterance.

This concludes the explanation of the product detection example. As described above, there are various methods for detecting products, and these methods may be combined as appropriate. For example, the product detection unit 202 may detect a product that the customer wants to acquire based on the analyzed conversation and the analyzed eye movement.

<Example of changing the orientation of a product in an image or video>
Here, an example will be described in which, while an image or video of a product is being displayed on the aerial display 22, the display orientation of the product in the image or video is changed.

For example, the video analysis unit 205 detects the customer's hand movement with respect to the aerial display 22 by analyzing the customer's video. Specifically, for example, the video analysis unit 205 may detect a hand movement such as turning a product. For example, the video analysis unit 205 may detect whether the hand movement is turning the product to the left or the hand movement is turning the product to the right.

Next, the shopping support system 20 performs control such that the orientation of the product in the image or video displayed on the aerial display 22 is changed based on the hand movement. There are various examples of control methods. Here, we will explain using a product video as an example.

First, an example will be described in which at least one of the imaging position and orientation of the imaging device 25 installed in a store is controlled based on hand movements. For example, if the product image is a product image captured in real time by an image capture device 25 installed in a store, the image capture device control unit 207 determines the position of the image capture device 25 and At least one of the orientations of the imaging device 25 is controlled. This allows the imaging device 25 to take images of the product in different orientations. More specifically, for example, when a hand movement such as turning the product to the left is detected, the imaging device control unit 207 changes the imaging position and orientation of the imaging device 25 so that the left side of the product currently being imaged can be imaged. control at least one of the following. The imaging position and direction may be specified according to the movement of the hand. Note that in a store, products may be placed on a stand, and the imaging device 25 may take an image of the product placed on the stand.

Here, the method of controlling the imaging position and orientation of the imaging device 25 is not particularly limited. For example, when the imaging device 25 is attached to a robot arm, the imaging device control unit 207 can control the imaging position of the imaging device 25 by controlling the robot arm.

FIG. 12 is an explanatory diagram showing an example of changing the orientation of a displayed product by changing the position and orientation of the imaging device 25 that images the product. First, at the store, the imaging device 25 captures an image of a product placed on a stand 27. In FIG. 12, an apple is placed on a stand 27 as a product.

Then, the second output control unit 203 displays the image of the product on the aerial display 22. In FIG. 12, an apple is displayed as a product on the aerial display 22.

In FIG. 12, the video analysis unit 205 analyzes the customer's hand movements by analyzing the customer's video captured by the imaging device 23.

In FIG. 12, the imaging device control unit 207 controls the position and orientation of the imaging device 25 based on the movement of the customer's hand. In FIG. 12, the position and orientation of the imaging device 25 have changed. As a result, an image of the product whose orientation has been changed is displayed on the aerial display 22. For example, in FIG. 12, other parts of the apple are displayed on the aerial display 22.

Returning to the explanation of FIG. 7. For example, when the second output control unit 203 causes the aerial display 22 to display an image of a product captured in real time by an imaging device 25 selected from a plurality of imaging devices 25 installed in a store, the second output control unit 203 A new imaging device 25 is selected based on the hand movement. Then, the second output control unit 203 causes the aerial display 22 to display the video imaged by the newly selected imaging device 25. In this manner, by switching the imaging device 25, the orientation of the product displayed on the aerial display 22 can be changed.

FIG. 13 is an explanatory diagram showing an example of changing the orientation of the displayed product by switching the imaging device 25. First, an imaging device 25-1 and an imaging device 25-2 are installed in a store. The imaging device 25-1 and the imaging device 25-2 capture images of the products placed on the stand 27 in the store. In FIG. 13, an apple is placed on a stand 27 as a product.

For example, the second output control unit 203 causes the aerial display 22 to display an image of the product captured by the imaging device 25-1. In FIG. 13, an apple is displayed as a product on the aerial display 22.

In FIG. 13, the video analysis unit 205 analyzes the customer's hand movements by analyzing the customer's video captured by the imaging device 23.

In FIG. 13, the second output control unit 203 selects the imaging device 25-2 from the imaging device 25-1 and the imaging device 25-2 based on the customer's hand movement, and selects the imaging device 25-2 from the imaging device 25-1 and the imaging device 25-2. 25-2 is displayed. As a result, the aerial display 22 displays an image of the product whose orientation has been changed. For example, in FIG. 13, other parts of the apple are displayed on the aerial display 22.

Returning to the explanation of FIG. 7. For example, if the product image is a product image captured in real time by an imaging device 25 installed in a store, the product to be displayed on the aerial display 22 is placed on the stand 27 by a store employee, and the imaging device 25 There are cases where an image of the product placed on the stand 27 is taken. For example, the stand 27 may be a rotatable stand 27. In such a case, the stand control unit 208 controls the rotation of the stand 27. This allows the orientation of the product to be imaged to be changed. The table control unit 208 controls the rotation of the table 27 based on the customer's hand movements. This allows the imaging device 25 to take images of the product in different orientations. More specifically, for example, when a hand movement such as turning the product to the left is detected, the imaging device control unit 207 rotates the stand 27 to the left so that the left side of the product currently being imaged can be imaged. . The amount of rotation may be specified according to the movement of the hand. Moreover, the table control unit 208 may control the height of the table 27 without being limited to the example of rotating the table 27. If the position of the stand 27 can be changed, the stand control unit 208 changes the position of the stand 27. Thereby, by controlling the height, position, and rotation of the stand 27, the stand control unit 208 can change the orientation of the product imaged by the imaging device 25.

FIG. 14 is an explanatory diagram showing an example of changing the orientation of the displayed product by rotating the table 27. First, the imaging device 25 captures an image of a product placed on a stand 27 in a store. In FIG. 14, an apple is placed on a stand 27 as a product.

Then, the second output control unit 203 displays the image of the product on the aerial display 22. In FIG. 14, an apple is displayed as a product on the aerial display 22.

In FIG. 14, the video analysis unit 205 analyzes the customer's hand movements by analyzing the customer's video captured by the imaging device 23.

Then, in FIG. 14, the table control unit 208 rotates the table 27 based on the customer's hand movement. As a result, the aerial display 22 displays an image of the product whose orientation has been changed. For example, in FIG. 14, other parts of the apple are displayed on the aerial display 22.

Returning to the explanation of FIG. 7. For example, if the image of the product is an image that has been captured in advance, the second output control unit 203 outputs an image of the product that is oriented in accordance with the movement of the hand from among the images of the product that has been captured in advance. Display.

In the above example, control is performed such that the direction of the product in the image displayed on the aerial display 22 is changed based on the customer's hand movement. Control may be performed such that the orientation of the product is changed. For example, first, the second output control unit 203 displays one of a plurality of images of the product taken from different directions. Then, when the movement of the customer's hand is detected, the second output control unit 203 displays an image taken in an orientation corresponding to the movement of the customer's hand from among the plurality of images.

Furthermore, when newly capturing an image of the product to be displayed, at least one of the imaging position and orientation of the imaging device 25 may be controlled by the imaging device control unit 207, similarly to the video example. For example, the imaging device control unit 207 controls at least one of the position and orientation of the imaging device 25, and after the position and orientation are controlled, the imaging device 25 newly captures an image of the product. Then, the second output control unit 203 displays the newly captured image of the product.

Furthermore, when newly capturing an image of the product to be displayed, the imaging device 25 may be switched by the second output control unit 203, similarly to the video example. When the second output control unit 203 causes the aerial display 22 to display an image of the product captured by the imaging device 25 selected from the plurality of imaging devices 25 installed in the store, the second output control unit 203 displays the hand movement from the plurality of imaging devices 25. A new imaging device 25 is selected based on. The newly selected imaging device 25 then captures an image of the product. Then, the second output control unit 203 causes the aerial display 22 to display the image captured by the newly selected imaging device 25.

Furthermore, when a new image of the product to be displayed is captured, the stand 27 may be controlled by the stand control unit 208, similarly to the video example. The table control unit 208 controls the rotation of the table 27 based on the customer's hand movements. The imaging device 25 then takes a new image of the product in a different orientation. Then, the second output control unit 203 causes the aerial display 22 to display the image newly captured by the imaging device 25.

Further, in the above example, control is performed such that the direction of the product is changed in the image displayed on the aerial display 22 based on the customer's hand movement, but the control is not limited to the hand movement. In response to an operation on the aerial display 22, control may be performed such that the orientation of the product in the video or image displayed on the aerial display 22 is changed.

For example, the aerial display 22 can accept user operations. Therefore, the operation reception unit 211 may accept an operation on the aerial display 22.

Similarly to the hand movement example, the imaging device control unit 207 may control at least one of the imaging position and orientation of the imaging device 25 based on the received operation. Alternatively, similar to the hand movement example, the second output control unit 203 newly selects an imaging device 25 from the plurality of imaging devices 25 based on the hand movement based on the received operation. Then, the second output control unit 203 causes the aerial display 22 to display the video imaged by the newly selected imaging device 25. Alternatively, similarly to the example of hand movement, the stand control unit 208 controls the rotation of the stand 27 based on the received operation. Alternatively, as in the example of hand movements, if the image of the product is an image that has been captured in advance, the first output control unit 201 may select a product corresponding to the received operation from among the images that have been captured in advance. Display an image that is facing the same direction.

Note that a detailed explanation of the specific control example of each functional unit based on the received operation is omitted because it is the same as the example of hand movement.

<Controlling the display of the dome-shaped display 21 while the aerial display 22 is displaying information about the product>
For example, the background of the aerial display 22 may be visible through it. When a customer views the product displayed on the aerial display 22, the dome-shaped display 21 is in front of the customer's line of sight. The customer can see the display on the dome-shaped display 21 in the background of the product and beyond the transparent product. In this way, while the information regarding the product is being displayed on the aerial display 22, it may be difficult for the customer to see the display on the dome-shaped display 21.

Therefore, the first output control unit 201 changes the display on the dome-shaped display 21 so that the product displayed on the aerial display 22 does not become difficult to see. Thereby, when the aerial display 22 and the dome-shaped display 21 are combined, it is possible to prevent the product from becoming difficult to see. To change the display on the dome-shaped display 21, the first output control unit 201 may change the entire display on the dome-shaped display 21, or may change the display on a part of the dome-shaped display 21. As an example of changing the overall display, the first output control unit 201 may turn off the display of the store video, or display a video corresponding to the product instead of the store video. For example, if the product is food, the image corresponding to the product may be a plate or the like.

In addition, when changing the display of a part of the dome-shaped display 21, the first output control unit 201 controls the dome-shaped display 21 that overlaps the product when the customer views the product in the video or image displayed on the aerial display 22. You may erase some of the display or change the color. Thereby, when the aerial display 22 and the dome-shaped display 21 are combined, it is possible to prevent the product from becoming difficult to see.

As an example of changing a part of the display, for example, while the aerial display 22 is displaying information regarding the product, the first output control unit 201 changes the display at the fixed position of the dome-shaped display 21. The fixed position is a position corresponding to a display area where the aerial display 22 displays information regarding the product. As a method of changing the display of the fixed position, the first output control unit 201 may erase the display of the fixed position. Alternatively, the first output control unit 201 sets the display color of the fixed position to a predetermined color. The predetermined color may be a color that makes the product easier to see, such as black or white, or a color that corresponds to the product.

Furthermore, although a fixed position is taken as an example, the position may be specified in more detail. The superimposition position specifying unit 209 identifies the customer based on the positional relationship between the customer's position and the position of the aerial display 22, and the positional relationship between the display area where the aerial display 22 displays information regarding the product and the position of the dome-shaped display 21. When a person views information regarding a product displayed in a display area, the relative position of the dome-shaped display 21 that overlaps with the display area is specified. For example, the display area is the aerial imaging area described with reference to FIG. Note that the customer's location may be specified by a three-dimensional (3D) sensor or the like. The 3D sensor may be installed on the dome-shaped display 21, for example. While the aerial display 22 is displaying information regarding the product, the first output control unit 201 changes the display of the specified position on the dome-shaped display 21. The method of changing the display is as described above.

Alternatively, while the aerial display 22 is displaying information regarding the product, the first output control unit 201 causes the dome-shaped display 21 to display an image different from the image of the sales floor. For example, the images may vary depending on the product, such as plates, other shelves, and the like. Here, the explanation will be given using a video as an example, but an image may also be used.

Here, an example has been given in which there is one customer for one dome-shaped display 21, but multiple customers may be shopping together. When multiple customers are present on the dome-shaped display 21, the first output control unit 201 may control the display on the dome-shaped display 21 for each of the multiple customers.

Furthermore, when a plurality of aerial displays 22 are installed, the first output control unit 201 may control the display on the dome-shaped display 21 for each of the plurality of aerial displays 22.

This concludes the explanation of the example of controlling the display of the dome-shaped display 21 while displaying information about products on the aerial display 22.

<Control of product display size and color in images and videos>
When an image or video of a product is displayed on the aerial display 22, the size of the product may appear to be different depending on the position from which the customer views the displayed product.

Therefore, the second output control unit 203 may display a measuring tape on the aerial display 22 that allows the dimensions to be confirmed, along with the image of the product. This allows the customer to check the actual size of the product by comparing the tape measure with the size of the product.

Additionally, the customer's viewing position may be specified so that a full-size image or video of the product can be displayed to the customer. For example, a 3D sensor may be installed on the dome-shaped display 21. A 3D sensor then measures the position of the customer's head and hands. The acquisition unit 204 then acquires the measurement results of the 3D sensor. The visual position specifying unit 210 specifies the customer's visual position based on the positional relationship between the head position and the hand position based on the measurement results. The second output control unit 203 displays either an image or a video of the product at a size based on the actual size of the product and the visible position. As a result, images and videos of the product are displayed so that the product can be seen in its actual size.

Additionally, the second output control unit 203 may cause the aerial display 22 to display the product in an enlarged manner. Alternatively, the second output control unit 203 may display the product on the aerial display 22 so as to reduce the size of the product. Here, the operation reception unit 211 may receive an instruction to enlarge or an instruction to reduce by accepting an operation on the aerial display 22. Based on the received operation, the second output control unit 203 may display the product in the image or video in an enlarged manner, or may display the product in the image or video in a reduced size.

Furthermore, for example, if the product is a fresh food such as vegetables or fruits, it is assumed that the color and shape will differ depending on the product. For example, when an image such as an illustration of a product or an image of a product reproduced in virtual space is displayed on the aerial display 22, if the same image is displayed every time, the customer will feel that it is different from realistic shopping. It is assumed that Here, the explanation will be given using an image of a product as an example, but the same applies to a video of a product. It is desirable to be closer to realistic shopping. Therefore, for example, the second output control unit 203 causes the aerial display 22 to display an image of the product based on at least one of information regarding the color of the product and the shape of the product. The color of the product and the shape of the product may be obtained from a conversation between a store clerk and a customer. For example, the voice analysis unit 206 detects at least one of information regarding the color of the product and information regarding the shape of the product by performing voice recognition on a conversation between a store clerk and a customer. Then, the second output control unit 203 displays an image of the product in at least one of a color and a shape depending on the conversation. Specifically, for example, when a conversation between a store clerk and a customer saying "Today's apples are redder than usual" is detected, the second output control unit 203 displays an image of an apple in a color redder than a predetermined standard. Display. For example, when a conversation between a store clerk and a customer saying "Today's apples are bigger than usual" is detected, the second output control unit 203 controls the display so that the apples are displayed a predetermined size larger than a predetermined standard. Display an image of an apple. Thereby, the image of the product can be displayed on the aerial display 22 in the color and shape of the product that are close to the actual product.

However, if either the images or videos of the individual products are available, the second output control unit 203 may display either the image or the video according to the individual product.

Also, for example, when images of the actual store are displayed in real time, if the shelf labels of the products are captured together with the products, customers can check the prices and names of the products. . However, if there is no shelf label or if the shelf label is not imaged, the second output control unit 203 may display the name of the product and the price of the product on the aerial display 22 along with the video and image of the product.

<Example of ending the display of product information on the aerial display 22>
The timing at which the display of product information on the aerial display 22 ends can be changed in various ways depending on how the application program is used.

For example, when the customer is interested in another product or a store employee, the display of information regarding the product on the aerial display 22 may end. For example, the video analysis unit 205 may detect the customer's line of sight from the customer's video. For example, the video analysis unit 205 detects that the customer's line of sight is upward and not looking at the aerial display 22. Then, when it is detected that the person is not looking at the aerial display 22, the second output control unit 203 ends displaying the information regarding the product on the aerial display 22. Further, when the product detection unit 202 detects a new product, such as when the customer extends his hand toward another product, the second output control unit 203 sends information about the product to the aerial display 22. End the display. However, in this case, the second output control unit 203 may cause the aerial display 22 to display information regarding the newly detected product.

For example, the display of information regarding the product on the aerial display 22 may be terminated by the customer's operation. For example, the customer's operation may be an operation indicating the end of display, or an operation to add the item to a shopping cart. The customer's operation can be detected by the customer's voice, customer's gesture, etc. Specifically, for example, the video analysis unit 205 detects a predetermined gesture of the customer from the customer's video. Then, when a predetermined gesture is detected, the second output control unit 203 ends displaying information regarding the product on the aerial display 22. Alternatively, for example, the voice analysis unit 206 detects a predetermined keyword from the customer's voice. Then, when a predetermined keyword is detected, the second output control unit 203 ends displaying information regarding the product on the aerial display 22.

Furthermore, when the customer disappears from the dome-shaped display 21, the display of information regarding the product on the aerial display 22 may end. For example, the customer disappearing from the dome-shaped display 21 means that the customer disappears from a place where the dome-shaped display 21 can be viewed or operated. Specifically, for example, when a customer disappears from the dome-shaped display 21, there is a case where the customer leaves the chair installed in front of the dome-shaped display 21. For example, the video analysis unit 205 may detect that the customer is gone by detecting from the video imaged by the imaging device 23 whether or not the customer is captured. Then, when it is detected that the customer is gone, the second output control unit 203 ends the display of information regarding the product on the aerial display 22.

(flowchart)
FIG. 15 is a flowchart showing an example of the operation of the shopping support system 20 according to the second embodiment. Here, although not specifically shown in the flowchart, the acquisition unit 204 acquires the video imaged by the imaging device 23 and the audio acquired from the recording device 24. Further, the video analysis unit 205 analyzes the customer's behavior from the video captured by the imaging device 23. Furthermore, the voice analysis unit 206 analyzes the customer's behavior from the voice acquired from the recording device 24.

The first output control unit 201 causes the dome-shaped display 21 to display information regarding the sales floor of the store (step S201).

Next, the product detection unit 202 determines whether a product that the customer wants to acquire is detected based on the customer's behavior (step S202). If no product is detected (step S202: No), the product detection unit 202 returns to step S202.

On the other hand, if the product is detected (step S202: Yes), the second output control unit 203 causes the aerial display 22 to display information regarding the product (step S203). The first output control unit 201 also changes the display on the dome-shaped display 21 (step S204). In step S204, the first output control unit 201 may change the display of the fixed position on the dome-shaped display 21, or may change the display of the specified position on the dome-shaped display 21. Note that the order of processing in step S203 and step S204 is not particularly limited. For example, step S203 and step S204 may be performed at the same timing.

Next, the second output control unit 203 changes the display of information regarding the product based on the customer's behavior (step S205). In step S205, the second output control unit 203 may change the display orientation of the product in the video or image based on the customer's behavior. Furthermore, regarding step S205, the control is not limited to the second output control section 203, and the control of the platform 27 by the platform control section 208 or the control of the imaging device 25 by the imaging device control section 207 may be performed. For example, the imaging device control unit 207 may change the orientation of the product displayed on the aerial display 22 by changing the imaging position and orientation of the imaging device 25. Alternatively, for example, the stand control unit 208 may control the stand 27 to change the orientation of the product displayed on the aerial display 22. Note that the display control by the second output control section 203, the control of the platform 27 by the platform control section 208, and the control of the imaging device 25 by the imaging device control section 207 may be combined as appropriate.

The second output control unit 203 determines whether to end the display of information regarding the product (step S206). If the display of information regarding the product is not finished (step S206: No), the second output control unit 203 returns to step S205. If the display of information regarding the product is to be ended (step S206: Yes), the first output control unit 201 returns to step S201.

Note that the flowchart can be ended as appropriate.

As described above, in the second embodiment, the shopping support system 20 detects the customer's hand movement from the customer's video or image, and detects the product that the customer wants to acquire based on the detected customer's hand movement. . More specifically, for example, the shopping support system 20 detects the product that the customer wants to acquire based on the position where the customer extends his/her hand. For example, the customer can display the desired product on the aerial display 22 by simply making an intuitive movement such as taking a product from a shelf. Furthermore, when a customer visits an actual store to shop, it is assumed that the customer grabs a product on a shelf and brings it close to his or her face to check it. Therefore, according to the shopping support system 20 in which information regarding products detected from hand movements is displayed on the aerial display 22, it is possible to provide a shopping experience closer to actual shopping.

Additionally, the shopping support system 20 analyzes the conversation between the customer and the store clerk from the audio, and detects the product the customer wants to acquire based on the conversation. As a result, information regarding the product the customer wants to check is displayed on the aerial display 22 simply by having a conversation with the store clerk. Furthermore, for example, when a customer visits an actual store to shop, it is assumed that the customer will have a conversation with the store staff, and the store staff will recommend and show products to the customer, or hand the product to the customer. be done. Therefore, according to the shopping support system 20 in which information regarding products detected from conversations is displayed on the aerial display 22, it is possible to provide a shopping experience closer to actual shopping.

Additionally, the shopping support system 20 causes the aerial display 22 to display an image or video of the product in at least one of the color and shape depending on the conversation. Thereby, the image or video of the product can be displayed on the aerial display 22 in the color and shape of the product that are close to the actual product.

Additionally, the shopping support system 20 causes the dome-shaped display 21 to display information regarding the store staff. This allows customers to shop in a manner that is more similar to that of an actual store.

Additionally, the shopping support system 20 changes the display of the product in the video based on the customer's hand movements. For example, the shopping support system 20 controls an imaging device 25 installed in a store based on the customer's hand movements. When a product is placed on the rotating table 27 and the imaging device 25 takes an image of the product placed on the table 27, the shopping support system 20 controls the rotation of the table 27 based on the customer's hand movement. This allows the customer to change the display orientation of the product by moving his or her hand. In this way, the orientation of the product can be changed intuitively.

Additionally, as described above, the information regarding the product displayed on the aerial display 22 is transparent. Therefore, it may be difficult for customers to see the products. Therefore, the shopping support system 20 changes the display on the dome-shaped display 21 while the aerial display 22 is displaying information regarding the product. Specifically, for example, the shopping support system 20 may change the display of the fixed position of the dome-shaped display 21. Alternatively, the shopping support system 20 may specify the position of the dome-shaped display 21 that is on the line of sight of the customer when viewing the product, and change the display at the specified position. Thereby, it is possible to prevent the customer from having difficulty viewing the product.

Further, the store may be a store in a virtual space or may be an actual store. If there is an actual store, the shopping support system 20 may display an image of the store's sales floor captured in real time by the imaging device 25 on the dome-shaped display 21. As a result, customers can be supported to have an experience as if they were in an actual store, even if they are not in the actual store.

This concludes the description of each embodiment. Each embodiment may be modified and used. Each embodiment may be used in combination as appropriate. Furthermore, in each embodiment, the shopping support system 20 may have a configuration in which each functional unit and a part of information are included.

Further, the form of each embodiment is not limited to the above-mentioned example, and can be modified in various ways. Further, the configuration of the shopping support system 20 in each embodiment is not particularly limited. For example, the shopping support system 20 may be realized by one device such as one server. When each functional part of the shopping support system 20 is realized by one device, for example, one device may be called a shopping support device, an information processing device, etc., and is not particularly limited. Alternatively, the shopping support system 20 in each embodiment may be realized by different devices depending on functions or data. For example, each functional unit of the shopping support system 20 may be configured with a plurality of servers, and the shopping support system 20 including the plurality of servers may be realized. For example, the shopping support system 20 may be realized by a database server including each DB and a server having each functional unit.

FIG. 16 is an explanatory diagram showing an example of implementation of the shopping support system 20. This will be explained using an example where there is an actual store.

In FIG. 16, the shopping support system 20 includes, for example, an edge terminal device 31 and a server 32.

For example, the edge terminal device 31, the aerial display 22, the dome-shaped display 21, the imaging device 23, and the recording device 24 are installed in a house, a shared space, or the like. The edge terminal device 31, the aerial display 22, the dome-shaped display 21, the imaging device 23, and the recording device 24 are connected via a communication network.

For example, the imaging device 25 is installed in a store. The imaging device 25, the edge terminal device 31, and the server 32 are connected via a communication network.

Additionally, the shopping support system 20 may be configured as an entire system including an edge terminal device 31, a server 32, an aerial display 22, a dome-shaped display 21, an imaging device 23, a recording device 24, and an imaging device 25.

In FIG. 16, for example, each functional unit in each embodiment is realized by an edge terminal device 31 and a server 32. For example, the edge terminal device 31 includes an acquisition section 204, a video analysis section 205, an audio analysis section 206, and an operation reception section 211. For example, since analysis of video and audio requires a large load, an edge terminal device may be configured to perform these analyses. On the other hand, for example, the server 32 includes a first output control section 201, a product detection section 202, a second output control section 203, an acquisition section 204, an imaging device control section 207, a stand control section 208, a superimposition position specifying section 209, and a visual recognition section 209. A position specifying section 210 is provided. Further, the server 32 may be a plurality of servers. In this way, each functional unit of the

shopping support systems

10 and 20 may be realized by a plurality of devices, and the plurality of devices may be installed at different locations.

Furthermore, in each embodiment, each piece of information and each DB may include part of the above-mentioned information. Moreover, each piece of information and each DB may include information other than the above-mentioned information. Each piece of information and each DB may be divided into a plurality of DBs and a plurality of pieces of information in more detail. In this way, the method of implementing each piece of information and each DB is not particularly limited.

Furthermore, each screen is an example and is not particularly limited. In each screen, buttons, lists, check boxes, information display fields, input fields, etc. (not shown) may be added. Also, the background color of the screen, etc. may be changed.

Further, for example, in each embodiment, the process of generating information etc. to be displayed on the aerial display 22 may be performed by the second output control unit 203 of the shopping support system 20. Further, this processing may be performed by the aerial display 22.

Furthermore, for example, in each embodiment, the process of generating information and the like to be displayed on the dome-shaped display 21 may be performed by the first output control unit 201 of the shopping support system 20. Further, this processing may be performed by the dome-shaped display 21.

Furthermore, in each of the embodiments, a display in which at least a portion of the screen is curved is used as an example of the first display, and specifically, the dome-shaped display 21 is used as an example of the first display. The first display is not limited to a display in which at least a portion of the screen is curved, as long as it is a display for the purpose of displaying a virtual space, video, images, etc. Alternatively, for example, as described above, the first display may be a device configured to cover the user's field of vision.

Furthermore, in each of the embodiments, a non-contact display was used as an example of the second display. As described above, the second display may be a contact type display and is not particularly limited.

(Example of computer hardware configuration)
Next, an example of the hardware configuration when each device such as the

shopping support systems

10 and 20 described in each embodiment is realized by a computer will be described. FIG. 17 is an explanatory diagram showing an example of the hardware configuration of a computer. For example, part or all of each device can be realized using any combination of a computer 80 and a program as shown in FIG. 17, for example.

The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. Further, the computer 80 has a communication interface 805 and an input/output interface 806. Each component is connected to each other via a bus 807, for example. Note that the number of each component is not particularly limited, and each component is one or more.

A processor 801 controls the entire computer 80. Examples of the processor 801 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The computer 80 includes a ROM 802, a RAM 803, a storage device 804, and the like as storage units. Examples of the storage device 804 include semiconductor memory such as flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), and the like. For example, the storage device 804 stores OS (Operating System) programs, application programs, programs according to each embodiment, and the like. Alternatively, the ROM 802 stores application programs, programs according to each embodiment, and the like. The RAM 803 is used as a work area for the processor 801.

Additionally, the processor 801 loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. Furthermore, the processor 801 may download various programs via the communication network NT. Further, the processor 801 functions as part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) through a wireless or wired communication line. Note that the communication network NT may be composed of a plurality of communication networks NT. Thereby, the computer 80 is connected to an external device or an external computer 80 via the communication network NT. Communication interface 805 serves as an interface between communication network NT and the inside of computer 80 . The communication interface 805 controls input and output of data from external devices and the external computer 80.

Further, the input/output interface 806 is connected to at least one of an input device, an output device, and an input/output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Further, examples of the input/output device include a touch panel display. Note that the input device, output device, input/output device, etc. may be built into the computer 80 or may be externally attached.

The hardware configuration of the computer 80 is an example. Computer 80 may include some of the components shown in FIG. Computer 80 may include components other than those shown in FIG. For example, the computer 80 may include a drive device or the like. The processor 801 may then read programs and data stored in a recording medium attached to a drive device or the like to the RAM 803. Examples of non-temporary tangible recording media include optical disks, flexible disks, magneto-optical disks, USB (Universal Serial Bus) memories, and the like. Further, as described above, for example, the computer 80 may include an input device such as a keyboard and a mouse. Computer 80 may have an output device such as a display. Further, the computer 80 may each have an input device, an output device, and an input/output device.

Additionally, the computer 80 may include various sensors (not shown). The type of sensor is not particularly limited. Further, the computer 80 may include an imaging device capable of capturing images and videos.

This concludes the explanation of the hardware configuration of each device. Furthermore, there are various variations in the method of implementing each device. For example, each device may be realized by an arbitrary combination of a computer and a program, each of which is different for each component. Further, the plurality of components included in each device may be realized by an arbitrary combination of one computer and a program.

Additionally, some or all of the components of each device may be realized by application-specific circuits. Moreover, a part or all of each component of each device may be realized by a general-purpose circuit including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of application-specific circuits, general-purpose circuits, and the like. Also, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. Further, the plurality of integrated circuits may be configured by being connected via a bus or the like.

Further, in the case where a part or all of each component of each device is realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be arranged centrally or in a distributed arrangement.

The shopping support method described in each embodiment is realized by being executed by the shopping support system. Further, for example, the shopping support method is realized by a computer such as a server or a terminal device executing a program prepared in advance. The programs described in each embodiment are recorded on a computer-readable recording medium such as an HDD, SSD, flexible disk, optical disk, flexible disk, magneto-optical disk, or USB memory. Then, the program is executed by being read from the recording medium by the computer. The program may also be distributed via the communications network NT.

The functions of each component of the shopping support system in each embodiment described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

Although the present disclosure has been described above with reference to each embodiment, the present disclosure is not limited to the above embodiments. The configuration and details of each present disclosure may include embodiments in which various changes that can be grasped by those skilled in the art within the scope of the present disclosure are applied. The present disclosure may include embodiments in which the matters described in this specification are appropriately combined or replaced as necessary. For example, matters described using a particular embodiment may be applied to other embodiments to the extent that no contradiction occurs. For example, although a plurality of operations are described in order in the form of a flowchart, the order in which they are described does not limit the order in which the plurality of operations are executed. Therefore, when implementing each embodiment, the order of the plurality of operations can be changed within a range that does not interfere with the content.

Part or all of the above embodiments can also be described as in the following additional notes. However, some or all of the above embodiments are not limited to the following.

(Additional note 1)
a first output control means that causes the first display to display information regarding the department of the store where the products are displayed;
product detection means for detecting a product that the customer wants to acquire based on the customer's behavior;
a second output control means for displaying information regarding the detected product on a second display;
A shopping support system equipped with
(Additional note 2)
the customer's action is a hand movement of the customer;
The shopping support system described in Appendix 1.
(Additional note 3)
The product detection means detects the product based on the position where the customer extends his/her hand.
The shopping support system described in Appendix 2.
(Additional note 4)
The customer's behavior is a conversation between the customer and a store clerk.
The shopping support system according to any one of Supplementary Notes 1 to 3.
(Appendix 5)
The first output control means further causes the first display to display information regarding the store clerk.
The shopping support system described in Appendix 4.
(Appendix 6)
The information regarding the product is a video of the product or an image of the product,
The second output control means displays information regarding the product in at least one of a color and a shape depending on the conversation between the customer and the store clerk.
The shopping support system according to any one of Supplementary Notes 1 to 5.
(Appendix 7)
The information regarding the product is an image of the product;
The shopping support system according to any one of Supplementary Notes 1 to 5.
(Appendix 8)
The second output control means changes the display of the product in the image of the product based on the customer's hand movement.
The shopping support system described in Appendix 7.
(Appendix 9)
The image of the product is an image of the product captured by an imaging device installed in the store,
Imaging device control means for controlling at least one of the imaging position and orientation of the imaging device based on the customer's hand movement;
The shopping support system described in Appendix 7.
(Appendix 10)
The image of the product is an image of the product captured by an imaging device installed in the store,
a table control means for controlling the rotation of a rotary table on which the product can be placed in the store based on the customer's hand movement;
The shopping support system described in Appendix 7.
(Appendix 11)
The first output control means changes the display of the fixed position of the first display when the second display displays information regarding the product.
The shopping support system according to any one of Supplementary Notes 1 to 10.
(Appendix 12)
Based on the positional relationship between the customer's position and the position of the second display, and the positional relationship between the display area of information regarding the product on the second display and the position of the first display, the customer specifying means for specifying a position of the first display that overlaps the display area when viewing information regarding the product displayed on the display area;
Equipped with
The first output control means changes the display of the specified position on the first display when the second display displays information regarding the product.
The shopping support system according to any one of Supplementary Notes 1 to 10.
(Appendix 13)
The store is a store in a virtual space,
The shopping support system according to any one of Supplementary Notes 1 to 12.
(Appendix 14)
The information regarding the sales floor is an image of the sales floor of the store captured by an imaging device installed in the store.
The shopping support system according to any one of Supplementary Notes 1 to 12.
(Appendix 15)
The first display is a device structured to cover the customer's field of view.
The shopping support system according to any one of Supplementary Notes 1 to 14.
(Appendix 16)
the first display is a dome-shaped display;
The shopping support system according to any one of Supplementary Notes 1 to 15.
(Appendix 17)
The second display is a non-contact display.
The shopping support system according to any one of Supplementary Notes 1 to 16.
(Appendix 18)
the second display is an aerial display;
The shopping support system described in Appendix 17.
(Appendix 19)
the second display is installed between the customer and the first display;
The shopping support system according to any one of Supplementary Notes 1 to 18.
(Additional note 20)
the second display is installed closer to the customer than the first display;
The shopping support system according to any one of Supplementary Notes 1 to 18.
(Additional note 21)
the first display is larger than the second display;
The shopping support system according to any one of Supplementary Notes 1 to 20.
(Additional note 22)
displaying information regarding the department of the store where the product is displayed on the first display;
detecting products that the customer wants to acquire based on the customer's behavior;
displaying information regarding the detected product on a second display;
Shopping support method.
(Additional note 23)
to the computer,
displaying information regarding the department of the store where the product is displayed on the first display;
detecting products that the customer wants to acquire based on the customer's behavior;
displaying information regarding the detected product on a second display;
The computer-readable non-transitory recording medium that records a program for executing processing.
(Additional note 24)
to the computer,
displaying information regarding the department of the store where the product is displayed on the first display;
detecting products that the customer wants to acquire based on the customer's behavior;
displaying information regarding the detected product on a second display;
A program that executes processing.

10, 20 Shopping support system 21 Dome-shaped display 22 Aerial display 23 Imaging device 24 Recording device 25, 25-1, 25-2 Imaging device 26 Recording device 27 Unit 31 Edge terminal device 32 Server 80

Computer

101, 201 First

output control Units

102 and 202

Product detection units

103 and 203 Second output control unit 204 Acquisition unit 205 Video analysis unit 206 Audio analysis unit 207 Imaging device control unit 208 Stand control unit 209 Superposition position identification unit 210 Visual recognition position identification unit 211 Operation reception unit 801 Processor 802 ROM
803 RAM
804 Storage device 805 Communication interface 806 Input/output interface 807 Bus 2101 Screen 2102 Projection device 2103 Table NT Communication network 2001 Customer DB
2002 Store DB
2003 Product DB

Claims

a first output control means that causes the first display to display information regarding the department of the store where the products are displayed;
product detection means for detecting a product that the customer wants to acquire based on the customer's behavior;
a second output control means for displaying information regarding the detected product on a second display;
A shopping support system equipped with
the customer's action is a hand movement of the customer;
The shopping support system according to claim 1.
The product detection means detects the product based on the position where the customer extends his/her hand.
The shopping support system according to claim 2.
The customer's behavior is a conversation between the customer and a store clerk.
A shopping support system according to any one of claims 1 to 3.
The first output control means further causes the first display to display information regarding the store clerk.
The shopping support system according to claim 4.
The information regarding the product is a video of the product or an image of the product,
The second output control means displays information regarding the product in at least one of a color and a shape depending on the conversation between the customer and the store clerk.
A shopping support system according to any one of claims 1 to 5.
The information regarding the product is an image of the product;
A shopping support system according to any one of claims 1 to 5.
The second output control means changes the display of the product in the image of the product based on the customer's hand movement.
The shopping support system according to claim 7.
The image of the product is an image of the product captured by an imaging device installed in the store,
Imaging device control means for controlling at least one of the imaging position and orientation of the imaging device based on the customer's hand movement;
The shopping support system according to claim 7, comprising:
The image of the product is an image of the product captured by an imaging device installed in the store,
a table control means for controlling the rotation of a rotary table on which the product can be placed in the store based on the customer's hand movement;
The shopping support system according to claim 7, comprising:
The first output control means changes the display of the fixed position of the first display when the second display displays information regarding the product.
A shopping support system according to any one of claims 1 to 10.
Based on the positional relationship between the customer's position and the position of the second display, and the positional relationship between the display area of information regarding the product on the second display and the position of the first display, the customer specifying means for specifying a position of the first display that overlaps the display area when viewing information regarding the product displayed on the display area;
Equipped with
The first output control means changes the display of the specified position on the first display when the second display displays information regarding the product.
A shopping support system according to any one of claims 1 to 10.
The store is a store in a virtual space,
A shopping support system according to any one of claims 1 to 12.
The information regarding the sales floor is an image of the sales floor of the store captured by an imaging device installed in the store.
A shopping support system according to any one of claims 1 to 12.
The first display is a device structured to cover the customer's field of view.
A shopping support system according to any one of claims 1 to 14.
the first display is a dome-shaped display;
A shopping support system according to any one of claims 1 to 15.
The second display is a non-contact display.
A shopping support system according to any one of claims 1 to 16.
the second display is an aerial display;
The shopping support system according to claim 17.
the second display is installed between the customer and the first display;
A shopping support system according to any one of claims 1 to 18.
the second display is installed closer to the customer than the first display;
A shopping support system according to any one of claims 1 to 18.
the first display is larger than the second display;
A shopping support system according to any one of claims 1 to 20.
displaying information regarding the department of the store where the product is displayed on the first display;
detecting products that the customer wants to acquire based on the customer's behavior;
displaying information regarding the detected product on a second display;
Shopping support method.
to the computer,
displaying information regarding the department of the store where the product is displayed on the first display;
detecting products that the customer wants to acquire based on the customer's behavior;
displaying information regarding the detected product on a second display;
The computer-readable non-transitory recording medium that records a program for executing processing.