JP7369600B2 - User terminal, control method and computer program - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies Event date: September 6, 2020 Meeting name: Hackathon, Event location: Otemachi Place West Tower 28th floor, 2-3-1 Otemachi, Chiyoda-ku, Tokyo

The present invention relates to technology for conveying information regarding products to users.

2. Description of the Related Art In recent years, a technology has been proposed as an electronic menu device for restaurants that displays three-dimensional images of menu items on a screen in stereoscopic view. For example, with the technology disclosed in Patent Document 1, it is possible to display three-dimensional images of the same product in different sizes in a semi-transparent manner. Such technology allows users to compare the volume of menus.

Japanese Patent Application Publication No. 2011-186830

However, with conventional devices, it is necessary to create three-dimensional images of products for each of a plurality of menus in advance and register them in a database. The effort required to create such three-dimensional images is no small feat. Therefore, the time and work costs for preparing three-dimensional images of products in advance and managing them in a database have increased. Such a problem does not necessarily relate only to products in restaurants, but is a common problem in various industries where products of multiple sizes exist.
In view of the above circumstances, it is an object of the present invention to provide a technology that can complete preparations for allowing users to more easily grasp the sizes of products of a plurality of sizes with a simpler operation.

One aspect of the present invention includes an information providing unit that acquires product identification information indicating a product and a size specification value indicating the size of the product; a size changing unit that generates image data of a modified image transformed to a size indicated by the size specification value by transforming the image data of the image based on the size specification value, The providing unit is a user terminal that displays the changed image on a screen.

One aspect of the present invention is the above user terminal, further comprising an imaging device, and the information providing unit superimposing the initial image and the changed image on a real-world video captured by the imaging device. Display on the screen.

One aspect of the present invention is the user terminal described above, wherein the information providing unit includes the initial image displayed on the screen and the changed image when compared with an object photographed in the real world video. Display the image so that its size is approximately the actual size.

One aspect of the present invention includes the step of acquiring product identification information indicating a product and a size specification value indicating the size of the product, and an initial image generated in advance as an image of a product of a predetermined size. generating image data of a modified image transformed to a size indicated by the designated size value by transforming the image data based on the designated size value; and displaying the modified image on a screen. This is a control method having the following.

One aspect of the present invention is a computer program for causing a computer to function as the user terminal described above.

According to the present invention, it is possible to complete preparations for allowing users to more easily understand the sizes of products of multiple sizes with a simpler operation.

1 is a schematic block diagram showing the system configuration of an information providing system 100 of the present invention. 2 is a diagram showing a specific example of the functional configuration of the user terminal 10. FIG. 3 is a diagram showing a specific example of the functional configuration of a product information management server 30. FIG. It is a diagram showing a specific example of product information. It is a diagram showing a specific example of product information. 3 is a flowchart showing an example of the operation of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG. 3 is a diagram showing a specific example of a screen of the user terminal 10. FIG.

Hereinafter, specific configuration examples of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram showing the system configuration of an information providing system 100 of the present invention. The information providing system 100 includes a user terminal 10, an administrator terminal 20, and a product information management server 30. The user terminal 10, the administrator terminal 20, and the product information management server 30 are connected to a network 50. The network 50 is, for example, the Internet or a mobile communication network.

The user terminal 10 is used by a user who wishes to know the size of an item. When the user terminal 10 is set to an image display mode (for example, AR (Augmented Reality) mode) and an operation is performed to display an image of a desired product, the screen of the user terminal 10 is set in advance. An image of the product (hereinafter referred to as "initial image") of the size is displayed. The initial image size is displayed to reproduce the actual product size. Such reproduction technology may be realized by applying conventional AR technology. In this case, the image data of the initial image is generated and stored as AR image data.

After that, when the user specifies the size and performs an operation to change the size of the image, the displayed product image will be changed to an image of the size specified by the user (hereinafter referred to as the "changed image"). ) will be changed. The size is specified by inputting a value indicating the size (hereinafter referred to as "size specified value"). The size of the changed image is displayed by processing the data of the initial image so that it becomes the size of the actual product specified by the user.

Because it operates in this way, the information providing system 100 allows the user to easily understand the actual product size in terms of multiple sizes. Furthermore, when displaying an image whose size has been changed (changed image), data of the changed image is generated by processing the data of the initial image. Therefore, there is no need to generate image data in advance for each product of a plurality of sizes. Therefore, it is possible to complete preparations for allowing the user to more easily understand the sizes of products of a plurality of sizes with a simpler task. The information providing system 100 will be described in detail below.

The user terminal 10 is an information device such as a tablet device, a mobile phone device, a smartphone, a game device, a camera, or a wearable terminal (for example, a smart watch). The user terminal 10 may be a device configured as a dedicated device for a specific purpose (for example, acquiring information regarding a specific product or placing an order regarding a specific product). A predetermined application (hereinafter referred to as "product viewing application") is installed in the user terminal 10 in advance. FIG. 2 is a diagram showing a specific example of the functional configuration of the user terminal 10. As shown in FIG. The user terminal 10 includes a communication section 11, a storage section 12, an input section 13, a display section 14, an imaging device 15, and a control section 16.

The communication unit 11 is configured using a network interface device. The communication unit 11 communicates with other information devices via the network 50. The communication unit 11 may be connected to the network 50 via a local network (for example, a LAN (Local Area Network)) not shown.

The storage unit 12 is configured using a storage device such as a magnetic hard disk or a semiconductor storage device. The storage unit 12 functions as a product information storage unit 121. The product information storage unit 121 stores product information. Product information is information that includes image data of a product displayed on the display unit 14 of the user terminal 10. The product information may be related to the product to be displayed and may be downloaded from another information device and recorded in the product information storage unit 121, or may be related to the product handled at the store where the user is located. It may be downloaded from another information device and recorded in the product information storage unit 121, or information related to products handled in the area where the user is located may be downloaded from another information device and recorded in the product information storage unit 121. may be done. Furthermore, if it is assumed in advance that the user terminal 10 will be used at a specific store, product information regarding products handled at that store may be recorded in the product information storage section 121. The product information recorded in the product information storage section 121 of the user terminal 10 is product information generated by the product information management server 30.

The input unit 13 is configured using existing input devices such as a keyboard, pointing device (mouse, tablet, etc.), buttons, touch panel, etc. The input unit 13 is operated by the user when inputting the user's instructions into the user terminal 10 . The input unit 13 may be an interface for connecting an input device to the user terminal 10. In this case, the input unit 13 inputs to the user terminal 10 an input signal generated in response to a user's input using the input device. The input unit 13 may be configured using a microphone and a voice recognition device. In this case, the input unit 13 performs voice recognition on the words uttered by the user, and inputs character string information resulting from the recognition into the user terminal 10 . The input unit 13 may be configured in any manner as long as it can input user instructions into the user terminal 10.

The display unit 14 is an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The display unit 14 displays image data included in product information. The display unit 14 may be an interface for connecting the image display device to the user terminal 10. In this case, the display unit 14 generates a video signal for displaying image data, and outputs the video signal to an image display device connected to itself.

The imaging device 15 is a camera configured using an imaging element and an optical system (lens). The imaging device 15 captures an image under the control of the control unit 16 and outputs image data obtained by capturing the image to the control unit 16.

The control unit 16 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The control unit 16 functions as an information providing unit 161 and a size changing unit 162 when the processor executes a product viewing application program. Note that all or part of each function of the control unit 16 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The product viewing application program described above may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (such as SSDs: Solid State Drives), and hard disks and semiconductor storages built into computer systems. It is a storage device such as a device. The product viewing application program described above may be transmitted via a telecommunications line.

The information providing unit 161 acquires the product information generated by the product information management server 30 and records it in the product information storage unit 121. The information providing unit 161 acquires identification information indicating a product (hereinafter referred to as "product identification information"). The product identification information may be acquired, for example, by the user operating the user terminal 10 and inputting information such as product identification information and a product name. The product identification information may be acquired, for example, by the information providing unit 161 displaying a plurality of candidates on the display unit 14, and the user operating the input unit 13 to select from among the candidates. The displayed candidates may be product identification information registered in the product information stored in the product information storage unit 121. The product identification information may be acquired, for example, by activating the imaging device 15 provided in the user terminal 10 and based on an image of a specific marker taken by the imaging device 15. Specifically, product identification information may be obtained by decoding information shown in a marker image (for example, a barcode image). The decoding process for the marker image may be performed by the information providing unit 161, or may be performed by a decoding program installed in the user terminal 10 in advance. Product identification information may be acquired by any other method.

The information providing unit 161 reads image data associated with the acquired product identification information from the product information storage unit 121. The information providing section 161 displays the read image data on the display section 14. The image displayed at this time is an image of the product (initial image) of a preset size. When the information providing unit 161 receives an instruction to specify the size and change the size of the image from the user via the input unit 13, the information providing unit 161 generates an image (changed image) according to the specified size. An instruction is given to the size changing unit 162. The information providing unit 161 then displays the changed image generated by the size changing unit 162 on the display unit 14.

The size changing unit 162 generates image data of a changed image from the image data of the initial image in response to an instruction received from the information providing unit 161. The size changing unit 162 performs transformation processing (enlargement processing or reduction processing) on the image data of the initial image so that it has the size specified by the user. For example, when the image display performed by the information providing unit 161 is an AR display, the size change unit 162 initially changes the size of the changed image displayed in AR to the size of the actual product. Performs transformation processing on image data. Such processing may be performed, for example, as follows.

When an initial image or a modified image is displayed in response to imaging a specific object such as a marker by the imaging device 15, the scale of the initial image is determined based on the size of the image of the specific object being imaged. is determined. That is, the smaller the size of the image of the specific object, the smaller the display size of the initial image is determined, and the larger the size of the image of the specific object, the larger the display size of the initial image is determined. Next, it is determined how to change the display size of the initial image according to the size specified by the user. For example, if the size specified by the user indicates a volume magnification, the vertical magnification, horizontal magnification, and height magnification are determined according to the magnification. Specifically, when 800% is specified, each of the vertical magnification, horizontal magnification, and height magnification is determined to be twice. In this case, the size changing unit 162 determines the display size of the changed image by multiplying the display size of the initial image by a vertical magnification, a horizontal magnification, and a height magnification. The size changing unit 162 performs transformation processing on the image data of the initial image so that it has the determined size. The deformation process may be, for example, affine transformation or other image processing.

Note that the user terminal 10 may be configured so that the user can specify any size, or the user terminal 10 may be configured so that the user can specify a size from among predetermined candidate sizes. It's okay. The predetermined candidates may be registered in the product information storage section 121 in advance, for example. For example, it may be registered in advance as attribute information in the product information storage section 121.

Next, the administrator terminal 20 will be explained. The administrator terminal 20 is an information device such as a tablet device, a mobile phone device, a smartphone, a game device, a personal computer, or a camera. The administrator terminal 20 includes an imaging device. The administrator terminal 20 is used when creating image data of a product. A predetermined application (hereinafter referred to as "administrator application") is installed in advance on the administrator terminal 20.

The administrator operates the administrator terminal 20 to start the administrator application. The administrator application activates the imaging device in response to an operation by the administrator. By activating the imaging device, the administrator terminal 20 becomes ready to take an image of the product. By operating the administrator terminal 20, the administrator images the product for which product information is to be generated. The administrator terminal 20 generates image data in response to an image capturing operation performed by the administrator.

For example, the administrator may photograph the product by moving the administrator terminal 20 while the imaging device is activated. In this case, for example, the manager may move 360 degrees around the product with the lens of the imaging device of the manager terminal 20 facing the product. By the administrator operating in this way, the administrator terminal 20 may generate imaging data having information for 360 degrees. Thereafter, the administrator terminal 20 generates image data of an initial image based on the generated imaging data in response to the administrator's operation. When the initial image or the changed image is displayed as an AR image, the administrator terminal 20 generates AR image data based on the imaging data. The administrator terminal 20 transmits the generated image data to the product information management server 30.

Next, the product information management server 30 will be explained. The product information management server 30 is an information processing device such as a personal computer or a server device. FIG. 3 is a diagram showing a specific example of the functional configuration of the product information management server 30. The product information management server 30 includes a communication section 31, a storage section 32, and a control section 33.

The communication unit 31 is configured using a network interface device. The communication unit 31 communicates with other information devices via the network 50. The communication unit 31 may be connected to the network 50 via a local network (for example, LAN) not shown.

The storage unit 32 is configured using a storage device such as a magnetic hard disk or a semiconductor storage device. The storage unit 32 functions as a product information storage unit 321. The product information storage unit 321 stores product information. The product information is generated by the control unit 33.

4 and 5 are diagrams showing specific examples of product information. The product information may be configured as a product information table as shown in FIGS. 4 and 5, for example. The product information table has a plurality of product information records 91. Each product information record 91 has information for each product. The product information record 91 has, for example, each value of product identification information, product image data, size information, product name, price, and attribute information. The product identification information represents the product identification information of the product registered in the product information record 91. Product image data represents image data of products registered in the product information record 91. The size information represents the size information of the product registered in the product information record 91. This size information indicates the size of the product in the image (initial image) displayed based on the product image data. The product name represents the name of the product registered in the product information record 91. The price represents the price of the product registered in the product information record 91. This price indicates the price of the product of the image (initial image) displayed based on the product image data. The attribute information represents attribute information of the product registered in the product information record 91. Any information may be registered in the attribute information. For example, information indicating raw materials used in the product, toppings attached to the product, type of product, etc. may be registered as attribute information.

The control unit 33 is configured using a processor such as a CPU and a memory. The control unit 33 functions as a data management unit 331 when the processor executes a program for product information management. Note that all or part of each function of the control unit 33 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The above product information management program may be recorded on a computer-readable recording medium. Computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, semiconductor storage devices (such as SSDs: Solid State Drives), and hard disks and semiconductor storages built into computer systems. It is a storage device such as a device. The product information management program described above may be transmitted via a telecommunications line.

The data management section 331 manages product information stored in the product information storage section 321. For example, upon receiving image data generated by the administrator terminal 20, the data management unit 331 issues product identification information according to the received image data. The data management section 331 registers the image data and the product identification information in association with each other as new product data in the product information storage section 321. For example, the data management unit 331 generates a new product information record 91 in the product information table, and registers new image data and new product identification information in the new product information record 91. FIG. 4 is a diagram showing a specific example of the product information table at the timing when the new product information record 91 is registered in this way.

Upon receiving a request from the administrator terminal 20 to register size information, product name, price, or attribute information in association with product identification information, the data management unit 331 converts the received information into the product information indicated by the product identification information. Register in record 91. FIG. 5 is a diagram showing a specific example of the product information table when each piece of information is further registered after the product information record 91 is registered in this manner. Values registered as attribute information such as "diameter 26 cm: 75%" and "diameter 36 cm: 144%" are used as candidates when the user specifies the size. In this case, a diameter value may be displayed on the display section 14 of the user terminal 10 as a changeable size candidate, or a size percentage may be displayed on the display section 14 of the user terminal 10. When size candidates are set in advance in this way, it is possible for the product provider to narrow down the products that can be provided, making it easier to provide products and maintain quality. You can get the effect of For example, if the product is a dish, a recipe can be prepared for a predetermined candidate size, so that the quality of the dish can be maintained and it can be easily served.

FIG. 6 is a flowchart showing an example of the operation of the user terminal 10. 7 to 12 are diagrams showing specific examples of screens of the user terminal 10. Hereinafter, an example of the operation of the user terminal 10 will be described while showing a specific example of a screen. First, a user operates the user terminal 10 to start a product viewing application (step S101). When the product viewing application is activated, the information providing unit 161 activates the imaging device 15 of the user terminal 10 (step S102). When the imaging device 15 is activated, the information providing unit 161 displays the video imaged by the imaging device 15 on the display unit 14. The information providing unit 161 periodically monitors whether product identification information is acquired from the captured image.

When a marker printed in advance on a menu or the like enters the imaging range of the imaging device 15 by the user changing the position or orientation of the user terminal 10, the information providing unit 161 acquires product identification information according to the marker. (Step S103). In the screen example shown in FIG. 7, a marker 51 and a mobile phone 52 are shown in the image captured by the imaging device 15. Further, at the bottom of the screen, an input field 101 for inputting a value indicating the changed size and an enter button 102 are displayed. When such a video is obtained, the information providing unit 161 obtains product identification information by decoding the image of the marker 51 displayed on the screen. The decryption process may be executed by another program, as described above.

The information providing unit 161 reads image data stored in the product information storage unit 121 in association with the acquired product identification information. The information providing unit 161 displays the initial image on the screen by displaying the read image data on the display unit 14 (step S104). In the screen example shown in FIG. 8, the image of the initial image 103 is displayed superimposed on the image of the marker 51 captured by the imaging device 15. In the example of FIG. 8, the product selected by the user is curry and rice. The size of the initial image 103 of curry and rice is approximately the same as the size of the actual product (curry and rice) when compared with the size of the marker 51 and mobile phone 52 in the real world captured by the imaging device 15. It will be displayed as follows. Such a display may be realized by changing the size of the initial image 103 based on the size of the marker 51, for example. In addition, in such a display, the distance between the position (especially height) of the user terminal 10 held by the user and the table is assumed in advance, and the size of the image data of the initial image is adjusted according to the assumed distance. This may be realized by setting .

Thereafter, when the user issues an instruction to change the size (step S105 - YES), the information providing unit 161 requests the resizing unit 162 to generate a changed image of the changed size. In the screen example shown in FIG. 9, a value indicating the size of "60%" is input into the input field 101 by the user. Then, the user presses the enter button 102. The size changing unit 162 generates a changed image by performing a transformation process on the image data of the initial image of the product (curry and rice in this case) specified by the user (step S106). The size changing unit 162 displays the generated changed image on the display unit 14 (step S107). In the screen example shown in FIG. 10, the image of the changed image 14 is displayed superimposed on the image of the marker 51 captured by the imaging device 15. The size of the changed image 104 of curry and rice is 60% of the size of the actual product (curry and rice) when compared with the size of the marker 51 and mobile phone 52 in the real world captured by the imaging device 15. It is displayed so that it is approximately the same as the size of the amount.

In the information providing system 100 configured in this manner, the user can view images of products of a plurality of sizes by operating the user terminal 10. The image of the product displayed on the user terminal 10 is generated based on the image data of the initial image. The image data of the initial image is generated based on the image data obtained by capturing an image using the image capturing device of the administrator terminal 20, but the image data of the changed image is generated based on the image data of the initial image rather than the image data. Generated based on. Therefore, the administrator does not need to perform imaging to generate images of multiple sizes (multiple changed images). Generally, a lot of effort is required to take an image to generate image data of a product. The reason for this is that it is necessary to prepare actual products of each size, which requires a great deal of effort. Furthermore, in order to generate better image data, it takes a lot of effort to prepare the imaging environment, take into consideration the movement of the imaging device, and perform imaging work. Since there is no need to perform the labor-intensive work of capturing changed images, preparations for allowing users to more easily understand the sizes of products of multiple sizes can be completed with simpler work. becomes possible.

<Modified example>
The information providing unit 161 may acquire product information as follows.
First, the first acquisition method will be explained. When the product viewing application is started, the information providing unit 161 acquires information for specifying a store that handles the product (hereinafter referred to as "store information"). The store information may be information indicating the current location of the user terminal 10, for example. Information indicating the current position may be acquired by, for example, a GPS (Global Positioning System) provided in the user terminal 10. Store information may be acquired, for example, by the user operating the user terminal 10 and inputting a store name. Store information may be acquired, for example, by the user operating the user terminal 10 to select from among candidates presented to the user by a product viewing application. Store information may be acquired by any other method. When the store information is acquired, the information providing unit 161 downloads product information corresponding to the acquired store information from the product information management server 30. The information providing unit 161 records the downloaded product information in the product information storage unit 121.

Next, the second acquisition method will be explained. If the store where the product viewing application is used is fixed in advance, the product information generated by the product information management server 30 may be set in advance as part of the data of the product viewing application. In this case, product information may be registered in the product information storage section 121 when installing the product viewing application.

FIGS. 11 and 12 are diagrams showing examples of screen display in a modified example. In the screen example shown in FIG. 11, size information 105 is displayed when the changed image 104 is displayed. The size information 105 may be registered in the attribute information of the product information record 91, for example. In the example of the attribute information shown in FIG. 5, when the size is between 1% and 75%, size information 105 indicating "S size is the closest" is registered. Therefore, when the size specified by the user is "60%", the character string "S size is closest" is displayed based on this size and the attribute information. With this configuration, even if 60% of the product does not actually exist, the user can easily find out the product with the closest size to the modified image he or she is viewing. .

In the example screen shown in FIG. 12, instead of the input field 101, a candidate display button 106 and a candidate display field 107 are displayed. When the candidate display button 106 is pressed by the user, candidates of selectable sizes are displayed side by side in the candidate display field 107. Information indicating size candidates may be registered in the attribute information of the product information record 91, for example. With this configuration, it is possible to prevent the user from selecting an amount that does not actually exist. Therefore, the user can only view images of the product in the size that actually exists.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

DESCRIPTION OF SYMBOLS 100... Information provision system, 10... User terminal, 20... Administrator terminal, 30... Product information management server, 11... Communication department, 12... Storage part, 121... Product information storage part, 13... Input part, 14... Display part , 15...Imaging device, 16...Control unit, 161...Information providing unit, 162...Size changing unit, 31...Communication unit, 32...Storage unit, 321...Product information storage unit, 33...Control unit, 331...Data management unit , 101...Input field, 102...Confirm button, 103...Initial image, 104...Change image, 105...Size information, 106...Candidate display button, 107...Candidate display field, 91...Product information record

Claims (5)

an information providing unit that acquires product identification information indicating a product and a size specification value indicating a size of the product;
By transforming the image data of an initial image generated as an image of a product of a predetermined size that is associated with the product identification information based on the size specification value, the size indicated by the size specification value is determined. a resizing unit that generates image data of a modified image that has been transformed to a size;
The information providing unit displays the changed image on a screen,
The information providing unit further displays a character string that is associated with the specified size value and indicates a size that is closest to the specified size value. Further equipped with an imaging device,
The user terminal according to claim 1, wherein the information providing unit displays the initial image and the changed image on the screen in a superimposed manner on a real-world image captured by the imaging device. The information providing unit displays the initial image and the modified image displayed on the screen so that the sizes thereof are approximately the actual size when compared with the object photographed in the real world video. , the user terminal according to claim 2. a step in which the computer obtains product identification information indicating the product and a size specification value indicating the size of the product;
The computer transforms image data of an initial image, which has been generated as an image of a product of a predetermined size that is associated with the product identification information, based on the size designation value, thereby obtaining the size designation. generating image data of the modified image transformed to a size indicated by the value;
a computer displaying the modified image on a screen;
the computer further displaying a character string associated with the specified size value and indicating a size closest to the specified size value;
A control method having A computer program for causing a computer to function as a user terminal according to any one of claims 1 to 3.

Images