JP2017182628A - Augmented reality user interface application device and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that a video photographed by a high-performance camera often contains more objects than otherwise that are objects of AR display and providing AR display for all of these would cause the visibility of essential information to be known to decrease significantly resulting in reduction of usability, for which reason the need arises for an RA user interface application device and control method which selectively provide AR display of information in which a user is interested and for which the user has greater need.SOLUTION: The present invention detects from images photographed by a user terminal an object which is to be presented by AR display, acquires static metadata and dynamic metadata of the detected object, calculates the interest value of the detected object on the basis of the dynamic metadata, determines a target to be displayed at the user terminal on the basis of the calculated interest value, and creates data for AR display for each object determined as display target, on the basis of the static metadata.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an augmented reality user interface application apparatus and a control method.

  With the downsizing and high performance of computers, the performance of smartphones and wearable devices used as augmented reality (AR) user interface application devices has also improved dramatically. In software technology for controlling the AR user interface, not only the conventional technology for determining the AR display target from AR markers such as two-dimensional barcodes and position information, but also display from image information using object recognition technology. The accuracy of the markerless AR technique for detecting an object (object detection) and performing AR display has also been improved. Further, due to the improvement in camera performance, even when a farther object is photographed, it is possible to detect the object as an AR display target (object). Note that the object here is a signboard or an advertisement of a store to be subjected to AR display, and can recognize and detect the object without using an AR marker from a specific shape of the object. However, as a matter of course, detection of an object by an AR marker is also included.

  However, there are cases where there are more objects that are targets for AR display in video shot with a high-performance camera, and performing AR display on all of these significantly reduces the visibility of information that is originally intended to be known. There is a possibility that it may lead to a decrease in usability. Therefore, in the markerless AR technology, even when there are a plurality of objects in the captured video, the AR display is not performed for all display targets, and the user is interested and needs more. There is a need for an AR user interface application apparatus and a control method for selecting information and displaying an AR.

In order to achieve such an object, the present invention provides a computer device that causes a user terminal to perform AR display for an object that is an object of augmented reality (AR) display based on the user's interest. Remember the static metadata that is
From the user terminal, an image or video taken at the user terminal, and additional information related to the image or the video are received,
Obtaining the static metadata corresponding to the object contained in the received image or video;
When the static metadata is acquired, dynamic metadata that is dynamic information related to the user's interest with respect to the object is acquired from the received image or video and the additional information,
Calculating an interest value of the object based on the dynamic metadata;
Based on the calculated interest value, a display target on the user terminal is determined from the objects,
Based on the static metadata, create display data that is AR display data for each object that is the display target that has been determined,
The created display data is transmitted to the user terminal.

Further, in the computer according to the preceding paragraph, the additional information includes at least one of data relating to photographing of the image or the video, data relating to a terminal of the user terminal, and physical characteristic data of the user,
The dynamic metadata includes at least one of a position coordinate, a size, and a continuous display time of the object in the image or the video of the object, and the dynamic metadata is captured in the past by the user. Including historical data for the recorded image or the video,
In calculating the interest value,
Determining whether the position coordinates included in the dynamic metadata relating to the user are closer to the center of the image than the position coordinates in the past data of the object;
Determining whether the size included in the dynamic metadata relating to the user is larger than the size of the object in the past data, and the continuous display time included in the dynamic metadata relating to the user , Executing at least one of determining whether or not a predetermined threshold value is exceeded,
The interest value is calculated based on a determination regarding at least one of the executed position coordinates, the size, and the continuous display time.

  Further, in the invention described in the preceding paragraph, in calculating the interest value, the user's eye line and the history data of the eye line are set as an observation event, and the user's interest stage with respect to the object is determined based on the observation event. The interest value is estimated based on a hidden Markov model as a node.

In the invention described in the previous three paragraphs, in determining the display target on the user terminal,
Determining, based on the calculated interest value, the upper predetermined number of objects having a high interest value as a display target;
Determining the object for which the calculated interest value is equal to or greater than a predetermined threshold as a display target;
Based on the calculated interest value and the display amount of the AR display for the object, at least one of determining a display target on the user terminal from among the objects is executed.

Furthermore, in the invention described in the preceding four paragraphs, the static metadata includes a name, a genre, and detailed information of the object, and in creating the display data,
Based on the calculated interest value, a detailed level of the display data is determined,
Based on the detail level, determine at least one of the name, the genre, and the detailed information as a display target,
Generating the display data including at least one of the name, the genre, and the detailed information.

According to another aspect of the present invention, there is provided a method executed by a computer device that causes a user terminal to perform AR display on an object that is an object of augmented reality (AR) display, based on user interest.
Storing static metadata which is information specific to the object;
Receiving from the user terminal an image or video taken at the user terminal and additional information relating to the image or video;
Obtaining the static metadata corresponding to the object included in the received image or video;
Acquiring the static metadata, which is dynamic information related to the user's interest in the object, from the received image or video and the additional information when acquiring the static metadata; ,
Calculating an interest value of the object based on the dynamic metadata;
Determining a display target on the user terminal from among the objects based on the calculated interest value;
Creating display data that is data for AR display for each object that is the display target determined based on the static metadata;
Transmitting the created display data to the user terminal.

Furthermore, the present invention is a computer program for causing a computer device to cause a user terminal to perform AR display for an object that is an object of augmented reality (AR) display based on the user's interest, and is executed by the computer device. Then, in the computer device,
Storing static metadata which is information specific to the object;
From the user terminal, an image or video taken at the user terminal, and additional information related to the image or the video are received,
Obtaining the static metadata corresponding to the object included in the received image or video;
When the static metadata is acquired, dynamic metadata that is dynamic information related to the user's interest in the object is acquired from the received image or video and the additional information,
Based on the dynamic metadata, the interest value of the object is calculated,
Based on the calculated interest value, the display target on the user terminal is determined from the objects,
Based on the static metadata, display data that is data for AR display for each object that is the display target determined is created,
The created display data is transmitted to the user terminal.

  As described above, according to the present invention, in the AR technique (a technique using an AR marker and a technique using a markerless AR), an object and / or an AR marker is recognized from a captured image, and the object detected thereby is detected. Therefore, it is possible to provide an AR user interface application apparatus and a control method for performing AR display by selecting an object that the user is interested in and needs more.

It is a figure which shows the conventional markerless AR display image. It is a figure which shows the markerless AR display image at the time of using this invention. It is a figure which shows the system configuration | structure which concerns on one Embodiment of this invention. It is a flowchart which shows AR display processing which concerns on one Embodiment of this invention. It is a flowchart which shows the interest value calculation process which concerns on one Embodiment of this invention. It is a figure which shows the data stored in the static metadata memory | storage part which concerns on one Embodiment of this invention. It is a figure which shows the data stored in the dynamic metadata memory | storage part which concerns on one Embodiment of this invention. It is a figure which shows the data stored in the display data storage part which concerns on one Embodiment of this invention. It is a figure which shows the interest estimation model which concerns on one Embodiment of this invention.

  First, the conventional AR display image and the AR display image when the present invention is used will be described in comparison. FIG. 1 is a diagram showing a conventional markerless AR display image. In this way, since information is displayed without limitation for all detected objects, the mutual information overlaps, and it takes a very long time for users to search for information that they originally want to know. It will lead to decline.

  On the other hand, FIG. 2 is a diagram showing a markerless AR display image when the present invention is used. Transition from the upper left image to the right image and the lower left image as time elapses, more detailed display of what the user is interested in, and more concisely display what the user is not interested in (or You want to hide). More specifically, the upper left image is used as an initial display, and the system is determined that the user is watching “food” and interested in “food”. Then, the screen changes to the right image, and detailed information related to “food” is displayed. At this time, information related to “bank” which is not related to “food and drink” (that is, the user is not interested) is hidden. Next, with respect to the right image, the system determines that the user is very interested in “Chinese cuisine” and slightly interested in “Japanese cuisine”. Then, a transition is made to the image on the lower left, very detailed information related to “Chinese food” is displayed, and information related to “Japanese food” is also switched to a more detailed display (however, related to “Chinese food”) Not as detailed as information).

  Next, an overview of a system according to an embodiment of the present invention will be described. FIG. 1 is a diagram showing a system configuration according to an embodiment of the present invention. In FIG. 1, an AR display control server 300 installed in a data center or the like includes one or a plurality of smartphones 302a to 302n (hereinafter collectively referred to as “smartphones 302”) via a network 301 (for example, the Internet), And one or a plurality of wearable devices 303a to 303n (hereinafter collectively referred to as “wearable devices 303”). Smartphone 302 and wearable device 303 may be further collectively referred to as “user terminals”. Further, although the AR display control server 300 in FIG. 1 is described as a single server computer for the sake of convenience, it can also be configured as a distributed system by a plurality of server computers. Furthermore, the function of the AR display control server 300 can be mounted on the user terminal, and the present invention can be implemented on the user terminal.

  The AR display control server 300 is a server managed and operated by a company that provides an AR user interface. The AR display control server 300 holds information about an object to be displayed as an AR, and in response to a request from the user terminal, a user's interest value for each object (for example, 0.1.0 within a range of 0 to 1.0). A numerical value in increments of 1. An index indicating an object that the user is interested in as the numerical value is higher (hereinafter referred to as “interest value”). Further, the AR display control server 300 determines and creates information to be displayed on the user terminal based on the calculated interest value, and provides it to the user terminal.

  The smartphone 302 and the wearable device 303 (user terminal) are both terminals used by users who are general consumers. The user uses a camera function of the user terminal to photograph a landscape in the city including, for example, an object (a store signboard or the like) for which detailed information is desired (the user is interested). The captured image or video is transmitted to the AR display control server 300 by the user terminal. In addition to the image and video, as additional information, the user terminal sends the AR display control server 300 with shooting data such as the shooting position and shooting time related to the image and video, the terminal data of the user terminal, and the user terminal. You may transmit the physical characteristic data etc. which the user who is using measured. Thereby, the user terminal receives the detailed information of each object edited by the AR display control server 300 from the AR display control server 300, and displays the AR. Further, since the detailed information of each object needs to be dynamically changed according to the user's interest, the user terminal periodically (for example, once every 10 seconds) takes captured images and videos, and additional information as AR. The AR display is updated by transmitting to the display control server 300 and receiving the detailed information of the object each time.

  Next, the configuration of the AR display control server 300 will be described in detail. In FIG. 1, only a necessary functional configuration is shown assuming a single computer system. In the AR display control server 300, a RAM 311, an input device 312, an output device 313, a communication control device 314, and a storage device 316 that is a non-volatile storage medium (ROM, HDD, etc.) are connected to a CPU 310 via a system bus 315. Have a configuration. The storage device 316 includes a program storage area that stores a software program for performing each function of the present system, and a data storage area that stores data handled by the software program. Each means of the program storage area described below is actually an independent software program, its routine, component, etc., and is called from the storage device 316 by the CPU 310 and expanded in the work area of the RAM 311 while appropriately accessing a database or the like. Each function is performed by being executed sequentially.

  Next, only the software programs stored in the program storage area in the storage device 316 that are related to the present invention are listed. Data transmission / reception means 320, object detection means 321, metadata management means 322, interest value calculation means. 323 and display data management means 324. These means are executed by the CPU 310.

  The data transmission / reception means 120 performs data transmission / reception with other computers such as the smartphone 302 and the wearable device 303.

  The object detection unit 321 detects an object that is an AR display target from the captured video (image) information.

  The metadata management unit 322 acquires static metadata for the detected object from the static metadata storage unit 330. Here, static metadata refers to metadata that is uniquely determined for an object regardless of the user's behavior and properties, surrounding environment, shooting timing, etc. (for example, store name, store description) ). Further, the metadata management unit 322 acquires and calculates dynamic metadata and stores it in the dynamic metadata storage unit 331. Here, the dynamic metadata is related to the object, but depends on the user's actions and properties including the user's physical characteristics such as the user's eyes and heart rate, the surrounding environment including the position information, the timing of shooting, etc. Metadata that changes (eg, the position of the object in the image, the time that the object has been displayed in the image, the user's interest in the object). In the present invention, metadata for an object can be classified into two types: static metadata and dynamic metadata.

  The interest value calculation means 323 calculates the interest value of the target object based on the dynamic metadata. Moreover, the interest value calculation means 323 estimates the interest value for the target object based on the past data of each object.

  The display data management unit 324 determines an object to be displayed on the user terminal from the calculated interest value, further creates display data for each object determined as the display target, and stores the display data in the display data storage unit 332.

  Next, the data storage area in the storage device 316 includes a static metadata storage unit 330, a dynamic metadata storage unit 331, and a display data storage unit 332 when only those related to the present invention are listed. Both are fixed storage areas secured in the storage device 316.

  The static metadata storage unit 330 stores data unique to each object regardless of the user's operation and properties, surrounding environment, shooting timing, and the like. FIG. 6 is a diagram illustrating data stored in the static metadata storage unit 330 according to an embodiment of the present invention. It is assumed that this data is held in advance by the AR display control server 300 for each object. The static metadata in FIG. 6 includes an “object ID” for uniquely identifying the object, “genre 1” and “genre 2” indicating the genre of the object, “name” indicating the name of the object, and detailed object information. “Detailed information” is stored. “Object ID” is, for example, a sequential number. For “genre 1” and “genre 2”, “genre 2” is a more detailed genre than “genre 1”. 6 shows “genre 1” and “genre 2” and two different genres, but in another embodiment, “genre 3”, “genre 4”, etc. A more detailed genre can also be set. In any case, it is assumed that more detailed information such as “genre 1”, “genre 2”, “genre 3”... “Detailed information” is displayed when the user shows more interest. ing.

  The dynamic metadata storage unit 331 stores dynamic data for each object, which varies depending on the user's operation and properties, surrounding environment, shooting timing, and the like. FIG. 7 is a diagram showing data stored in the dynamic metadata storage unit 331 according to an embodiment of the present invention. It is assumed that this data is created by the AR display control server 300 every time an object is detected. The dynamic metadata in FIG. 7 includes a “user ID” that uniquely identifies the user, an “object ID” that uniquely identifies the object, a “shooting date and time” that indicates the date and time when the object was shot, and the display position of the object on the user terminal "Coordinates in sight" indicating, "continuous display time" indicating the elapsed time that the object has been displayed on the user terminal, "size" indicating the size of the object on the user terminal, and the degree of user interest in the object Stores “interest value”. The “user ID” may be an identifier in units of user terminals (for example, a MAC address), or an identifier in units of users using the AR service (AR service login ID). The “object ID” is linked to the “object ID” in the static metadata (FIG. 6). However, since this data is dynamic data, “user ID”, “object ID”, and “shooting” It becomes unique data by three data items of “date and time”. Therefore, this data can also be used as past data such as whether a user has been interested in a certain object in the past. “In-view coordinates” are, for example, xy coordinates indicating the position of an object when the upper left of the image is (0.0), and a predetermined range from the center of the image is defined as within the user's view (hereinafter referred to as “in-view coordinates”). , “In the field of view”) may be position coordinates within a predetermined range (considering that the end of the image is a reflected part but is not viewed by the user). “Continuous display time” is an elapsed time during which an object continues to be displayed in an image or field of view. “Size” is, for example, the total number of pixels occupied by an object in an image. The “interest value” is, for example, a numerical value in increments of 0.1 within a range of 0 to 1.0. A higher numerical value indicates that the user is interested in the target object.

  The display data storage unit 332 stores data for controlling AR display on the user terminal. FIG. 8 is a diagram illustrating data stored in the display data storage unit 332 according to an embodiment of the present invention. The display data in FIG. 8 includes an “object ID” for uniquely identifying the object, a “display name” indicating the name of the object displayed on the user terminal, and “display information” indicating the detailed information of the object displayed on the user terminal. Store. The display data in FIG. 8 indicates data for AR display in the lower left image of the display image in FIG. In addition to the above data items, the display data can also include items for specifying display characters and balloon display styles.

  Next, the AR display processing according to the embodiment of the present invention will be described along the flow with reference to the flowchart of FIG. 4 and the data of FIGS. 6-8. FIG. 4 is a flowchart showing an AR display process according to an embodiment of the present invention. In this process, for example, a user captures a downtown area as shown in FIGS. 1 and 2 using a camera mounted on the user terminal, and the captured video (image) is transmitted to the AR display control server 300. Assumes after. Since the AR display according to the present invention assumes real-time processing, transmission of captured video and additional information from the user terminal to the AR display control server 300 is performed periodically (for example, once every 10 seconds).

  When the captured video is transmitted from the user terminal, the data transmitting / receiving unit 320 receives the video (step 401). Next, the object detection means 321 detects an object from the received video (image) (step 402). The object is detected by recognizing an object in the image by a general method and detecting it as an object. Of course, in most cases, a plurality of objects are detected from one image. However, in step 402, when one object is detected, the process proceeds to the next step.

  When the object is detected, the metadata management unit 322 searches the static metadata storage unit 330 and acquires the static metadata (FIG. 6) for the detected object (step 403). When static metadata cannot be acquired (when the target object is not an AR display target), the process proceeds to the No route of step 404 and transitions to processing for the next object. If the next object exists, the process proceeds to the Yes route in step 405, and the acquisition of static metadata and dynamic metadata is repeated until there are no more objects (steps 403 to 408). If no static metadata for the detected object can be acquired, the process proceeds with the No route in Step 404, the No route in Step 405, and the No route in Step 406, and the process ends. In this case, that is, it means that no AR display target object exists.

  On the other hand, if the target static metadata can be acquired in step 403, the process proceeds to a Yes route in step 404, and the metadata management unit 322 acquires dynamic metadata (FIG. 7) for the detected object. This is stored in the target metadata storage unit 331 (step 407).

  Here, a method for acquiring dynamic metadata will be described with reference to FIG. As described above, the “user ID” is an identifier in units of user terminals (for example, a MAC address) or an identifier in units of users using the AR service (AR service login ID). It is transmitted from the user terminal to the AR display control server 300. The “object ID” is the same as the “object ID” of the static metadata acquired in step 403. “Shooting date / time” is, for example, the creation date / time of a shot video (image). That is, the shooting dates and times of a plurality of objects acquired from the same image are the same. As described above, the “in-view coordinates” are, for example, the position coordinates of the object when the upper left of the captured image is (0.0), but within a predetermined range (within the view) from the center coordinates of the captured image. It may be a position coordinate. The position coordinates of the object may be the midpoint of the object, or may be a feature point detected when the object is detected. “Continuous display time” is an elapsed time during which an object continues to be displayed in an image or field of view, but depends on a shooting period in the user terminal. For example, when the shooting cycle is once every 10 seconds, the “continuous display time” of the object reflected in the previous and current captured images is 10 seconds (in addition, if the image is captured before the previous time, +10 seconds will be added). For example, even if it is out of the image or view once in the 10 seconds, the “continuous display time” is 10 seconds. For this reason, the “continuous display time” is obtained by searching for this data using the “object ID” (and “user ID”) of the target object as a search key. It can be calculated by adding the imaging period to the “continuation display time” (for example, +10 seconds). The “size” can be calculated from the number of pixels of the object in the image. The calculation of the “interest value” will be described in step 409 and the interest value calculation process in FIG. Furthermore, in another embodiment, using the wearable device 303 or the smartphone 302 installed with the measurement application, the user's physical characteristic data such as the user's line of sight and heart rate is acquired as dynamic data. You can also.

  After the dynamic metadata is acquired in step 407, if there is a next object, the process proceeds to the Yes route in step 408, and acquisition of static metadata and dynamic metadata is repeated until there are no more objects ( Steps 403-408).

  If the next object does not exist, the interest value calculation means 323 calculates the interest value of each object (step 409). Note that the object that is the target of interest value calculation is limited to the object (object for AR display) in which static metadata exists. The interest value calculation process in step 409 will be described later with reference to FIG.

  After calculating the interest value of each object, the display data management unit 324 determines an object to be displayed on the user terminal based on the calculated interest value (step 410). For example, the upper predetermined number (for example, four) objects having a high calculated interest value can be displayed. Alternatively, an object having an interest value equal to or higher than a threshold value (for example, 0.5 or higher) can be displayed. Further, the determination of the object to be displayed can be performed in conjunction with the generation of display data in the next step 411. For example, the display amount of AR display (for example, the number of display characters and the display area of the balloon (number of pixels)) is summed in order from the object with the highest interest value until the total value of the display amount falls within a predetermined amount. An object can be determined as an object to be displayed.

  When the display target object is determined, the display data management unit 324 creates display data for the determined object (FIG. 8) and stores it in the display data storage unit 332 (step 411). The display data is created based on the static metadata acquired in step 403 (FIG. 6) and the interest value calculated in step 409. In this case, the detail level of the display data can be changed by the interest value (an object that the user is more interested in displays the display data in more detail). For example, the detail level of the display data is divided into several stages, and the detail level of the display data is determined depending on whether the interest value exceeds the threshold corresponding to each of the stages.

  More specifically, when the interest value is 0.9 or more, the detail level is determined as “highest”, and “genre 2”, “name”, and “detailed information” in the static metadata (FIG. 6) are targeted. The display data of the object is determined (for example, Sichuan cuisine XX: Chinese cuisine, mapo tofu is a specialty / lunch time 11: 30-13: 30).

  When the interest value is 0.6 or more and less than 0.9, the detail level is determined as “high”, and “genre 2” and “name” in the static metadata (FIG. 6) are determined as the display data of the target object. (Eg, Sichuan cuisine XX: Chinese cuisine).

  When the interest value is 0.3 or more and less than 0.6, the detail level is determined as “medium”, and “genre 2” in the static metadata (FIG. 6) is determined as the display data of the target object (for example, Chinese cuisine). Note that when the detail level is “medium” or lower, display data may overlap with other objects, so that only one display data with exactly the same content can be displayed.

  When the interest value is less than 0.3, the detail level is determined to be “low”, and “genre 1” in the static metadata (FIG. 6) is determined as the display data of the target object (eg, food and drink). This can also be controlled so that only one object is displayed when the display data overlaps with another object. From the above, it is possible to display in more detail what the user is interested in, and more concisely indicate what the user is not interested in.

  When the display data is created, the data transmission / reception means 320 transmits the created display data to the user terminal (step 412). Thereafter, the user terminal performs AR display based on the display data. Then, at the next shooting cycle, the user terminal transmits a newly shot video (image) to the AR display control server 300, and the processing is repeated from step 401. After step 412, the process ends.

  Next, the interest value calculation process (step 409) in FIG. 4 will be described along the flow with reference to the flowchart of FIG. FIG. 5 is a flowchart showing an interest value calculation process according to an embodiment of the present invention. In the process of FIG. 5, various determinations are made based on past data, and it is determined that the user is interested in the various determinations, and an interest value (for example, an initial value of 0. 1), a predetermined value is added. However, instead of the addition method, for example, various determinations are prioritized and processed, and if any of the determinations is satisfied, a predetermined value is set as an interest value, or a predetermined value is set for each determination. It is also possible to adopt a weighting method in which the interest value is added or set by weighting. Moreover, the process of FIG. 5 is only one embodiment, and the execution order of each determination and the type of determination to be executed can be changed.

  The process of FIG. 5 will be described. First, the interest value calculation means 323 determines whether or not the detected object has been detected last time (step 501). Specifically, the dynamic metadata (FIG. 7) is stored in the dynamic metadata storage unit 331 using the “user ID” of the target user terminal, the “object ID” of the target object, and the previous “shooting date and time” as search keys. ) Exists. When the corresponding dynamic metadata does not exist, it can be determined that the object is the object that the user is interested in for the first time and is not particularly interested in other detected objects. In this case, the process proceeds to the No route in step 501 and the process ends (the interest value of the target object remains the initial value 0.1). Note that the search key when searching for dynamic metadata can include not only the previous “shooting date / time”, but also the previous “shooting date / time” and the previous “shooting date / time” (multiple “shooting date / time”). OR search by “date and time”). This is because it may not be appropriate to set the interest value of the target object as the initial value, such as when the target object is out of the field of view in the previous shooting, but has been displayed all the time until the previous time.

  On the other hand, if the corresponding dynamic metadata exists, the process proceeds to the Yes route in step 501, and the metadata management unit 322 determines the current value of “in-view coordinates” in the dynamic metadata (FIG. 7) of the detected object. The previous value is acquired (step 502).

  Next, the interest value calculation unit 323 compares the acquired current value of the “in-view coordinates” with the previous value, and determines whether or not the detected object is closer to the center than the previous value (step 503). Thereby, when it is determined that the detected object is closer to the center than the previous time, it can be determined that the user is more interested in the target object. In this case, the process proceeds to the Yes route in Step 503, and the interest value calculation unit 323 adds a predetermined value (for example, 0.1) to the interest value of the target object (Step 504).

  On the other hand, if it is not determined that the detected object is closer to the center than the previous time, the process proceeds to the No route in step 503, and the process proceeds to the next process without changing the interest value. However, in another embodiment, when it is determined that the detected object is closer to the end than the previous time, a predetermined value can be subtracted from the interest value.

  In still another embodiment, the predetermined value to be added or subtracted can be changed according to the degree of approach to the center or edge of the detected object. For example, even if the difference between the current value and the previous value of “in-view coordinates” is the same, it is determined that the user's interest is different when approaching the center slightly and when approaching the center from near the center. This is because it can be determined that the case of approaching the center from near the center is more interested in the target object than the case of approaching the edge a little.

  After step 503 or step 504, the metadata management unit 322 acquires the continuous display time of the detected object (step 505). This can be acquired by adding (+10 seconds) a shooting cycle (for example, once every 10 seconds) to the previous “shooting date and time” in the dynamic metadata (FIG. 7).

  Next, the interest value calculation means 323 determines whether or not the acquired continuous display time is equal to or longer than a predetermined threshold and the detected object is displayed for a long time (step 506). Thereby, when it is determined that the detected object is displayed for a long time, it can be determined that the user continues to be interested in the target object. In this case, the process proceeds to the Yes route in step 506, and the interest value calculation unit 323 adds a predetermined value to the interest value of the target object (step 507).

  On the other hand, if it is not determined that the detected object is displayed for a long time, the process proceeds to the No route in step 506, and the process proceeds to the next process without changing the interest value. In another embodiment, when it is not determined that the detected object is displayed for a long time, a predetermined value can be subtracted from the interest value. In yet another embodiment, the predetermined threshold to be compared with the acquired continuous display time is further subdivided (for example, a threshold for long-time display, a threshold for medium-time display, ...), and depending on which threshold is exceeded In step 507, the predetermined value to be added to or subtracted from the interest value can be changed.

  After step 506 or step 507, the metadata management unit 322 acquires the current value and the previous value of “size” in the dynamic metadata (FIG. 7) (step 508).

  Next, the interest value calculation unit 323 compares the acquired current value of “size” with the previous value, and determines whether or not the detected object is larger than the previous value (step 509). Thereby, when it is determined that the detected object is larger than the previous time, it can be determined that the user is more interested in the target object. In this case, the process proceeds to the Yes route in step 509, and the interest value calculation unit 323 adds a predetermined value to the interest value of the target object (step 510). After step 510, the process ends.

  On the other hand, if it is not determined that the detected object has become larger than the previous time, the process proceeds to No route in step 509, and the present process ends without changing the interest value. However, in another embodiment, if it is determined that the detected object has become smaller than the previous time, a predetermined value can be subtracted from the interest value.

  In yet another embodiment, the predetermined value to be added (subtracted) can be changed depending on the degree of increase (decrease) in the detected object. For example, it is possible to judge that the user's interest is different between when it grows at a stretch and when it grows a little (when it grows all at once, it becomes more interesting than when it grows a little You can determine that you are more interested in the object).

  In another embodiment, in the interest value calculation process (step 409) in FIG. 4, the interest value can be estimated (calculated) using the estimation model. FIG. 9 is a diagram showing an interest estimation model according to an embodiment of the present invention. The estimation model divides the psychological stages of user interest for all objects detected in step 402 into four stages of AIDA (from the least interesting stage to “Attention” (attention stage), “Interest” (appeal stage), "Desire" (desired stage), "Action" (behavior stage)), a hidden Markov model with each object at each stage as a node. The transition between each node uses an observation event. An observation event is a user's physical characteristics, such as a user's eyes and a heart rate, for example. It is possible to determine which object the user is interested in based on which object in the captured image the user's eye is directed to, and which object has a change in heart rate. In the estimation model of FIG. 9, for the sake of convenience, two detected objects (an object related to “Chinese cuisine” and an object related to “Japanese cuisine”) are shown. However, in reality, the estimation model including more objects It is assumed that the interest value of each object is estimated. In the estimation model of FIG. 9, the interest in the object related to “Chinese cuisine” and the object related to “Japanese cuisine” reflected in the captured image (that is, considered to be interested by the user) is the history data of the user's eyes. From the above, it is an estimation model showing how the transition may occur.

  In FIG. 9, for example, an object related to “Chinese cuisine” in the Attention stage (attention stage, for example, an interest stage with an interest value of 0.1 to 0.2) has a probability of 0.4 (= 40 from the history data of the user's eyes. %), It is estimated that the interest is deepened and the transition to the Interest stage (the appeal stage, for example, the interest stage having an interest value of 0.3 to 0.5) is assumed (in this case, for example, “Chinese Chinese”). The object's interest value for “cooking” is estimated to be 0.3). In addition, the object related to “Chinese cuisine” in the Attention stage in FIG. 9 is transferred to the object related to “Japanese food” with a probability of 0.3, the interest remains unchanged with a probability of 0.2, and “Other” with a probability of 0.1. This means that it has been estimated that interest has shifted to the object. Here, the “other” object is an object other than the detected object related to “Chinese food” and “Japanese food” (that is, an object not reflected in the image or view).

  Each transition probability in the estimation model of FIG. 9 is learned by using historical data of observation events (user's gaze or user's physical characteristics such as heart rate) and a general estimation algorithm such as the Baum-Welch algorithm. At this time, it is possible to limit the history data of the user's eyes used for learning. For example, even if it is history data relating to the same object as the detected object, data taken at a completely different place is not used for learning, or history data within a predetermined period ("shooting date and time" is within a predetermined period) Historical data (even if it is the same user, based on the idea that very old data does not affect user interest) can also be used. Further, in the estimation model of FIG. 9, it is shown that the interest stage changes only in one stage, but it can be estimated that the stage changes in two stages or more.

  Then, using the learned transition probability, the history data of the current observation event, and a general estimation algorithm such as the Viterbi algorithm, it is possible to uniquely determine the most likely node where the current user interest is. . As a result, the interest value of the determined node (object) is defined by, for example, the interest value defined for each interest stage (for example, the Attention stage is 0.1, the Interest stage is 0.3, the Desire stage is 0.6, Action) The stage can be determined as 0.9).

  As described above, according to the present invention, in the AR technique (a technique using an AR marker and a technique using a markerless AR), an object and / or an AR marker is recognized from a captured image, and a user is detected from the detected objects. It is possible to provide an AR user interface application apparatus and a control method for performing an AR display by selecting an object that is more interested and more necessary.

300 AR display control server 301 Network 302 Smartphone 303 Wearable device 310 CPU
311 RAM
312 Input device 313 Output device 314 Communication control device 315 System bus 316 Storage device 320 Data transmission / reception means 321 Object detection means 322 Metadata management means 323 Interest value calculation means 324 Display data management means 330 Static metadata storage unit 331 Dynamic meta Data storage unit 332 Display data storage unit 332

Claims (7)

Based on the user's interest, in a computer device that causes a user terminal to perform AR display for an object that is an object of augmented reality (AR) display, static metadata that is information unique to the object is stored;
From the user terminal, an image or video taken at the user terminal, and additional information related to the image or the video are received,
Obtaining the static metadata corresponding to the object contained in the received image or video;
When the static metadata is acquired, dynamic metadata that is dynamic information related to the user's interest with respect to the object is acquired from the received image or video and the additional information,
Calculating an interest value of the object based on the dynamic metadata;
Based on the calculated interest value, a display target on the user terminal is determined from the objects,
Based on the static metadata, create display data that is AR display data for each object that is the display target that has been determined,
A computer apparatus, wherein the created display data is transmitted to the user terminal. The additional information includes at least one of data related to shooting of the image or the video, data related to a terminal of the user terminal, and physical characteristic data of the user,
The dynamic metadata includes at least one of a position coordinate, a size, and a continuous display time of the object in the image or the video of the object, and the dynamic metadata is captured in the past by the user. Including historical data for the recorded image or the video,
In calculating the interest value,
Determining whether the position coordinates included in the dynamic metadata relating to the user are closer to the center of the image than the position coordinates in the past data of the object;
Determining whether the size included in the dynamic metadata relating to the user is larger than the size of the object in the past data, and the continuous display time included in the dynamic metadata relating to the user , Executing at least one of determining whether or not a predetermined threshold value is exceeded,
The computer apparatus according to claim 1, wherein the interest value is calculated based on a determination regarding at least one of the executed position coordinates, the size, and the continuous display time.   In calculating the interest value, the user's eyes and the history data of the eyes are observation events, and based on the observation events, based on a hidden Markov model having the user's interest stage for the object as a node, The computer apparatus according to claim 1, wherein an interest value is estimated. In determining the display target on the user terminal,
Determining, based on the calculated interest value, the upper predetermined number of objects having a high interest value as a display target;
Determining the object for which the calculated interest value is equal to or greater than a predetermined threshold as a display target;
At least one of determining the display target on the user terminal from among the objects is executed based on the calculated interest value and a display amount of AR display for the object. Item 4. The computer apparatus according to any one of Items 1 to 3. The static metadata includes the name, genre, and detailed information of the object, and in creating the display data,
Based on the calculated interest value, a detailed level of the display data is determined,
Based on the detail level, determine at least one of the name, the genre, and the detailed information as a display target,
The computer apparatus according to any one of claims 1 to 4, wherein the display data including the determined at least one of the name, the genre, and the detailed information is created. A method executed by a computer device that causes a user terminal to perform AR display for an object that is an object of augmented reality (AR) display based on user interests, comprising:
Storing static metadata which is information specific to the object;
Receiving from the user terminal an image or video taken at the user terminal and additional information relating to the image or video;
Obtaining the static metadata corresponding to the object included in the received image or video;
Acquiring the static metadata, which is dynamic information related to the user's interest in the object, from the received image or video and the additional information when acquiring the static metadata; ,
Calculating an interest value of the object based on the dynamic metadata;
Determining a display target on the user terminal from among the objects based on the calculated interest value;
Creating display data that is data for AR display for each object that is the display target determined based on the static metadata;
Transmitting the created display data to the user terminal. A computer program that causes a computer device to cause a user terminal to perform AR display on an object that is an object of augmented reality (AR) display based on the user's interest, and when executed by the computer device, the computer device In addition,
Storing static metadata which is information specific to the object;
From the user terminal, an image or video taken at the user terminal, and additional information related to the image or the video are received,
Obtaining the static metadata corresponding to the object included in the received image or video;
When the static metadata is acquired, dynamic metadata that is dynamic information related to the user's interest in the object is acquired from the received image or video and the additional information,
Based on the dynamic metadata, the interest value of the object is calculated,
Based on the calculated interest value, the display target on the user terminal is determined from the objects,
Based on the static metadata, display data that is data for AR display for each object that is the display target determined is created,
A computer program for transmitting the created display data to the user terminal.

Images