JP4282091B1 - Terminal device, information processing method, and program - Google Patents

Abstract

An object of the present invention is to analyze a behavior factor by identifying an object of interest by a moving person, determining an area of interest in the object, and calculating an objective response to obtain an objective result. Realization of high accuracy.
A behavior analysis / application control unit 43 of a measuring device 4 identifies a target object 1 that a user 2 is paying attention to based on the position information and direction of the user 2 and the position information of the target object 1. . Then, the behavior analysis / application control unit 43 identifies the marker in the visual field image, and from the viewpoint position of the user 2 based on the object information, the area where the user 2 is interested in the object 1 And emotional reaction is calculated. And the function control part 51 of the mobile terminal 5 starts the application matched with the area where the user 2 is interested. By obtaining the conversion information of the user 2 accompanying the activation of the application, the behavior of the user 2 can be analyzed.
[Selection] Figure 4

Description

The present invention relates to a terminal device, an information processing method, and a program used for analyzing a user's behavior, and in particular, to identify an object that a moving user is interested in , determine an area in the object, and relate to the area. The present invention relates to a technique for starting an application and collecting information on user behavior .

  Conventionally, a system based on human eye movement is known as a system for evaluating the degree of interest in content (see Patent Document 1). This system focuses on the fact that human emotional responses appear in eye movements, and calculates the user's viewpoint dwell time or pupil diameter for the content displayed on the display device, thereby quantitatively determining the degree of interest in the content. The content is calculated and evaluated.

  In such a system, the user's viewpoint dwell time and pupil diameter are measured based on the user's line of sight and eyeball image. In order to detect the user's line of sight, for example, a difference image between an eyeball image captured by irradiating the eyeball with infrared rays and an eyeball image captured when the infrared rays are not irradiated is used (see Patent Document 2). reference). In addition, a technique using an eyeball shape model is also known in order to detect a user's line of sight with higher accuracy (see Non-Patent Document 1).

  By the way, it is known that a camera for photographing a user's eyeball and a camera for photographing a content actually viewed by the user are provided (see Non-Patent Document 2). The eyeglasses are also provided with a recording medium for storing eyeball images taken by a camera and visual field images such as contents. When the user wears the glasses, the eyeball image and the field-of-view image captured by the camera are stored in the recording medium. Then, the control device reads the eyeball image and the visual field image from the recording medium and performs image processing to calculate the viewpoint position, the viewpoint locus, the residence time, the pupil diameter, the number of blinks, and the like in the visual field image.

  On the other hand, a navigation system using a mobile terminal such as a mobile phone is known (see Patent Document 3). This system collects information from a mobile terminal, analyzes and processes the collected information, generates various types of information related to navigation, and transmits the information to the mobile terminal. For example, user location information is collected from the mobile terminal as an action result, and the difference between the preset action schedule and the collected action result is analyzed. The result analyzed in this way is used for tabulation such as marketing.

JP 2004-282471 A JP 2007-4448 A JP 2003-196284 A Takehiko Ohno and two others, “Gaze Measurement Method Based on Eyeball Shape Model”, 8th Image Sensing Symposium, 2002, pp. 307-312 “Mobile Eye Mark Recorder EMR-9”, [online], NAC Image Technology, Inc. [searched on August 1, 2008], Internet <URL: http://www.eyemark.jp/product/emr_9/ index.html>

  In the system disclosed in Patent Document 1 described above, when a user views a screen of content displayed on a display device, the degree of interest is calculated and content is evaluated. For this reason, there is a problem in that it is necessary to prepare a screen on which content is displayed in advance, and the user needs to view the content in front of the display device, so that preparation for evaluating the content takes time. In addition, when a user moves to various places, for example, when he sees a signboard at the destination, the content or the like noticed in the signboard is specified, and the behavior of the user corresponding to the content or the like is analyzed Then, the analysis result can be useful for users and content providers.

  For example, when the system using the glasses of Non-Patent Document 2 described above is assumed, it is possible to obtain a view image actually viewed by a moving user with the glasses, and interest in the viewpoint position in the view image. The degree can be calculated. However, when the user looks at the signboard, for example, in this system, the position where the view image exists in the signboard, the location in the signboard where the user is interested, and the user It is not possible to determine which content is noticed. Therefore, even when a system in which the glasses of Non-Patent Document 2 are applied to the system of Patent Document 1 is assumed, the behavior of the moving user cannot be analyzed, and useful information may be generated. Can not.

  In general, in order to analyze human behavior, for example, it is necessary to obtain answers to a questionnaire and tabulate the answers. However, since the behavioral analysis by the questionnaire is subjective, objective analysis results are not always obtained.

Therefore, the present invention has been made in order to solve the above-described problem, and its purpose is to identify an object that the person is paying attention to when performing behavior analysis on a moving person , Terminal device and information that can achieve high-precision behavioral factor analysis by determining the area within the target object, collecting information about user behavior by starting applications related to the area, and obtaining objective results It is to provide a processing method and a program.

In order to achieve the above object, the invention of claim 1 is an apparatus that is carried by a moving user, identifies an object that the moving user is paying attention to, and is arranged in the object. In a terminal device that determines an area surrounded by a plurality of markers and activates an application related to the area, position information indicating the position of the target object, position information indicating the position of the marker arranged in the target object, And object information storage means for storing object information including relationship information indicating the relationship between the area surrounded by a plurality of markers and the marker, and an area in the object and an application related to the area Application information storage means for storing the application information in association with each other, and a moving distance that changes as the user moves is a predetermined distance. If it is above, an acquisition request including information defining a predetermined range based on the current location of the user is transmitted to a server that manages object information and application information via the network, and the server Receiving the object information about the object installed within the predetermined range in the acquisition request and the application information associated with the area in the object, and the object information storage means and the application information storage means The object information and application information collecting means to be stored respectively in the object information and the position information of the object stored in the object information storage means are read out, and the user based on the current location of the user is suitable for the user. When the presence position indicated by the position information of the read target object is included within a predetermined range in the direction in which the target object is It is arranged on the object specified by the object specifying means from the object specifying means for specifying the object that the user is paying attention to and the visual field image in which the front facing the user is taken. The position of the identified marker is read out from the object information storage means, the viewpoint position that the user is viewing, and the position of the viewpoint indicated by the position information of the read marker A plurality of markers that surround the target object information storage means, read out the relationship information between the markers and the area stored in the object information storage means, specify the area that the plurality of markers surround, and the user's viewpoint in the specified area Area determination means for determining the identified area as an area within the object that the user is paying attention to when the position remains for a predetermined time or more; and the application Read out a plurality of pieces of application information associated with the area determined by the area determination unit from the application information storage unit, and specify one of the plurality of application information according to the viewpoint dwell time in the area And an application activation unit that activates the application, and the user's input operation is collected as action information for the area in the target object that the user has noticed as the application is activated by the application activation unit. And an information collecting means .

In addition, according to a second aspect of the present invention, in the terminal device according to the first aspect, the area determining means has the viewpoint position of the user in the specified area, and a pupil diameter of the user's eyeball is predetermined. When the length is equal to or greater than the length, the identified area is determined as an area in the target object that the user is paying attention to.

Further, the invention according to claim 3 is the terminal device according to claim 1 or 2, wherein the application activation unit is associated with a plurality of areas associated with the areas determined by the area determination unit from the application information storage unit. Application information is read out, and one of the plurality of application information is specified and the application is activated in accordance with the pupil diameter of the eyeball when the viewpoint stays in the area.

According to a fourth aspect of the present invention, in the terminal device according to any one of the first to third aspects, the application information is a URL designating a web page.

According to a fifth aspect of the present invention, there is provided an apparatus carried by a moving user, the position information indicating the position of the target object, the position information indicating the position of the marker arranged in the target object, and a plurality of Object information storage means for storing object information including relation information indicating the relationship between the area surrounded by the marker and the marker; application information for activating an area in the object and an application relating to the area; Application information storage means that associates and stores object information and application information collection means, object identification means, area determination means, application activation means, and information collection means, and the moving user Is identified, and an area surrounded by a plurality of markers arranged in the object is determined, and the error is determined. In the information processing method for activating an application related to a), the object information and application information collecting means, when the moving distance that changes with the movement of the user is a predetermined distance or more, the current presence of the user An acquisition request including information defining a predetermined range based on the position is transmitted to a server that manages object information and application information via the network, and is installed within the predetermined range in the acquisition request from the server. A first step of receiving object information about the object and application information associated with an area in the object, and storing the information in the object information storage unit and the application information storage unit; and The specifying means reads out the position information of the object stored in the object information storage means, When the existence position indicated by the position information of the read object is included in a predetermined range in the direction in which the user is facing based on the current existence position of the user, the object is A second step of specifying as an object that the user is paying attention to, and an object specified by the object specifying means from a field-of-view image in which the area determination means is taken in front of the user. A marker placed on an object is identified, position information of the identified marker is read from the object information storage means, a viewpoint position viewed by the user, and a marker position indicated by the position information of the read marker By specifying a plurality of markers surrounding the viewpoint position, reading the relationship information between the marker and the area stored in the object information storage means, specifying the area surrounded by the plurality of markers, A third step of determining the identified area as an area within the object that the user is paying attention to when the user's viewpoint position remains in the identified area for a predetermined time or more; An application activation unit reads a plurality of pieces of application information associated with the area determined by the area determination unit from the application information storage unit, and according to a viewpoint staying time in the area, A fourth step of activating an application by identifying one of them, and the information collecting means pays attention to the user's input operation as the application is activated by the application activation means. And a fifth step of collecting as behavior information for the area within the target object And wherein the door.

Further, the invention according to claim 6 is the information processing method according to claim 5, wherein the third step includes the use of the utilization point by the area determination unit, and the user's viewpoint position exists in the specified area. When the pupil diameter of the person's eyeball is greater than or equal to a predetermined length, the identified area is determined as an area within the object that the user is paying attention to.

The invention according to claim 7 is the information processing method according to claim 5 or 6, wherein the fourth step is determined by the area determining unit from the application information storing unit by the application starting unit. Reading a plurality of pieces of application information associated with the area, specifying one of the plurality of pieces of application information according to the pupil diameter of the eyeball when the viewpoint is retained in the area, and starting the application It is characterized by.

The invention according to claim 8 is the information processing method according to any one of claims 5 to 7, wherein the application information is a URL designating a web page.

A ninth aspect of the present invention is an information processing program that causes a computer to execute the information processing method according to any one of the fifth to eighth aspects.

As described above, according to the present invention, a target object that a moving user is interested in is identified, an area in the target object is determined, and an application related to the area is presented to collect information related to user behavior. It was to obtain an objective result by. Thereby, objective behavior analysis can be performed for a moving user. Therefore, since objective means are used instead of subjective means such as a questionnaire, it is possible to realize high accuracy of behavior factor analysis.

The best mode for carrying out the present invention will be described below with reference to the drawings.
[Outline of the system]
First, an outline of an entire system to which an embodiment of the present invention is applied will be described. FIG. 1 is a schematic diagram of the entire system. This system is intended to analyze the behavior of a user 2 who is interested in a specific area in the object 1. Specifically, this system identifies the target object 1 that the user 2 is paying attention from among a plurality of target objects, and determines the area in which the user 2 is interested in the target object 1. Then, information regarding the area is presented to the user 2, and information (conversion information) on the conversion (action) of the user 2 based on the information is collected and analyzed.

  The target object 1 is, for example, a POP (Point Of Purchase), a signboard, a poster, or a display shelf, and is provided with a marker for specifying an area in which the user 2 is interested. Details of the area and the marker will be described later. A user 2 wears glasses 3 and carries a measuring device 4 and a mobile terminal 5. The glasses 3 are provided with a field-of-view video camera 8 for photographing the object 1 in front of the user 2 and an eyeball image camera 9 for photographing the eyeball of the user 2. . The position of the visual field video camera 8 in the glasses 3 so that when the user 2 views the object 1, the video imaged by the visual field video camera 8 matches the video image of the user 2. Etc. are adjusted in advance. The eyeball image camera 9 is also adjusted in advance so that the eyeball of the user 2 can be correctly photographed and the line of sight etc. can be analyzed from the eyeball image by the measuring device 4. Data captured by the visual field image camera 8 and the eyeball image camera 9 is transmitted to the measuring device 4 via a cable or wirelessly.

  The measuring device 4 and the mobile terminal 5 constitute a behavior analysis terminal device. Hereinafter, the measurement device 4 and the mobile terminal 5 will be described as separate devices, but may be an integrated device as a behavior analysis terminal device.

  The measuring device 4 is, for example, attached to a belt, and sequentially stores various data such as information on the target object 1 noted by the user 2, position information (latitude and longitude) and direction of the user 2, and a dwell time of the viewpoint. . Data is transmitted and received between the measuring device 4 and the mobile terminal 5 by ToruCa (registered trademark), human body communication, Bluetooth, or the like. The measuring device 4 stores various data in an SD (Secure Digital) card, and the user 2 removes the SD card from the measuring device 4 and attaches it to the mobile terminal 5 so that the data is delivered. Good.

  The mobile terminal 5 is, for example, a mobile phone, a mobile terminal, or a PDA (Personal Digital Assistant), and automatically starts an application, starts an application according to the intention or selection of the user 2, or measures the mobile terminal 5. A function of starting an application, a function of transmitting / receiving an e-mail, a function of displaying a web page, and the like are realized by approaching the device 4. Data is transmitted and received between the mobile terminal 5 and the service center 6 by wireless communication.

  The service center 6 collects information about the target object 1 that has been noticed by the user 2 and information (behavior results) such as the dwell time at the viewpoint, and conversion information of the user 2 based on the presented information. Analyze behavior. Further, the service center 6 transmits the analysis result to the mobile terminal 5 carried by the user 2 or presents it to the operator as marketing information.

[System configuration]
FIG. 2 is a block diagram showing a configuration of the entire system shown in FIG. First, the measuring device 4 will be described. The measuring device 4 includes an eye movement analysis unit 41, a position / direction analysis unit 42, and a behavior analysis / application control unit 43.

  The eye movement analysis unit 41 inputs a view image from the view image camera 8 provided in the glasses 3 and also receives an eye image from the eye image camera 9. Then, the eye movement analysis unit 41 analyzes the visual field image and the eyeball image, and outputs the viewpoint and pupil diameter, which are eye movements, and the visual field image associated with the viewpoint to the behavior analysis / application control unit 43.

  The position / direction analysis unit 42 inputs GPS (Global Positioning System) data from an antenna (not shown in FIG. 1) provided in the measuring device 4 and also includes an acceleration sensor (see FIG. 1) provided in the measuring device 4. Acceleration sensor data is input from (not shown). Then, the position / direction analysis unit 42 analyzes the GPS data and the acceleration sensor data, and outputs the position information (latitude and longitude) and direction of the user 2 to the behavior analysis / application control unit 43.

  The behavior analysis / application control unit 43 transmits a request (target object information acquisition request) for acquiring information (target object information) related to the target object 1 to the mobile terminal 5. Then, the behavior analysis / application control unit 43 receives the object information from the mobile terminal 5 and stores it in the storage medium.

  In addition, the behavior analysis / application control unit 43 inputs the viewpoint, pupil diameter, and visual field image of the user 2 from the eye movement analysis unit 41, and the position information (latitude and longitude) of the user 2 from the position direction analysis unit 42. And enter the direction. The behavior analysis / application control unit 43 stores the input analysis data in a storage medium in time series. In addition, the behavior analysis / application control unit 43 identifies the target object 1 that the user 2 is paying attention to, and determines a line-of-sight matching area that is an area in which the user 2 is interested in the target object 1. The degree of interest in the line-of-sight matching area is calculated, and these action results are stored in the storage medium and transmitted to the mobile terminal 5.

  Next, the mobile terminal 5 will be described. The mobile terminal 5 includes a function control unit 51. The function control unit 51 receives the object information acquisition request from the measuring device 4 and sends the object information and the application information acquisition request to the server of the service center 6 in order to acquire the application information in addition to the object information to be acquired. 10 to send. Further, the function control unit 51 receives object information and application information from the server 10 of the service center 6 and stores the application information in a recording medium. Then, the function control unit 51 transmits the object information to the measuring device 4.

  Further, the function control unit 51 receives the analysis data and the action result from the measuring device 4 and transmits the analysis data to the server 10 of the service center 6. Moreover, the function control part 51 specifies the application corresponding to the received action result from application information, and starts the application.

  The user 2 acts based on information displayed on the screen of the mobile terminal 5 or operates the mobile terminal 5 in accordance with the activated application. The function control unit 51 collects the action, action, operation, etc. of the user 2 at this time as conversion information reflecting the operation of the mobile terminal 5, and transmits the action result and the conversion information to the server 10 of the service center 6. .

  Next, the server 10 of the service center 6 will be described. The server 10 of the service center 6 includes an information management / analysis unit 61. The information management / analysis unit 61 receives the object information and application information acquisition request from the mobile terminal 5 and transmits the object information and application information corresponding to this request to the mobile terminal 5.

  The information management / analysis unit 61 receives analysis data, behavior results, and conversion information from the function control unit 51 of the mobile terminal 5, analyzes the behavior of the user 2, and is useful for the user 2 and the operator. Generate and present information. In addition, the information management / analysis unit 61 changes various parameters necessary for the processing of the measuring device 4 and the mobile terminal 5.

[System functions]
Next, the functional flow of the system shown in FIGS. 1 and 2 will be described. FIG. 3 is a functional flow for explaining the overall processing, and shows the main processing performed by the system in time series. First, the behavior analysis / application control unit 43 of the measuring device 4 manages the object information required according to the location information (latitude and longitude) of the user 2 via the mobile terminal 5 and information management of the service center 6. Obtained from the analysis unit 61 and stored in a storage medium (step S301). In addition, the function control unit 51 of the mobile terminal 5 obtains application information required according to the position information (latitude and longitude) of the user 2 in accordance with a request from the behavior analysis / application control unit 43 of the measuring device 4. The information is acquired from the information management / analysis unit 61 of the service center 6 and stored in the storage medium (step S301).

  Specifically, the behavior analysis / application control unit 43 of the measuring device 4 applies these objects to all the objects 1 provided within a predetermined range centering on the position information (latitude and longitude) of the user 2. Object information is always stored in a storage medium. This is because the object 1 that the user 2 will pay attention to as the user 2 moves can be estimated in advance based on the position information (latitude and longitude) of the user 2. That is, the measuring device 4 acquires the object information of the object 1 before specifying the object 1.

  The behavior analysis / application control unit 43 of the measuring device 4 identifies the target object 1 that the user 2 is paying attention to (step S302). Specifically, the behavior analysis / application control unit 43 of the measuring device 4 includes the position information (latitude and longitude) and direction of the user 2, the position information of the target object 1 (position information included in the target object information), Based on the above, the positional relationship between the user 2 and the target object 1 is calculated, and the target that the user 2 is paying attention to the target object 1 that the user 2 will see according to the direction of the user 2 Identified as object 1.

  The behavior analysis / application control unit 43 of the measuring device 4 determines an area in which the user 2 is interested in the object 1 that the user 2 is paying attention to as a line-of-sight matching area (step S303). As described above, the object 1 is provided with a marker for determining an area where the object 1 is divided. The behavior analysis / application control unit 43 of the measurement device 4 identifies a marker in the visual field image, and determines a line-of-sight matching area where the viewpoint is retained based on the marker. Details will be described later.

  The behavior analysis / application control unit 43 of the measurement device 4 generates behavior result data and stores it in the storage medium (step S304). Specifically, the behavior analysis / application control unit 43 of the measuring device 4 calculates, for example, the viewpoint dwell time in the line-of-sight matching area, and uses this as the degree-of-interest degree data as an action including the line-of-sight matching area and the degree of interest. Generate the resulting data.

  The function control unit 51 of the mobile terminal 5 receives the behavior result from the behavior analysis / application control unit 43 of the measuring device 4, identifies the application associated with the line-of-sight matching area included in the behavior result from the application information, The identified application is activated (step S305).

  The function control unit 51 of the mobile terminal 5 provides information for the user 2 to act based on information displayed on the screen of the mobile terminal 5 or to operate the mobile terminal 5 with the activated application. collect. Then, the operation result of the mobile terminal 5 reflecting the behavior, action, operation, etc. of the user 2 at this time is converted as conversion information, and the behavior result and the conversion information are transmitted to the server 10 of the service center 6 (step S306).

  The information management / analysis unit 61 of the service center 6 receives the behavior result and the conversion information from the function control unit 51 of the mobile terminal 5, analyzes the behavior of the user 2, and is useful information for the user 2 or the business operator. Is generated and presented (step S307). Although not in the functional flow, analysis data (time, viewpoint position, pupil diameter, user position information, and user direction) generated by the behavior analysis / application control unit 43 of the measurement apparatus 4 is also transmitted from the measurement apparatus 4. It is transmitted to the server 10 of the service center 6 via the mobile terminal 5.

[Measurement equipment]
Next, the measuring device 4 shown in FIGS. 1 and 2 will be described in detail. FIG. 4 is a block diagram showing the configuration of the measuring device 4. The measuring device 4 includes the eye movement analysis unit 41, the position / direction analysis unit 42, and the behavior analysis / application control unit 43 as described above. The eye movement analysis unit 41 includes a viewpoint / pupil diameter analysis unit 411 and a visual field image analysis unit 412, and the position / direction analysis unit 42 includes a latitude / longitude analysis unit 421 and a direction analysis unit 422. The behavior analysis / application control unit 43 includes an application management control unit 431, a storage medium 432, a data synchronization management control unit 433, an image analysis / area interest analysis unit 434, a data transfer management control unit 435, and an inter-function communication unit 436. I have.

  The viewpoint / pupil diameter analysis unit 411 of the eye movement analysis unit 41 inputs the eyeball image of the user 2 from the eyeball image camera 9 of the glasses 3 and analyzes the eyeball image so that a visual field image to be described later is made flat. The viewpoint (positional coordinate in the visual field image) and pupil diameter are calculated, and the viewpoint and pupil diameter are output to the image analysis / area interest analysis unit 434 of the behavior analysis / application control unit 43. The eyeball image analysis method is already known, and is described in, for example, Patent Documents 1 and 2 and Non-Patent Document 1 described above.

  The visual field image analysis unit 412 receives the visual field image of the user 2 from the visual field image camera 8 of the glasses 3 and is calculated by the visual field image and the viewpoint / pupil diameter analysis unit 411 that are input every 1/30 seconds, for example. Perform temporal association with the viewpoint and the pupil diameter, and view the visual field image temporally corresponding to the viewpoint and the pupil diameter output from the viewpoint / pupil diameter analysis unit 411. Image analysis / area of the application control unit 43 The result is output to the interest analysis unit 434. Therefore, the viewpoint and pupil diameter output by the viewpoint / pupil diameter analysis unit 411 and the visual field image output by the visual field image analysis unit 412 are synchronized data.

  The latitude / longitude analysis unit 421 of the position / direction analysis unit 42 receives GPS data from the antenna, analyzes the GPS data, generates position information (latitude and longitude) of the user 2, and the position information (latitude and longitude). ) To the data synchronization management control unit 433 of the behavior analysis / application control unit 43. GPS data is data obtained by communication with GPS satellites.

  The direction analysis unit 422 receives the acceleration sensor data from the acceleration sensor, generates data of the direction (direction) of the user 2 by analyzing the acceleration sensor data, and determines the direction of the data of the behavior analysis / application control unit 43. The data is output to the synchronization management control unit 433. Specifically, the direction analysis unit 422 analyzes acceleration sensor data in the x, y, and z directions measured by the acceleration sensor, and generates data in the direction in which the user 2 is facing. For details of the method for generating the data of the direction of the user 2, refer to, for example, Japanese Patent Application Laid-Open No. 2006-320666.

  The application management control unit 431 of the behavior analysis / application control unit 43 controls the processing of the storage medium 432, the data synchronization management control unit 433, the image analysis / area interest analysis unit 434, and the data transfer management control unit 435 in an integrated manner. to manage. For example, the application management control unit 431 outputs the start and end of behavior analysis / application processing to each unit, or outputs a synchronization signal so that the processing of each unit operates in synchronization.

  The data synchronization management control unit 433 inputs the position information (latitude and longitude) of the user 2 from the latitude / longitude analysis unit 421 of the position / direction analysis unit 42, and inputs the direction of the user 2 from the direction analysis unit 422. The position information (latitude and longitude) and direction are synchronized, and the synchronized position information (latitude and longitude) and direction are output to the image analysis / area interest analysis unit 434 and the data transfer management control unit 435. Note that synchronized position information (latitude and longitude) and direction may be output in accordance with requests from the image analysis / area interest analysis unit 434 and the data transfer management control unit 435.

  The data transfer management control unit 435 receives the synchronized position information (latitude and longitude) and direction from the data synchronization management control unit 433, and the storage medium 432 has a predetermined range centered on the position information (latitude and longitude) of the user 2. In order to always store the object information of all the objects 1 in the mobile terminal, an object information acquisition request including position information indicating the range of the acquisition object is sent via the inter-function communication unit 436. To 5. Specifically, as the user 2 moves, the data transfer management control unit 435 compares the latest position information (latitude and longitude) already input with the newly input position information (latitude and longitude). When the moving distance is equal to or greater than the predetermined distance, the object information acquisition request is transmitted. The data transfer management control unit 435 receives the target object information of all the target objects 1 installed within the acquisition target range from the mobile terminal 5 via the inter-function communication unit 436, and stores the target object information. Store in the medium 432.

  For example, the data transfer management control unit 435 transmits the object acquisition request using the position information (latitude and longitude) of the user 2 and position information of a predetermined range centering on the position information as the position information of the acquisition target range. It may be. In this case, when receiving the object information from the mobile terminal 5, the data transfer management control unit 435 deletes the object information already stored in the storage medium 432 and newly stores the received object information.

  In addition, when the predetermined range centering on the location information (latitude and longitude) of the user 2 changes as the user 2 moves, the data transfer management control unit 435 changes the changed range (new range). Alternatively, the object acquisition request may be transmitted using only the acquisition target range. In this case, when receiving the object information from the mobile terminal 5, the data transfer management control unit 435 deletes the object information out of the predetermined range from the object information stored in the storage medium 432, and receives the object information. The object information within the new range is stored in the storage medium 432. Thereby, the storage medium 432 stores object information within a predetermined range centered on the position information (latitude and longitude) of the user 2. The data transfer management control unit 435 does not need to receive all object information within a predetermined range centered on the position information (latitude and longitude) of the user 2, and object information within a new range accompanying movement. Therefore, the amount of communication can be suppressed, and efficient processing can be realized.

  FIG. 9 is a diagram illustrating a configuration of the object information. As shown in FIG. 9, the object information includes an object ID, object position information (latitude and longitude), object type (for example, POP, signboard, poster, display shelf), marker format (for example, LED, QR code, checkerboard pattern), marker position information (position coordinates when the object is viewed from the front), and information indicating the relationship between the area and the marker (relation information). Such configuration data exists for each object. Here, the marker position information indicates, for example, the coordinates at which the marker exists on the plane when the user 2 views the object. Further, when the area is an area surrounded by, for example, four markers, the relationship information between the area and the marker is information indicating the relationship between one area and the four markers. Note that numbers are assigned in advance to the areas and markers. These data are stored in the storage medium 432 by the data transfer management control unit 435, and read from the storage medium 432 by the image analysis / area interest analysis unit 434.

  The image analysis / area interest analysis unit 434 receives the viewpoint, pupil diameter, and visual field image from the eye movement analysis unit 41, and receives the synchronized position information (latitude and longitude) and direction from the data synchronization management control unit 433, These data are stored in the storage medium 432 as analysis data together with time data.

  FIG. 11 is a diagram showing a configuration of analysis data. As shown in FIG. 11, the analysis data includes time, viewpoint position, pupil diameter, user 2 position information (latitude and longitude), and user 2 direction data. As described above, these data are stored in the storage medium 432 by the image analysis / area interest analysis unit 434 and read out from the storage medium 432 by the data transfer management control unit 435. The visual field image is included in the analysis data, but is omitted for convenience of explanation.

  The image analysis / area interest analysis unit 434 reads out the object information from the storage medium 432, the position information (latitude and longitude) and direction of the user 2, and the position information (latitude and latitude) of the object 1 included in the object information. The positional relationship between the user 2 and the object 1 is calculated based on the longitude, and the object 1 that the user 2 is paying attention to is specified. Specifically, the image analysis / area interest analysis unit 434 determines a predetermined direction in which the user 2 is looking from the position information (latitude and longitude) and direction of the user 2 around the position where the user 2 exists. An area within the distance is set, and the target object 1 within the range is specified from the position information (latitude and longitude) of the target object 1. When there are a plurality of objects 1 within the range, the closest object 1 is specified.

  Then, the image analysis / area interest analysis unit 434 identifies, in the identified object 1, an area (line-of-sight matching area) in which the user 2 is interested by identifying a marker included in the visual field image, The determination is made based on the object information of the identified object 1, the degree of interest of the user 2, and the like. This will be described in detail below.

  FIG. 7 is a diagram for explaining the object 1, the visual field image, the line-of-sight matching location, and the marker. As shown in FIG. 7, a plurality of markers (in the form of circles in FIG. 7) are embedded or pasted on the target object 1 at predetermined positions, and each marker has unique identification information. Yes. For example, when the marker is a QR code, the marker includes information indicating 0, 1, 2, 3,. The marker is made of a high-intensity infrared light LED, a phosphor, or the like, and may be one in which LEDs are arranged and represented as numbers, or may be a checkered pattern.

  As shown in FIG. 7, it is assumed that the user 2 pays attention to the object 1 and the field-of-view video is captured by the field-of-view video camera 8 provided in the glasses 3. In this case, the visual field image includes markers embedded in the object 1 (four markers in FIG. 7), and the viewpoint of the user 2 exists in the visual field image.

  FIG. 8 is a diagram illustrating processing for determining a line-of-sight matching area. As in FIG. 7, the user 2 pays attention to the object 1, and includes four markers M 3, 1, M 3, 2, M 4, 1 and M 4, 2 by the visual field image camera 8 provided on the glasses 3. It is assumed that a field of view image has been taken. Here, M3, 1 and the like indicate marker numbers.

  The image analysis / area interest analysis unit 434 performs marker identification processing according to the format of the marker included in the object information, and identifies the marker included in the visual field image input from the eye movement analysis unit 41. In the example of FIG. 8, four markers M3, 1, M3, 2, M4, 1, and M4, 2 are identified. Then, the image analysis / area interest analysis unit 434 extracts the position information of the identified marker from the object information of the identified object 1, and receives the position information of the viewpoint input from the eye movement analysis unit 41 and the identified marker. Based on the position information, four markers surrounding the viewpoint are specified.

  For example, the marker position information included in the object information shown in FIG. 9 is information such as (M3, 1: x3, y1) (M3, 2: x3, y2). Here, M3, 1 and the like indicate the marker number, and (x3, y1) and the like indicate the planar coordinates of the marker in the object 1. In this case, as shown in FIG. 8, the image analysis / area interest analysis unit 434 extracts the position information (x3, y1) and the like of the identified marker M3, 1 from the object information, and extracts the position information and viewpoint of the extracted marker. The markers M3, 1, M3, 2, M4, 1, and M4, 2 surrounding the viewpoint are specified based on the position information.

  Then, the image analysis / area interest analysis unit 434 specifies an area surrounded by the specified marker from the relationship information between the area and the marker included in the target object information of the specified target object 1. In the example of FIG. 8, the area A3, 1 surrounded by the markers M3, 1, M3, 2, M4, 1 and M4, 2 is specified.

  When the image analysis / area interest analysis unit 434 determines that the viewpoint of the user 2 has stayed in the specified area for a predetermined time or more, the image analysis / area interest analysis unit 434 determines that the user 2 has shown interest in the area. The area is determined as a line-of-sight matching area. In this case, instead of the determination based on the viewpoint dwell time, the image analysis / area interest analysis unit 434 is used when it is determined that the pupil diameter of a predetermined length or longer continues for a predetermined time for the specified area. It may be determined that the person 2 has shown interest in the area, and the area is determined as the line-of-sight matching area.

  The image analysis / area interest analysis unit 434 calculates how long and in which time the viewpoint has stayed in which area. That is, as long as the viewpoint stays in the same area, the image analysis / area interest analysis unit 434 counts the time and calculates the stay time in the area. The longer the viewpoint stays, the higher the degree of interest in the line-of-sight matching area. The image analysis / area interest analysis unit 434 determines the object ID for identifying the object 1, the area, the dwell time of the viewpoint in the area, the dwell start time, the dwell end time, and the line-of-sight matching area. An action result including the result is generated, and the action result is stored in the storage medium 432.

  FIG. 12 is a diagram illustrating a configuration of an action result. As shown in FIG. 12, the action result is composed of the object ID, the area, the staying time of the viewpoint in the area, the staying start time, the staying end time, and the determination result as to whether or not the line-of-sight matching area. These data are stored in the storage medium 432 by the image analysis / area interest analysis unit 434 and read from the storage medium 432 by the data transfer management control unit 435. The image analysis / area interest analysis unit 434 may store an average value of pupil diameters instead of the viewpoint dwell time. In this case, the greater the average value of the pupil diameter, the higher the degree of interest in the line-of-sight matching area.

  The image analysis / area interest analysis unit 434 determines the line-of-sight matching area, and when the viewpoint deviates from the line-of-sight matching area, the behavior result when the user 2 is interested (the object ID, the area, in the area) The data transfer management control unit 435 is instructed via the application management control unit 431 to transmit the viewpoint retention time, residence start time, residence end time, and line-of-sight matching area). Then, the data transfer management control unit 435 transmits the behavior result when the user 2 is interested to the mobile terminal 5 via the inter-function communication unit 436. Further, the data transfer management control unit 435 reads the analysis data and the action result from the storage medium 432 at a predetermined cycle, and transmits the analysis data and the action result to the mobile terminal 5 via the inter-function communication unit 436.

[Mobile terminal]
Next, the mobile terminal 5 shown in FIGS. 1 and 2 will be described in detail. FIG. 5 is a block diagram showing a configuration of the mobile terminal 5. The mobile terminal 5 includes the function control unit 51 as described above. Further, the function control unit 51 includes a central control unit 511, a storage medium 512, an inter-function communication unit 513, and a mobile communication unit 514, as shown in FIG.

  The central control unit 511 receives the object information acquisition request including the position information indicating the range of the acquisition target from the measurement device 4 via the inter-function communication unit 513, and adds the application information in addition to the target object information in the acquisition target range. In order to acquire information, the object information and application information acquisition request is transmitted to the service center 6 via the mobile communication unit 514. Then, the central control unit 511 receives the object information and application information in the acquisition target range from the service center 6 via the mobile communication unit 514, and stores the application information in the storage medium 512.

  FIG. 10 is a diagram illustrating a configuration of application information. As illustrated in FIG. 10, the application information includes an object ID, an area number, an application associated with the area, and an application activation method. The application information stored in the storage medium 512 is the area of the target object 1 in all the target objects 1 set within a predetermined range centered on the position information (latitude and longitude) of the user 2. It is information such as associated applications. The application associated with the area includes, for example, browsing a web page with a specified URL, sending an e-mail to a specified address, and specifying an application. The application activation methods include, for example, immediate activation, activation according to the degree of interest, and activation when the mobile terminal 5 is brought close to the measuring device 4. The activation when the mobile terminal 5 is brought close to the measuring device 4 is provided with a sensor that reacts when the measuring device 4 is brought close to the mobile terminal 5. The sensor does not react when the distance is equal to or greater than the predetermined distance, and the sensor reacts when the distance between the mobile terminal 5 and the measuring device 4 is shorter than the predetermined distance, and the application is activated.

  The same application may be assigned to a plurality of area numbers, or a plurality of applications may be assigned to one area number. In addition, an application corresponding to the degree of interest (for example, viewpoint dwell time, pupil diameter) is allocated, and for example, when the degree of interest is greater than a predetermined set value, the first application (for example, the user 2 A second application (for example, a user) is activated when an application that displays a product purchase screen for a product displayed in an area that has shown interest and the degree of interest is less than a predetermined set value. For the product displayed in the area where 2 has shown interest, an application for displaying a detailed explanation screen of the product may be assigned to be activated. Moreover, you may make it allocate an application and its starting method so that the application according to each conditions, such as the positional information (latitude and longitude) of the user 2, the direction of the user 2, and time, may start.

  Further, the central control unit 511 receives the analysis data and the action result from the measurement device 4 via the inter-function communication unit 513 and transmits these data to the service center 6 via the mobile communication unit 514.

  In addition, the central control unit 511 performs an action result when the user 2 is interested from the measurement device 4 via the inter-function communication unit 513 (object ID, area, dwell time of viewpoint in the area, dwell start) Time, staying end time, and line-of-sight matching area). The central control unit 511 reads the application information from the storage medium 512, and uses the object ID and the area (area number) included in the received action result as keys, and the application associated with the area included in the application information and Get the application launch method.

  The central control unit 511 activates the application associated with the line-of-sight matching area using the activation method. The user 2 determines the next action or the like by looking at the information displayed on the screen of the mobile terminal 5 as the application is started. The central control unit 511 collects the action, action, and operation by the user 2 at that time as conversion information including time data via an operation input to the mobile terminal 5. Then, the central control unit 511 transmits the action result, the activated application (including the case where it has not been activated), the activation method and the conversion information to the service center 6 via the mobile communication unit 514.

[Service Center]
Next, the server 10 of the service center 6 shown in FIG. 2 will be described in detail. FIG. 6 is a block diagram illustrating a configuration of the server 10. The server 10 includes an information management / analysis unit 61. The information management / analysis unit 61 includes a mobile communication unit 62, an object information management unit 63, an information analysis unit 64, and a parameter setting unit 65.

  The object information management unit 63 includes a management unit 631, a setting unit 632, and a storage medium 633. The setting unit 632 sets each piece of data constituting the object information shown in FIG. 9 and the application information shown in FIG. 10, and stores these pieces of information in the storage medium 633. In addition, the setting unit 632 reads the object information and application information from the storage medium 633, corrects each data constituting the information, and stores the data in the storage medium 633. That is, the storage medium 633 stores object information (see FIG. 9) and application information (see FIG. 10) corresponding to position information (latitude and longitude).

  The management unit 631 receives the object information and application information acquisition request including the position information indicating the range of the acquisition target from the mobile terminal 5 via the mobile communication unit 62. The management unit 631 searches the storage medium 633 using the position information indicating the acquisition target range included in the target information and application information acquisition request as a key, reads the target information and application information in the acquisition target range, It transmits to the mobile terminal 5 via the mobile communication unit 62. Thereby, the measurement apparatus 4 acquires the object information of the object 1 provided in the predetermined range centering on the positional information (latitude and longitude) of the user 2 as the user 2 moves, and stores it. It can be stored on medium 432. Similarly, the mobile terminal 5 can also acquire application information of the target 1 provided within a predetermined range centered on the position information (latitude and longitude) of the user 2 and store it in the storage medium 512. it can.

  The information analysis unit 64 includes an information collection unit 641, an object effect measurement unit 642, a behavior analysis unit 643, a recommended information generation unit 644, and a storage medium 645. The information collection unit 641 receives the analysis data, the behavior result, the behavior result when the user 2 is interested, the activated application, the activation method, the conversion information, and the like from the mobile terminal 5 via the mobile communication unit 62. Receive and store in storage medium 645.

  The object effect measurement unit 642 quantifies whether the object 1 has an influence on the action of the user 2 by using an action result or the like when the user 2 shows interest. That is, the object effect measuring unit 642 reads out data such as behavior results when the user 2 is interested from the storage medium 645, and for each line-of-sight matching area that shows interest in the object that the user 2 has paid attention to. Then, the object effect measurement result is calculated, organized, and stored in the storage medium 645. Specifically, the object effect measurement unit 642 calculates the viewpoint dwell time included in the action result when the user 2 is interested and the pupil diameter included in the analysis data at that time to start dwelling of the viewpoint. The integration process is performed between the time and the residence end time, and the degree of interest that is the integration value is calculated. The object effect measurement unit 642 identifies the name of the application that is activated first, and identifies conversion information indicating the behavior, action, etc. of the user 2. If there is no activated application, the name of the unactivated application is specified. The conversion information includes, for example, an operation for acquiring detailed information about a product displayed on the screen by starting up an application, an application for the product, and an operation such as purchase settlement.

  In addition, the object effect measurement unit 642 calculates a time (T1) from when the line-of-sight matching area in which the user 2 has shown interest is determined to when the application is started, The time (T2) until the action is performed is calculated. In the calculation of T2, the predetermined action refers to, for example, an operation of purchasing a product displayed on the screen by starting an application. The object effect measurement unit 642 calculates the time T2 when the predetermined action is included in the conversion information, and does not calculate the time T2 when the predetermined action is not included. In the above-described example, the time T2 is not calculated when an operation for purchasing a product is not performed.

  The object effect measurement unit 642 calculates an evaluation score (0 to 100 points) based on the degree of interest, the identified application and conversion information, time T1 and time T2. For example, the data such as the degree of interest is respectively weighted and normalized so that the total becomes 100 points or less. These data are stored in the storage medium 645 as the object effect measurement results, transmitted to the mobile terminal 5 for notification to the user 2, and provided to the operator as useful marketing information.

  FIG. 13 is a diagram illustrating a configuration of the object effect measurement result. As shown in FIG. 13, the object effect measurement result is data for each user 2 and includes an object, an area, an interest level, an application, conversion information, time T1, time T2, and an evaluation point. As described above, these data are stored in the storage medium 645 by the object effect measurement unit 642.

  Thus, the object effect measuring unit 642 can analyze the usefulness of the object 1 with respect to the action of the user 2 by digitizing the action result of the user 2 for each object 1.

  The behavior analysis unit 643 digitizes the behavior result of the user 2 for each user 2 in order to analyze the behavior of the consumer who is the user 2. That is, the behavior analysis unit 643 reads the object effect measurement results from the storage medium 645, organizes evaluation points for each object 1 for each user 2, and stores them in the storage medium 645 as behavior analysis results.

  FIG. 14 is a diagram illustrating a configuration of a behavior analysis result. As shown in FIG. 14, the behavior analysis result is composed of evaluation points of the respective objects 1 for each user 2. For example, it is shown that the consumer 1 who is the user 2 has an evaluation score of 85 for the object a and an evaluation score of 60 for the object b. Such behavior analysis results are stored in the storage medium 645 by the behavior analysis unit 643.

  As described above, the behavior analysis unit 643 generates the behavior analysis result shown in FIG. 14, whereby the purchase pattern and the consumption behavior pattern of the product indicated by the target object 1 for the consumer who is the user 2 and the consumer monitor. Can be analyzed. Further, by using the behavior analysis result of the first user 2 for another user 2 belonging to the group (age, gender, personality, etc.) of the first user 2, for example, the same application is started. By presenting the same product information, the other users 2 can view the desired product information, and as a result, the purchase of the product can be promoted.

  In addition, although the example which analyzes the consumption action with respect to goods was shown in the said embodiment, it can apply also in facilities with strong recreation property, such as an amusement park.

  The recommended information generation unit 644 organizes the behavior analysis result of the consumer who is the user 2 as recommended information and presents it to the user 2 in real time. That is, the recommendation information generation unit 644 reads the object effect measurement result and the behavior analysis result from the storage medium 645, specifies the product name and the store name from the activated application and conversion information, determines the purchase point, Recommendation information including the total score (evaluation score) is generated, stored in the storage medium 645, and transmitted to the mobile terminal 5 carried by the user 2. For example, it is assumed that the product name and the store name are included in a plurality of data displayed on the screen when the application is started, and the user 2 is selected by an operation. Purchase points are determined from conversion information that has been previously scored. For example, when the operation for acquiring detailed information about the product displayed on the screen by starting up the application, the score when the application for the product is operated, the purchase settlement etc. A score and the like are prepared in advance, and the recommended information generation unit 644 determines a score corresponding to actual conversion information as a purchase point. Further, the degree of interest and the total score (evaluation score) are extracted from the object effect measurement result.

  FIG. 15 is a diagram showing a configuration of recommendation information. As shown in FIG. 15, the recommended information is composed of the interest level, the purchase level, and the total score (evaluation score) for each user 2, the product name and the store name. Such recommendation information is stored in the storage medium 645 by the recommendation information generation unit 644.

  As described above, the recommendation information generation unit 644 generates recommendation information for the user 2 and presents the recommendation information to the user 2, so that the user 2 can recognize a product, a store, and the like according to his / her own interest. . For example, the recommendation information generation unit 644 can present to the user 2 recommendation information limited to a shopping street and a shopping center in the target area where the target object 1 is installed.

  The parameter setting unit 65 includes an emotion reaction weight value setting unit 651, a significant time threshold value setting unit 652, and a storage medium 653. The parameter setting unit 65 sets various parameters necessary for the measurement device 4 and the mobile terminal 5 to operate, and transmits the parameters via the mobile communication unit 62. Here, in the setting of various parameters by the parameter setting unit 65, in addition to setting parameters necessary for causing the measuring device 4 and the mobile terminal 5 to perform predetermined operations, measurement in which certain operations are performed. In order to cause the device 4 and the mobile terminal 5 to perform a new operation, changing parameters that have already been set is also included. Various set parameters are stored in the storage medium 653. For example, the data transfer management control unit 435 of the behavior analysis / application control unit 43 in the measurement device 4 acquires object information within a predetermined range centered on the position information (latitude and longitude) of the user 2. In this case, the parameter setting unit 65 sets and transmits a distance that defines the predetermined range. Further, the image analysis / area interest analysis unit 434 of the behavior analysis / application control unit 43 in the measurement device 4 sets and transmits the set value when determining the line-of-sight matching area in which the user 2 is interested. As described above, the parameter setting unit 65 sets and changes various parameters necessary for the measurement device 4 and the mobile terminal 5 to operate. Therefore, the measurement device 4 and the mobile terminal 5 are provided from the service center 6 side. Can be controlled. The details of the emotion reaction weight value setting unit 651, the significant time threshold value setting unit 652, and the storage medium 653 are the entire system to which another embodiment of the present invention using FIGS. 16 and 17 described later is applied. Will be described.

  As described above, according to the embodiment of the present invention, the measuring device 4 includes the position information (latitude and longitude) and direction of the user 2 and the position information (latitude and longitude) of the object 1 included in the object information. ) To identify the target object 1 that the user 2 is interested in. Thereby, even if the user 2 is moving, the target object 1 which the user 2 is paying attention to can be specified from among a plurality of target objects.

  Further, according to the embodiment of the present invention, the measuring device 4 identifies the marker in the visual field image, the marker type and position information included in the object information, the relationship information between the area and the marker, the degree of interest, From the viewpoint data of the user 2, it is possible to determine an area where the viewpoint exists in the target object 1 and to determine a line-of-sight matching area where the user 2 is interested.

  Further, according to the embodiment of the present invention, the mobile terminal 5 specifies and activates an application associated with an area in the object 1 that the user 2 is interested in based on the application information. . Thereby, with the start of the application, for example, conversion information such as the behavior of the user 2 based on the information displayed on the screen can be obtained, and the behavior of the user 2 can be analyzed.

  As described above, according to the embodiment of the present invention, the target object 1 that the moving user 2 is interested in is identified, the area showing interest in the target object 1 is determined, and the user 2 The objective response was obtained by calculating the emotional response based on the gaze. Thereby, objective behavior analysis can be performed on the moving user 2. Therefore, since objective means are used instead of subjective means such as a questionnaire, it is possible to realize high accuracy of behavior factor analysis.

[Modification]
Next, a modification of the system shown in FIG. 2 will be described. FIG. 16 is a block diagram showing an overall system to which another embodiment of the present invention is applied. As in the system shown in FIG. 2, this system uses the measurement device 4, the mobile terminal 5, and the server 10 of the service center 6 to provide a user 2 who is interested in a specific area in the object 1. The purpose is to analyze behavior. When the system shown in FIG. 2 is compared with the system shown in FIG. 16, the measurement device 4 in FIG. 16 has an eye movement analysis unit 41, a position / direction analysis unit 42, and behavior analysis / application control in the measurement device 4 in FIG. 2. In addition to the unit 43, it further differs in that it includes a physiological reaction analysis unit 44 and an external condition analysis unit 45. In addition, with this difference, processing by the behavior analysis / application control unit 43 of the measuring device 4 and the information management / analysis unit 61 of the server 10 is different.

  In FIG. 16, the measuring device 4 includes an eye movement analysis unit 41, a position / direction analysis unit 42, a behavior analysis / application control unit 43, a physiological reaction analysis unit 44, and an external condition analysis unit 45. The eye movement analysis unit 41 and the position / direction analysis unit 42 are the same as those shown in FIG.

  The physiological reaction analysis unit 44 inputs various sensor data from the armpit body temperature sensor and the body surface sensor attached to a predetermined location of the user 2, analyzes the input various sensor data, and calculates the body temperature, pulse, body surface temperature, and The amount of sweating is output to the behavior analysis / application control unit 43. The armpit body temperature sensor is attached to the armpit of the user 2 and outputs data for detecting body temperature. The body surface sensor is attached to an underwear or a wristwatch, for example, and outputs data for detecting a pulse, a body surface temperature, and a sweat rate. A pulse meter may be attached to the earlobe.

  The external condition analysis unit 45 inputs various sensor data from the temperature sensor, the humidity sensor, and the light quantity sensor held by the user 2, analyzes the input various sensor data, and relates to the external conditions of the ambient environment of the user 2. The outside air temperature, humidity and light quantity as data are output to the behavior analysis / application control unit 43.

  The behavior analysis / application control unit 43 transmits an object acquisition request for acquiring object information of the object 1 to the mobile terminal 5, receives the object information, and stores the object information in a storage medium. Further, the behavior analysis / application control unit 43 obtains the viewpoint and pupil diameter and visual field image of the user 2 from the eye movement analysis unit 41, and the position information (latitude and longitude) and direction of the user 2 from the position / direction analysis unit 42. The body temperature, pulse, body surface temperature and perspiration amount of the user 2 are input from the physiological reaction analysis unit 44, and the outside air temperature, humidity and light intensity are input from the external condition analysis unit 45, respectively. Then, the behavior analysis / application control unit 43 stores these input data in a storage medium in time series.

  In addition, the behavior analysis / application control unit 43 identifies the target object 1 that the user 2 is paying attention to, and determines a line-of-sight matching area that is an area in which the user 2 is interested in the target object 1. The emotional reaction in the line-of-sight matching area is calculated, a time zone in which the calculated emotional response value is greater than a predetermined threshold is set as a significant time, and these action results are stored in the storage medium and transmitted to the mobile terminal 5 . The behavior analysis / application control unit 43 flags each data in the significant time as an interest induction factor.

  Hereinafter, processing such as calculation of emotional reaction and setting of significant time by the behavior analysis / application control unit 43 will be described in detail. Since the other processes by the behavior analysis / application control unit 43 have already been described with reference to FIGS. 2 to 4, detailed description thereof is omitted here.

  The data synchronization management control unit 433 of the behavior analysis / application control unit 43 inputs the position information (latitude and longitude) and direction of the user 2 from the position / direction analysis unit 42, and the outside air temperature and humidity from the external condition analysis unit 45. Then, the input data are synchronized, and the input data are synchronized, and the synchronized position information (latitude and longitude), direction, outside temperature, humidity and light amount are output to the image analysis / area interest analysis unit 434.

  The image analysis / area interest analysis unit 434 receives the viewpoint, pupil diameter, and visual field image from the eye movement analysis unit 41, the body temperature, pulse, body surface temperature, and sweating amount from the physiological reaction analysis unit 44, and the data synchronization management control unit 433. Synchronized position information (latitude and longitude), direction, outside air temperature, humidity and light quantity are input.

Then, as described above, the image analysis / area interest analysis unit 434 identifies the target object 1 and determines the line-of-sight matching area in which the user 2 is interested in the identified target object 1 as well as the pupil diameter, Emotional responses are calculated based on time series data of body temperature, pulse, body surface temperature, and sweat rate (each data is synchronized). Specifically, the image analysis / area interest analysis unit 434 sets the time series data of the pupil diameter to p (t), the time series data of the body temperature to a (t), the time series data of the pulse to b (t), the body When the time series data of the table temperature is c (t), the time series data of the sweating amount is d (t), the weight values for each time series data are P, A, B, C, D, and the constant is S, The emotional reaction E (t) is calculated by the following equation.
E (t) = P * p (t) + A * a (t) + B * b (t) + C * c (t) + D * d (t) + S

The image analysis / area interest analysis unit 434 compares the calculated emotional response E (t) with a predetermined threshold th, and sets the time zone as a significant time when the emotional response E (t) is equal to or greater than the threshold th. Then, the data of the time zone is labeled as an interest inducing factor. That is, t when J (t) = 1 is set as a significant time according to the following equation.
J (t) = 1 (if E (t) ≧ th)
= 0 (else)

  The image analysis / area interest analysis unit 434 then identifies an object ID for identifying the identified object 1, area, viewpoint, pupil diameter, visual field image, body temperature, pulse, body surface temperature, perspiration, user 2 Position information (latitude and longitude), direction of user 2, outside air temperature, humidity, light quantity, weighted pupil diameter time series data P · p (t), etc., emotional response E (t) and label together with time data The data is stored in the storage medium 432 as analysis data. The analysis data is transmitted to the service center 6 via the mobile terminal 5.

  FIG. 17 is a diagram for explaining the significant time. As shown in FIG. 17, when the time zone in which the emotional reaction E (t) is larger than the threshold th is t1 to t2 and t3 to t4, the significant time is t1 to t2, t3 to t4. Therefore, since J (t) = 1 at times t1 to t2 and t3 to t4, the analysis data is labeled with each data of significant times t1 to t2 and t3 to t4, and significant times t1 to t2 and t3. Each data at ~ t4 becomes an interest induction factor.

  Further, as described above, the image analysis / area interest analysis unit 434 determines the object ID for identifying the specified object 1, the determined line-of-sight matching area, the dwell time of the viewpoint in the line-of-sight matching area, the dwell start time, and An action result including the residence end time is generated, and the action result is stored in the storage medium 432. The action result is transmitted to the service center 6 via the mobile terminal 5.

  Further, as described above, the data transfer management control unit 435 determines the line-of-sight coincidence area, and when the viewpoint deviates from the line-of-sight coincidence area, the action result (object ID, area) when the user 2 shows interest. The data transfer management control unit 435 is instructed via the application management control unit 431 to transmit the staying time of the viewpoint in the area, the staying start time, the staying end time, and the line-of-sight matching area. Then, the data transfer management control unit 435 transmits the behavior result when the user 2 is interested to the mobile terminal 5 via the inter-function communication unit 436. As described above, when the mobile terminal 5 receives an action result when the user 2 is interested from the measuring device 4, the mobile terminal 5 specifies an application corresponding to the action result from the application information and starts the application. Further, the data transfer management control unit 435 reads the analysis data and the action result from the storage medium 432 at a predetermined cycle, and transmits the analysis data and the action result to the mobile terminal 5 via the inter-function communication unit 436.

  Next, the parameter setting unit 65 of the information management / analysis unit 61 in the server 10 shown in FIG. 6 will be described. As described above, the parameter setting unit 65 includes the emotion reaction weight value setting unit 651, the significant time threshold value setting unit 652, and the storage medium 653. The emotional response weight value setting unit 651 sets weight values of time series data of pupil diameter, body temperature, pulse, body surface temperature, and sweating amount used when calculating the emotional response E (t), and the mobile communication unit 62 To send through. The set weight value is stored in the storage medium 653. The emotion reaction weight value setting unit 651 reads out and corrects the weight value from the storage medium 653, transmits it through the mobile communication unit 62, and stores it in the storage medium 653.

  The significant time threshold setting unit 652 sets a threshold th used when calculating the significant time, and transmits the threshold th through the mobile communication unit 62. The set threshold th is stored in the storage medium 653. The significant time threshold value setting unit 652 reads and corrects the threshold value th from the storage medium 653, transmits the threshold value th through the mobile communication unit 62, and stores it in the storage medium 653.

  Further, the parameter setting unit 65 sets items of time series data used when calculating the emotional reaction E (t), transmits the items via the mobile communication unit 62, and stores them in the storage medium 653. Thereby, the emotional reaction E (t) is calculated by the time series data item set by the parameter setting unit 65 in the behavior analysis / application control unit 43 of the measuring device 4. As described above, since the parameter setting unit 65 sets and changes various parameters such as the weight value and the threshold value th necessary for the operation of the measuring device 4 and the mobile terminal 5, from the service center 6 side. The measuring device 4 and the mobile terminal 5 can be controlled.

  As described above, according to another embodiment of the present invention, the object 1 to which the moving user 2 is interested is specified, and the area in which the object 1 is interested is determined and used. The emotional reaction E (t) based on the gaze of the person 2 was calculated to obtain an objective result. Thereby, objective behavior analysis can be performed on the moving user 2. Therefore, since objective means are used instead of subjective means such as a questionnaire, it is possible to realize high accuracy of behavior factor analysis.

  Further, according to another embodiment of the present invention, the information management / analysis unit 61 of the server 10 sets the weight value P and the like of each time-series data necessary for calculating the emotional reaction E (t). I made it. In addition, the information management / analysis unit 61 sets a threshold th necessary for calculating a significant time. Thereby, since the weight value P and the threshold value th can be set according to the user 2, it is possible to calculate the emotional reaction E (t) with high accuracy according to the physical condition, constitution, environment, etc. of the user 2. it can. Therefore, it is possible to achieve high accuracy in behavior factor analysis.

  As described above, the embodiment of the present invention has been described using the system shown in FIGS. 2 and 16, but the zoom-up degree and resolution of the visual field image are changed according to the degree of interest. Also good. Specifically, the image analysis / area interest analysis unit 434 displays the degree of interest (for example, viewpoint dwell time, pupil diameter size, deformation of FIG. 16) when the user 2 is actually looking at the object 1. The emotional reaction in the example may be calculated as described above. A zoom control unit (not shown) inputs the degree of interest from the image analysis / area interest analysis unit 434, and when it is determined that the degree of interest is high by the threshold determination, the zoom-up degree is increased so that the zoom-up degree is increased. To control the zoom of the camera 8 for the visual line direction video. As a result, the line-of-sight image analysis unit 412 and the image analysis / area interest analysis unit 434 can input a visual field image based on a zoom-up degree controlled by a zoom control unit (not shown). Therefore, behavior analysis corresponding to the degree of interest can be performed. In other words, when the degree of interest is high, it is possible to obtain a clear visual field image and further clarify the portion that the user 2 is paying attention to, so that more accurate behavior analysis can be performed.

  The line-of-sight image analysis unit 412 inputs the degree of interest from the image analysis / area interest analysis unit 434, and sets the resolution so that the resolution increases when the degree of interest is determined to be large by the threshold determination. , Generate and output a visual field image with the resolution. Accordingly, the image analysis / area interest analysis unit 434 can input a high-resolution visual field image when the degree of interest is high. Therefore, behavior analysis corresponding to the degree of interest can be performed. In other words, when the degree of interest is high, it is possible to obtain a clear visual field image and further clarify the portion that the user 2 is paying attention to, so that more accurate behavior analysis can be performed.

  In this case, the parameter setting unit 65 of the information management / analysis unit 61 in the server 10 can set and change the threshold parameters in the zoom control unit and the line-of-sight image analysis unit 412 (not shown). Also, it is possible to set and change the zoom-up degree and resolution according to the degree of interest, including the normal zoom-up degree and resolution, and the zoom-up degree and resolution when the degree of interest is large.

  The measuring device 4, the mobile terminal 5, and the server 10 are each configured by a computer having a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, and an interface. Each function of the eye movement analysis unit 41, the position / direction analysis unit 42, and the behavior analysis / application control unit 43 provided in the measurement device 4 shown in FIG. 2, and the eye movement analysis unit 41, the position / direction analysis unit shown in FIG. 42, each function of the behavior analysis / application control unit 43, the physiological reaction analysis unit 44, and the external condition analysis unit 45 is realized by causing the CPU to execute a program describing these functions. The function of the function control unit 51 provided in the mobile terminal 5 is realized by causing the CPU to execute a program describing this function. The function of the information management / analysis unit 61 provided in the server 10 is realized by causing the CPU to execute a program describing this function. These programs can also be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, or the like.

  The present invention has been described with reference to the embodiment. However, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the technical idea thereof. For example, in the above embodiment, the glasses 3 are used, but the glasses 3 are not necessarily required. For example, a hat provided with a camera may be used. In short, the eyeball image and field-of-view image of the user 2 may be taken by the camera.

  In the embodiment, the measurement apparatus 4 generates analysis data including the pupil diameter and stores the analysis data in the storage medium 432. However, the measurement apparatus 4 does not necessarily include the pupil diameter. It may be time. In short, any data relating to the degree of interest or emotional reaction of the user 2 obtained from the eyeball image may be used.

  In the embodiment, the measurement device 4 generates an action result including the viewpoint dwell time and stores it in the storage medium 432. However, the measurement result is not necessarily the viewpoint dwell time. For example, the pupil diameter, blink It may be the number of times. In short, any data relating to the degree of interest or emotional reaction of the user 2 obtained from the eyeball image may be used.

  Moreover, in the said embodiment, although the measuring device 4 handled the body temperature, the pulse, the body surface temperature, and the amount of perspiration as physiological reaction data, it is not necessary to be these data, and it is related to the physiological reaction of the user 2. Any data can be used. Moreover, in the said embodiment, although the measuring device 4 handled external temperature, humidity, and light quantity as data of the external condition which is environmental conditions, it does not necessarily need to be these data, and the circumference | surroundings of the user 2 around Any data related to the environment may be used.

1 is a schematic diagram of an entire system to which an embodiment of the present invention is applied. It is a block diagram which shows the structure of the whole system with which embodiment of this invention is applied. It is a functional flowchart explaining the whole process. It is a block diagram which shows the structure of a measuring device. It is a block diagram which shows the structure of a mobile terminal. It is a block diagram which shows the structure of a server. It is a figure explaining a target object, a visual field image, a gaze coincidence place, and a marker. It is a figure explaining the process which determines a visual line coincidence area. It is a figure which shows the structure of target object information. It is a figure which shows the structure of application information. It is a figure which shows the structure of analysis data. It is a figure which shows the structure of an action result. It is a figure which shows the structure of a target object effect measurement result. It is a figure which shows the structure of an action analysis result. It is a figure which shows the structure of recommendation information. It is a block diagram which shows the structure of the whole system with which other embodiment of this invention is applied. It is a figure explaining significant time.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Object 2 User 3 Glasses 4 Measuring device 5 Mobile terminal 6 Service center 7 Network 8 Field of view image camera 9 Eyeball image camera 10 Server 41 Eye movement analysis unit 42 Position direction analysis unit 43 Behavior analysis / application control unit 44 Physiological reaction analysis unit 45 External condition analysis unit 51 Function control unit 61 Information management / analysis unit 62 Mobile communication unit 63 Object information management unit 64 Information analysis unit 65 Parameter setting unit 411 Viewpoint / pupil diameter analysis unit 412 Visual field image analysis unit 421 Latitude / longitude analysis unit 422 Direction analysis unit 431 Application management control unit 432 Storage medium 433 Data synchronization management control unit 434 Image analysis / area interest analysis unit 435 Data transfer management control unit 436 Inter-function communication unit 511 Central control unit 512 Storage medium 513 Inter-function communication unit 514 Mobile communication unit 631 Management unit 32 setting unit 633 storage medium 641 information collecting unit 642 object effect measuring unit 643 behavioral analysis unit 644 recommendation information generating unit 645 storage medium 651 emotional response weighting value setting unit 652 significantly time threshold setting unit 653 storage medium

Claims (9)

A device carried by a moving user, which identifies an object of interest by the moving user, determines an area surrounded by a plurality of markers arranged in the object, and relates to the area In the terminal device that starts the application,
Object including position information indicating the position of the object, position information indicating the position of the marker arranged in the object, and relation information indicating the relationship between the area surrounded by a plurality of markers and the marker Object information storage means for storing object information;
Application information storage means for storing an area in the object and application information for starting an application related to the area in association with each other;
When the moving distance that changes with the movement of the user is equal to or greater than a predetermined distance, an acquisition request including information defining a predetermined range based on the current location of the user is sent via the network. Object information and application information transmitted to a server that manages the object information about the object installed within a predetermined range in the acquisition request from the server, and application information associated with an area in the object And object information and application information collection means for storing the object information storage means and application information storage means, respectively,
Read the position information of the object stored in the object information storage means, and within the predetermined range in the direction the user is facing, based on the current location of the user Object specifying means for specifying the object as the object that the user is paying attention to when the presence position indicated by the position information is included;
Identifying a marker placed on the object specified by the object specifying means from the field-of-view image taken in front of the user, and position information of the identified marker on the object information storage means From the viewpoint position that the user is viewing and the marker position indicated by the read marker position information, a plurality of markers that surround the viewpoint position are identified and stored in the object information storage means When the relationship information between the marker and the area is read, the area surrounded by the plurality of markers is specified, and when the user's viewpoint position stays in the specified area for a predetermined time or more, the specified area is Area determination means for determining the area within the target object that the user is paying attention to;
A plurality of pieces of application information associated with the area determined by the area determination unit are read out from the application information storage unit, and one of the plurality of pieces of application information is selected according to the viewpoint dwell time in the area. An application launching means for identifying and launching the application;
Information collection means for collecting an input operation by the user as action information for an area in the target object that the user has been paying attention to in accordance with the activation of the application by the application activation means;
A terminal device comprising: The terminal device according to claim 1,
The area determination means is configured to allow the user to pay attention to the specified area when the viewpoint position of the user exists in the specified area and the pupil diameter of the user's eyeball is equal to or larger than a predetermined length. The terminal device is characterized in that it is determined as an area within the target object. In the terminal device according to claim 1 or 2,
The application activation unit reads a plurality of pieces of application information associated with the area determined by the area determination unit from the application information storage unit, and according to the pupil diameter of the eyeball when the viewpoint stays in the area. A terminal device characterized in that one of the plurality of pieces of application information is specified and an application is activated. In the terminal device according to any one of claims 1 to 3,
A terminal device characterized in that the application information is a URL designating a web page. A device carried by a moving user includes position information indicating the position of an object, position information indicating the position of a marker placed in the object, an area surrounded by a plurality of markers, and the marker. Object information storage means for storing object information including relation information indicating the relationship between the area, and an application information storage means for storing an area in the object and application information for activating an application related to the area in association with each other And object information and application information collection means, object identification means, area determination means, application activation means, and information collection means, and identify the object that the moving user is paying attention to. , Determining an area surrounded by a plurality of markers arranged in the object, and an application related to the area In the information processing method for start-up,
The object information and application information collecting means, when the moving distance that changes with the movement of the user is a predetermined distance or more, information that defines a predetermined range based on the current location of the user Including the acquisition request including the object information and the application information via the network to the server that manages the object information and the application information, and the object information about the object installed within the predetermined range in the acquisition request, and the object A first step of receiving application information associated with an area in the object and storing the information in the object information storage unit and the application information storage unit, respectively;
The object specifying means reads out the position information of the object stored in the object information storage means, and is within a predetermined range in the direction in which the user is facing based on the current location of the user. A second step of identifying the object as an object that the user is paying attention to when the presence position indicated by the position information of the read object is included;
The area determining means identifies a marker placed on the object specified by the object specifying means from a visual field image taken in front of the user, and position information of the identified marker is obtained. A plurality of markers surrounding the viewpoint position are identified by the viewpoint position read from the object information storage means and viewed by the user and the marker position indicated by the position information of the read marker, and the object information When the relationship information between the marker and the area stored in the storage unit is read, the area surrounded by the plurality of markers is specified, and the user's viewpoint position stays in the specified area for a predetermined time or more, A third step of determining the identified area as an area within the object that the user is paying attention to;
The application activation unit reads a plurality of pieces of application information associated with the area determined by the area determination unit from the application information storage unit, and the plurality of pieces of application information according to a viewpoint staying time in the area A fourth step of identifying one of the applications and launching the application;
A fifth step in which the information collecting means collects an input operation by the user as action information for an area in the target object that the user has noticed as the application is started by the application starting means; ,
An information processing method characterized by comprising: The information processing method according to claim 5,
In the third step, the area determination unit determines the specified area when the viewpoint position of the user exists in the specified area and the pupil diameter of the user's eyeball is equal to or larger than a predetermined length. The information processing method is characterized in that it is determined as an area in the target object that the user is paying attention to. The information processing method according to claim 5 or 6,
In the fourth step, the application activation unit reads a plurality of pieces of application information associated with the area determined by the area determination unit from the application information storage unit, and a viewpoint is retained in the area. An information processing method comprising: starting an application by specifying one of the plurality of pieces of application information according to a pupil diameter of an eyeball. In the information processing method according to any one of claims 5 to 7,
An information processing method, wherein the application information is a URL designating a web page. An information processing program for causing a computer to execute the information processing method according to any one of claims 5 to 8.

Images