JP2020135737A - Gaze action search system, and control program - Google Patents

Abstract

To provide a gage action search system that acquires a location of terminals provided in users, and can readily grasp a check-up situation of facilities within a premise.SOLUTION: A gaze action search system according to the present invention includes: a wearable device that acquires shooting information shot from a plurality of different shooting locations by an imaging unit, and acquires visual line information indicative of a visual line direction of a user in a timing at a time of acquiring the shooting information; and a server that calculates the shooting location in a virtual three-dimensional space for each shooting information, calculates a gaze point location where the user gazes, and a gaze time in the virtual three-dimensional space on the basis of the visual line information, stores a virtual three-dimensional space model indicative of a three-dimensional shape of a peripheral environment of the user in the virtual three-dimensional space, and causes a display unit to display the shooting location in the identifiable gaze point location according to the gaze time and/or in the visual line information serving the basis of the calculation of the gaze point location, and the virtual three-dimensional space model.SELECTED DRAWING: Figure 1

Description

The present invention relates to a gaze behavior investigation system and a control program.

Conventionally, a system that includes a mobile terminal owned by a user and a server capable of communicating with the mobile terminal and tracks the position of the user has been known. In such a system, the server can acquire the position information and various information of the mobile terminal from the mobile terminal, and accumulate and analyze the acquired information.

For example, in Patent Document 1, a sensor device that emits radio waves such as RFID (Radio Frequency IDentifier) is installed in equipment in the facility, and a mobile terminal owned by an inspection worker is based on the strength of radio waves from the sensor device. The technology for identifying the position of the mobile terminal is described. In this system, the mobile terminal transmits the inspection result together with the position of the mobile terminal, and the server identifies which equipment the transmitted inspection result is for based on the transmitted position. As described above, in the system disclosed in Patent Document 1, since the specified inspection results are accumulated by the server, the time and effort for the inspection worker to register the work results is omitted, and the work efficiency is improved.

Japanese Unexamined Patent Publication No. 2014-146217

However, the layout in facilities such as factories and plants and commercial facilities may be changed as necessary, and when the system disclosed in Patent Document 1 is used, system management is performed every time the layout is changed. The person had to re-install the sensor equipment. In addition, when the facility is vast or the system is used in a wide range outside the facility, the amount of sensor equipment to be installed becomes enormous, which causes a problem that the installation cost and the maintenance cost increase.

Furthermore, when this system is used in a facility where there is a place where GPS (Global Positioning System) signals cannot be received, even if the user uses a terminal with a GPS signal receiver, the captured position should be stably acquired. I couldn't.

The present invention has been made to solve such a problem, and is a gaze behavior survey system that can stably acquire the position of a terminal provided by a user and easily grasp an object that the user is gaze at. , And to provide a control program.

The gaze behavior investigation system according to the present invention is a gaze behavior investigation system including a wearable device worn on the user's head and a server capable of communicating with the wearable device, and the wearable device includes a photographing unit and a photographing unit. It has an acquisition unit that acquires shooting information shot from a plurality of different shooting positions and acquires line-of-sight information indicating the user's line-of-sight direction at the timing of acquiring the shooting information in association with the shooting information. The server has a display unit, a first calculation unit that calculates a shooting position in the virtual three-dimensional space for each shooting information, and a gazing point position and a gazing time that the user is gazing at in the virtual three-dimensional space based on the line-of-sight information. A second calculation unit that calculates, a storage unit that stores a virtual 3D space model that shows the three-dimensional shape of the user's surrounding environment in the virtual 3D space, and a gaze point position and / or note that can be identified according to the gaze time. It has a shooting position in the line-of-sight information that is the basis for calculating the viewpoint position, and a server display processing unit that displays the virtual three-dimensional space model on the display unit.

Further, in the gaze behavior investigation system according to the present invention, it is preferable that the second calculation unit calculates the gaze time for each imaging position from which the gaze position is calculated.

Further, in the gaze behavior investigation system according to the present invention, it is preferable that the first calculation unit calculates the shooting position as the same shooting position within a predetermined region in the virtual three-dimensional space.

Further, in the gaze behavior investigation system according to the present invention, it is preferable that the second calculation unit calculates the gaze position as the same gaze position in a predetermined region in the virtual three-dimensional space.

Further, in the gaze behavior investigation system according to the present invention, it is preferable to further have a virtual three-dimensional space model generation unit that generates a virtual three-dimensional space model from shooting information.

Further, in the gaze behavior investigation system according to the present invention, the second calculation unit calculates a line segment extending in the corresponding line-of-sight direction from each shooting position for each shooting position, and the calculated line segment is calculated for each shooting position. It is preferable to calculate the intersection of the line segment and the outer surface of the virtual three-dimensional space model as the gazing point.

The control program according to the present invention is a control program of a server capable of communicating with a wearable device mounted on the user's head, and includes shooting information shot from a plurality of different shooting positions by a shooting unit included in the wearable device. The line-of-sight information indicating the user's line-of-sight direction at the timing of acquiring the shooting information is received from the wearable device, the shooting position in the virtual three-dimensional space is calculated for each shooting information, and based on the line-of-sight information, in the virtual three-dimensional space. The gaze position and gaze time that the user is gazing at are calculated, and the gaze position that can be identified according to the gaze time and / or the shooting position in the gaze information that is the basis for calculating the gaze position, and the virtual 3 The server is made to display a virtual three-dimensional space model showing the three-dimensional shape of the user's surrounding environment in the three-dimensional space.

By the gaze behavior investigation system and the control program according to the present invention, the position of the terminal provided to the user can be stably acquired, and the target gaze by the user can be easily grasped.

It is a function explanatory drawing for demonstrating the outline of the function which a gaze behavior investigation system 1 has. It is a figure which shows an example of the schematic structure of the gaze behavior investigation system 1. (A) is a perspective view showing an example of the appearance of the wearable device 2, and (b) is a diagram showing an example of a schematic configuration of the wearable device 2. It is a figure which shows an example of the schematic structure of the server 3. (A) is a schematic diagram showing a usage example of the wearable device 2, and (b) and (c) are diagrams showing an example of shooting information captured by the wearable device 2. (A) is a schematic diagram showing a three-dimensional model image, a shooting position, and line-of-sight information, and (b) is a diagram showing an example of displaying a three-dimensional model image and a gazing point. (A) is a diagram showing an example of the data structure of the equipment arrangement table, and (b) is a diagram showing an example of the data structure of the inspection result table. It is a figure which shows an example of the operation sequence of the gaze behavior investigation system 1. It is a figure which shows an example of the operation flow of 3D information calculation processing. It is a figure which shows an example of the operation flow of the gaze point calculation process. It is a figure which shows an example of the operation flow of inspection record calculation processing. It is a figure which shows an example of the schematic structure of the gaze behavior investigation system 10.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings. However, it should be noted that the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

(Overview of Gaze Behavior Survey System 1)
FIG. 1 is a function explanatory diagram for explaining an outline of the function of the gaze behavior survey system 1 of the present embodiment. The gaze behavior survey system 1 of the present embodiment is used to identify the position and gaze behavior of the user. The gaze behavior survey system 1 includes a wearable device 2 and a server 3.

The wearable device 2 is a terminal worn on the user's head. The wearable device 2 is, for example, a goggle-type wearable terminal or a glasses-type wearable terminal. The wearable device 2 has an imaging function for photographing the surrounding environment and acquiring the photographed information, an eye tracking function for detecting the direction of the user's line of sight, a storage function for storing the photographing information acquired by the photographing function, and various information. Any form may be used as long as it is a device having at least a medium storage function for storing in a storage medium. The wearable device 2 may have a function of reading various information from the storage medium. The storage medium is an SD memory card, SmartMedia (registered trademark), USB (Universal Serial Bus) memory, or the like. The wearable device 2 may have a communication function with the server 3 and a display function of various information. The wearable device 2 having a communication function with the server 3 does not have to have a storage function for storing various information in a storage medium.

The server 3 is a computer capable of acquiring information from the wearable device 2, and has a function of acquiring various information stored in a storage medium, a display function, and the like. The server 3 may have a function of storing various information in the storage medium. The server 3 may have a communication function with the wearable device 2, and in this case, the server 3 does not have a function of acquiring various information stored in the storage medium and a function of storing various information in the storage medium. May be good. Hereinafter, the server 3 will be described as a computer capable of communicating with the wearable device 2, but the computer capable of communicating with the wearable device 2 is a terminal device such as a multifunctional mobile phone (so-called “smartphone”), a tablet terminal, or a tablet PC (Personal). It may be a computer) or the like. The wearable device 2 and the computer capable of acquiring information from the wearable device 2 may both have a short-range wireless communication function such as Bluetooth (registered trademark).

Hereinafter, the outline of the function of the gaze behavior survey system 1 will be described with reference to FIG.

An imaging unit such as a camera included in the wearable device 2 attached to the user photographs a space in a specific direction at predetermined time intervals (for example, 1/30 second, 1/15 second, or 1 second). The photographing unit may take an image at an arbitrary timing specified by the user. The specific direction is the vertical direction and the subject direction of the image sensor provided in the photographing unit.

In FIG. 1 (1), the wearable device 2 mounted on the user shoots at the shooting point S1 to acquire the shooting information IM1, and shoots at the shooting point S2 to acquire the shooting information IM2. Is shown. The shooting information IM1 and IM2 are information indicating a still image. The shooting information IM1 and IM2 may be information indicating a moving image.

The line-of-sight detection unit included in the wearable device 2 detects line-of-sight information indicating the user's line-of-sight direction or gaze position at predetermined time (for example, 1/30 second, 1/15 second, or 1 second). The imaging timing by the imaging unit and the detection timing by the line-of-sight detection unit may be the same.

The wearable device 2 includes a slot for holding the storage medium detachably. The wearable device 2 stores the acquired shooting information and the line-of-sight information corresponding to the shooting information in the storage medium held in the slot. The line-of-sight information corresponding to the shooting information is the line-of-sight information acquired at the timing closest to the timing (acquisition time) at which the shooting information was acquired. The wearable device 2 may store the acquisition time corresponding to the shooting information in the storage medium in association with the shooting information and the line-of-sight information.

The wearable device 2 may transmit the acquired shooting information and the line-of-sight information corresponding to the shooting information to the server 3 each time the shooting information is acquired by the shooting unit. In this case, the wearable device 2 may associate the shooting information with the line-of-sight information and transmit the acquisition time corresponding to the shooting information to the server 3. The wearable device 2 may simultaneously transmit a plurality of untransmitted shooting information and line-of-sight information to the server 3 at a predetermined transmission timing.

The server 3 acquires the shooting information stored by the wearable device 2 via the storage medium. The server 3 has a position information which is a three-dimensional coordinate of the surrounding environment in the virtual three-dimensional space from the acquired shooting information, a virtual three-dimensional space model which shows the three-dimensional shape by the position information, and a shooting position in the virtual three-dimensional space. The 3D information calculation process for calculating the shooting position information indicating the 3D coordinates and the shooting direction is executed. The virtual three-dimensional space model is, for example, a surface model that shows the three-dimensional shape of the surrounding environment by the three-dimensional coordinates of a predetermined coordinate system (model coordinate system or world coordinate system) of the virtual three-dimensional space. The server 3 may acquire the shooting information by receiving the shooting information acquired from the wearable device 2. The three-dimensional information calculation process is based on, for example, a known SfM (Structure from Motion) process that uses a plurality of consecutive shooting information.

FIG. 1 (2) shows an example of the calculation result by the three-dimensional information calculation process. For example, the server 3 stores the calculation result and displays it on the display unit. When the wearable device 2 has various information display functions, the calculation result acquired from the server 3 by the wearable device 2 via the storage medium (or the calculation result transmitted from the server 3) is displayed by the display unit of the wearable device 2. May be done.

As shown in (2) of FIG. 1, feature points on the outer shape of the surrounding environment included in both the shooting information IM1 and the shooting information IM2 are extracted, and the epipolar geometric constraints that the feature points assumed to be the same should satisfy. Based on the conditions, the respective shooting positions and shooting directions of the shooting information IM1 and the shooting information IM2, and the three-dimensional coordinates in the virtual three-dimensional space for each feature point are restored.

Next, based on the respective shooting positions and shooting directions of the restored shooting information IM1 and shooting information IM2, the position information which is the three-dimensional coordinates in the virtual three-dimensional space of each pixel included in the shooting information of both is calculated. To. This position information is three-dimensional point cloud data having three-dimensional coordinates in a virtual three-dimensional space for showing the three-dimensional shape of the surrounding environment. By restoring the shooting position and shooting direction for all shooting information, it is possible to specify the 3D coordinates of each pixel included in all shooting information in the virtual 3D space, and within the range of all shooting information. The position information of various objects in is calculated. Further, a virtual three-dimensional space model, which is a surface model of the surrounding environment, is generated based on the three-dimensional point cloud data showing the three-dimensional shapes of various objects in the virtual three-dimensional space, which is the calculated position information. The shooting position and shooting direction of the restoration target may correspond to some shooting information.

The shooting position information indicating the three-dimensional coordinates of the shooting position in the virtual three-dimensional space shown in FIG. 1 (2) is, for example, the shooting position S1 in which the shooting was performed first and the shooting position S2 in which the shooting was performed next. Calculated based on the translation vector and rotation matrix between.

When the server 3 calculates the virtual 3D space model and the shooting position information based on a plurality of shooting information, the user gazes at each shooting position based on the line-of-sight information and the calculated virtual 3D space model and the shooting position information. The gaze point calculation process for calculating the gaze point position information indicating the three-dimensional coordinates in the virtual three-dimensional space of the gaze point position of the surrounding environment is executed.

In the gazing point calculation process, the server 3 first calculates a line segment extending from the shooting position in the line-of-sight direction based on the line-of-sight information based on the shooting position information and the line-of-sight information. Then, based on the generated virtual three-dimensional space model, the server 3 sets the first intersection of the outer surface of the surrounding environment and the line segment as the gazing point, and sets the position coordinates of the gazing point position in the virtual three-dimensional space. Calculated as gazing point position information.

FIG. 1 (3) shows an example of a display screen including a three-dimensional model image that gives a bird's-eye view of the surrounding environment of the target from an arbitrary viewpoint position and the position of the gazing point based on the virtual three-dimensional space model. There is. The display screen is displayed by, for example, the server 3. The display screen may be transmitted from the server 3 to the wearable device 2 and displayed by the display unit of the wearable device 2. When the wearable device 2 displays a three-dimensional model image, the virtual three-dimensional space model in the virtual three-dimensional space is displayed by perspective projection from the viewpoint position in the virtual three-dimensional space corresponding to the current position of the wearable device 2. The 3D model image projected on the virtual screen is displayed.

In this way, the gaze behavior survey system 1 can stably acquire the shooting position of the wearable device 2 and the gaze point of the user at any position in the facility by using a plurality of shooting information and a plurality of line-of-sight information. At the same time, it becomes possible to generate a virtual three-dimensional space model of the user's surrounding environment. Further, since it is not necessary to newly install a position identification device even in an indoor facility where GPS cannot be used, the gaze behavior survey system 1 inexpensively calculates the shooting position of the wearable device 2 and the position of the user's gaze point. It becomes possible to execute. Further, the gaze behavior investigation system 1 displays a three-dimensional model image showing the three-dimensional shape of the surrounding environment based on a plurality of shooting information and a plurality of gaze information, and displays the user's gaze point on the three-dimensional model image. can do.

The above description of FIG. 1 is merely a description for deepening the understanding of the contents of the present invention. Specifically, the present invention may be carried out in each of the embodiments described below, and may be carried out by various modifications without substantially exceeding the principles of the present invention. All such variations are included within the scope of the present invention and the present specification.

(Gaze behavior survey system 1)
FIG. 2 is a diagram showing an example of a schematic configuration of the gaze behavior survey system 1. The gaze behavior survey system 1 has a wearable device 2 and a server 3. Hereinafter, the gaze behavior survey system 1 will be described as an example when a user such as an inspection worker is used to inspect a plurality of inspection targets in a predetermined facility. As shown in FIG. 2, the transfer of information between the wearable device 2 and the server 3 is performed via the storage medium M.

Prescribed facilities include petroleum, petroleum refining, petroleum chemistry, gas, general chemistry, electricity, steelmaking, nuclear power, coal, water production, agriculture, feed, biochemistry, food, pharmaceuticals, medical care, information, communications, transportation, distribution, Facilities related to stockpiling, urban development, water and sewage, industrial pollution prevention, disaster prevention, environmental conservation, space development, construction, civil engineering, surveying, etc., facilities related to paper / pulp, ceramics / cement, metal refining, building materials, etc. And so on. Equipment inspection targets include health monitoring of leaks, wall thinning or damage from equipment (piping, equipment, machinery, etc.) in the facility, or vibration, instruments, temperature, humidity, specific gas concentration, etc. It is a point to measure vital life sensors for worker health management. Hereinafter, a predetermined facility may be simply referred to as a "facility".

(Wearable device 2)
FIG. 3A is a perspective view showing an example of the appearance of the wearable device 2.

The wearable device 2 shown in FIG. 3A is a spectacle-type wearable terminal worn on the head of a user who inspects a plurality of facilities arranged in a predetermined facility. The wearable device 2 includes a storage unit 21 that stores various information, a photographing unit 22 that photographs the inside of the facility and acquires the photographed information, a line-of-sight detection unit 23 that detects the user's line-of-sight direction and gaze point position, and a storage medium. A mounting unit 24 that is detachably held and a processing unit 25 that processes various information are provided. Further, the wearable device 2 may include a head-mounted display (HMD) type display unit for displaying various information. Further, the wearable device 2 may include a communication unit for communication with the server 3. Further, the storage unit 21, the photographing unit 22, the line-of-sight detection unit 23, the mounting unit 24, and the processing unit 25 of the wearable device 2 are stored in the same housing. The storage unit 21, the photographing unit 22, the line-of-sight detection unit 23, the mounting unit 24, and the processing unit 25 may be stored in different housings, or may be stored in at least two housings.

FIG. 3B is a diagram showing an example of a schematic configuration of the wearable device 2.

The storage unit 21 has, for example, a semiconductor memory. The storage unit 21 stores a driver program, an operating system program, an application program, data, and the like used for processing in the processing unit 25. For example, the storage unit 21 stores as a driver program a photographing device driver program that controls the photographing unit 22, a detection device driver program that controls the line-of-sight detection unit 23, an information recording / reading device driver program that controls the mounting unit 24, and the like. To do. Further, the storage unit 21 stores a storage processing program or the like for storing information in the storage medium M as an application program. In addition, the storage unit 21 stores the image captured by the photographing unit 22 as data.

The photographing unit 22 includes an imaging optical system, an imaging element, and the like. The imaging optical system is, for example, an optical lens, and a light flux from a subject is imaged on an imaging surface of an image pickup device. The image pickup device is a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or the like, and outputs an image of a subject image formed on the image pickup surface. The photographing unit 22 creates and outputs moving image data in a predetermined file format at predetermined intervals from the images continuously generated by the image sensor. Alternatively, the photographing unit 22 creates and outputs still image data in a predetermined file format from the image generated by the image sensor. When the image is output, the photographing unit 22 may associate the acquisition time of the image with the image.

The line-of-sight detection unit 23 uses, for example, a corneal reflex method in which the user's eyeball is irradiated with infrared rays, a reflected image on the corneal surface is photographed, and the line of sight is obtained from the distance between the center of the pupil and the reflected image, and the direction of the user's line of sight is obtained. Is detected. The line-of-sight detection unit 23 may detect the direction of the user's line of sight by using an eye potential method using an electro-oculography sensor. The line-of-sight detection unit 23 is not limited to the above two methods, and any method that can detect the direction of the user's line of sight may be used, and the line-of-sight detection unit 23 realizes the method. Various devices may be provided.

As shown in FIG. 3A, the line-of-sight detection unit 23 detects the direction of the line-of-sight for each of the left and right eyes of the user, calculates the point where the two lines of sight intersect as the gazing point, and is the center of the wearable device 2. The direction of a straight line connected from a position (for example, an intermediate position between the left and right eyeballs) toward the gazing point is detected as the user's line-of-sight direction. The imaging timing by the photographing unit 22 and the detection timing by the line-of-sight detection unit 23 may be simultaneous.

The line-of-sight direction detected by the line-of-sight detection unit 23 is represented by, for example, a unit vector based on the device coordinate system shown in the x-axis, y-axis, and z-axis shown in FIG. 3A. The z-axis direction of the device coordinate system is the photographing direction. The line-of-sight detection unit 23 outputs line-of-sight information indicating the detected line-of-sight direction at predetermined time intervals. When the line-of-sight detection unit 23 outputs the line-of-sight information, at least one of the shooting information acquired at the same timing and the acquisition time of the line-of-sight information may be associated with the line-of-sight information.

The mounting unit 24 includes a card slot that holds the storage medium M detachably. For example, when the storage medium M is an SD memory card, the mounting unit 24 includes an SD card slot. The mounting unit 24 has a function of reading various information stored in the mounted storage medium M and a function of recording various information in the mounted storage medium M.

The processing unit 25 includes one or more processors and peripheral circuits thereof. The processing unit 25 comprehensively controls the overall operation of the wearable device 2. For example, the CPU (Central Processing), which is a circuitry that executes various processes and controls devices by various programs. Unit). The processing unit 25 includes the photographing unit 22, the line-of-sight detection unit 23, and the mounting unit 24 so that various processes of the wearable device 2 are executed in an appropriate procedure according to the output from the program stored in the storage unit 21. Etc. are controlled. The processing unit 25 executes processing based on a program (driver program, operating system program, application program, etc.) stored in the storage unit 21. Further, the processing unit 25 can execute a plurality of programs (application programs and the like) in parallel.

The processing unit 25 includes an acquisition unit 251 and a storage processing unit 252. Each of these units included in the processing unit 25 is a functional module implemented by a program executed on the processor included in the processing unit 25. Alternatively, each of these units included in the processing unit 25 may be mounted on the wearable device 2 as an independent integrated circuit, microprocessor, or firmware.

(Server 3)
FIG. 4 is a diagram showing an example of a schematic configuration of the server 3.

In particular, the server 3 acquires data from the wearable device 2, executes three-dimensional information calculation processing, gaze point calculation processing, and inspection record calculation processing, and stores and / or displays data such as the results of various calculation processes. Has the function of In order to realize these functions, the server 3 includes a server storage unit 31, a server display unit 32, a server mounting unit 33, and a server processing unit 34.

The server storage unit 31 has, for example, at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The server storage unit 31 stores driver programs, operating system programs, application programs, data, and the like used for processing by the server processing unit 34. For example, the server storage unit 31 stores, as a driver program, a display device driver program that controls the server display unit 32, an information recording / reading device driver program that controls the server mounting unit 33, and the like. The various programs may be installed in the server storage unit 31 from a computer-readable portable recording medium such as a CD-ROM or a DVD-ROM using a known setup program or the like. The server storage unit 31 stores various tables and the like, which will be described later, as data.

The server display unit 32 may be any device as long as it can output a moving image, a still image, or the like, and is, for example, a touch panel type display device, a liquid crystal display, an organic EL display, or the like. The server display unit 32 displays a moving image according to the moving image data supplied from the server processing unit 34, a still image corresponding to the still image data, and the like.

The server mounting unit 33 includes a card slot that holds the storage medium M detachably. For example, when the storage medium M is an SD memory card, the server mounting unit 33 includes an SD card slot. The server mounting unit 33 has a function of reading various information stored in the mounted storage medium M and a function of recording various information in the mounted storage medium M.

The server processing unit 34 includes one or more processors and peripheral circuits thereof. The server processing unit 34 is a CPU that comprehensively controls the overall operation of the server 3, and is, for example, a circuitry that executes various processes and controls devices by various programs. The server processing unit 34 controls the operations of the server display unit 32, the server mounting unit 33, and the like so that various processes of the server 3 are executed in an appropriate procedure based on a program or the like stored in the server storage unit 31. To do. The server processing unit 34 executes processing based on a program (operating system program, driver program, application program, etc.) stored in the server storage unit 31. Further, the server processing unit 34 can execute a plurality of programs (application programs and the like) in parallel.

The server processing unit 34 includes a server acquisition unit 341, a calculation unit 342, a server display processing unit 343, and a server storage processing unit 344. Each of these units included in the server processing unit 34 is a functional module implemented by a program executed on the processor included in the server processing unit 34. Alternatively, each of these units included in the server processing unit 34 may be mounted on the server 3 as an independent integrated circuit, microprocessor, or firmware.

FIG. 5A is a schematic diagram showing a usage example of the wearable device 2.

As shown in FIG. 5A, the user wears the wearable device 2 when inspecting a plurality of inspection targets in a predetermined facility. The photographing unit 22 of the wearable device 2 mounted on the user's head takes an image at predetermined time intervals.

5 (b) and 5 (c) are examples of two consecutive shooting information shot by the shooting unit 22 of the wearable device 2. The shooting information shown in FIG. 5B was first shot by the shooting unit 22 of the wearable device 2, and the shooting information shown in FIG. 5C was later shot by the shooting unit 22 of the wearable device 2. It was taken.

When the server 3 acquires two consecutive shooting information shown in FIGS. 5B and 5C from the wearable device 2, the server 3 obtains the known SfM based on the two consecutive shooting information acquired. Execute 3D information calculation processing such as processing.

In the three-dimensional information calculation process, first, the server 3 extracts a plurality of feature points from each of the two shooting information by using a known local feature amount extraction method or the like, and local features of each feature point. Calculate the amount. The local feature extraction method is a SURF (Speeded Up Robust Features) method, a SIFT (Scale-Invariant Feature Transform) method, or the like. The extraction results of the feature points are shown in FIGS. 5 (b) and 5 (c). The method for extracting feature points is not limited to the above-mentioned method, and may be a known method such as a Hessian matrix.

Next, the server 3 comprises an X-axis, a Y-axis, and a Z-axis as shown in FIG. 6A based on a plurality of feature points extracted from each of the two shooting information under the constraint condition of epipolar geometry. The translation vector t and the rotation matrix R of the shooting position in the virtual three-dimensional space are calculated. The constraint conditions of epipolar geometry are as shown in the following equation (1).

The image plane coordinates of the feature points included in the above shooting information are as shown in the following equation (2).

The image plane coordinates of the feature points included in the subsequent shooting information are as shown in the following equation (3).

Next, the server 3 has an internal parameter matrix K1 of the photographing unit 22 corresponding to the earlier shooting information, an internal parameter matrix K2 of the photographing unit 22 corresponding to the later shooting information, and an elementary matrix derived by the equation (1). Using F, the basic matrix E is calculated according to the following equation (4).

Next, the server 3 decomposes the fundamental matrix E into a translation vector t and a rotation matrix R by using a known decomposition method. The translation vector t and the rotation matrix R indicate the relative positions in three dimensions for each of the positions of the plurality of feature points, the shooting position corresponding to the earlier shooting information, and the shooting position corresponding to the later shooting information.

Next, the server 3 calculates the translation vector t and the rotation matrix R for all the combinations of the continuous two shooting information in the plurality of shooting information acquired from the wearable device 2 as described above.

For example, the server 3 sets the position of the first shooting information among the plurality of shooting information acquired from the wearable device 2 as the origin coordinates (X, Y, Z) = (0,0,0) in the virtual three-dimensional space. To do. Next, the server 3 sequentially accumulates the translation vector t calculated for the combination of two consecutive shooting information according to the order of the shooting information, so that the subsequent shooting information of the combination of the two consecutive shooting information is shot. The coordinate value of the position in the virtual three-dimensional space is calculated. Next, the server 3 calculates the posture of the photographing unit 22 (the amount of rotation between the coordinate system in the virtual three-dimensional space and the device coordinate system) by accumulating the rotation matrix R. That is, since the z-axis of the device coordinate system of the imaging unit 22 is the imaging direction, the server 3 captures each imaging information using the rotation matrix R calculated for the combination of the two consecutive imaging information. The z-axis direction of the coordinate system (for example, a unit vector in the z-axis direction) is converted into a coordinate system in a virtual three-dimensional space. Next, the server 3 calculates the direction of the z-axis after conversion starting from the coordinate value in the virtual three-dimensional space of the shooting position as the shooting direction. Next, the server 3 stores the information indicating the shooting position and the information indicating the shooting direction in the virtual three-dimensional space calculated for each of the plurality of shooting information in the server storage unit 31 as the shooting position information and the shooting direction information, respectively. .. The server 3 calculates the height direction of the equipment included in the shooting information from the shooting information, and shoots the shooting position and shooting so that the height direction is the calculated Z-axis direction of the coordinate system of the virtual three-dimensional space. You may rotate the direction.

Then, the server 3 uses a known technique of multi-viewpoint image measurement to measure a three-dimensional point cloud of the target facility from a shooting position and a shooting direction corresponding to a plurality of shooting information. The server 3 may perform three-dimensional point group measurement of the target facility by using the translation vector t and the rotation matrix R corresponding to a plurality of shooting information instead of the shooting position and the shooting direction. The server 3 automatically matches each pixel of a plurality of shooting information (images) from the shooting position and shooting direction of each shooting information, and calculates the three-dimensional coordinates in the virtual three-dimensional space corresponding to each pixel. The three-dimensional coordinates calculated for the pixels of the plurality of shooting information are the three-dimensional coordinates in the virtual three-dimensional space that defines the outer surface of the facility. A plurality of three-dimensional coordinates that define the outer surface of the facility are three-dimensional point cloud data indicating the outer surface of the facility, and the server 3 stores the three-dimensional point cloud data as equipment position information in the server storage unit 31. Remember. Further, the server 3 recognizes the surface of the three-dimensional shape from the equipment position information which is the three-dimensional coordinates in the virtual three-dimensional space that defines the outer surface of the facility, thereby showing the three-dimensional shape of the equipment in the facility. A virtual three-dimensional space model, which is a model, is generated, and the generated virtual three-dimensional space model is stored in the server storage unit 31. With the above, the three-dimensional information calculation process is completed.

In the example shown in FIG. 6A, the server 3 creates a virtual three-dimensional space model showing the three-dimensional shape of the equipment in the facility in the virtual three-dimensional space based on the equipment position information. Next, the server 3 generates a three-dimensional model image of the target equipment from an arbitrary viewpoint position based on the virtual three-dimensional space model, and displays it on the server display unit 32. When displaying the three-dimensional model image, the server 3 may paste an image of shooting information corresponding to the position on the surface of the surface model of the equipment which is a virtual three-dimensional space model and display the image.

When the three-dimensional information calculation process is completed, the gaze point calculation process is executed. The gaze point calculation process will be described below.

First, the server 3 calculates a line segment extended in the line-of-sight direction indicated by the line-of-sight information corresponding to the shooting information from the shooting position of each shooting information stored in the server storage unit 31. The line-of-sight information corresponding to the shooting information is the line-of-sight information acquired at the timing closest to the timing (acquisition timing of the shooting information) at which the shooting was performed at the shooting position indicated by the shooting information. By calculating the line segment extended in the line-of-sight direction in this way, for example, when the line-of-sight information is information expressed by a unit vector, the surface and line-of-sight of the outer shape of each facility shown by the virtual three-dimensional space model. It is possible to prevent the vector indicated by the information from not intersecting.

Then, the server 3 calculates the first intersection of the outer surface of each facility and the line segment as a gazing point based on the virtual three-dimensional space model. The server 3 stores the calculated information indicating the virtual three-dimensional coordinates of the gazing point as the gazing point position information.

The server 3 generates a vector image from the shooting position indicated by the shooting information to the gazing point indicated by the line-of-sight information corresponding to each shooting information, and displays it on the server display unit 32. As shown in FIG. 6B, the shooting position and the gazing point are displayed together with the 3D model image based on the virtual 3D space model, so that the user is gazing at which position of each facility and from which location. You can figure out what it is.

Further, the server 3 extracts the position where the gazing point is concentrated on the surface of the target equipment based on the gazing point position information, and displays a three-dimensional model image highlighting the position on the server display unit 32. You may. For example, the server 3 calculates the distance between the gazing points, calculates the number of other gazing points whose calculated distance is within a predetermined distance for each gazing point, and sets the calculated number of other gazing points to the calculated number of gazing points. Each gaze point may be displayed on the server display unit 32 based on the corresponding display mode. Alternatively, the server 3 divides the outer surface of the virtual three-dimensional space model into grid-like regions at predetermined intervals, totals the number of gaze points included in each of the divided regions, and responds to the number of gaze points. The area may be displayed in gradation expression.

For example, in FIG. 6B, a star mark is displayed large in a portion where the gazing point is concentrated, and a small mark is displayed in other locations according to the degree of concentration of the gazing point. This is because the number of gaze points generated at predetermined time intervals is totaled for any place on the facility, and the gaze time of the place is calculated. That is, by multiplying the number of gaze points by a predetermined time interval, the gaze time of the portion corresponding to the gaze point is calculated. Therefore, it can be understood that the place where the gaze point is concentrated is an important inspection point because the place is gaze at the place for a long time or frequently. In this way, by displaying the display mode of the position of gaze more clearly than other gaze points, it is possible to easily present an important part.

The server 3 may generate inspection route information when the user inspects the target equipment by using the plane coordinates based on the shooting position information. The inspection route information is generated as line information in which the plane coordinates of the shooting position information are continuous. The shooting position information used when the inspection route information is generated is all or a part of the shooting position information calculated by the three-dimensional information calculation process, and the plane coordinates are the height included in the shooting position information. Two-dimensional coordinates (coordinate values in the X-axis direction and Y-axis direction) other than the coordinates of the information (coordinate values in the Z-axis direction). For example, the calculation unit X342 extracts all or a part of the shooting position information calculated by the three-dimensional information calculation process, and specifies the plane coordinates included in each of the extracted shooting position information. Then, the calculation unit 342 generates inspection route information indicating a polygonal line generated by connecting two continuous plane coordinates with a line segment according to the time of each shooting position information. By displaying the generated inspection route information on the three-dimensional model image of the server display unit 32, the server 3 can clearly indicate the route information of the inspector. Alternatively, the wearable device 2 displays the route information of the inspector on the display unit (not shown), so that the user can use the wearable device 2 of the present embodiment as a guide for the route information at the time of equipment inspection. Is possible.

Further, the server 3 calculates the residence time at an arbitrary point on the inspection route by totaling the number of shooting positions generated at predetermined time intervals. Since the places where the shooting positions are concentrated in some points are the places where the user needs time for inspection, the server 3 extracts the points as priority inspection points and highlights the points. The three-dimensional model image may be displayed on the server display unit 32. For example, the server 3 calculates the distance between a plurality of shooting positions, calculates the number of other shooting positions whose distance is within a predetermined distance for each shooting position, and sets the calculated number of other shooting positions. Each shooting position may be displayed on the server display unit 32 based on the corresponding display mode. Further, the wearable device 2 may display each shooting position on a display unit (not shown) based on a display mode according to the number of other shooting positions at each shooting position acquired from the server 3.

Further, when the inspection work is performed on the same floor, the place where the height information (coordinate value in the Z-axis direction) of the shooting position information is different from the height information of other shooting position information is when the user inspects. It can be seen that this is a place where inspections are carried out by changing the posture such as bending over or using a ladder or stepping stone. Therefore, the server 3 extracts the corresponding place (a place different from the height information of other shooting position information) from the shooting position information, and displays each shooting position on the server in a display mode different from the other shooting positions. It may be displayed in unit 32. Further, the wearable device 2 displays each shooting position on a display unit (not shown) in a display mode different from that of the other shooting positions in a place different from the height information of the other shooting position information acquired from the server 3. May be good.

Further, the server 3 extracts the position on the target facility where the gaze point is concentrated from the corresponding line-of-sight information as the priority inspection point for each extracted priority inspection point, and the user operates the priority inspection point and the priority inspection point. It may be displayed on the server display unit 32 in a display mode associated with the inspection location. Further, the wearable device 2 may be displayed on a display unit (not shown) in a display mode in which the priority inspection points and the priority inspection points are associated with each other based on the priority inspection points and the priority inspection points acquired from the server 3. ..

Traditionally, in facilities such as large-scale factories and plants, workers regularly inspect equipment, but the objects that workers should inspect are wide-ranging and the patrol inspection work is complicated. Nevertheless, there are not always many cases where detailed inspection manuals are prepared. For this reason, it is desired to visualize the inspection method of the equipment including the patrol route, the inspection point, and the inspection point when the skilled worker performs the patrol inspection, and to prepare the inspection manual. On the other hand, as described above, by using the gaze behavior survey system 1 in facilities such as large-scale factories and plants, it becomes possible to clearly indicate the user's inspection route, priority inspection point, and priority inspection point. The gaze behavior survey system 1 contributes to visualization of inspection methods for large-scale factories and / or facilities such as plants.

FIG. 7A shows an equipment layout table. In the equipment arrangement table, the equipment ID of the equipment, the three-dimensional design model information, the equipment name, the equipment drawing, the coordinate system, and the like are stored in association with each other for each equipment. The three-dimensional design model information is design model information indicating the arrangement position and the outer shape of a plurality of facilities arranged in a predetermined facility. In addition to the equipment, the equipment arrangement table may store information related to instruments and points for measuring temperature, humidity, a specific gas concentration, and the like. In this case, the position information of the instruments or the measurement points is stored as the three-dimensional design model information. The target for creating the equipment layout table may be the priority inspection points.

FIG. 7B shows an inspection result table. In the inspection result table, shooting position information, line-of-sight information, gaze point position information, and the like are stored in association with each other for each time.

(Operation sequence of gaze behavior survey system 1)
FIG. 8 is a diagram showing an example of an operation sequence of the gaze behavior investigation system 1. This operation sequence is based on a program stored in the server storage unit 31 and the storage unit 21 in advance, and is mainly performed by the server processing unit 34 and the processing unit 25 in cooperation with each element of the server 3 and the wearable device 2. Will be executed.

First, the photographing unit 22 of the wearable device 2 mounted on the user's head photographs the inside of the facility in a specific direction at predetermined time intervals. Further, the line-of-sight detection unit 23 of the wearable device 2 detects the line-of-sight direction of the user at predetermined time intervals. The acquisition unit 251 acquires the shooting information from the shooting unit 22, and also acquires the line-of-sight information from the line-of-sight detection unit 23, and the storage processing unit 252 attaches the acquired shooting information and the line-of-sight information to the mounting unit 24. Store in the storage medium M (step S101).

Next, when the server acquisition unit 341 of the server 3 acquires the shooting information and the line-of-sight information from the wearable device 2 via the storage medium M, the server acquisition unit 341 passes the shooting information and the line-of-sight information corresponding to the acquisition time to the calculation unit 342 and 3 The calculation unit 342 is instructed to execute the dimension information calculation process. When the server acquisition unit 341 instructs the calculation unit 342 to execute the calculation process, the calculation unit 342 executes the three-dimensional information calculation process (step S102). The details of the three-dimensional information calculation process will be described later.

Next, when the three-dimensional information calculation process is completed, the calculation unit 342 executes the gazing point calculation process (step S103). The details of the gazing point calculation process will be described later.

Next, when the gazing point calculation process is completed, the calculation unit 342 executes the inspection record calculation process (step S104). Details of the inspection record calculation process will be described later.

The server display processing unit 343 of the server 3 may display the three-dimensional model image and the gazing point on the server display unit 32 in response to an instruction from the system administrator or the like. The server display processing unit 343 displays at least one of the inspection route information, the priority inspection point, and the priority inspection point calculated by the inspection record calculation process in place of the 3D model image or together with the 3D model image. You may. Next, the server storage processing unit 344 includes the calculated inspection route information, priority inspection points, and priority inspection points together with the acquired shooting information and line-of-sight information, or various types included in the inspection result table and / or the equipment arrangement table. The information is stored in the storage medium M mounted on the server mounting unit 33 (step S105).

Next, when the acquisition unit 251 of the wearable device 2 acquires various information from the server 3 via the storage medium M, the storage processing unit 252 stores the acquired various information in the storage unit 21 (step S106).

Hereinafter, a case where the wearable device 2 includes a VR display of a glasses-type wearable terminal will be described as an example. The processing unit 25 of the wearable device 2 uses a known image matching technique to match the shape of the equipment included in the photographing information with the shape of the equipment included in the equipment position information. Next, the processing unit 25 calculates the position on the VR display of each facility when viewed from the position of the user's eyeball based on the matching result. Then, the processing unit 25 displays an image showing the gazing point at the intersection of the user's line of sight and the display when the user sees the position of the gazing point. Alternatively, an image showing at least one of the calculated inspection route information, the priority inspection point, and the priority inspection point may be displayed.

In addition, the processing unit 25 displays the 3D model image on the VR display of the eyeglass-type wearable terminal, and the gaze point indicated by the gaze point position information and / or the calculated inspection on the shape of the equipment of the 3D model image. An image showing route information, a priority inspection point, or at least one of the priority inspection points may be displayed.

(3D information calculation processing)
FIG. 9 is a diagram showing an example of an operation flow of the three-dimensional information calculation process. The three-dimensional information calculation process shown in FIG. 9 is executed in step S102 of FIG.

First, the calculation unit 342 acquires two consecutive captured images out of the captured information received from the server acquisition unit 341 (step S201).

Next, the calculation unit 342 extracts the feature points of the equipment commonly included in the two shooting information based on the known extraction method (step S202).

Next, the calculation unit 342 calculates the translation vector t of the photographing position and the rotation matrix R by using SfM processing or the like (step S203).

Then, the calculation unit 342 sets the shooting position and shooting direction corresponding to the earlier shooting information and the shooting position and shooting direction corresponding to the later shooting information in a virtual three-dimensional space based on the translation vector t and the rotation matrix R. It is converted into the three-dimensional coordinates in the above, and the shooting position information indicating the converted three-dimensional coordinates and the shooting direction are calculated. The same process is performed for all the shooting information, and the shooting position and the shooting direction corresponding to all the shooting information are calculated (step S204).

Using the shooting position and shooting direction corresponding to all shooting information, each pixel of multiple shooting information (images) is automatically matched from the shooting position and shooting direction of each shooting information, and virtual 3 corresponding to each pixel By calculating the equipment position information indicating the three-dimensional coordinates in the dimensional space, a virtual three-dimensional space model showing the three-dimensional shape of the equipment in the facility is generated (step S205), and a series of steps are completed. The calculation unit 342 stores the calculated various information in the inspection result table or the like in association with the acquisition time. Further, the calculation unit 342 stores the equipment position information and / or the virtual three-dimensional space model in the server storage unit 31. The calculation unit 342 is an example of a storage processing unit.

(Gaze point calculation process)
FIG. 10 is a diagram showing an example of an operation flow of the gaze point calculation process. The gaze point calculation process shown in FIG. 10 is executed in step S103 of FIG.

First, the calculation unit 342 calculates a line segment extended in the line-of-sight direction indicated by the line-of-sight information acquired at the same timing from the three-dimensional coordinates of the shooting position corresponding to each shooting information in the virtual three-dimensional space (step). S301).

Next, the calculation unit 342 calculates the first intersection of the outer surface of each equipment and the line segment as a gazing point based on the virtual three-dimensional space model of the equipment (step S302), and ends a series of steps. To do.

(Inspection record calculation process)
FIG. 11 is a diagram showing an example of an operation flow of the inspection record calculation process for calculating the inspection record including the inspection route information, the priority inspection point, the priority inspection point, and the like. The inspection record calculation process shown in FIG. 11 is executed in step S104 of FIG.

When the shooting position and the gazing point position of all the shooting information are calculated, the calculation unit 342 extracts the position where the gazing point is concentrated on the surface of the target equipment. Gaze point per unit area (for example, a plurality of grid-like areas obtained by dividing the outer surface of the virtual 3D space model at predetermined intervals) with respect to the outer surface of the virtual 3D space model indicating the equipment. Aggregate the number of. The calculation unit 342 sets a unit area in which the total number of gazing points exceeds a predetermined number as a priority inspection point (step S401).

Next, the calculation unit 342 generates inspection route information using the plane coordinates based on the shooting position information (step S402). The inspection route information is generated as line information in which the plane coordinates of the shooting position information are continuous.

Next, the calculation unit 342 extracts the locations on the inspection route where the shooting positions are concentrated as the priority inspection points. For example, the calculation unit 342 calculates the distance between a plurality of shooting positions, calculates the number of other shooting positions whose distance is within a predetermined distance for each shooting position, and performs a priority inspection of those exceeding the predetermined number. Let it be a point. In addition, the calculation unit 342 also extracts a location where the height information (coordinate values in the Z-axis direction) of the shooting position information is different from the height information of other shooting position information (step S403).

Next, the calculation unit 342 creates an equipment arrangement table. The equipment layout table is created by storing the equipment ID, 3D design model information, equipment name, equipment drawing, coordinate system, etc. of the equipment in association with each other for each equipment, but from the calculated priority inspection points. , The priority inspection points corresponding to the shooting location may be created (step S404).

The server display processing unit 343 of the server 3 sets at least one of the inspection route information, the priority inspection points, and the priority inspection points created as in steps S401 to S404 according to the instruction of the system administrator or the like. The three-dimensional model image and the gazing point are superimposed and displayed on the server display unit 32 (step S405), and a series of steps is completed. The server 3 wears at least one of the inspection route information, the priority inspection point, and the priority inspection point, and the three-dimensional model image and the gazing point so that it can be displayed on the display unit (not shown) of the wearable device 2. You may send to 2.

As described above, the gaze behavior investigation system 1 can stably acquire the shooting position of the wearable device 2 and the gaze point of the user at any position by using a plurality of shooting information and a plurality of line-of-sight information. At the same time, it becomes possible to generate a virtual three-dimensional space model around the user. Further, since the gaze behavior survey system 1 does not need to use a sensor other than an optical sensor such as a camera, it is possible to inexpensively execute the calculation process of the shooting position of the wearable device 2 and the position of the user's gaze point. .. In addition, the gaze behavior survey system 1 displays a three-dimensional model image showing the three-dimensional shape of the user's surrounding environment based on a plurality of shooting information and a plurality of gaze information, and also displays the user's gaze point of view and inspection route information. Priority inspection points and priority inspection points can be displayed on the 3D model image.

(Modification example 1)
In the above description, the case where the gaze behavior survey system 1 is used when a user such as an inspection worker inspects a plurality of inspection targets in a predetermined facility has been described as an example, but the place where the gaze behavior survey system 1 is applied is , Not limited to facilities inspected by inspection workers.

For example, the wearable device 2 of the gaze behavior survey system 1 may be used on a daily basis by a general user. In this case, the server 3 can identify an object (for example, a product watched by the user in a commercial facility, an advertisement watched by the user in the street) with respect to the gazing point in the surrounding environment when the user moves. Become.

(Modification 2)
Further, as a virtual three-dimensional space model of the equipment, a three-dimensional design model created in advance by CAD (Computer Aided Design) or the like may be used. In this case, the server 3 applies the position coordinates of the equipment in the three-dimensional design model to the position coordinates in the virtual three-dimensional space, and then stores the position coordinates in the server storage unit 31. Further, the gazing point may be an intersection of the three-dimensional design model information and a line segment extending in the line-of-sight direction based on the line-of-sight information from the shooting position.

For example, the calculation unit 342 uses a known three-dimensional model collation process to obtain the outer shape of each equipment defined by the equipment position information and the outer shape of a plurality of equipment indicated by the three-dimensional design model information. The coordinate axes of the virtual three-dimensional space model in the equipment position information or the coordinate axes of the three-dimensional design model information are moved by a predetermined distance and rotated at a predetermined rotation angle so as to substantially match.

Next, in the case of movement and rotation of the coordinate axes in the virtual three-dimensional space model, the calculation unit 342 also moves the shooting position indicated by the shooting position information and the line-of-sight direction indicated by the line-of-sight information by a predetermined distance. Rotate at a predetermined rotation angle. Then, the calculation unit 342 determines the intersection of the outer surface of the plurality of facilities indicated by the three-dimensional design model information and the line segment extending in the line-of-sight direction after the movement and rotation from the shooting position after the movement and rotation. It is calculated as a gaze point, and the position information indicating the calculated gaze point is stored in the server storage unit 31 as the gaze point position information. When the coordinate axes of the 3D design model information are moved and rotated, this process is not performed.

(Modification 3)
Further, the gaze behavior survey system 10 may have a wearable device 20 and a server 30 that are connected to each other via a communication network. For example, as shown in FIG. 12, the wearable device 20 and the server 30 included in the gaze behavior survey system 10 are connected to each other via the access point 4, the router 5, and the Internet 6. The access point 4 is a wireless LAN (Local Area Network) access point constructed in a predetermined facility. The server 30 may be installed inside the facility or outside the facility. The gaze behavior survey system 1 may have an external computer (not shown) for controlling the server 30.

In this case, the program executed by the server 30 (for example, a control program) and the program executed by the wearable device 20 (for example, a communication program) are communication protocols such as Hypertext Transfer Protocol (HTTP). Communicate using.

The wearable device 20 has the same configuration as the wearable device 2 shown in FIG. 3, and further includes a communication unit. The server 30 has the same configuration as the server 3 shown in FIG. 4, and further includes a server communication unit.

The communication unit of the wearable device 20 has a communication interface circuit including an antenna whose sensitive band is mainly a 2.4 GHz band, a 5 GHz band, or the like. The communication unit performs wireless communication with the access point 4 of the wireless LAN based on the wireless communication method of the IEEE (The Institute of Electrical and Electronics Engineers, Inc.) 802.11 standard. The frequency band of the communication unit is not limited to the frequency band described above. Then, the communication unit supplies the data received from the access point 4 to the processing unit 25, and transmits the data supplied from the processing unit 25 to the access point 4.

The communication unit may have a communication interface circuit including an antenna whose sensitive band is mainly 2.1 GHz, and may connect the wearable device 20 to a communication network (not shown). In this case, the communication unit establishes a wireless signal line with the base station by a CDMA (Code Division Multiple Access) method or the like via a channel assigned by the base station (not shown), and communicates with the base station. Communicate with. The communication method with the base station is not limited to the CDMA method, and other communication methods such as W-CDMA (Wideband Code Division Multiple Access) method and LTE (Long Term Evolution) method may be used in the future. Communication method may be used. Further, the frequency band of the communication unit is not limited to the frequency band described above. The communication unit supplies the data received from the base station to the processing unit 25, and transmits the data supplied from the processing unit 25 to the base station.

The server communication unit of the server 30 has a communication interface circuit for connecting the server 30 to the Internet 6. The server communication unit supplies the data received from the wearable device 20 and the like to the server processing unit 34. Further, the server communication unit transmits the data supplied from the server processing unit 34 to the wearable device 2 and the like.

(Modification example 4)
Further, the wearable device 2 or the wearable device 20 may include a display unit. The display unit is an HMD-type display device capable of outputting moving images, still images, and the like. The display unit may be any device as long as it can output a moving image, a still image, or the like, and may be, for example, a touch panel type display device, a liquid crystal display, an organic EL (Electro-Luminescence) display, or the like. The display unit displays a moving image according to the moving image data supplied from the processing unit 25, a still image corresponding to the still image data, and the like.

It will be appreciated by those skilled in the art that various changes, substitutions, and modifications can be made to this without departing from the spirit and scope of the invention.

1,10 Gaze behavior investigation system 2,20 Wearable device 21 Storage unit 22 Imaging unit 23 Line-of-sight detection unit 24 Mounting unit 25 Processing unit 251 Acquisition unit 252 Storage processing unit 3,30 Server 31 Server storage unit 32 Server display unit 33 Server mounting Unit 34 Server processing unit 341 Server acquisition unit 342 Calculation unit 343 Server display processing unit 344 Server storage processing unit 4 Access point 5 Router 6 Internet

Claims (7)

A gaze behavior survey system including a wearable device worn on the user's head and a server capable of communicating with the wearable device.
The wearable device is
With the shooting department
An acquisition unit that acquires shooting information shot from a plurality of different shooting positions by the shooting unit, and acquires line-of-sight information indicating the line-of-sight direction of the user at the timing of acquiring the shooting information in association with the shooting information. Have,
The server
Display and
The first calculation unit that calculates the shooting position in the virtual three-dimensional space for each shooting information,
Based on the line-of-sight information, a second calculation unit that calculates the gaze position and gaze time that the user is gazing at in the virtual three-dimensional space
A storage unit that stores a virtual three-dimensional space model showing the three-dimensional shape of the user's surrounding environment in the virtual three-dimensional space.
The display unit displays the shooting position in the line-of-sight information that is the basis for calculating the gaze position and / or the gaze position that can be identified according to the gaze time, and the virtual three-dimensional space model. Has a server display processing unit,
A gaze behavior survey system characterized by this. The gaze behavior investigation system according to claim 1, wherein the second calculation unit calculates the gaze time for each imaging position on which the gaze position is calculated. The gaze behavior investigation system according to claim 1 or 2, wherein the first calculation unit calculates the shooting position as the same shooting position in a predetermined area in the virtual three-dimensional space. The gaze behavior investigation system according to any one of claims 1 to 3, wherein the second calculation unit calculates the gaze position as the same gaze position in a predetermined area in the virtual three-dimensional space. The gaze behavior investigation system according to any one of claims 1 to 4, further comprising a virtual three-dimensional space model generation unit that generates the virtual three-dimensional space model from the shooting information. The second calculation unit
For each shooting position, a line segment extended in the corresponding line-of-sight direction is calculated from each shooting position.
The gaze behavior survey according to any one of claims 1 to 5, wherein the intersection of the calculated line segment and the outer surface of the virtual three-dimensional space model is calculated as a gaze point for each shooting position. system. A server control program that can communicate with a wearable device worn on the user's head.
The wearable device receives the shooting information shot from a plurality of different shooting positions by the shooting unit included in the wearable device and the line-of-sight information indicating the line-of-sight direction of the user at the timing when the shooting information is acquired.
The shooting position in the virtual three-dimensional space is calculated for each of the shooting information.
Based on the line-of-sight information, the gaze point position and gaze time that the user is gazing at in the virtual three-dimensional space are calculated.
The shooting position in the line-of-sight information that is the basis for calculating the gaze position and / or the gaze position that can be identified according to the gaze time, and the three-dimensional shape of the user's surrounding environment in the virtual three-dimensional space. Display the virtual 3D space model shown,
A control program characterized by causing the server to execute the above.