JP2020177066A - Observation state display system, observation state display method and program - Google Patents

Abstract

To provide an observation state display system capable of intuitively grasping a movement of a head of an observer for changing an observation direction as a behavior of the observer in an observation space and also capable of easing a visually induced motion sickness for the observer caused by a change in a visual field in a visual field pick-up image or by a camera shake.SOLUTION: An observation state display system displays a state of an observer. The observation state display system includes: a visual field image acquisition unit configured to acquire an image of a visual field in an observation direction as a visual field pick-up image; a head posture information acquisition unit configured to detect a head posture being a posture of a head of the observer; and a display frame display control unit configured to move, within a display screen, a display frame in which the visual field pick-up image is to be displayed, correspondingly to the head posture.SELECTED DRAWING: Figure 1

Description

The present invention relates to an observation state display system, an observation state display method, and a program for displaying the observation state of an observer.

An observer moves while observing the above object in a three-dimensional space (hereinafter referred to as an observation space) such as a display space where the objects to be observed are displayed in a store or an exhibition hall where various observation objects exist. As a device for measuring the line-of-sight direction of the observer, which can also be used in the case of The wearable eye-gaze measuring device has been put into practical use (see, for example, Patent Document 1 and Patent Document 2).
Generally, in the case of a head-mounted line-of-sight measuring device (hereinafter referred to as a head-mounted measuring device) fixed to the observer's head, the line-of-sight data obtained from the above-mentioned line-of-sight measurement function is stored on the observer's head. On the other hand, it shows the line-of-sight direction in the fixed head coordinate system.

The field-of-view camera is provided at a position in the center of both eyes in the head-mounted measuring device. When measuring the line-of-sight direction in the field of view in the observation direction of the observer, the field-of-view camera captures the field-of-view image as the image of the field of view in the observation direction of the observer.
The line-of-sight direction is uniquely the position of the viewpoint detected by the line-of-sight measuring device, based on geometric parameters such as the mounting position and the angle of view of the field-of-view camera mounted on the head of the field-of-view camera, and the pixel position on the field-of-view image. Can be associated.

As a result, the gazing point marker is superimposed and displayed as the position of the gazing point of the observer (hereinafter referred to as the gazing point) on the field-of-view captured image without using the information of the position and the posture of the observer. The position of the observer's gaze point can be visually shown.

Japanese Unexamined Patent Publication No. 06-189906 Japanese Unexamined Patent Publication No. 2012-146199

As described above, by using the wearable line-of-sight measuring device, when the observer observes the object while moving in the observation space, the observation operation in the observer's observation space is not hindered.
Therefore, by using the wearable line-of-sight measuring device of Patent Document 1 and Patent Document 2, the experimenter can visually and in real time view the gazing point in the observation space of the observer by the observer's free movement in the observation space. Can be asked for.

However, when changing the observation direction in the observation space, the observer moves the observation position or changes the line of sight in the field of view in the observation direction because of the need to face the object of interest, that is, the observation direction in which the gazing point exists. The posture of the head is changed to change it.
At this time, the relationship between the coordinate system fixed to the observation space (hereinafter referred to as the global coordinate system) and the head coordinate system of the visual field camera that captures the visual field observed by the observer also changes.

On the other hand, when analyzing the psychological state of the observer with high accuracy, the relationship between the movement of the head for changing the observation direction (change in the posture of the head) and the line of sight in the visual field in the observation direction is investigated. Need to be analyzed.
However, in Patent Document 1 and Patent Document 2, the gazing point measured in the head coordinate system is converted into a global coordinate system and plotted as a gazing point marker, and the observation direction, that is, the observer's head No consideration is given to analyzing motion.
Therefore, in Patent Document 1 and Patent Document 2, since the gaze point corresponding to the line of sight of the observer is analyzed, the line of sight to the object can be obtained, while the posture of the head in which the observation direction is changed. Information cannot be shown visually.

When observing the psychological state of the observer, it is necessary to analyze the posture information of the head when the observation method is changed, and it is necessary to analyze the posture information of the head directly with respect to the field-of-view image taken at the time of imaging in the head coordinate system. It is conceivable to plot the gaze marker on.
As a result, each time the observation direction is changed, the field-of-view image changes according to the movement of the head, and when the line-of-sight direction changes, the position of the gazing point marker in the field-view image changes, so that the observation direction and note Information with the position of the viewpoint can be shown individually.

As described above, since the visual field moves in response to the change in the visual field captured image, it is possible to observe the movement of the head from the direction of movement of the visual field independently of the gazing point.
However, it is difficult to intuitively grasp the direction of movement of the observer's visual field, that is, which direction the head has changed, from the change of the image in the visual field captured image.
In addition, the visual field captured image has a large amount of camera shake due to the movement of the observer's head and the change in posture. Therefore, the experimenter may have a change in the visual field in the image captured in the visual field and camera shake, that is, image sickness due to the irregularly shaken image, which may make it difficult for the observer to analyze for a long time.

The present invention has been made in view of such a situation, and it is possible to intuitively grasp the movement of the observer's head for changing the observation direction as the behavior of the observer in the observation space, and the visual field. Provided are an observation state display system, an observation state display method, and a program capable of reducing an experimenter's image sickness due to a change in the visual field in a captured image and camera shake.

In order to solve the above-mentioned problems, the observation state display system of the present invention is an observation state display system that displays the state of the observer, and is a field of view that acquires an image of the field of view in the observation direction of the observer as a field of view image. The position of the image acquisition unit, the head posture information acquisition unit that detects the head posture that is the posture of the observer's head, and the display frame on which the field-of-view image is displayed correspond to the head posture. It is characterized by including a display frame display control unit for moving within the display screen.

In the observation state display system of the present invention, the display frame display control unit moves the display frame in accordance with the amount of change in the posture angle of the head posture in the movement angle of the display frame in the display screen. It is characterized by.

In the observation state display system of the present invention, the display frame display control unit performs projective conversion on the display frame and the field-of-view image captured in the display frame in accordance with each of the directions and sizes of the moving angles. It is characterized by.

In the observation state display system of the present invention, a line-of-sight measuring unit that measures the direction of the line of sight of the observer and a gaze point marker indicating the position of the gaze point corresponding to the direction of the line of sight are superimposed on the field-of-view image. It is characterized by further including a gazing point marker display unit for displaying.

In the observation state display system of the present invention, the head posture information acquisition unit is attached to the observer's head by a predetermined device, and is an acceleration sensor or an acceleration sensor that measures rotational movement in the observer's head posture. It is characterized in that either or both of the angular velocity sensors are provided.

The observation state display system of the present invention further includes an image pickup device provided in the observation space where the observer observes, and the head posture information acquisition unit uses the moving image of the observer captured by the image pickup device. , The feature is to detect the head posture of the observer.

The observation state display system of the present invention is attached to the observer's head by a predetermined device at a position where the observer's visual field can be imaged in the vicinity of the position of the eye on the head, and the visual field image is captured. It is characterized in that the field-of-view image acquisition device further includes a field-of-view image pickup device for capturing the image, and the field-of-view image acquisition unit acquires the field-of-view image image from the field-of-view image pickup device.

The observation state display method of the present invention is an observation state display method for displaying the state of the observer, and includes a visual field image acquisition process in which the visual field image acquisition unit acquires an image of the visual field in the observation direction as a visual field image acquisition image. The head posture detection process in which the part posture information acquisition unit detects the head posture, which is the posture of the observer's head, and the display frame display control unit determine the position of the display frame on which the visual field captured image is displayed. It is characterized by including a display frame display control process of moving in the display screen in correspondence with the head posture.

The program of the present invention is a program that causes a computer to execute an operation of an observation state display system that displays the state of an observer, and is a field image acquisition means for acquiring the image of the field of view in the observation direction as a field image image. , The head posture detecting means for detecting the head posture, which is the posture of the observer's head, and the position of the display frame on which the visual field captured image is displayed are moved in the display screen in correspondence with the head posture. It is a program for functioning as a display frame display control means.

As described above, according to the present invention, as the behavior of the observer in the observation space, the movement of the observer's head for changing the observation direction can be intuitively grasped, and the visual field in the visual field captured image. It is possible to provide an observation state display system, an observation state display method, and a program capable of reducing an observer's image sickness due to changes in the image and camera shake.

It is a block diagram which shows the structural example of the observation state display system by this Embodiment. It is a conceptual diagram which shows the structural example of the observer information acquisition apparatus 20 in the observation state display system 1 by this Embodiment. It is a conceptual diagram which shows an example of the observation direction and the line of sight of an observer in an observation space 310. FIG. 5 is a conceptual diagram illustrating control of a display position of a display frame on which a field-of-view captured image is displayed on the display screen of the image display unit 16. It is a conceptual diagram which shows an example of the observation direction and the line of sight of an observer in an observation space 310_1. FIG. 5 is a conceptual diagram illustrating control of a display position of a display frame on which a field-of-view captured image is displayed on the display screen of the image display unit 16. It is a flowchart explaining the operation example of the line-of-sight measurement processing of the observation state display system 1 by this Embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of an observation state display system according to the present embodiment. In FIG. 1, the observation state display system 1 is a system for displaying the state of the observer, and includes each of the observation state display device 10 and the observer information acquisition device 20. Each of the observation state display device 10 and the observer information acquisition device 20 is connected by a predetermined network (not shown).
The observer information acquisition device 20 provides the observation state display device 10 with information regarding the observation field of the observer, including each of the line-of-sight measurement information, the image information, and the head posture information. Supply.

FIG. 2 is a conceptual diagram showing a configuration example of the observer information acquisition device 20 in the observation state display system 1 according to the present embodiment. The observer information acquisition device 20 includes a head mounting jig 210, a line-of-sight measurement unit 220, a visual field imaging device 230, and a posture information acquisition device 240, respectively.
The head mounting jig 210 is a jig that is mounted on the head by the observer 200, for example, in the shape of a helmet or glasses, and is fixed so as not to deviate from the mounting position with respect to the movement of the head. It is a fixture of the configuration to be used. Here, when the HMD (Head Mounted Display) is used as the head mounting jig 210, the image of the virtual space displayed on the HMD is used as the field of view image image instead of the field of view image captured by the field image imager 230. Can be done. Then, each of the line-of-sight measurement unit 220, the visual field imaging device 230, and the posture information acquisition device 240 is attached to the HMD (for example, built-in) to display the visual field captured image.

The line-of-sight measurement unit 220 measures the line of sight of the observer 200, and the line-of-sight measurement information is information indicating the position of the gaze point, which is the point of view of the observer 200, within the visual field in the observation direction corresponding to the line of sight. Ask for.
Then, the line-of-sight measurement unit 220 supplies the obtained line-of-sight measurement information to the observation state display device 10. In the present embodiment, the line-of-sight measurement information is information indicating the coordinate position of the gazing point in the two-dimensional plane of the field-of-view captured image described later.

For example, the lower left position coordinate of the field image captured on the rectangle is (0.0, 0.0), and the upper right position coordinate is (1.0, 1.0). In this coordinate range, the coordinates of the gazing point. The position is shown. Here, the relationship between the line-of-sight direction of the observer 200 and the coordinate position of the gazing point in the field-of-view image is taken into consideration in advance with respect to the relative positions of the line-of-sight measurement unit 220 and the field-of-view imager 230 on the head mounting jig 210. Then, it is calibrated in advance for each observer.

The visual field imaging device 230 captures a moving image (60 frames / sec) of the visual field in the observation direction of the observer 200, and supplies the captured image as the visual field captured image (image information) to the observation state display device 10.
In the present embodiment, the posture information acquisition device 240 is, for example, a posture sensor, and is composed of each of a gyro sensor and an acceleration sensor. Here, the gyro sensor acquires rotation information of the head of the observer 200 (angle change amount in each of the pan, tilt, and roll of the head). The accelerometer acquires the acceleration for calculating the rotation information of the head of the observer 200 from the direction of gravity. The posture information acquisition device 240 supplies each of the acquired movement information and rotation information to the observation state display device 10 as head posture information. The rotation information in the head posture information is, for example, time-series data of the rotation angle represented by a quaternion (corresponding to a time stamp described later).

Returning to FIG. 1, the observation state display device 10 includes a line-of-sight information acquisition unit 11, a visual field image acquisition unit 12, a gazing point marker superimposition unit 13, a head posture information acquisition unit 14, a display frame display control unit 15, and an image display unit 16. And each of the storage units 17.
The line-of-sight information acquisition unit 11 writes and stores the line-of-sight measurement information, which is supplied from the line-of-sight measurement unit 220 and indicates the coordinate position of the gazing point in the visual field captured image, in the storage unit 17 in correspondence with the time stamp. This time stamp is information indicating in chronological order the order in which the visual field captured images captured in frame units are captured.
The field-of-view image acquisition unit 12 writes and stores the field-of-view image captured in frame units supplied from the field-of-view image pickup device 230 in the storage unit 17 in correspondence with the time stamp.

The gazing point marker superimposing unit 13 reads out the visual field captured image from the storage unit 17, and superimposes the gazing point marker image on the coordinate position of the gazing point in the read visual field captured image. Then, the gazing point marker superimposing unit 13 writes and stores the visual field captured image on which the gazing point marker is superimposed in the storage unit 17 (for example, overwrites the image data of the read visual field captured image).

The head posture information acquisition unit 14 inputs the head posture information supplied from the posture information acquisition device 240 and outputs it to the display frame display control unit 15.
The display frame display control unit 15 detects a change in the observation direction from the head posture information of the observer 200, and controls the display position of the display frame on the display screen of the image display unit 16. This display frame is a frame of a sub screen on which a field-of-view image is displayed.

FIG. 3 is a conceptual diagram showing an example of the observation direction and line of sight of the observer in the observation space 310. FIG. 3 is a plan view seen from the upper part of the observation space 310. Each of the target object 340 and the target object 350 is arranged in the observation space 310.
The observation direction 330 is obtained based on the head posture information (first) supplied from the posture information acquisition device 240. The visual field imaging device 230 captures an image of the visual field of the observer 200 in the observation direction 330 in the visual field as a visual field imaging image. The line-of-sight measurement unit 220 measures the line-of-sight 360 of the observer 200, obtains the position coordinates of the gazing point in the field-of-view image, and outputs the line-of-sight measurement information.

FIG. 4 is a conceptual diagram illustrating control of a display position of a display frame on which a field-of-view captured image is displayed on the display screen of the image display unit 16.
FIG. 4A shows a field-of-view image 410 in which the gazing-point marker 420 is superimposed on the field-of-view image captured in the observation direction 330 in FIG. The gazing point marker 420 is, for example, a cross-shaped mark, which is superimposed on the image 450 of the target object 350. That is, the line of sight of the observer 200 is directed toward the target object 350.

In FIG. 4B, the field-of-view captured image 410 of FIG. 4A is displayed in the display frame 510 displayed at the display position based on the head posture information in the display screen 500 of the image display unit 16. Indicates the state of being.
In FIG. 4B, the display frame 510 is centered in the display screen 500 because the observer is facing the front in the observation space 310, for example, that is, the observation direction is the front to the observer. It is displayed in.
At this time, the display frame display control unit 15 makes the viewing angle of the field of view observed by the observer 200 and the viewing angle of the display frame 510 visually recognized by the experimenter observing the display screen 500 the same. Each viewing angle is displayed in correspondence with each other.

FIG. 5 is a conceptual diagram showing an example of the observation direction and line of sight of the observer in the observation space 310_1. FIG. 5 is a plan view of the observation space 310 as viewed from above, as in FIG. In the observation space 310_1, each of the target object 340 and the target object 350 is arranged as in FIG.
The observation direction 330_1 is obtained based on the head posture information (second) supplied from the posture information acquisition device 240. The visual field imaging device 230 captures an image of the visual field of the observer 200 in the observation direction 330_1 as a visual field imaging image. The line-of-sight measurement unit 220 measures the line-of-sight 360_1 of the observer 200, obtains the position coordinates of the gazing point in the field-of-view image, and outputs the line-of-sight measurement information.

FIG. 6 is a conceptual diagram illustrating control of a display position of a display frame on which a field-of-view captured image is displayed on the display screen of the image display unit 16.
FIG. 6A shows a field-of-view image 410_1 in which the gazing-point marker 420_1 is superimposed on the field-of-view image captured in the observation direction 330_1 in FIG. The gazing point marker 420_1 is, for example, a cross-shaped mark, which is superimposed on the image 450 of the target object 350. That is, the line of sight of the observer 200 is directed toward the target object 350, as in FIG. 4A.

In FIG. 6B, the field-of-view captured image 410_1 of FIG. 6A is displayed in the display frame 510_1 displayed at the display position based on the head posture information in the display screen 500 of the image display unit 16. Indicates the state of being.
In FIG. 6B, the observer 200 rotates the head in the pan direction (for example, to the left in FIG. 5) by a predetermined angle change amount with respect to the position (and posture) in the initial state of FIG. It shows the state of the head after the posture change.

At this time, the display frame display control unit 15 determines the viewing angle of the field of view observed by the observer 200 and the display frame visually recognized by the experimenter observing the display screen 500, as in the case of FIG. 4B. The respective viewing angles are displayed in association with each other so that the viewing angles of 510_1 are the same. Further, the display frame display control unit 15 corresponds to the movement (parallel movement, rotational movement) of the head of the observer 200 indicated by the head posture information, that is, the change in the observation direction indicated by the head posture information, in the display screen 500. Controls the display position of the display frame 510_1 in.

For example, in the case of FIG. 6B, when the observer 200 rotates his head to the left in the pan direction by a predetermined amount of angle change, the observation direction of the observer 200 changes to the left, so that the display frame is displayed. The control unit 15 moves the display position of the display frame 510_1 on the display screen 500 to the left in accordance with the amount of change in the angle. At this time, the display frame display control unit 15 displays that the amount of angle change due to the rotation of the observer's head is equal to the amount of angle change due to the rotation of the experimenter's head when visually recognizing the display frame 510_1 on the display screen. The display frame 510_1 on the display screen 500 is moved to the position. Further, when the observer 200 rotates the head to the right in the pan direction by a predetermined angle change amount, the observation direction of the observer 200 changes to the right, so that the display frame display control unit 15 changes the angle. The display position of the display frame 510_1 on the display screen 500 is moved to the right in accordance with the above.

Similarly, when the observer 200 rotates the head upward in the tilt direction by a predetermined angle change amount, the observation direction of the observer 200 changes upward, so that the display frame display control unit 15 changes the angle. The display position of the display frame 510_1 on the display screen 500 is moved upward according to the amount. Further, when the observer 200 rotates the head downward in the tilt direction by a predetermined angle change amount, the observation direction of the observer 200 changes downward, so that the display frame display control unit 15 changes the angle. Corresponding to, the display position of the display frame 510_1 on the display screen 500 is moved downward.

Similarly, when the observer 200 rotates the head clockwise in the roll direction by a predetermined angle change amount, the observation field of view of the observer 200 changes clockwise, so that the display frame display control unit 15 , The display angle of the display frame 510_1 on the display screen 500 is rotated clockwise according to the amount of change in the angle. Further, when the observer 200 rotates the head counterclockwise in the roll direction by a predetermined angle change amount, the observation field of view of the observer 200 changes counterclockwise, so that the display frame display control unit 15 can be used. The display angle of the display frame 510_1 on the display screen 500 is rotated counterclockwise according to the amount of change in the angle.
As a result, the amount of angle conversion in the rotation of the observer's head can be made to correspond to the movement angle visually recognized by the experimenter 200 on the display screen 500. Therefore, when the display position of the display frame 510 (FIG. 4 (b)) changes to the display position of the display frame 510_1 (FIG. 6 (b)), the visual field captured image in the display frame is stationary so as to be stationary. It can make the experimenter feel.

Further, the display frame display control unit 15 sets the coordinate position for displaying the display frame 510_1 (FIG. 6B) in accordance with the amount of angle change according to the coordinate conversion conditions set in advance. Coordinate conversion is performed from the coordinate position shown in FIG. 4 (b).
As a result, the visual angle of the object in the visual field captured image observed by the experimenter on the display screen 500 is visually recognized by the experimenter as the same visual angle as the visual angle of the object observed by the observer in the visual field in the observation space. The visual field captured image can be visually recognized by the experimenter as a visual field intuitively similar to the visual field observed by the observer.

Further, when moving the display position of the display frame 510_1 on the display screen 500, the display frame display control unit 15 is far from the center of the display frame 510_1 as shown in FIG. 6B in accordance with the amount of angle change. Increase the length of the side and shorten the length of the near side (in other words, increase the length of the near side on the side facing each side of the display screen 500 and increase the length of the far side. In order to shorten the length of the display frame 510_1, projection conversion such as keystone correction may be performed on the display frame 510_1.

By this projective transformation, the shape of the object in the display frame 510_1 can be kept constant for the experimenter observing the display screen 500, and the display frame 510_1 corresponding to the field of view of the observer 200 on the display screen 500 can be maintained. A sense of perspective can be given to the experimenter on the display screen 500.

That is, when it is desired that the apparent shape of the object in the display frame 510_1 moved on the display screen 500 be visually recognized as the same shape as when observed in front of the experimenter, the center of the display screen 500 on the display frame 510_1 is desired. It is necessary to perform a projective transformation that makes the side farther away from. Unless the above-mentioned projective transformation is performed, the objects in the visual field captured image in the display frame 510_1 cannot be visually recognized by the experimenter as having the same shape.
Therefore, as shown in FIG. 6B, the display frame display control unit 15 compares the side far from the center of the display screen 500 with the side closer to the opposite side in the display frame 510_1. Perform projective transformation to lengthen.

When the observer 200 does not change the movement and posture of the head, that is, the observation direction beyond a predetermined time, the display frame display control unit 15 displays the display frame 510 on the display screen 500 at a preset movement speed. The reset process (movement amount and movement amount) of gradually returning to the center of the display screen 500 and using the image captured in the observation space of the display frame 510 of the display screen 500 as the field of view in the observation direction at the initial value of the head posture information of the head of the observer 200. The process of setting the amount of change in angle to 0) is performed.
As a result, the movement range of the arrangement position of the display frame 510 on the restricted display screen 500 is substantially expanded, and the change in the observation direction of the observer 200 in the observation space can be changed in the display screen 500 in which the experimenter is restricted. The display frame 510 can be prevented from protruding from the display screen 500, and the observer's observation direction and visual field can be effectively observed.

Further, the display frame display control unit 15 does not make the amount of change in the angle of the head of the observer 200 and the moving angle of the arrangement position of the display frame 510 on the display screen 500 the same, and displays by the experimenter on the display screen 500. In consideration of the viewing angle of the frame 510 and the visibility of the display frame 510 of the experimenter observing the display screen 500, the configuration may be arbitrarily increased or decreased at a predetermined ratio.

FIG. 7 is a flowchart illustrating an operation example of the line-of-sight measurement process of the observation state display system 1 according to the present embodiment.
Step S101:
The field of view image acquisition unit 12 inputs the field of view image data supplied from the field of view image pickup device 230 at a predetermined cycle. Then, the field of view image acquisition unit 12 adds a time stamp to the input field of view image capture image data, writes it in the storage unit 17, and stores it.

Step S102:
The head posture information acquisition unit 14 obtains head posture information that measures the movement of the observer's head, which is supplied from the posture information acquisition device 240 at the timing when the visual field image acquisition unit 12 inputs the visual field captured image data. input. Then, the head posture information acquisition unit 14 makes the input head posture information correspond to the time stamp, writes it in the storage unit 17, and stores it.

Step S103:
The line-of-sight information acquisition unit 11 inputs the line-of-sight measurement information supplied from the line-of-sight measurement unit 220 at the timing when the field-of-view image acquisition unit 12 inputs the field-of-view image image data. Then, the line-of-sight information acquisition unit 11 associates the input line-of-sight measurement information with the time stamp, writes it in the storage unit 17, and stores it.

Step S104:
The head posture information acquisition unit 14 determines whether or not each of the movement amount and the angle change amount of the newly input head posture information has changed as compared with the head posture information previously input in the time stamp described above. Is determined.
At this time, when the input head posture information changes in the movement amount and the angle change amount as compared with the head posture information input immediately before, the head posture information acquisition unit 14 performs the process in step S104. Proceed to. On the other hand, when the input head posture information does not change in the movement amount and the angle change amount as compared with the head posture information input immediately before, the head posture information acquisition unit 14 steps the process. Proceed to S106.

Step S105:
The display frame display control unit 15 obtains a display position of the display frame 510 in the display screen 500 corresponding to each of the new movement amount and the angle change amount, and has a shape of the display frame 510 corresponding to the display position. Perform projective transformation.

Step S106:
The gazing point marker superimposing unit 13 reads out the visual field captured image data of the latest time stamp from the storage unit 17, and superimposes the gazing point marker indicating the gazing point of the observer on the coordinate position corresponding to the line-of-sight measurement data (for example, 4 (a) and 6 (a)).
Then, the gazing point marker superimposing unit 13 writes and stores the visual field captured image data on which the gazing point marker is superimposed in the storage unit 17 in correspondence with the time stamp.

Step S107:
The display frame display control unit 15 reads out the visual field captured image data on which the gazing point marker is superimposed from the storage unit 17, performs the same projection conversion as the display frame 510, and converts the shape into the shape included in the display frame 510.
Then, the display frame display control unit 15 displays the display frame 510 in which the field-of-view captured image data is fitted at the already determined display position on the display screen 500 (for example, FIGS. 4 (b) and 6 (b)). ). At this time, the display frame display control unit 15 writes and stores the display image data of the display screen 500 on which the display frame 510 is displayed in the storage unit 17 in correspondence with the latest time stamp.

Step S108:
The observation state display device 10 detects whether or not the experimenter has input to end the measurement process from an input means (not shown). At this time, the observation state display device 10 ends the process when the input for ending the measurement process is performed, and returns the process to step S101 when the input for ending the measurement process is not performed.

As described above, in the present embodiment, when the experimenter analyzes the observer's interest in the object in the observation space, the display frame 510 displays the observer in the display screen 500 in the behavior of the observer observing the object. Because it changes in response to the movement of the head (change in the observation direction), the experimenter observing the display screen 500 moves the observer's head and changes the posture (object) in order to change the observation direction. The state of the observer observing an object) can be grasped intuitively.

As a result, according to the present embodiment, the observer can observe the object of interest, as compared with the conventional case where only the line-of-sight direction of the observer can be displayed with respect to the object. When the observer turns his gaze on the object first and then turns his head toward the object (observation direction), or the observer turns his head toward the object (observation direction). It is possible to visually and intuitively acquire information for analyzing the detailed relationship between the observation direction and the line of sight, such as whether to direct the line of sight to the object after turning the) first.

Further, in the present embodiment, the experimenter observes the movement of the display frame 510 corresponding to the change in the observation direction of the observer, that is, visually observes a visual field image similar to the visual field observed by the observer. You can get the feeling that the experimenter is moving his or her head.
Therefore, according to the present embodiment, as in the conventional case, the display frame on which the visual field captured image is displayed is fixed regardless of the observation direction of the observer, and the image in the display frame changes significantly. It is possible to suppress image sickness and facilitate the observation of the movement of the observer's head and the change in the gazing point by the experimenter.

Further, in the above-described embodiment, the posture information acquisition device 240 has been described as a configuration in which the movement of the observer's head and the amount of change in angle are acquired by an acceleration sensor or a gyro sensor, but a moving image of the observer is captured. The movement of the observer's head may be analyzed from the captured image data of the above to obtain the amount of movement of the head and the amount of change in angle.
At this time, the head posture information is supplied to the observation state display device 10 as data of a moving image captured by the observer from the image pickup device provided for observing the observer in the observation space. Then, the head posture information acquisition unit 14 extracts an image of the observer's head from the moving image captured by the observer supplied from the imaging device by image processing, and moves the image of the head. By analyzing, the amount of movement of the observer's head and the amount of change in angle are detected.

Further, the line-of-sight measurement unit 220 detects a change in the line-of-sight of the observer from images obtained by a plurality of near-infrared illuminations and an imaging device arranged for observing the observer's eyes in the observation space, and makes a visual field. It may be configured to acquire the line-of-sight measurement information corresponding to the captured image. Furthermore, in the observation space, features around the eyes such as the inner and outer corners of the eyes and pupils, which are important for detecting the line of sight of the observer, are extracted from images obtained from a plurality of image pickup devices arranged for observing the observer's eyes. , The angle of the line of sight in the field of view in the observation direction may be calculated to obtain the line of sight.

Further, as another configuration of the head posture information acquisition unit 14, a depth camera or the like is used for the posture information acquisition device 240, and the three-dimensional shape of the observer's head is extracted from the depth map supplied from the depth camera. A configuration may be used in which the change in the three-dimensional shape of the head is analyzed to detect the amount of movement and the amount of change in the angle of the observer's head.
As described above, it may be used as a posture information acquisition device 240 for obtaining head posture information as long as it can detect changes in the observer's head. Similarly, if the configuration is such that a change in the direction of the line of sight of the observer in the observation space can be detected, an electro-oculography measuring device or the like may be used as the line-of-sight measuring unit 220 for obtaining line-of-sight measurement information.

Further, as another configuration of the observation state display system 1, the observer information acquisition device 20 may be replaced with a storage device inside or outside the observation state display device 10. In this case, the storage device stores the time-series information of the observer's line of sight, head posture, and visual field image in advance, and the line-of-sight information acquisition unit 11 stores the time-series information of the line of sight in the head posture information acquisition unit. Reads the time-series information of the head posture and the field image acquisition unit sequentially reads the time-series information of the field image from the storage device. As a result, the state of the observer is displayed on the image display unit 16 as in the case of using the observer information acquisition device 20.
Further, when the storage device is inside the observation state display device 10, the storage device may be the storage unit 17.
Further, when the storage device is outside the observation state display device 10, the storage device may be connected to the observation state display device 10 via a network such as the Internet.

Further, in the present embodiment, the observation space to be observed by the observer has been described as a real space, but in a virtual space such as CG (Computer Graphics) or VR (Virtual Reality), which is observed by wearing an HMD. , It can also be used as a configuration for analyzing the observation direction and gaze point of the observer in this virtual space. In this case, the image of the virtual space is displayed on the display screen 500, and the display frame 510 is displayed at a position including the visual field captured image in the observation direction of the observer.

Further, in the present embodiment, the display screen 500 observed by the experimenter to analyze each of the observer's observation direction and gazing point is set as a two-dimensional plane (FIGS. 4 (b), 6 (b), etc.). As described above, in order to expand the moving range of the display frame 510, a display screen having a curved surface shape such as a cylinder or a sphere is used as the display screen on which the display frame 510 is superposed and arranged. You may.

The program for realizing the function of the observation state display device 10 in the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by the computer system and executed for observation. The observation process of the observer's observation direction and the gazing point in the state display may be performed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices.
Further, the "computer system" shall also include a WWW system provided with a homepage providing environment (or display environment). Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Furthermore, a "computer-readable recording medium" is a volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, it shall include those that hold the program for a certain period of time.

Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, a so-called difference file (difference program) may be used, which can realize the above-mentioned functions in combination with a program already recorded in the computer system.

1 ... Observation state display system 10 ... Observation state display device 11 ... Line-of-sight information acquisition unit 12 ... Field image acquisition unit 13 ... Gaze point marker superimposition unit 14 ... Head posture information acquisition unit 15 ... Display frame display control unit 16 ... Image display Unit 17 ... Storage unit 20 ... Observer information acquisition device 210 ... Head mounting jig 220 ... Line-of-sight measurement unit 230 ... Field image imaging device 240 ... Attitude information acquisition device

Claims (9)

It is an observation status display system that displays the status of the observer.
A field image acquisition unit that acquires a field image of the field of view in the observation direction of the observer as a field image image.
A head posture information acquisition unit that acquires the head posture, which is the posture of the observer's head,
An observation state display system including a display frame display control unit that moves the position of the display frame on which the field-of-view image is displayed in the display screen according to the head posture. The display frame display control unit
The observation state display system according to claim 1, wherein the movement angle of the display frame in the display screen is moved according to the amount of change in the posture angle of the head posture. The display frame display control unit
The observation state display system according to claim 2, wherein a projection conversion is performed on the display frame and the field-of-view image captured in the display frame in accordance with each of the direction and the size of the movement angle. A line-of-sight measuring unit that measures the direction of the line-of-sight of the observer,
Claims 1 to 3 further include a gazing point marker display unit that superimposes and displays the gazing point marker indicating the position of the gazing point corresponding to the direction of the line of sight on the visual field captured image. The observation status display system according to any one of the above. The head posture information acquisition unit
It is attached to the observer's head by a predetermined device, and is provided with either or both of an acceleration sensor and an angular velocity sensor that measure each of translation and rotational movement in the observer's head posture. The observation state display system according to any one of claims 1 to 4, wherein the observation state display system is characterized. An image pickup device provided in the observation space where the observer observes is further provided.
Any one of claims 1 to 4, wherein the head posture information acquisition unit detects the head posture of the observer from a moving image of the observer captured by the imaging device. Observation status display system described in. A visual field imaging device is further provided on the observer's head by a predetermined device so that the observer's visual field can be imaged in the vicinity of the eye position on the head. ,
The observation state display system according to any one of claims 1 to 5, wherein the field image acquisition unit acquires the field image image from the field image imager. It is an observation state display method that displays the state of the observer.
A field image acquisition process in which the field image acquisition unit acquires an image of the field of view in the observation direction of the observer as a field image image.
The head posture detection process in which the head posture information acquisition unit acquires the head posture, which is the posture of the observer's head,
An observation state in which the display frame display control unit includes a display frame display control process in which the position of the display frame on which the field-of-view image is displayed is moved in the display screen in accordance with the head posture. Display method. A program that causes a computer to execute the operation of the observation status display system that displays the status of the observer.
The computer
A field image acquisition means for acquiring an image of the field of view in the observation direction of the observer as a field image image.
A head posture detecting means for acquiring a head posture, which is the posture of the observer's head.
A program for making the position of the display frame on which the field-of-view image is displayed correspond to the head posture and functioning as a display frame display control means for moving the image in the display screen.

Images