JP7342409B2 - Observation status display system, observation status display method and program - Google Patents

Description

The present invention relates to an observation state display system, an observation state display method, and a program for displaying an observer's observation state.

An observer moves through a three-dimensional space (hereinafter referred to as an observation space) such as a display space where objects to be observed are displayed in a store or exhibition hall where various observation objects exist, and observes the objects. As a device for measuring the direction of the observer's line of sight, which can also be used when A wearable line of sight measurement device has been put into practical use (see, for example, Patent Document 1 and Patent Document 2).
In general, in the case of a head-mounted gaze measuring device that is fixed to the observer's head (hereinafter referred to as a head-mounted measuring device), the gaze data obtained from the above-mentioned gaze measurement function is attached to the observer's head. The line of sight direction is shown in a fixed head coordinate system.

The above-mentioned visual field camera is provided at the center position of both eyes in the head-mounted measurement device. When measuring the line-of-sight direction in the visual field of the observer in the observation direction, a visual field camera captures a visual field captured image as an image of the visual field in the observation direction of the observer.
The direction of the line of sight is uniquely determined as the position of the viewpoint detected by the line-of-sight measurement device, based on the mounting position of the field-of-view camera on the head-mounted measurement device, geometric parameters such as the angle of view, and the pixel position on the field-of-view captured image. Can be matched.

As a result, a gaze point marker is superimposed and displayed as the position of the observer's gaze point (hereinafter referred to as the gaze point) on the visual field captured image, without using information about the observer's position or posture, and the experimenter It is possible to visually indicate the position of the observer's gaze point.

Japanese Patent Application Publication No. 06-189906 Japanese Patent Application Publication No. 2012-146199

As described above, by using the wearable line-of-sight measuring device, when the observer observes an object while moving in the observation space, the observation movement of the observer in the observation space is not obstructed.
Therefore, using the wearable gaze measuring devices of Patent Documents 1 and 2, the experimenter can visually and in real time measure the gaze point of the observer in the observation space by moving the observer's free will in the observation space. can be asked for.

However, when changing the observation direction in the observation space, the observer needs to face the observation direction where the object of interest, that is, the gaze point, is located, so the observer moves the observation position or changes the line of sight in the field of view in the observation direction. In order to change the posture of the head, the posture of the head may be changed.
At this time, the relationship between the coordinate system fixed to the observation space (hereinafter referred to as global coordinate system) and the head coordinate system of the visual field camera that images the visual field observed by the observer also changes.

On the other hand, when analyzing the psychological state of an observer with high accuracy, we need to investigate the relationship between head movements (changes in head posture) to change the observation direction and the line of sight within the field of view in the observation direction. Needs to be analyzed.
However, in Patent Document 1 and Patent Document 2, the gaze point measured in the head coordinate system is converted to the global coordinate system and plotted as a gaze point marker. No consideration is given to analyzing movement.
For this reason, in Patent Document 1 and Patent Document 2, since the gaze point corresponding to the observer's line of sight is analyzed, the line of sight to the object can be determined, while the posture of the head when the observation direction is changed. Information cannot be presented visually.

When observing the psychological state of an observer, it is necessary to analyze the head posture information when changing the observation method, and it is necessary to analyze the head posture information when changing the observation method. It is conceivable to plot a gaze point marker on the
As a result, each time the observation direction changes, the visual field image changes in accordance with the movement of the head, and when the line of sight direction changes, the position of the gaze point marker in the visual field image changes, thereby changing the observation direction and attention. Information regarding the position of the viewpoint can be shown individually.

As described above, since the visual field of the captured image moves in response to a change in the observation direction, it is possible to observe the movement of the head from the direction of movement of the visual field independently of the point of gaze.
However, it is difficult to intuitively grasp the direction of movement of the observer's field of view, that is, the direction in which the head has changed, from changes in images in the visual field captured images.
In addition, the field-of-view captured image has large camera shake due to movement of the observer's head and changes in posture. Therefore, the experimenter may experience video sickness due to changes in the field of view and camera shake in the visual field captured image, that is, images that shake irregularly, making it difficult for the observer to perform analysis over a long period of time.

The present invention has been made in view of the above situation, and it is possible to intuitively grasp the movement of the observer's head to change the observation direction as the behavior of the observer in the observation space, and it is possible to change the field of view. Provided are an observation state display system, an observation state display method, and a program that can reduce visual sickness of an experimenter due to changes in the field of view in captured images and camera shake.

In order to solve the above-mentioned problems, the observation state display system of the present invention displays an image of the visual field of the observer in the observation direction in real space as a visual field captured image, and includes an image display unit having a display screen for observation by the experimenter. An observation state display system comprising : a visual field image acquisition unit that acquires the visual field captured image ; a head posture information acquisition unit that acquires a head posture that is a posture of the observer's head; The present invention is characterized by comprising a display frame display control unit that moves the position of the display frame in which the image is displayed within the experimenter's display screen in correspondence with the head posture.

In the observation state display system of the present invention, the display frame display control unit moves the display frame within the display screen by making the movement angle of the display frame correspond to the amount of change in the posture angle of the head posture. It is characterized by

In the observation state display system of the present invention, the display frame display control section performs projective transformation on the display frame and the visual field captured image within the display frame in correspondence with each of the directions and magnitudes of the movement angle. It is characterized by

The observation state display system of the present invention includes a line-of-sight measurement unit that measures the direction of the observer 's line of sight , and a point-of-gaze marker that indicates the position of the point of gaze corresponding to the direction of the line-of-sight, superimposed on the visual field captured image. The present invention is characterized in that it further includes a gaze point marker display section that displays the point of interest marker.

In the observation state display system of the present invention, the head posture information acquisition section is attached to the head of the observer with a predetermined brace, and the head posture information acquisition section is provided with an acceleration sensor or an acceleration sensor that measures rotational movement in the head posture of the observer. It is characterized by being provided with one or both of the angular velocity sensors.

The observation state display system of the present invention further includes an imaging device provided in an observation space in which the observer performs observation, and the head posture information acquisition unit collects a moving image of the observer captured by the imaging device. , is characterized by detecting the head posture of the observer.

The observation state display system of the present invention is attached to the observer's head with a predetermined equipment at a position near the eye position on the head where the visual field of the observer can be imaged, and the visual field image is The device further includes a visual field imaging device that images the visual field image, and the visual field image acquisition unit acquires the visual field captured image from the visual field imaging device.

The observation state display method of the present invention displays an image of the visual field of an observer in the observation direction in real space as a visual field captured image, and the observation state in an observation state display system comprising an image display unit having a display screen for observation by an experimenter. The display method includes a visual field image acquisition step in which the visual field image acquisition unit acquires the visual field captured image , and a head posture information acquisition unit in which the head posture information acquisition unit acquires a head posture that is the posture of the head of the observer. a posture detection step; and a display frame display control step in which the display frame display control unit moves the position of the display frame in which the visual field captured image is displayed within the experimenter's display screen in correspondence with the head posture. It is characterized by containing.

The program of the present invention causes a computer to display an image of the field of view of an observer in the observation direction in real space as a field-of-view captured image, and to cause a computer to operate an observation state display system including an image display section having a display screen for observation by an experimenter. The computer program includes: a visual field image acquisition unit that acquires the visual field captured image ; a head posture detection unit that acquires a head posture that is the posture of the observer's head; This program functions as a display frame display control means for moving the position of the display frame within the display screen of the experimenter in accordance with the head posture.

As explained above, according to the present invention, it is possible to intuitively understand the movement of the observer's head to change the observation direction as the behavior of the observer in the observation space, and the visual field in the visual field captured image can be understood intuitively. It is possible to provide an observation state display system, an observation state display method, and a program that can reduce the viewer's motion sickness caused by changes in the image and camera shake.

FIG. 1 is a block diagram showing an example of the configuration of an observation state display system according to the present embodiment. FIG. 2 is a conceptual diagram showing a configuration example of an observer information acquisition device 20 in the observation state display system 1 according to the present embodiment. 3 is a conceptual diagram showing an example of an observation direction and line of sight of an observer in an observation space 310. FIG. FIG. 3 is a conceptual diagram illustrating control of the display position of a display frame in which a visual field captured image is displayed on the display screen of the image display unit 16. FIG. It is a conceptual diagram showing an example of an observer's observation direction and line of sight in an observation space 310_1. FIG. 3 is a conceptual diagram illustrating control of the display position of a display frame in which a visual field captured image is displayed on the display screen of the image display unit 16. FIG. It is a flowchart explaining an example of the operation of the line of sight measurement processing of the observation state display system 1 according to the present embodiment.

Hereinafter, one embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of an observation state display system according to this embodiment. In FIG. 1, an observation state display system 1 is a system that displays the state of an observer, and includes an observation state display device 10 and an observer information acquisition device 20. The observation state display device 10 and the observer information acquisition device 20 are each connected through a predetermined network (not shown).
The observer information acquisition device 20 transmits information regarding the visual field observed by the observer, including each of gaze measurement information, image information, and head posture information, to the observation state display device 10. supply

FIG. 2 is a conceptual diagram showing a configuration example of the observer information acquisition device 20 in the observation state display system 1 according to the present embodiment. The observer information acquisition device 20 includes a head mounting jig 210, a line of sight measurement section 220, a visual field imaging device 230, and a posture information acquisition device 240.
The head mounting jig 210 is, for example, a helmet-shaped or glasses-shaped jig that is mounted on the head of the observer 200, and is fixed so that it does not shift from the position when worn due to movement of the head. This is an orthosis with a configuration that allows Here, when an HMD (Head Mounted Display) is used as the head mounting jig 210, an image of the virtual space displayed on the HMD is used as the visual field image in place of the visual field image captured by the visual field imaging device 230. I can do it. Then, each of the line-of-sight measurement unit 220, the visual field imaging device 230, and the posture information acquisition device 240 is attached to (for example, built into) the HMD, and the above-mentioned visual field image is displayed.

The line-of-sight measurement unit 220 measures the line-of-sight of the observer 200, and generates line-of-sight measurement information that is information indicating the position of the point of gaze, which is the viewpoint that the observer 200 is gazing at, within the field of view in the observation direction corresponding to this line of sight. seek.
Then, the line-of-sight measurement unit 220 supplies the obtained line-of-sight measurement information to the observation state display device 10. In this embodiment, the line of sight measurement information is information indicating the coordinate position of a gaze point on a two-dimensional plane of a visual field captured image, which will be described later.

For example, if the lower left position coordinates of a rectangular visual field captured image are (0.0, 0.0) and the upper right position coordinates are (1.0, 1.0), then in this coordinate range, the coordinates of the gaze point are The location is indicated. Here, the relationship between the gaze direction of the observer 200 and the coordinate position of the gaze point in the visual field captured image is determined in advance by considering the relative positions of the visual field measurement unit 220 and the visual field imaging device 230 in the head mounting jig 210. and calibrate and obtain it in advance for each observer.

The visual field imaging device 230 captures a moving image (60 frames/sec) of the visual field of the observer 200 in the observation direction, and supplies the captured image to the observation state display device 10 as the visual field captured image (image information).
In the present embodiment, the posture information acquisition device 240 is, for example, a posture sensor, and includes a gyro sensor and an acceleration sensor. Here, the gyro sensor acquires rotation information of the head of the observer 200 (the amount of angular change in each of panning, tilting, and rolling of the head). The acceleration sensor acquires acceleration for calculating rotation information of the observer's 200 head from the direction of gravity. The posture information acquisition device 240 supplies each of the acquired movement information and rotation information to the observation state display device 10 as head posture information. The rotation information in the head posture information is, for example, time series data of rotation angles expressed in quaternions (corresponding to time stamps described later).

Returning to FIG. 1, the observation state display device 10 includes a line of sight information acquisition section 11, a visual field image acquisition section 12, a gaze point marker superimposition section 13, a head posture information acquisition section 14, a display frame display control section 15, and an image display section 16. and a storage unit 17.
The line-of-sight information acquisition unit 11 writes the line-of-sight measurement information, which is supplied from the line-of-sight measurement unit 220 and indicates the coordinate position of the gaze point in the visual field captured image, in the storage unit 17 in correspondence with the time stamp, and stores the information. This time stamp is information indicating in chronological order the order in which visual field images captured in frame units were captured.
The visual field image acquisition unit 12 writes and stores the visual field captured image in units of frames, which is supplied from the visual field imaging device 230, in correspondence with the time stamp in the storage unit 17.

The gaze point marker superimposing unit 13 reads the visual field captured image from the storage unit 17 and superimposes the image of the gaze point marker on the coordinate position of the gaze point in the read visual field captured image. Then, the gaze point marker superimposing unit 13 writes the visual field captured image on which the gaze point marker is superimposed into the storage unit 17 and stores it (for example, overwrites the image data of the read visual field captured image).

The head posture information acquisition section 14 inputs the head posture information supplied from the posture information acquisition device 240 and outputs it to the display frame display control section 15 .
The display frame display control section 15 detects a change in the viewing direction from the head posture information of the observer 200, and controls the display position of the display frame on the display screen of the image display section 16. This display frame is a sub-screen frame on which the visual field captured image is displayed.

FIG. 3 is a conceptual diagram showing an example of the observation direction and line of sight of an observer in the observation space 310. FIG. 3 is a plan view of the observation space 310 seen from above. In the observation space 310, a target object 340 and a target object 350 are arranged.
The observation direction 330 is determined based on the head posture information (first) supplied from the posture information acquisition device 240. The visual field imaging device 230 captures an image of the visual field of the observer 200 in the observation direction 330 as a visual field captured image. The line-of-sight measuring unit 220 measures the line-of-sight 360 of the observer 200, determines the positional coordinates of the point of gaze in the visual field captured image, and outputs the position coordinates as line-of-sight measurement information.

FIG. 4 is a conceptual diagram illustrating control of the display position of the display frame in which the visual field captured image is displayed on the display screen of the image display unit 16.
FIG. 4A shows a visual field captured image 410 in which a gaze point marker 420 is superimposed on the visual field captured image in the observation direction 330 in FIG. The gaze point marker 420 is, for example, a cross-shaped mark, and is superimposed on the image 450 of the target object 350. That is, the line of sight of the observer 200 is directed toward the target object 350.

4(b) shows that the visual field captured image 410 of FIG. 4(a) is displayed within a display frame 510 displayed at a display position based on head posture information on a display screen 500 in the image display unit 16. It shows the state of being.
In FIG. 4B, the initial state is such that the observer is facing forward in the observation space 310, that is, the observation direction is the front for the observer, so that the display frame 510 is located at the center of the display screen 500. is displayed.
At this time, the display frame display control unit 15 controls the display frame display control unit 15 so that the visual angle of the visual field observed by the observer 200 and the visual angle of the display frame 510 viewed by the experimenter observing the display screen 500 are the same. are displayed in correspondence with each viewing angle.

FIG. 5 is a conceptual diagram showing an example of the observation direction and line of sight of the observer in the observation space 310_1. Similar to FIG. 3, FIG. 5 is a plan view of the observation space 310 seen from above. In the observation space 310_1, a target object 340 and a target object 350 are arranged, as in FIG. 3.
The observation direction 330_1 is determined based on the head posture information (second) supplied from the posture information acquisition device 240. The visual field imaging device 230 captures an image of the visual field of the observer 200 in the observation direction 330_1 as a visual field captured image. The line-of-sight measurement unit 220 measures the line-of-sight 360_1 of the observer 200, determines the positional coordinates of the gaze point in the visual field captured image, and outputs the position coordinates as line-of-sight measurement information.

FIG. 6 is a conceptual diagram illustrating control of the display position of the display frame in which the visual field captured image is displayed on the display screen of the image display unit 16.
FIG. 6A shows a visual field captured image 410_1 in which a gaze point marker 420_1 is superimposed on the visual field captured image in the observation direction 330_1 in FIG. The gaze point marker 420_1 is, for example, a cross-shaped mark, and is superimposed on the image 450 of the target object 350. That is, the line of sight of the observer 200 is directed toward the target object 350, as in FIG. 4(a).

FIG. 6(b) shows that the visual field captured image 410_1 of FIG. 6(a) is displayed within a display frame 510_1 displayed at a display position based on the head posture information on the display screen 500 of the image display unit 16. It shows the state of being.
In FIG. 6(b), the observer 200 rotates his head in the panning direction (for example, to the left in FIG. 5) by a predetermined angle change amount with respect to the initial position (and posture) in FIG. The figure shows the state after the head posture has changed.

At this time, the display frame display control unit 15 determines the viewing angle of the field of view observed by the observer 200 and the display frame visually recognized by the experimenter observing the display screen 500, as in the case of FIG. 4(b). The respective viewing angles are displayed in correspondence so that the viewing angles are the same as those of 510_1. The display frame display control unit 15 also controls the display screen 500 to correspond to the movement (parallel movement, rotational movement) of the observer's 200 head indicated by the head posture information, that is, the change in the observation direction indicated by the head posture information. The display position of the display frame 510_1 is controlled.

For example, in the case of FIG. 6(b), when the observer 200 rotates his head to the left in the panning direction by a predetermined angle change amount, the observation direction of the observer 200 changes to the left, so the display frame is displayed. The control unit 15 moves the display position of the display frame 510_1 on the display screen 500 to the left in accordance with the amount of change in angle. At this time, the display frame display control unit 15 displays a display in which the amount of angular change due to the rotation of the observer's head is equal to the amount of angular change due to the rotation of the experimenter's head when viewing the display frame 510_1 on the display screen. The display frame 510_1 on the display screen 500 is moved to the position. Furthermore, when the observer 200 rotates his or her head to the right in the panning direction by a predetermined angle change amount, the viewing direction of the observer 200 changes to the right. In response to this, the display position of the display frame 510_1 on the display screen 500 is moved to the right.

Similarly, when the observer 200 rotates his or her head upward in the tilt direction by a predetermined angle change amount, the viewing direction of the observer 200 changes upward, so the display frame display control unit 15 controls the angle change. The display position of the display frame 510_1 on the display screen 500 is moved upward in accordance with the amount. Further, when the observer 200 rotates his or her head downward by a predetermined angle change amount in the tilt direction, the viewing direction of the observer 200 changes downward, so the display frame display control unit 15 controls the angle change amount. In response to this, the display position of the display frame 510_1 on the display screen 500 is moved downward.

Similarly, when the observer 200 rotates his head clockwise in the roll direction by a predetermined angle change amount, the observation field of the observer 200 changes clockwise, so the display frame display control unit 15 , the display angle of the display frame 510_1 on the display screen 500 is rotated clockwise in accordance with the angle change amount. Further, when the observer 200 rotates his head counterclockwise in the roll direction by a predetermined angle change amount, the observation field of the observer 200 changes counterclockwise, so the display frame display control unit 15 The display angle of the display frame 510_1 on the display screen 500 is rotated counterclockwise in accordance with the amount of change in angle.
Thereby, the amount of angular change in the rotation of the observer's head can be made to correspond to the movement angle visually recognized by the experimenter on the display screen 500. Therefore, when the display position of the display frame 510 (FIG. 4(b)) changes to the display position of the display frame 510_1 (FIG. 6(b)), the visual field captured image within the display frame remains stationary. It can be made to feel for the experimenter.

In addition, the display frame display control unit 15 changes the coordinate position for displaying the display frame 510_1 (FIG. 6(b)) according to the angle change amount according to preset coordinate conversion conditions. Coordinate transformation is performed from the coordinate position shown in FIG. 4(b)).
This allows the experimenter to visually recognize the visual angle of the object in the visual field captured image observed by the experimenter on the display screen 500 as the same visual angle as the visual angle of the object observed by the observer in the visual field in the observation space. This allows the experimenter to visually recognize the visual field captured image as a visual field that is intuitively similar to the visual field observed by the observer.

In addition, when moving the display position of the display frame 510_1 on the display screen 500, the display frame display control unit 15 moves the display frame 510_1 further away from the center of the display frame 510_1 as shown in FIG. The length of one side is made longer and the length of the nearer side is made shorter. In order to shorten the length of the display frame 510_1, projective transformation such as trapezoidal correction may be performed on the display frame 510_1.

Through this projective transformation, the shape of the object within the display frame 510_1 can be kept constant for the experimenter observing the display screen 500, and the shape of the object within the display frame 510_1 corresponding to the visual field of the observer 200 on the display screen 500 can be kept constant. A sense of perspective can be provided to the experimenter on the display screen 500.

That is, if you want the experimenter to see the apparent shape of the object in the display frame 510_1 that has moved on the display screen 500 as the same shape as when observing it from the front, the center of the display screen 500 in the display frame 510_1 It is necessary to perform a projective transformation to make the edges longer as they are further away from . Unless the above-mentioned projective transformation is performed, the objects in the visual field captured images within the display frame 510_1 will not be visually recognized by the experimenter as having the same shape.
Therefore, as shown in FIG. 6(b), the display frame display control unit 15 compares the side far from the center of the display screen 500 with the side near the opposite sides of the display frame 510_1. Perform projective transformation to lengthen.

If the observer 200 does not change the movement and posture of the head, that is, does not change the observation direction, for a predetermined period of time, the display frame display control unit 15 changes the display frame 510 to the display screen 500 at a preset movement speed. is gradually returned to the center of the display frame 510 of the display screen 500, and the field of view captured image in the observation space of the display frame 510 is reset to the field of view in the observation direction at the initial value of the head posture information of the head of the observer 200 (the amount of movement and Processing to set the angle change amount to 0) is performed.
As a result, the movement range of the display frame 510 on the limited display screen 500 is substantially expanded, and the experimenter can change the observation direction of the observer 200 within the limited display screen 500 within the limited display screen 500. , the display frame 510 can be prevented from protruding from the display screen 500, and the viewer can effectively observe the viewing direction and field of view.

The display frame display control unit 15 also controls the experimenter's display on the display screen 500 without making the amount of change in the angle of the head of the observer 200 the same as the movement angle of the arrangement position of the display frame 510 on the display screen 500. In consideration of the viewing angle of the frame 510 and the visibility of the display frame 510 by an experimenter observing the display screen 500, the number may be arbitrarily increased or decreased at a predetermined ratio.

FIG. 7 is a flowchart illustrating an example of the operation of the line-of-sight measurement process of the observation state display system 1 according to the present embodiment.
Step S101:
The visual field image acquisition unit 12 receives visual field captured image data supplied from the visual field imaging device 230 at a predetermined period. Then, the visual field image acquisition unit 12 adds a time stamp to the input visual field captured image data, and writes it into the storage unit 17 for storage.

Step S102:
The head posture information acquisition unit 14 receives head posture information obtained by measuring the movement of the observer's head and is supplied from the posture information acquisition device 240 at the timing when the visual field image acquisition unit 12 inputs the visual field captured image data. input. Then, the head posture information acquisition section 14 associates the input head posture information with the above-mentioned time stamp, and writes it into the storage section 17 for storage.

Step S103:
The line-of-sight information acquisition unit 11 inputs the line-of-sight measurement information supplied from the line-of-sight measurement unit 220 at the timing when the visual field image acquisition unit 12 inputs the visual field captured image data. Then, the line-of-sight information acquisition unit 11 associates the input line-of-sight measurement information with the above-mentioned time stamp, and writes it into the storage unit 17 for storage.

Step S104:
The head posture information acquisition unit 14 determines whether the newly input head posture information has changed in the amount of movement and the amount of angular change compared to the head posture information input immediately before at the above-mentioned timestamp. Make a judgment.
At this time, if the input head posture information has changed in the amount of movement and the amount of angular change compared to the head posture information input immediately before, the head posture information acquisition unit 14 executes the process in step S104. Proceed to. On the other hand, if the input head posture information has not changed in the amount of movement and the amount of angular change compared to the head posture information input immediately before, the head posture information acquisition unit 14 executes the process in step Proceed to S106.

Step S105:
The display frame display control unit 15 determines the display position of the display frame 510 within the display screen 500 in accordance with each new movement amount and angle change amount, and changes the shape of the display frame 510 in accordance with the display position. Perform projective transformation.

Step S106:
The gaze point marker superimposing unit 13 reads the visual field captured image data with the latest time stamp from the storage unit 17, and superimposes a gaze point marker indicating the observer's gaze point on the coordinate position corresponding to the gaze measurement data (for example, FIGS. 4(a) and 6(a)).
Then, the point-of-gaze marker superimposing section 13 writes and stores the field-of-view captured image data on which the point-of-gaze marker is superimposed in the storage section 17 in correspondence with the time stamp.

Step S107:
The display frame display control unit 15 reads the visual field captured image data on which the gaze point marker is superimposed from the storage unit 17, performs the same projective transformation as the display frame 510, and converts it into a shape included in the display frame 510.
Then, the display frame display control unit 15 displays the display frame 510 in which the visual field captured image data is fitted at the already determined display position on the display screen 500 (for example, as shown in FIG. 4(b) and FIG. 6(b)). ). At this time, the display frame display control unit 15 writes and stores the display image data of the display screen 500 on which the display frame 510 is displayed in the storage unit 17 in correspondence with the latest time stamp.

Step S108:
The observation state display device 10 detects whether or not the experimenter has inputted an input to end the measurement process from an input means (not shown). At this time, the observation state display device 10 ends the process if an input to end the measurement process is made, whereas if an input to end the measurement process is not made, the process returns to step S101.

As described above, in this embodiment, when an experimenter analyzes an observer's interest in a target object in an observation space, the display frame 510 is set to Because the experimenter who is observing the display screen 500 changes in response to the movement of the observer's head (change in the observation direction), the experimenter who is observing the display screen 500 can change the observer's head movement and posture change (change in the observation direction) to change the observation direction. It is possible to intuitively grasp the state of an observer observing an object.

As a result, according to the present embodiment, compared to conventional methods that can only display the direction of the observer's line of sight with respect to the object, according to the present embodiment, the observer can observe the object in which he is interested. When the observer first directs his or her line of sight toward the object, then turns the head in the direction of the object (observation direction), or the observer turns the head toward the object (observation direction). ), it is possible to visually and intuitively obtain information for analyzing the detailed relationship between the observation direction and the line of sight, such as whether the user directs his or her line of sight to the object after first pointing at the object.

In addition, in this embodiment, the experimenter observes the movement of the display frame 510 corresponding to a change in the observation direction of the observer, that is, visually observes a field-of-view captured image similar to the field of view observed by the observer, The experimenter can feel as if they are moving their own head.
Therefore, according to the present embodiment, the display frame in which the visual field captured image is displayed is fixed regardless of the observer's observation direction, and the image within the display frame changes greatly. Image sickness can be suppressed, and the experimenter can easily observe movements of the observer's head and changes in the gaze point.

Furthermore, in the above-described embodiment, the posture information acquisition device 240 has been described as having a configuration that acquires the movement and angular change amount of the observer's head using an acceleration sensor or a gyro sensor. A configuration may be adopted in which the movement of the observer's head is analyzed from the captured image data to determine the amount of movement and angle change of the head.
At this time, the head posture information is supplied to the observation state display device 10 as data of a moving image of the observer from an imaging device provided for observing the observer in the observation space. Then, the head posture information acquisition unit 14 extracts an image of the observer's head from the moving image of the observer supplied from the imaging device through image processing, and calculates the movement of the head image. Through analysis, the amount of movement and angle change of the observer's head is detected.

In addition, the line of sight measurement unit 220 detects changes in the line of sight of the observer from images obtained by a plurality of near-infrared illumination and imaging devices arranged for observing the eyes of the observer in the observation space, and detects changes in the line of sight of the observer. A configuration may also be adopted in which line of sight measurement information associated with a captured image is acquired. Furthermore, features around the eyes, such as the inner and outer corners of the eyes and pupils, which are important for detecting the observer's line of sight, are extracted from images obtained from multiple imaging devices arranged to observe the observer's eyes in the observation space. , the angle of the line of sight in the field of view in the observation direction may be calculated to determine the line of sight.

In addition, as another configuration of the head posture information acquisition unit 14, a depth camera or the like is used as the posture information acquisition device 240, and the three-dimensional shape of the observer's head is extracted from the depth map supplied from the depth camera. A configuration may be used in which the amount of movement and angular change of the observer's head is detected by analyzing changes in the three-dimensional shape of the head.
As described above, any configuration that can detect changes in the observer's head may be used as the posture information acquisition device 240 for obtaining head posture information. Similarly, an electro-oculography measuring device or the like may be used as the line-of-sight measurement unit 220 for obtaining line-of-sight measurement information, as long as it is configured to be able to detect changes in the direction of the observer's line of sight in the observation space.

Further, as another configuration of the observation state display system 1, the observer information acquisition device 20 may be replaced with a storage device inside or outside the observation state display device 10. In this case, the storage device stores in advance time-series information of the observer's line of sight, head posture, and visual field image, and the line-of-sight information acquisition unit 11 receives the line-of-sight time-series information from the head posture information acquisition unit. The time-series information of the head posture is read out by the visual field image acquisition unit, and the time-series information of the visual field images is sequentially read out by the visual field image acquisition unit in synchronization with each other from the storage device. Thereby, the condition of the observer is displayed on the image display section 16 in the same way as when the observer information acquisition device 20 is used.
Further, when the storage device is inside the observation state display device 10, the storage device may be the storage section 17.
Further, when the storage device is located outside the observation state display device 10, the storage device may be connected to the observation state display device 10 via a network such as the Internet.

In addition, in this embodiment, the observation space that is the object of observation by the observer is described as a real space, but in a virtual space such as CG (Computer Graphics) or VR (Virtual Reality) that is observed by wearing an HMD. , it can also be used as a configuration for analyzing the observation direction and gaze point of the observer in this virtual space. In this case, an image of the virtual space is displayed on the display screen 500, and a display frame 510 is displayed at a position that includes the visual field captured image in the observation direction of the observer.

Furthermore, in this embodiment, the display screen 500 that the experimenter observes in order to analyze each of the observer's observation direction and gaze point is a two-dimensional plane (such as FIG. 4(b) and FIG. 6(b)). As described above, in order to expand the movement range of the display frame 510, a display screen having a curved shape such as a cylinder or a sphere, instead of a two-dimensional planar shape, is used as the display screen on which the display frame 510 is arranged in an overlapping manner. It's okay.

Note that a program for realizing the functions of the observation status display device 10 according to the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is loaded into a computer system and executed to perform observation. Measurement processing of the observer's observation direction and gaze point in the status display may also be performed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices.
Furthermore, the term "computer system" includes a WWW system equipped with a home page providing environment (or display environment). Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" refers to volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. This also includes programs that are retained for a certain period of time.

Further, the program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Moreover, the above program may be for realizing a part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

1... Observation state display system 10... Observation state display device 11... Line of sight information acquisition section 12... Visual field image acquisition section 13... Gaze point marker superimposition section 14... Head posture information acquisition section 15... Display frame display control section 16... Image display Part 17...Storage unit 20...Observer information acquisition device 210...Head mounting jig 220...Line of sight measurement unit 230...Visual field imaging device 240...Posture information acquisition device

Claims (9)

An observation state display system comprising an image display unit having a display screen for displaying an image of the field of view in the observation direction of an observer in real space as a field-of-view captured image and for observation by an experimenter,
a visual field image acquisition unit that acquires the visual field captured image;
a head posture information acquisition unit that acquires a head posture that is a head posture of the observer;
An observation state display system comprising: a display frame display control unit that moves a position of a display frame in which the visual field captured image is displayed within a display screen of an experimenter in accordance with the head posture. The display frame display control section is
The observation state display system according to claim 1, wherein the display frame is moved by making a movement angle of the display frame within the display screen correspond to an amount of change in a posture angle of a head posture. The display frame display control section is
The observation state display system according to claim 2, wherein projective transformation is performed on the display frame and the visual field captured image within the display frame in correspondence with each of the direction and magnitude of the movement angle. a line-of-sight measurement unit that measures the direction of the observer's line of sight;
Claims 1 to 3, further comprising: a gaze point marker display unit that displays a gaze point marker indicating the position of the gaze point corresponding to the direction of the line of sight, superimposed on the visual field captured image. Observation status display system according to any one of the items. The head posture information acquisition unit,
Either or both of an acceleration sensor or an angular velocity sensor is attached to the observer's head with a predetermined equipment and measures each of parallel and rotational movement in the observer's head posture. The observation state display system according to any one of claims 1 to 4, characterized by: further comprising an imaging device provided in an observation space where the observer performs observation,
Any one of claims 1 to 4, wherein the head posture information acquisition unit detects the head posture of the observer from a moving image of the observer captured by the imaging device. Observation status display system described in . The method further includes a visual field imaging device that is attached to the observer's head by a predetermined equipment at a position where the visual field of the observer can be imaged near the position of the eye on the head, and that captures the visual field image. ,
The observation state display system according to any one of claims 1 to 5, wherein the visual field image acquisition unit acquires the visual field captured image from the visual field imaging device. An observation state display method in an observation state display system comprising an image display unit having a display screen for displaying an image of the field of view in the observation direction of an observer in real space as a field-of-view captured image, and for an experimenter to observe,
a visual field image acquisition step in which the visual field image acquisition unit acquires the visual field captured image ;
a head posture detection step in which the head posture information acquisition unit acquires a head posture that is a head posture of the observer;
and a display frame display control step in which the display frame display control unit moves the position of the display frame in which the visual field captured image is displayed within the experimenter's display screen in correspondence with the head posture. How to display observation status. A program that causes a computer to execute an operation of an observation state display system that displays an image of the visual field of an observer in the observation direction in real space as a visual field captured image, and includes an image display unit having a display screen for observation by an experimenter,
The computer,
visual field image acquisition means for acquiring the visual field captured image;
head posture detection means for acquiring a head posture that is the posture of the observer's head;
A program for functioning as a display frame display control means for moving a position of a display frame in which the visual field captured image is displayed within a display screen of an experimenter in correspondence with the head posture.

Images