JP2014093704A - Head mount display and movement detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology for detecting the movement of a user who is wearing a head mount display.SOLUTION: The casing of a head mount display 200 is configured to store a presentation part for presenting a three-dimensional video, and when it is mounted on the head of a user 500 as an observer, it is positioned ahead of both eyeballs of the user 500. A casing light emitting element 210 is configured to, when the casing is mounted on the head of the user 500, emit the rays of light in a vertical downward direction to the casing. An image acquisition part is configured to acquire the image of the rays of light of the casing light emitting element 210 imaged by a body imaging element 302 mounted on the body of the observer, and configured to image the rays of light to be emitted by the casing light emitting element 210. A head tracking part is configured to analyze the image of the rays of light acquired by the image acquisition part, and to track the relative movement of the casing light emitting element 210 to the body imaging element 302.

Description

  The present invention relates to a head mounted display and a motion detection method in an image presentation system including the head mounted display.

  In recent years, technological development for presenting stereoscopic images has progressed, and head mounted displays capable of presenting stereoscopic images with depth have become widespread. The head-mounted display is mounted so as to cover both eyes of the user who is an observer, and moves in conjunction with the movement of the user's head. For this reason, if the movement of the user's head is tracked and the result is fed back to the video of the virtual space presented by the head-mounted display, the user is presented with a higher sense of immersion and presence in the virtual space. It is thought that you can.

  Further, since the head mounted display is mounted so as to cover both eyes of the user who is an observer, it is generally difficult for the user to see an actual image of the outside world such as the hand with the head mounted display mounted. On the other hand, if the movement of the user's hand can be tracked, it is considered that an instruction determined according to the type of the movement can be used for controlling the head mounted display itself or a video to be presented.

  As a method for tracking the movement of the head mounted display, for example, a 9-axis method for measuring 3-axis acceleration, 3-axis angular velocity, and 3-axis geomagnetism is known. In this method, depending on the region, it is difficult to track the movement due to the geomagnetic dip, and it may be easily influenced by the surrounding magnetic field such as a steel frame of a building or a motor. In addition, in order to feed back the movement of the user's hand to the head-mounted display, it is desired to develop a small and light device that detects the movement of the user's hand and that can be intuitively operated by the user.

  The present invention has been made in view of these problems, and an object thereof is to provide a technique for detecting the movement of a user wearing a head-mounted display.

  In order to solve the above problems, an aspect of the present invention is a head mounted display. This head-mounted display houses a presentation unit that presents a three-dimensional image, and a housing that is positioned in front of the viewer's eyes when mounted on the viewer's head; When mounted on the head, the housing light-emitting element that irradiates light vertically downward with respect to the casing, and the light that is mounted on the body of the observer and irradiated by the housing light-emitting element are imaged An image acquisition unit that acquires an image of the light of the housing light-emitting element, which is imaged by the trunk imaging device configured as described above, and an image of the light acquired by the image acquisition unit, and the imaging of the trunk A head tracking unit that tracks a relative movement of the housing light emitting element with respect to the element;

  Another aspect of the present invention includes a head-mounted display, a torso imaging device configured to be communicable with the head-mounted display and mounted on the torso of the observer, and movement of the observer's hand on the head-mounted display. It is a motion detection method in a video presentation system including a light emitting element that moves in conjunction. The method includes the steps of: imaging the light emitted by the light emitting element that moves in conjunction with the movement of the observer's hand; analyzing the imaged light image emitted by the light emitting element; Detecting a relative movement of the light emitting element with respect to the display; and irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on a head of an observer. A step of acquiring an image captured by the torso image sensor with respect to light irradiated in the vertically downward direction; and an analysis of the acquired image by the torso image sensor to analyze the head mount with respect to the torso image sensor And detecting the relative movement of the display.

  Yet another embodiment of the present invention is a program that causes a computer to implement the steps of the above method.

  This program may be provided as part of firmware incorporated in the device in order to perform basic control of hardware resources such as video, audio, game device, and head mounted display. This firmware is stored in a semiconductor memory such as a ROM (Read Only Memory) or a flash memory in the device, for example. In order to provide the firmware or to update a part of the firmware, a computer-readable recording medium storing the program may be provided, and the program may be transmitted through a communication line.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which detects the motion of the user who mounts a head mounted display can be provided.

It is a figure which shows typically the outline of the whole structure of the video presentation system which concerns on embodiment. It is a figure which shows typically the utilization scene of the video presentation system which concerns on embodiment. It is a figure which shows typically the internal structure of the image | video presentation system which concerns on embodiment. It is a figure which shows typically the function structure of the head mounted display which concerns on embodiment. It is a flowchart which shows the flow of the tracking process by the image | video presentation system which concerns on embodiment. It is a figure which shows typically the outline of the whole structure of the hand tracking which concerns on embodiment. It is a figure which shows typically the utilization scene of the hand tracking of the video observation system which concerns on embodiment. It is a figure which shows typically an example of the image which the housing | casing imaging part which concerns on embodiment imaged. It is a figure which shows typically the function structure of the hand part tracking part which concerns on embodiment. It is a flowchart which shows the flow of a hand part tracking part process which concerns on embodiment. It is a figure which shows typically the outline of the whole structure of the head tracking which concerns on embodiment. It is a figure which shows typically the utilization scene of the head tracking of the video observation system which concerns on embodiment. It is a figure which shows typically an example of the image which the trunk | drum imaging part which concerns on embodiment imaged. It is a figure which shows typically the function structure of the head tracking part which concerns on embodiment. It is a flowchart which shows the flow of the head tracking part process which concerns on embodiment. It is the figure which expanded a part of head mounted display concerning an embodiment. It is a flowchart which shows the flow of the bright spot selection process by the bright spot selection part which concerns on embodiment. It is a flowchart which shows the flow of the process which detects the signal which the position designation apparatus which concerns on embodiment emits. It is a state transition diagram which shows the internal state of a luminescent spot detection part when detecting the signal which the position designation | designated apparatus which concerns on embodiment emits.

(Overall configuration of video presentation system 100)
FIG. 1 is a diagram schematically showing an overview of the overall configuration of a video presentation system 100 according to an embodiment. The video presentation system 100 according to the embodiment includes a head mounted display 200, a body fixing device 300, and a position specifying device 400.

  The head mounted display 200 is configured so that the casing thereof can be freely mounted on the head of a user who is an observer, and presents a three-dimensional image to the user using a presentation unit stored inside the casing. The head mounted display 200 includes, for example, a first button 202 that serves as an interface for turning on / off the power, a housing imaging unit 204 that is provided in the housing of the head mounted display 200 and images light outside the housing, A housing light emitting element 210 that is provided in the housing of the mount display 200 and irradiates light outside the housing is further provided. The housing imaging unit 204 further includes a first imaging element 206 and a second imaging element 208.

  The torso fixing device 300 is configured to be installed near the torso of the user who wears the head mounted display 200, for example, near the chest of the user, and the case light emitting element 210 of the head mounted display 200 irradiates. Imaging light. Therefore, the torso fixing device 300 includes a torso image pickup element 302 and an element support part 314 configured to be attachable to the user 500. As shown in FIG. 1, the body fixing device 300 and the head mounted display 200 are connected by wire, and the head mounted display 200 can acquire an image captured by the body image sensor 302. The head mounted display 200 can specify the relative position of the casing light emitting element 210 with respect to the body fixing device 300 by analyzing the acquired image. The body fixing device 300 and the head mounted display 200 may be wirelessly connected using a technique such as infrared communication. The element support unit 314 also includes an adjustment unit 312 that adjusts the inclination of the body image sensor 302.

  The position designation device 400 includes a gripping unit 402 for a user to grip, a position designation light emitting element 404 that is a light emitting element attached to the gripping unit 402, and a first method for controlling how the position designation light emitting element 404 emits light. 2 buttons 406. Depending on whether the user turns on or off the second button, the manner in which the position-designated light emitting element 404 emits light changes. The housing imaging unit 204 of the head mounted display 200 captures and images the light emitted from the position designation light emitting element 404. The head mounted display 200 can specify the position of the position designation light emitting element 404 and whether or not the second button is pressed by analyzing the image. Note that the position specifying device 400 does not necessarily need to include the grip portion 402, and may be any configuration as long as the position specifying light emitting element 404 moves in conjunction with the movement of the user's hand. For example, the position specifying device 400 may be configured to include the position specifying light emitting element 404 on a ring or a bracelet.

  FIG. 2 is a diagram schematically illustrating a usage scene of the video presentation system 100 according to the embodiment. As shown in FIG. 2, the head mounted display 200 is attached to the head of the user 500 when in use, and moves in conjunction with the movement of the head of the user 500. The torso fixing device 300 is suspended by a string-like member hung on the user's neck like a necklace, and is installed, for example, in the vicinity of the user's chest from the center of the abdomen. Although not shown, the body fixing device 300 may be fixed to the user's body using a fixing member such as a belt. The position specifying device 400 is gripped by the user 500 when in use. For this reason, when the user 500 moves the hand holding the position specifying device 400, the position specifying light emitting element 404 also moves in conjunction with the movement of the user's hand.

  FIG. 3 is a diagram schematically illustrating an internal configuration of the video presentation system 100 according to the embodiment. As described above, the video presentation system 100 according to the embodiment includes the head mounted display 200, the body fixing device 300, and the position specifying device 400.

  The head mounted display 200 includes a storage unit 212, a communication unit 214, a recording medium interface 216, a presentation unit 218, an audio output unit 220, a CPU (Central Processing Unit), in addition to the first button 202 and the housing imaging unit 204 described above. 222, an acceleration sensor 224, a first LED (Light Emitting Diode) 226, a power source 228, an internal path 230, and an I / O (Input / Output) control unit 232.

  The first image sensor 206 is configured to image light in an invisible light region such as infrared light. Specifically, the first image sensor 206 images light that has passed through an optical filter that attenuates light outside the invisible light region to be imaged. In contrast, the second image sensor 208 includes an optical filter that attenuates light outside the visible light region, and images light in the visible light region. Both the first image sensor 206 and the second image sensor 208 can be realized by a known solid-state image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  Further, both the first image sensor 206 and the second image sensor 208 are provided with fisheye lenses. When the head mounted display 200 is mounted on the head of the user 500, the first image sensor 206 and the second image sensor 208 are vertically below the casing of the head mounted display 200. The light in the direction and in the direction of the sight line of the user 500 is imaged. As indicated by the line segment connecting the reference sign S1 and the reference sign S2 in FIG. 1, the user 500 attaches the head-mounted display 200 to the lower surface of the casing of the head-mounted display 200, and the viewing direction from the user 500 side. There is a slope region that is inclined vertically upward. Along with the first image sensor 206 and the second image sensor 208, it is provided in a slope region existing on the lower surface of the housing of the head mounted display 200. As a result, the first image sensor 206 and the second image sensor 208 are vertically downward with respect to the housing of the head mounted display 200 and the user 500 when the head mounted display 200 is mounted on the head of the user 500. It is possible to take an image of light ahead in the line-of-sight direction.

  The storage unit 212 stores firmware including an operation program for comprehensively controlling the operation of the head mounted display 200. The storage unit 212 also serves as a work area for storing images captured by the case imaging unit 204 and images acquired from the torso fixing device 300 and processing executed by the CPU 22. The storage unit 212 can be realized by using, for example, known volatile and nonvolatile memories.

  The communication unit 214 is an interface for the head mounted display 200 to communicate with an external device of the video presentation system 100. The communication unit 214 can be realized by using a known wireless communication technology such as WiFi (registered trademark) or Bluetooth (registered trademark).

  The recording medium interface 216 is an interface for reading a recording medium such as a DVD (Digital Versatile Disc), a Bluray (registered trademark) disk, or a flash memory that stores contents such as movies and games. The content read by the recording medium interface 216 is processed by the CPU 222 and provided to the user via the presentation unit 218 and the audio output unit 220. The recording medium interface 216 can be realized using a known technique such as an optical drive.

  The presentation unit 218 is a device for presenting video to the user 500. Although not shown, the presentation unit 218 includes a left-eye presentation unit and a right-eye presentation unit, and can independently present the left-eye parallax image and the right-eye parallax image to the left eye and right eye of the user 500, respectively. Thereby, it becomes possible to present a three-dimensional image having depth to the user 500. The presentation unit 218 can be realized using a known display technology such as an LCD (Liquid Crystal Display) or an OELD (Organic Electro-Luminescence Display).

  The audio output unit 220 is a device for presenting audio to the user 500. Although not shown, the audio output unit 220 is connected to the left earphone and the right earphone, and can provide stereo sound to the user.

  The CPU 222 processes information acquired from each unit in the head-mounted display 200 via the internal path 230, and executes processing such as content provision and head tracking processing and hand tracking processing described later.

  The acceleration sensor 224 detects the acceleration of the head mounted display 200. More specifically, the acceleration sensor 224 detects the direction of gravity in order to detect the tilt of the head mounted display 200. The acceleration sensor 224 can be realized using, for example, a known low g acceleration sensor. The first LED 226 is an LED light source that irradiates light to the outside of the head mounted display 200, and irradiates light in an invisible region such as infrared light. The housing light emitting element 210 described above is realized by the first LED 226.

  The I / O control unit 232 controls input from the first button 202, the housing imaging unit 204, and the acceleration sensor 224 and light irradiation by the first LED 226. The I / O control unit 232 also controls transmission / reception of information to / from the I / O control unit 304 in the body fixing device 300. The power source 228 includes a battery (not shown), and supplies power for driving to each part of the head mounted display 200.

  The body fixing device 300 includes an I / O control unit 304 and a third image sensor 306. The third image sensor 306 corresponds to the above-described body image sensor 302 and can be realized by a known image sensor such as a CCD or a CMOS. The third image sensor 306 includes a perspective projection lens. As will be described in detail later, the third image sensor 306 includes an optical filter that transmits light in the vicinity of the wavelength of light emitted by the first LED 226 in the head mounted display 200 and attenuates light of other wavelengths. Thereby, the light emitted by the first LED 226 can be selectively imaged. Electric power for driving each part of the torso fixing device 300 is supplied from a power source 228 by a wire (not shown).

  The I / O control unit 304 outputs an image captured by the third image sensor 306 to the I / O control unit 232 in the head mounted display 200. Further, the I / O control unit 304 receives an instruction to start and stop imaging of the third imaging element 306 from the I / O control unit 232 in the head mounted display 200 and controls the imaging operation of the third imaging element 306. .

  The position specifying device 400 includes a second button 406, an I / O control unit 408, a second LED 410, and a power source 412.

  The second LED 410 corresponds to the position-designated light emitting element 404 described above, and emits light in an invisible region such as infrared light under the control of the I / O control unit 408. The I / O control unit 408 controls how the second LED 410 emits light according to whether the second button 406 is on or off. Specifically, the I / O control unit 408 always turns on the second LED 410 when the second button 406 is on. When the second button 406 is off, the I / O control unit 408 causes the second LED 410 to blink at a predetermined blinking frequency. Assuming that the blinking frequency is F [Hz], when the second button 406 is off, the I / O control unit 408 lights the second LED 410 for 1 / (2 × F) seconds, and continues to 1 / (2 XF) The control for turning off the second LED 410 for 2 seconds is repeated. The power source 412 includes a battery (not shown) and supplies power for driving to each part of the position specifying device 400.

  FIG. 4 is a diagram schematically illustrating a functional configuration of the head mounted display 200 according to the embodiment. The head mounted display 200 according to the embodiment includes a fisheye lens 234, a housing imaging unit 204, a presentation unit 218, an image acquisition unit 250, a tracking unit 260, a content reproduction unit 270, and a synthesis unit 280. The image acquisition unit 250 further includes a first image acquisition unit 252 and a second image acquisition unit 254. The tracking unit 260 further includes a head tracking unit 600, a hand tracking unit 700, and an output unit 262.

  The fisheye lens 234 is a lens having an angle of view of about 180 degrees, and is used, for example, for imaging by an equidistant projection method. Both the first image sensor 206 and the second image sensor 208 described above capture an image through the fisheye lens 234. In FIG. 4, the first image sensor 206 and the second image sensor 208 are collectively referred to as a case image capturing unit 204. doing. Hereinafter, in the present specification, the first imaging element 206 and the second imaging element 208 are collectively referred to as a case imaging unit 204 unless it is necessary to distinguish between the first imaging element 206 and the second imaging element 208. . The first image sensor 206 and the second image sensor 208 may capture images via different fisheye lenses.

  As described above, the position specifying device 400 is held by the hand of the user 500 during use. For this reason, the position specifying light emitting element 404 is present vertically downward with respect to the housing of the head mounted display 200 and in front of the user 500 in the line of sight. When the head mounted display 200 is mounted on the head of the user 500, the housing imaging unit 204 captures light that is vertically downward with respect to the housing of the head mounted display 200 and forward in the line of sight of the user 500. Can do. For this reason, the housing imaging unit 204 can easily grasp the position-designated light emitting element 404 that moves in conjunction with the movement of the hand of the user 500 at the angle of view.

  A second image acquisition unit 254 in the image acquisition unit 250 acquires an image captured by the housing imaging unit 204. The hand tracking unit 700 in the tracking unit 260 acquires the image captured by the housing imaging unit 204 from the second image acquisition unit 254. The hand tracking unit 700 analyzes a position change of a bright spot derived from the position designation light emitting element 404 by analyzing the acquired image. Since the position designation light emitting element 404 moves in conjunction with the hand movement of the user 500, the hand tracking unit 700 can detect the hand movement of the user 500 as a result. The hand tracking unit 700 can be realized by the CPU 222 executing a program for realizing the hand tracking process stored in the storage unit 212 in FIG. Details of the hand tracking process in the hand tracking unit 700 will be described later.

  The first image acquisition unit 252 in the image acquisition unit 250 acquires an image captured by the body image sensor 302 of the body fixing device 300. The head tracking unit 600 acquires the image acquired by the first image acquisition unit 252.

  As described above, the torso imaging element 302 images a subject including light emitted from the casing light emitting element 210 of the head mounted display 200. Therefore, a bright spot derived from the casing light emitting element 210 is captured in the image acquired by the first image acquisition unit 252. Therefore, the head tracking unit 600 tracks the change in position of the position-designated light emitting element 404 by analyzing the acquired image. Thereby, the head tracking unit 600 can detect the relative movement of the housing light emitting element 210 with respect to the torso imaging element 302. As a result, the head tracking unit 600 can detect the orientation of the user 500's head relative to the user's 500 trunk, that is, the line-of-sight direction. The hand tracking unit 700 can be realized by the CPU 222 executing a program for realizing the head tracking process stored in the storage unit 212 in FIG. Details of the head tracking process in the head tracking unit 600 will be described later.

  The output unit 262 in the tracking unit 260 outputs the processing results of the head tracking unit 600 and the hand tracking unit 700. The content playback unit 270 plays back the content read by the recording medium interface 216. The content playback unit 270 changes the video presented to the user 500 according to the output result of the tracking unit 260 acquired from the output unit 262. For example, when the line of sight of the user 500 is directed to the left with respect to the torso, the content reproduction unit 270 causes the presentation unit 218 to display a 3D image corresponding to the left direction. The content reproduction unit 270 can be realized using the resources of the storage unit 212 and the CPU 222.

  The synthesizing unit 280 synthesizes the video generated by the content reproduction unit 270 and the video captured by the case imaging unit 204. Specifically, the synthesis unit 280 acquires the line-of-sight direction of the user 500 from the output unit 262. When the acquired line-of-sight direction of the user is the vertical downward direction, that is, the direction of the user's 500 hand or foot, the synthesizing unit 280 converts the visible light image captured by the housing imaging unit 204 into the video generated by the content reproduction unit 270. Synthesize. Video composition by the composition unit 280 can be realized by the CPU 222 executing a program for realizing a known image processing technique such as alpha blending. As another means for acquiring the hand direction, the direction of the position-designated light emitting element 404 with respect to the coordinate system 10 of the head mounted display 200 that can be acquired by hand tracking described later can be used.

  The head mounted display 200 according to the embodiment is configured to be positioned in front of both eyes of the user 500 when the head mounted display 200 is worn on the head of the user 500 and to block the surrounding visual field. Since the user 500 observes the virtual three-dimensional video presented by the presentation unit 218, the user 500 can be provided with a higher immersive feeling and presence. On the other hand, when the synthesizing unit 280 synthesizes a real image near the user's hand or under the foot, the user 500 can easily perform a hand operation or walking while wearing the head mounted display 200, and improve usability. be able to. Note that whether or not the combining unit 280 combines the images captured by the housing imaging unit 204 may be switched by pressing the first button 202, for example.

  FIG. 5 is a flowchart showing a flow of tracking processing by the video presentation system 100 according to the embodiment. The processing in this flowchart starts when the power of the head mounted display 200 is turned on, for example.

  The case imaging unit 204 captures and images the light emitted from the position designation light emitting element 404 of the position designation device 400 (S2). The hand tracking unit 700 analyzes the image captured by the housing imaging unit 204 and detects the hand movement of the user 500 (S4).

  The casing light emitting element 210 irradiates light in the vertically downward direction to the head mounted display 200 under the control of the I / O control unit 232 (S6). The torso imaging device 302 of the torso fixing device 300 captures the light emitted from the casing light emitting element 210 in the vertically downward direction and acquires an image (S8). The head tracking unit 600 analyzes the image of the torso imaging element 302 acquired by the first image acquisition unit 252 and detects the relative movement of the head mounted display 200 with respect to the torso imaging element 302 (S10).

  When the head tracking unit 600 detects the relative movement of the head mounted display 200 with respect to the torso image sensor 302, the processing in this flowchart ends. For convenience of explanation, the case where the head tracking detection by the head tracking unit 600 is performed after the hand movement detection by the hand tracking unit 700 has been described. However, the hand movement detection is performed after the head movement detection. It may be executed or both may be executed in parallel.

(Hand tracking)
The overall configuration of the video presentation system 100 and the outline of the tracking process have been described above. Next, hand tracking according to the embodiment will be described.

  FIG. 6 is a diagram schematically illustrating a configuration used for hand tracking in the video presentation system 100 according to the embodiment. As shown in FIG. 6, in the hand tracking according to the embodiment, a head mounted display 200 and a position specifying device 400 are used.

  FIG. 7 is a diagram schematically showing a use scene of hand tracking of the video presentation system 100 according to the embodiment. In the hand tracking according to the embodiment, the direction of the position-designated light-emitting element 404 with respect to the coordinate system 10 (Hx, Hy, Hz) of the head mounted display 200 is acquired as a set of pitch and yaw. The coordinate system 10 shown in FIG. 7 uses the position of the housing imaging unit 204 as the origin, sets the Hy axis vertically upward, and sets the Hx axis in the direction from the left hand side to the right hand side of the user 500's head. This is a coordinate system in which the Hz axis is set in a direction from the front to the back of the user's 500 head. In FIG. 7, the body fixing device 300 is omitted.

  FIG. 8 is a diagram schematically illustrating an example of an image 256 captured by the housing imaging unit 204 according to the embodiment. In FIG. 8, a point O is a center point of an image 256 captured by the housing imaging unit 204, and a point P is a bright spot derived from the position designation light emitting element 404. As described above, the case imaging unit 204 images using the equidistant projection method via the fisheye lens 234. Therefore, when the position-designated light emitting element 404 exists on the optical axis of the housing imaging unit 204, the bright spot P exists at the position of the center point O of the image 256.

  The longer the distance from the position-designated light emitting element 404 to the optical axis of the housing imaging unit 204, that is, the length of the perpendicular line drawn from the position-designated light emitting element 404 onto the optical axis of the housing imaging unit 204, the brighter the image 256. The distance between the point P and the center point O becomes longer. When the position-designated light-emitting element 404 is moved while keeping the distance from the position-designated light-emitting element 404 to the optical axis of the housing imaging unit 204 constant, the bright spot P in the image 256 is an arc centered on the center point O. Move up.

  From this, the pitch of the position-designated light emitting element 404 with respect to the coordinate system 10 can be acquired by measuring the length L of the vector when the line segment OP is the radial vector. Further, the yaw of the position-designated light emitting element 404 with respect to the coordinate system 10 can be acquired by measuring the deflection angle θ of the radial vector.

  FIG. 9 is a diagram schematically illustrating a functional configuration of the hand tracking unit 700 according to the embodiment. The hand tracking unit 700 includes a bright spot detection unit 710, a vector acquisition unit 720, a pitch acquisition unit 730, a yaw acquisition unit 740, and a gesture recognition unit 750.

  The bright spot detection unit 710 acquires an image captured by the housing imaging unit 204 from the second image acquisition unit 254. The bright spot detection unit 710 detects the bright spot P derived from the position designation light emitting element 404 from the acquired image. As described above, the position-designated light-emitting element 404 changes whether it is constantly lit or blinking according to the on / off state of the second button 406. Therefore, the bright spot detection unit 710 also detects whether the detected bright spot is always lit or blinking, and also detects whether the second button 406 has been pressed. The bright spot detection unit 710 can be realized using a known image processing technique such as pattern matching. Details of the detection process of whether or not the second button 406 is pressed by the bright spot detection unit 710 will be described later.

  The vector acquisition unit 720 acquires a radial vector starting from the bright spot P detected by the bright spot detection unit 710 and starting from the center point O of the image 256 captured by the housing imaging unit 204. The pitch acquisition unit 730 acquires the pitch of the position-designated light emitting element 404 with respect to the coordinate system 10 with the position of the housing imaging unit 204 as the origin from the length L of the radial vector acquired by the vector acquisition unit 720. The yaw acquisition unit 740 acquires the yaw of the position-designated light emitting element 404 with respect to the coordinate system 10 with the position of the housing imaging unit 204 as the origin, from the deviation angle θ of the radial vector acquired by the vector acquisition unit 720.

  The gesture recognizing unit 750 acquires an image captured by the second image sensor 208 in the housing image capturing unit 204. As described above, the second image sensor 208 images light in the visible light region. For example, when the user 500 does not have the position specifying device 400, the second imaging element 208 images the hand of the user 500. The gesture recognition unit 750 recognizes the shape of the hand of the user 500 imaged by the second image sensor 208 and outputs the information to the output unit 262.

  As an example, it is assumed that, for example, a gesture of making a ring with the thumb and forefinger corresponds to the second button 406 being turned on, and the fist is made to correspond to the second button 406 being turned off. At this time, when the gesture recognizing unit 750 detects a gesture that forms a ring with the thumb and index finger in the image captured by the second image sensor 208, the bright spot detecting unit 710 detects that the second button 406 is turned on. A similar signal is output. Similarly, when the gesture recognizing unit 750 detects a fist in the image captured by the second image sensor 208, the same signal as that when the bright spot detecting unit 710 detects that the second button 406 is turned on is output.

  Thereby, even when the user 500 does not have the position specifying device 400, the same information as when the user 500 presses the second button 406 can be acquired. The gesture recognition unit 750 may further recognize gestures other than those described above. As a result, more information can be acquired from the user 500 than when the second button 406 is pressed. The gesture recognition unit 750 can be realized by using a known image recognition technique, such as using pattern recognition using machine learning. The gesture recognition unit 750 and the use of the position specifying device 400 may be used in combination. The gesture recognizing unit 750 can also recognize the locus of hand tracking as a gesture by acquiring an image captured by the first image sensor 206 in the housing image capturing unit 204.

  FIG. 10 is a flowchart showing the flow of the hand tracking unit process according to the embodiment, and shows in detail step S5 in FIG.

  The second image acquisition unit 254 acquires an image captured by the housing imaging unit 204 from the infrared light emitted by the position-designated light emitting element 404 captured by the housing imaging unit 204 (S402). The bright spot detection unit 710 detects a bright spot derived from infrared light in the image acquired from the second image acquisition unit 254 (S404).

  The vector acquisition unit 720 acquires a vector having the bright spot detected by the bright spot detection unit 710 as an end point and the center of the image captured by the casing imaging unit 204 as a start point (S406). The pitch acquisition unit 730 acquires the length of the vector acquired by the vector acquisition unit 720 (S408). The yaw acquisition unit 740 acquires the deflection angle of the vector acquired by the vector acquisition unit 720 (S410).

(Head tracking)
The hand tracking according to the embodiment has been described above. Next, head tracking according to the embodiment will be described.

  FIG. 11 is a diagram schematically illustrating a configuration used for head tracking in the video presentation system 100 according to the embodiment. As shown in FIG. 11, in the hand tracking according to the embodiment, a head mounted display 200 and a body fixing device 300 are used.

  FIG. 12 is a diagram schematically showing a use scene for head tracking of the video presentation system 100 according to the embodiment. In the head tracking according to the embodiment, the relative posture angle of the coordinate system 10 (Hx, Hy, Hz) of the head-mounted display 200 with respect to the coordinate system 12 (Bx, By, Bz) of the torso fixing device 300 ( Hpitch, Hyaw, Hroll) changes are detected. Here, H pitch represents the pitch of the coordinate system 10 with respect to the coordinate system 12, Hyaw represents the yaw of the coordinate system 10 with respect to the coordinate system 12, and Hroll represents the roll of the coordinate system 10 with respect to the coordinate system 12. A change in the coordinate system 10 of the head mounted display 200 with respect to the coordinate system 12 of the torso fixing device 300 corresponds to a change in the position of the head with respect to the torso of the user 500.

  The coordinate system 12 shown in FIG. 12 uses the position of the torso imaging element 302 as the origin, sets the By axis upward in the vertical direction, sets the Bx axis in the direction from the left hand side to the right hand side of the user 500, and This is a coordinate system in which the Bz axis is set in the direction from the front to the back. In FIG. 12, the position specifying device 400 is omitted.

  In the coordinate system 12 of the trunk fixing device 300, the By axis is parallel to the direction of gravity, and the Bx axis and the Bz axis are perpendicular to the direction of gravity. Similarly, when the user 500 is facing the front, the Hy axis in the coordinate system 10 of the head mounted display 200 is parallel to the gravity direction, and the Hx axis and the Hz axis are perpendicular to the gravity direction.

  As indicated by the coordinate system 12 of the torso fixing device 300, when the user 500 moves the neck up and down, that is, when looking up at the floor or looking up at the ceiling, the pitch of the coordinate system 10 of the head-mounted display 200 with respect to the coordinate system 12 Changes. Further, when the user 500 tilts his / her neck to the left / right, that is, when the user 500 moves his / her neck, the roll of the coordinate system 10 with respect to the coordinate system 12 changes. When the pitch or roll of the coordinate system 10 with respect to the coordinate system 12 changes, the Hx axis and the Hz axis of the coordinate system 10 shift from being perpendicular to the direction of gravity. By detecting this “deviation” using the acceleration sensor 224 in FIG. 3, the pitch or roll of the coordinate system 10 relative to the coordinate system 12 can be detected.

  On the other hand, it is difficult to detect the change in the yaw of the coordinate system 10 relative to the coordinate system 12 from the change in the direction of gravity. For example, the yaw of the coordinate system 10 relative to the coordinate system 12 changes when the user 500 turns left and right while keeping his neck vertical. However, the Hy axis in the coordinate system 10 does not change while keeping parallel to the direction of gravity. Therefore, the video presentation system 100 according to the embodiment detects the change in the yaw of the coordinate system 10 relative to the coordinate system 12 by capturing the light emitted from the housing light emitting element 210 with the torso imaging element 302 and analyzing the image. To do. Hereinafter, the principle of detecting a change in the yaw of the coordinate system 10 with respect to the coordinate system 12 from the image captured by the trunk image sensor 302 will be described.

  FIG. 13 is a diagram schematically illustrating an example of an image 308 captured by the torso imaging device 302 according to the embodiment. Specifically, a locus of bright spots derived from the case light emitting element 210 when the user 500 turns the head to the left and right with the head mounted display 200 attached is shown. In general, the angle at which a human can turn his / her neck left and right is said to be around 60 degrees on each side. Therefore, when the user 500 turns the head to the left and right with the head mounted display 200 attached, the locus of the bright spot derived from the housing light emitting element 210 becomes a fan-shaped arc with a central angle of around 120 degrees. .

  In FIG. 13, a point P indicates a bright spot derived from the casing light emitting element 210 when the user 500 is facing the front. Point Q indicates a bright spot derived from the casing light emitting element 210 when the user 500 turns the neck to the right as much as possible. A point R indicates a bright spot derived from the housing light emitting element 210 when the user 500 turns the neck to the left as much as possible. Assuming that the locus of the bright spot derived from the casing light emitting element 210 draws an arc, the three points of the point P, the point Q, and the point R exist on the same circumference. In general, since a circle passing through three different points is uniquely determined, the arc can be identified by detecting the point P, the point Q, and the point R.

  Here, when the user 500 turns the neck left and right with the neck bent forward or backward, the radius of the arc drawn by the locus of the bright spot derived from the housing light emitting element 210 in the image 308 is the front of the user 500. It becomes shorter compared to the case of facing. This is because when the user 500 bends the neck forward or backward, the housing light emitting element 210 approaches the neck of the user 500 serving as the rotation axis.

  An arc CD in FIG. 13 illustrates a locus when the user 500 turns the neck left and right with the neck bent forward or backward. From this, the inventor of the present application has found that there is a “existing region of the casing light emitting element 210” in which a bright spot derived from the casing light emitting element 210 exists in the image 308. More specifically, it is an area surrounded by an arc having a central angle passing through point P, point Q, and point R in FIG. The region where the housing light emitting element 210 exists is a region surrounded by a line segment AB and an arc AB in FIG. Note that the point O in FIG. 13 is the center point of the arc.

  As described above, the trunk fixing device 300 includes the trunk imaging device 302 and the element support portion 314 that supports the trunk imaging device 302. The element support unit 314 also includes an adjustment unit 312 that adjusts the inclination of the body image sensor 302. When the user 500 uses the video presentation system 100, the torso imaging is performed using the adjustment unit 312 so that the “area where the casing light emitting element 210 exists” is included in the image 308 captured by the torso imaging element 302. Calibration for adjusting the tilt of the element 302 is executed. Thereby, the possibility that a bright spot derived from the housing light emitting element 210 exists in the image 308 can be increased.

  FIG. 14 is a diagram schematically illustrating a functional configuration of the head tracking unit 600 according to the embodiment. The head tracking unit 600 according to the embodiment includes an area setting unit 610 and a bright spot analysis unit 620. The region setting unit 610 acquires the image 308 captured by the trunk image sensor 302 from the first image acquisition unit 252. When the “existing area of the casing light emitting element 210” has already been determined, the area setting unit 610 sets the existing area of the casing light emitting element 210 in the image 308.

  The bright spot analysis unit 620 analyzes the image 308 and detects the yaw of the housing light emitting element 210 with respect to the body image sensor 302. Details of the bright spot analysis unit 620 will be described below.

  The bright spot analysis unit 620 includes a bright spot selection unit 630, a locus detection unit 640, a fitting unit 650, a vector acquisition unit 660, and a yaw acquisition unit 670. When there are a plurality of bright spots in the image 308, the bright spot selection unit 630 selects a bright spot derived from the casing light emitting element 210 from the plurality of bright spots.

  The case where there are a plurality of bright spots in the image 308 may be, for example, bright spots derived from the position-designated light emitting element 404 in addition to the bright spots derived from the casing light emitting element 210. Details of the sorting process by the bright spot sorting unit 630 will be described later.

  The locus detection unit 640 detects the locus of the bright spot selected by the bright spot selection unit 630 in the image 308 captured by the torso image sensor 302. The fitting unit 650 acquires an arc that fits the trajectory detected by the trajectory detection unit 640. The fitting unit 650 outputs the acquired arc to the region setting unit 610 as the existence region of the housing light emitting element 210 when executing calibration for adjusting the tilt of the torso image sensor 302.

  The vector acquisition unit 660 acquires a radial vector having the center point of the arc acquired by the fitting unit 650 as a start point and the bright point selected by the bright point selection unit 630 as an end point. Similarly to the yaw acquisition unit 740 in the hand tracking unit 700, the yaw acquisition unit 670 acquires the deflection angle of the radial vector acquired by the vector acquisition unit 660, and the housing light emitting element 210 with respect to the trunk image sensor 302. Detect yaw.

  FIG. 15 is a flowchart showing the flow of the head tracking unit process according to the embodiment, and shows in detail step S10 in FIG.

  The region setting unit 610 sets the presence region of the casing light emitting element 210 in the image 308 captured by the trunk image sensor 302 (S102). When there are a plurality of bright spots in the area set by the area setting section 610, the bright spot selecting section 630 identifies a bright spot caused by the casing light emitting element 210 (S104). The locus detection unit 640 detects the locus of the bright spot selected by the bright spot selection unit 630 in the image 308 captured by the torso image sensor 302 (S106).

  The fitting unit 650 acquires an arc that fits the trajectory detected by the trajectory detection unit 640 (S108). The vector acquisition unit 660 acquires a radial vector starting from the center point of the arc acquired by the fitting unit 650 and having the bright spot selected by the bright spot selection unit 630 as an end point (S110). The yaw acquisition unit 670 acquires the deflection angle of the radial vector acquired by the vector acquisition unit 660 (S112).

(Combination of hand tracking and head tracking)
The head tracking according to the embodiment has been described above. Next, a case where hand tracking and head tracking according to the embodiment are used together will be described.

(Positional relationship between the casing light emitting element 210 and the casing imaging unit 204)
FIG. 16 is an enlarged view of a part of the head mounted display 200 according to the embodiment, and is a diagram illustrating a positional relationship between the housing light emitting element 210 and the housing imaging unit 204.

  As described above, the first image sensor 206 and the second image sensor 208 in the housing imaging unit 204 are provided in the slope region of the housing of the head mounted display 200. In FIG. 16, the slope area is an area indicated by a line segment and a diagonal line connecting the reference sign S1 and the reference sign S2. Here, the housing imaging unit 204 is intended to image the position designation light emitting element 404. Therefore, it is preferable that the housing imaging unit 204 does not capture the light emitted by the housing light emitting element 210 used for head tracking.

  Therefore, the housing of the head mounted display 200 is configured to shield between the housing imaging unit 204 and the housing light emitting element 210. Specifically, as indicated by reference signs S <b> 1 and S <b> 3 in FIG. 16, at least a part of the casing is configured to exist between the casing imaging unit 204 and the casing light emitting element 210. As a result, it is possible to reduce that the light emitted from the housing light emitting element 210 enters the angle of view of the housing imaging unit 204 or at least enters the existence area of the housing light emitting element 210 set by the region setting unit 610. be able to.

  Further, as described above, the housing light emitting element 210 is intended to detect a change in yaw of the coordinate system 10 of the head mounted display 200 with respect to the coordinate system 12 of the torso fixing device 300. Here, the fitting unit 650 performs circle fitting when detecting a change in yaw. Generally, the larger the radius of the circle to be fitted, the higher the accuracy of the fitting. Therefore, it is preferable that the housing light emitting element 210 be positioned as far as possible from the head of the user 500 when the user 500 wears the head mounted display 200.

  Therefore, when the head mounted display 200 is mounted on the head of the user 500, the housing light emitting element 210 is installed forward of the user's 500 line of sight compared to the installation position of the housing imaging unit 204. Further, as described above, since the housing light emitting element 210 is configured to exist between the housing imaging unit 204 and the housing light emitting element 210, the housing light emitting element 210 has a vertical direction compared to the installation position of the housing imaging unit 204. Installed above. Thereby, the radius of the circular arc drawn by the locus of the bright spot derived from the casing light emitting element 210 in the image 308 captured by the trunk image sensor 302 can be increased, and the accuracy of the circular fitting by the fitting unit 650 is increased. It becomes possible.

(Bright spot selection)
Next, the bright spot selection process by the bright spot selection unit 630 will be described.

  As described above, the bright spot analysis unit 620 detects the movement of the case light emitting element 210 by analyzing the movement of the bright spot existing in the region where the case light emitting element 210 exists set by the area setting unit 610. When there are a plurality of bright spots in the region where the body light emitting element 210 exists, first, the bright spots derived from the casing light emitting element 210 are selected. For example, when hand tracking and head tracking are used together, a bright spot derived from the position-designated light emitting element 404 may be reflected in the region where the housing light emitting element 210 exists.

  Therefore, the bright spot selecting unit 630 uses the housing light emitting element based on the size of the plurality of bright spots detected in the existence region of the housing light emitting element 210 set in the image 308 captured by the trunk image sensor 302. The bright spots derived from 210 are selected.

  The light emitted from the position designation light emitting element 404 enters the angle of view of the torso image sensor 302 when the user 500 moves the position designation device 400 between the head mounted display 200 and the torso image sensor 302. it is conceivable that. In this case, the distance from the body image sensor 302 to the housing light emitting element 210 is longer than the distance from the body image sensor 302 to the position designation light emitting element 404. Here, the size of the housing light emitting element 210 according to the embodiment is the same as the size of the position specifying light emitting element 404. For this reason, when a bright spot derived from the casing light emitting element 210 and a bright spot derived from the position-designated light emitting element 404 are simultaneously present in the region where the casing light emitting element 210 exists, the bright spot derived from the casing light emitting element 210 is present. The size of the point is smaller than the size of the bright spot derived from the position designation light emitting element 404.

  Therefore, the bright spot selection unit 630 compares the sizes of a plurality of bright spots detected in the region where the housing light emitting element 210 exists, and selects a small bright spot as a bright spot derived from the housing light emitting element 210. Thereby, it is possible to easily determine the bright spot derived from the casing light emitting element 210 from the bright spot existing in the region where the casing light emitting element 210 exists.

  FIG. 17 is a flowchart showing a flow of the bright spot selection processing by the bright spot selection unit 630 according to the embodiment, and is a diagram for explaining step 104 in FIG. 15 in detail.

  The bright spot selection unit 630 checks whether there is a bright spot in the region where the housing light emitting element 210 set by the area setting unit 610 exists. If no bright spot exists (N in S20), the bright spot is not detected. (S38). When a bright spot exists (Y of S20), the bright spot selection unit 630 checks whether or not the number of bright spots in the presence region of the housing light emitting element 210 is two or more. When the number of bright spots is not 2 or more, that is, when the number of bright spots is 1 (N in S22), the bright spot selection unit 630 selects the detected bright spot as a bright spot derived from the casing light emitting element 210 ( S36).

  When the number of bright spots is 2 or more (Y in S22), the bright spot selection unit 630 checks the magnitude relationship of the detected bright spot sizes. When there is a magnitude relationship (Y in S24), the bright spot selection unit 630 selects the minimum bright spot as the bright spot derived from the casing light emitting element 210 (S34). When there is no magnitude relationship (N in S24), the bright spot selecting unit 630 obtains the distance to each bright spot from the position of the bright spot selected as the bright spot derived from the previous case light emitting element 210. When the obtained distance has a magnitude relationship (Y in S28), the bright spot selection unit 630 selects a bright spot located closest to the previous bright spot as a bright spot derived from the casing light emitting element 210 (S362). .

  When the obtained distance is not related to the magnitude (N in S28), the bright spot selecting unit 630 acquires the coordinates of the center of gravity of the position of each bright spot, and the bright spot derived from the housing light emitting element 210 exists at the coordinate position. This is considered (S30).

(Detection of signal from position specifying device 400)
The bright spot selection process by the bright spot selection unit 630 has been described above. Next, a detection process of whether or not the second button 406 of the position specifying device 400 is pressed by the bright spot detection unit 710 will be described.

  As described above, the position-designated light emitting element 404 is always lit when the second button 406 is on, and blinks at a predetermined blinking frequency F when the second button 406 is off. The bright spot selection unit 630 detects whether or not the second button 406 is pressed by analyzing the presence or absence of a bright spot derived from the position-designated light emitting element 404 in the image 256 captured by the housing imaging unit 204.

  More specifically, the housing imaging unit 204 analyzes an image 256 acquired by imaging at a frequency 2F that is at least twice the blinking frequency F of the position designation light emitting element 404. The case imaging unit 204 captures an image 256 at a frequency at least twice the blinking frequency of the position-designated light emitting element 404 and analyzes the image 256 for two consecutive frames, whereby the bright spot detection unit 710 performs the position designation. It is known that it is possible to identify whether the light emitting element 404 is always lit, blinks at a frequency F (period T = 1 / F), or no image exists in the image 256. The blinking frequency F may be determined by experiment in consideration of the time resolution of the housing imaging unit 204, and is 120 [Hz] as an example. In this case, when the second button 406 is off, the position-designated light emitting element 404 is repeatedly turned on and off every 1/240 seconds. The housing imaging unit 204 images a subject including the position designation light emitting element 404 at least at 240 [Hz].

  FIG. 18 is a flowchart showing a flow of processing for detecting a signal generated by the position specifying device 400 according to the embodiment, and is a diagram showing control of the position specifying light-emitting element 404.

  The I / O control unit 408 in the position specifying device 400 initializes the toggle (S40). Here, the “toggle” is a binary value indicating whether the second LED is turned on or off when the second button 406 is not depressed. In the embodiment, the toggle is inverted every 1/240 seconds and stored in a temporary recording unit (not shown) of the I / O control unit 408.

  The I / O control unit 408 confirms the on / off state of the second button 406 (S42). When the second button 406 is pressed (Y in S44), the I / O control unit 408 turns on the second LED 410 that is the position designation light emitting element 404 (S50). When the second button 406 is not pressed (N in S44) and the toggle is not 0 (N in S46), the I / O control unit 408 turns on the second LED 410 that is the position specifying light emitting element 404 (S50). .

  When the second button 406 is not pressed (N in S44) and the toggle is 0 (Y in S46), the I / O control unit 408 turns off the second LED 410 that is the position designation light emitting element 404 (S48). Thereafter, when the toggle is 0 (Y in S52), the I / O control unit 408 inverts the toggle to 1 (S56). If the toggle is not 0 (N in S52), the I / O control unit 408 inverts the toggle to 0 (S56). After inverting the toggle, the process returns to step S42 and the above process is repeated.

  The I / O control unit 408 repeats the above processing from step S42 to steps S54 and S56, for example, every 1/240 seconds. Accordingly, the I / O control unit 408 as a whole blinks the second LED 410 at a cycle of 120 [Hz] when the second button 406 is off, and continues the second LED 410 when the second button 406 is on. Can be realized.

  FIG. 19 is a state transition diagram showing an internal state of the bright spot detection unit 710 when a signal generated by the position specifying device 400 according to the embodiment is detected, and is a diagram for explaining the detection process of the bright spot detection unit 710. It is.

  The bright spot detection unit 710 changes the value of the “pressing flag” in accordance with the detected signal while detecting the signal emitted by the position specifying device 400 based on the image captured by the housing imaging unit 204. Here, the “press flag” is a working memory used to detect the presence / absence of a bright spot derived from the position-designated light emitting element 404 and the on / off state of the second button 406 when a bright spot is detected. Stored in the storage unit 212.

  More specifically, the bright spot detection unit 710 acquires and analyzes the image 256 captured by the housing imaging unit 204 for each frame. When the bright spot detection unit 710 detects that the bright spot derived from the position-designated light emitting element 404 is blinking, that is, when it is identified that the second button 406 is off, the value of the push flag is set to 0. To do. When the bright spot detection unit 710 detects that the bright spot derived from the position-designated light emitting element 404 is always on, that is, when it is identified that the second button 406 is on, the value of the pressing flag is set to 1. Set to. The bright spot detection unit 710 further sets the value of the pressing flag to 2 when the bright spot derived from the position-designated light emitting element 404 cannot be detected in the image captured by the casing imaging unit 204.

  Here, the value of the pressing flag corresponds to the internal state of the bright spot detection unit 710. That is, when the value of the pressing flag is 0, the bright spot detection unit 710 is in the off identification state (ST0). When the value of the pressing flag is 1, the bright spot detection unit 710 is in the on identification state (ST1), and when the value of the pressing flag is 2, the bright spot detection unit 710 is in the identification unknown state (ST2). The bright spot detection unit 710 analyzes the image 256 for two consecutive frames captured by the housing imaging unit 204 and changes the internal state according to the result.

  For example, assume that when the internal state of the bright spot detection unit 710 is in the off-identification state (ST0), a bright spot derived from the position-designated light emitting element 404 is detected in both images 256 for two consecutive frames. In this case, the bright spot detection unit 710 changes the value of the push flag from 0 to 1, and transitions the internal state to the on identification state ST1. This is because it is considered that the luminescent spots are always lit when both detect the bright spots in the images 256 for two consecutive frames.

  In FIG. 19, an arrow from the off state identification ST0 to the on identification state ST1 and (P → P) are used. Here, the symbol “P” represents the presence of a bright spot in the image 256, and is an acronym for English Present indicating the presence. (P → P) indicates an event in which a bright spot is detected in a certain image 256 and a bright spot is detected in a subsequent image 256.

  When the internal state of the bright spot detection unit 710 is the off identification state (ST1), it is assumed that a bright spot is detected in an image 256 that continues from a state in which no bright spot can be detected in the images 256 for two consecutive frames. In this case, the bright spot detection unit 710 changes the value of the push flag from 1 to 0 and changes the internal state to the on identification state ST1. In FIG. 19, an arrow and an event (A → P) from the on-state identification ST1 to the off-identification state ST0 are shown. Here, the symbol “A” represents the absence of a bright spot in the image 256, and is an acronym for English Absent that means it does not exist. The event (A → P) indicates that a bright spot is not detected in a certain image 256 and a bright spot is detected in the subsequent image 256. Similarly, the event (A → A) represents a case where no bright spot is detected in the image 256 for two consecutive frames, and the event (P → A) is detected as a bright spot in an image 256. In the subsequent image 256, no bright spot is detected.

  Similarly, when the internal state of the bright spot detection unit 710 is the off identification state (ST1) and the event (A → A) occurs, the bright spot detection unit 710 changes the value of the push flag from 1 to 2, The state transitions to the identification unknown state ST2. This is because if no bright spot is detected in the images 256 for two consecutive frames, the bright spot is not present in the image 256 and it is considered that the bright spot has been lost. When the internal state of the bright spot detection unit 710 is the identification unknown state (ST2) and the event (P → P) occurs, the bright spot detection unit 710 changes the value of the push flag from 2 to 1, and turns on the internal state Transition to the identification state ST2.

  When the internal state of the bright spot detection unit 710 is the identification unknown state (ST2), if an event (P → A) occurs, the bright spot detection unit 710 changes the value of the push flag from 2 to 0 and turns off the internal state Transition to the identification state ST2. When the internal state of the bright spot detection unit 710 is the off identification state (ST0) and the event (A → A) occurs, the bright spot detection unit 710 changes the value of the push flag from 0 to 2, and changes the internal state. Transition to the identification unknown state ST2.

  As described above, the bright spot detection unit 710 can identify the presence / absence of a bright spot derived from the second button 406 and the on / off state when the bright spot is detected.

  As described above, according to the video presentation system 100 according to the embodiment, it is possible to provide a technique for detecting the movement of the user wearing the head mounted display.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

(First modification)
In the above, based on the size of the plurality of bright spots detected by the bright spot selection unit 630 in the existence region of the casing light emitting element 210 set in the image 308 captured by the torso imaging element 302, the casing The case where the bright spot derived from the light emitting element 210 is selected has been described. The method of selecting bright spots by the bright spot selecting unit 630 is not limited to this. In addition to this, for example, a difference in wavelength of light emitted by a light emitting element that is a source of a bright spot may be used. This case will be described below.

  The wavelength of light emitted from the housing light emitting element 210 according to the first modification is different from the wavelength of light emitted from the position specifying light emitting element 404. Therefore, the body imaging element 302 captures an image through an optical filter in which the transmittance of the light emitted from the housing light emitting element 210 is larger than the transmittance of the light emitted from the position designation light emitting element 404. As a result, even if a bright spot derived from the housing light emitting element 210 and a bright spot derived from the position-designated light emitting element 404 exist in the region where the housing light emitting element 210 exists, the brightness value of these bright spots is different. Appears.

  Specifically, for example, when imaging is performed such that the luminance value of the body imaging element 302 increases as the light amount increases, the luminance value of the bright spot derived from the housing light emitting element 210 is the position designation light emitting element 404. It becomes larger than the luminance value of the bright spot derived from. This is because the light emitted from the position designation light emitting element 404 is attenuated by the optical filter. Therefore, the bright spot selection unit 630 compares the brightness values of the two bright spots in the image, and selects the bright spot having the higher brightness value as the bright spot derived from the casing light emitting element 210.

(Second modification)
In the above description, the case where the bright spot selection unit 630 selects the bright spot located closest to the previous bright spot as the bright spot derived from the casing light emitting element 210 has been described. If a bright spot derived from is detected, the current bright spot position may be predicted from that position. This can be realized, for example, by sampling the deviation angle of the radial vector acquired by the yaw acquisition unit 670 and estimating the angular velocity of the bright spot. As described above, since the bright spot derived from the housing light emitting element 210 can be expected to draw an arc-shaped locus, the current bright spot position can be accurately estimated from the previous bright spot position and the angular velocity of the bright spot. .

  Unlike the case where the optical filter described above is used, for example, even if the wavelength of light emitted from a television remote control or the like is close to the wavelength of light emitted from the casing light emitting element 210, the current casing light emitting element 210 This is advantageous in that the position of the bright spot derived from it can be accurately estimated.

(Third Modification)
In the above description, the case where the housing imaging unit 204 includes the first imaging element 206 that images light in the invisible light region such as infrared light and the second imaging element 208 that images light in the visible light region has been described. The housing imaging unit 204 may use a single imaging element that can simultaneously capture light in the visible light region and the invisible light region. Thereby, the weight of the head mounted display 200 can be reduced, and the burden on the user 500 wearing the head mounted display 200 can be reduced.

  The invention according to the present embodiment may be specified by the items described below.

(Item 1-1)
A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the viewer's eyes when worn on the viewer's head;
An image sensor that is provided in the housing and images light outside the housing;
The image pickup device picks up light vertically downward with respect to the housing and in front of the viewer's line of sight when the housing is mounted on an observer's head. .
(Item 1-2)
Of the surfaces constituting the housing, at least a part of the lower surface positioned vertically downward with respect to the housing when the housing is mounted on the observer's head is directed vertically upward. It has a slope area that slopes,
Item 11. The head mounted display according to Item 1-1, wherein the image sensor is provided in the slope region.
(Item 1-3)
Item 11. The head-mounted display according to Item 1-1 or Item 1-2, wherein the imaging device acquires light from the outside of the housing through a fisheye lens and images it.
(Item 1-4)
The head-mounted display according to any one of Items 1-1 to 1-3, wherein the imaging device images light in an invisible light region.
(Item 1-5)
The imaging element images light emitted from a light emitting element that moves in conjunction with the movement of an observer's hand,
Item 5. The head mounted display according to item 1-4, further comprising a hand tracking unit that analyzes an image captured by the image sensor and tracks a change in position of the light emitting element.
(Item 1-6)
Item 6. The head mounted display according to item 1-5, wherein the light emitting element that moves in conjunction with the movement of the observer's hand is attached to a gripping part that is gripped by the observer.
(Item 1-7)
The hand tracking unit is
A bright spot detection unit for detecting a bright spot of light in an image captured by the image sensor;
A vector acquisition unit for acquiring a vector starting from the bright spot detected by the bright spot detection unit and starting from the center of the image captured by the image sensor;
A pitch acquisition unit that sets the length of the vector acquired by the vector acquisition unit as a pitch of the light emitting element with respect to the imaging element;
Item 1-5 or Item 1-6, further comprising: a yaw acquisition unit that sets a deflection angle of the vector acquired by the vector acquisition unit as a yaw of the light emitting element with respect to the imaging element. Head mounted display.
(Item 1-8)
The imaging device generates a visible light image by imaging light in a visible light region,
4. The head mounted display according to any one of items 1-1 to 1-3, further comprising an image composition unit that synthesizes the visible light image and a virtual image to be presented to the presentation unit.
(Item 1-9)
Item 9. The head mounted display according to item 1-8, further comprising a gesture recognition unit that recognizes an observer's gesture imaged by the image sensor.
(Item 1-10)
An image acquisition unit for acquiring an image obtained by imaging an infrared light emitted by a light emitting element that moves in conjunction with a movement of a user's hand, by an imaging element attached to the user's head;
A bright spot detection unit for detecting a bright spot of infrared light in an image captured by the image sensor;
A vector acquisition unit for acquiring a vector starting from the bright spot detected by the bright spot detection unit and starting from the center of the image captured by the image sensor;
A pitch acquisition unit that sets the length of the vector acquired by the vector acquisition unit as a pitch of the light emitting element with respect to the imaging element;
A motion detection apparatus comprising: a yaw acquisition unit that sets a vector deflection angle acquired by the vector acquisition unit as a yaw of the light emitting element with respect to the imaging element.
(Item 1-11)
Acquiring an image obtained by imaging infrared light emitted by a light emitting element that moves in conjunction with the movement of a user's hand with an imaging element attached to the user's head;
Detecting a bright spot of infrared light in the acquired image;
Obtaining a vector having a detected bright spot as an end point and a start point at the center of an image captured by the image sensor;
Obtaining the length of the obtained vector;
A motion detection method, characterized by causing a processor to execute a step of acquiring an argument of an acquired vector.
(Item 1-12)
A function of acquiring an image obtained by imaging an infrared light emitted by a light emitting element that moves in conjunction with the movement of a user's hand with an imaging element attached to the user's head;
A function to detect the bright spot of infrared light in the acquired image;
A function of obtaining a vector having a detected bright spot as an end point and a center of an image captured by the image sensor as a start point;
A function to acquire the length of the acquired vector;
A program for causing a computer to realize a function of acquiring an argument of an acquired vector.
(Item 1-13)
In a video presentation system including a head-mounted display and a light emitting element that is attached to a gripping part that is gripped by an observer and emits infrared light,
The head mounted display is
A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the viewer's eyes when worn on the viewer's head;
An imaging device that is provided in the housing and images infrared light emitted by the light emitting device that moves in conjunction with the movement of the gripping portion;
An image presentation system comprising: a motion tracking unit that analyzes an infrared light image captured by the image sensor and tracks a relative position change of the light emitting element with respect to the image sensor.

(Item 2-1)
A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the viewer's eyes when worn on the viewer's head;
A housing light emitting element that emits light vertically downward with respect to the housing when the housing is mounted on the head of an observer;
An image acquisition unit that acquires an image of light of the casing light emitting element, which is imaged by a trunk imaging element that is mounted on an observer's trunk and configured to capture light emitted by the casing light emitting element. When,
A head-mounted display, comprising: a head tracking unit that analyzes a light image acquired by the image acquisition unit and tracks a relative movement of the housing light-emitting element with respect to the trunk image sensor.
(Item 2-2)
The housing further includes a housing imaging unit that images light emitted by the light emitting element that moves in conjunction with the movement of the observer's hand,
The case light emitting element is installed forward in the observer's line of sight and vertically above the installation position of the case imaging unit when the case is mounted on the head of the observer. ,
Item 2. The head mounted display according to Item 2-1, wherein the casing is configured to shield between the casing imaging unit and the casing light emitting element.
(Item 2-3)
The head-mounted display according to item 2-1 or item 2-2, wherein the head tracking unit detects a yaw of the housing light-emitting element with respect to the trunk image sensor.
(Item 2-4)
The head tracking unit includes:
An area setting unit for setting a presence area of the casing light emitting element defined as an area in which the casing light emitting element is imaged in an image captured by the trunk imaging element;
Item 2-1 to Item 2 further comprising a luminescent spot analysis unit for analyzing the movement of the luminescent spot existing in the region set by the region setting unit and detecting the movement of the casing light emitting element. The head mounted display in any one of -3.
(Item 2-5)
The bright spot analysis unit
A locus detection unit for detecting a locus of a bright spot of light in an image captured by the trunk image sensor;
A fitting unit that acquires an arc that fits the locus detected by the locus detector;
A vector acquisition unit for acquiring a radial vector starting from a center of a circle including the arc acquired by the fitting unit and having a bright spot on the image as an end point;
The head mounted display according to item 2-4, further including a yaw acquisition unit that acquires a deflection angle of the radial vector acquired by the vector acquisition unit.
(Item 2-6)
The bright spot analysis unit, when there are a plurality of bright spots in the region set by the region setting unit, at least one of the difference in the size of the bright spot in the image and the difference in the brightness value of the bright spot Item 6. The head mounted display according to Item 2-5, further including a luminescent spot selection unit for specifying a luminescent spot caused by the housing light emitting element.
(Item 2-7)
Item 7. The head mounted display according to item 2-6, wherein the bright spot selection unit compares the sizes of a plurality of bright spots in an image and selects a smaller bright spot.
(Item 2-8)
The wavelength of light emitted by the housing light emitting element is different from the wavelength of light emitted by the light emitting element that moves in conjunction with the movement of the observer's hand,
The trunk imaging element transmits an optical filter in which the transmittance of light emitted from the housing light-emitting element is larger than the transmittance of light emitted from the light-emitting element moving in conjunction with the movement of the observer's hand. The captured light,
Item 7. The head mounted display according to item 2-6, wherein the bright spot selection unit compares the brightness values of a plurality of bright spots in the image and selects a bright spot having a larger brightness value.
(Item 2-9)
The area setting unit sets a fan-shaped area centered on the center of a circle including the arc acquired by the fitting section as an existing area of the housing light emitting element. 9. The head mounted display according to any one of 8.
(Item 2-10)
A hand tracking unit that analyzes an image captured by the housing imaging unit and tracks a relative movement of the light emitting element that moves in conjunction with the movement of the observer's hand relative to the housing imaging unit; Item 2-2. The head mounted display according to item 2-2.
(Item 2-11)
The torso imaging device has an element support portion configured to be freely mounted on an observer's torso,
The head mounted display according to any one of Items 2-1 to 2-10, wherein the element support section includes an adjustment section that adjusts an inclination of the body image sensor.
(Item 2-12)
The head mounted display according to any one of Items 2-1 to 2-11, wherein the casing light emitting element emits light in an invisible light region.
(Item 2-13)
A head-mounted display; a torso imaging device configured to be communicable with the head-mounted display; and mounted on an observer's torso; and a light-emitting element that moves in conjunction with movement of the observer's hand of the head-mounted display; A motion detection method in a video presentation system including:
The processor of the head mounted display is
Imaging light emitted by a light emitting element that moves in conjunction with the movement of the observer's hand;
Analyzing the imaged light image of the light emitting element to detect relative movement of the light emitting element with respect to the head mounted display;
Irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on the head of an observer;
Obtaining an image obtained by imaging the light irradiated in the vertical downward direction by the torso imaging device;
Analyzing the acquired image by the torso image sensor and detecting a relative movement of the head mounted display with respect to the torso image sensor, and a motion detection method in a video presentation system.
(Item 2-14)
A head-mounted display; a torso imaging device configured to be communicable with the head-mounted display; and mounted on an observer's torso; and a light-emitting element that moves in conjunction with movement of the observer's hand of the head-mounted display; A program for realizing a motion detection function in a computer of a video presentation system including the motion detection function,
A function of irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on an observer's head;
A function of imaging light emitted by a light emitting element that moves in conjunction with the movement of the observer's hand;
A function of analyzing the image of the light emitted by the light emitting element that has been imaged and detecting the relative movement of the light emitting element with respect to the head mounted display;
A function of acquiring an image captured by the torso image sensor with respect to the light irradiated vertically downward;
A program for analyzing the acquired image of the torso image sensor and detecting a relative movement of the head mounted display with respect to the torso image sensor.

(Item 3-1)
In a video presentation system including a head mounted display, a first imaging device configured to be communicable with the head mounted display, and a first light emitting device that moves in conjunction with a movement of a user's hand of the head mounted display,
The head mounted display is
A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the user's eyes when worn on the user's head;
A second light emitting element that is provided in the housing and emits light vertically downward with respect to the housing when the housing is mounted on a user's head;
A second imaging element that is provided in the housing and images the light emitted by the first light emitting element;
The video presenting system, wherein the first image sensor is mounted on a user's torso and images the light emitted by the second light emitting element.
(Item 3-2)
Movement in a video presentation system including a head-mounted display, a first imaging device configured to be able to communicate with the head-mounted display, and a first light-emitting element that moves in conjunction with the movement of a user's hand of the head-mounted display A detection method,
The processor of the head mounted display is
Irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on a user's head;
Imaging the light emitted by the first light emitting element;
Analyzing the imaged light image emitted by the first light emitting element to detect relative movement of the first light emitting element with respect to the head mounted display;
Obtaining an image of the light irradiated in the vertically downward direction captured by the first image sensor mounted on a torso of a user;
Analyzing the acquired image by the first image sensor and detecting a relative movement of the head mounted display with respect to the first image sensor. 7. A motion detection method in a video presentation system, comprising:
(Item 3-3)
A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the user's eyes when worn on the user's head;
An image sensor that is provided in the housing and that images light emitted by the first light emitting element that moves in conjunction with the movement of the user's hand;
A second light emitting element that is provided in the housing and emits light vertically downward with respect to the housing when the housing is mounted on a user's head;
A head-mounted display comprising: an image acquisition unit that acquires an image of light emitted by the second light emitting element, which is imaged by an imaging element mounted on a user's torso.
(Item 3-4)
An image acquisition unit for acquiring an image detected by an image sensor mounted on the user's torso, the light emitted by the light emitting element that moves in conjunction with the movement of the user's head;
A locus detecting unit for detecting a locus of a bright spot of light in the image obtained by the image obtaining unit;
A fitting unit that acquires an arc that fits the locus detected by the locus detector;
A vector acquisition unit for acquiring a radial vector starting from a center of a circle including the arc acquired by the fitting unit and having a bright spot on the image as an end point;
A motion detection apparatus comprising: a yaw acquisition unit that uses a deflection angle of a radial vector acquired by the vector acquisition unit as a yaw of the light emitting element with respect to the imaging element.

  10, 12 coordinate system, 22 CPU, 100 video presentation system, 200 head mounted display, 202 first button, 204 housing imaging unit, 206 first imaging device, 208 second imaging device, 210 housing light emitting device, 212 storage Unit, 214 communication unit, 216 recording medium interface, 218 presentation unit, 220 audio output unit, 222 CPU, 224 acceleration sensor, 226 first LED, 228 power supply, 230 internal path, 232 I / O control unit, 234 fisheye lens, 250 image Acquisition unit, 252 first image acquisition unit, 254 second image acquisition unit, 260 tracking unit, 262 output unit, 270 content playback unit, 280 composition unit, 300 trunk fixing device, 302 trunk imaging device, 304 I / O Control unit, 30 6 third image sensor, 312 adjustment unit, 314 element support unit, 400 position designation device, 402 gripping unit, 404 position designation light emitting element, 406 second button, 408 I / O control unit, 410 second LED, 412 power supply, 500 User, 600 head tracking unit, 610 region setting unit, 620 bright spot analysis unit, 630 bright spot selection unit, 640 locus detection unit, 650 fitting unit, 660 vector acquisition unit, 670 yaw acquisition unit, 700 hand tracking unit, 710 bright spot detection unit, 720 vector acquisition unit, 730 pitch acquisition unit, 740 yaw acquisition unit, 750 gesture recognition unit.

Claims (14)

A housing that houses a presentation unit that presents a three-dimensional image and is positioned in front of the viewer's eyes when worn on the viewer's head;
A housing light emitting element that emits light vertically downward with respect to the housing when the housing is mounted on the head of an observer;
An image acquisition unit that acquires an image of light of the casing light emitting element, which is imaged by a trunk imaging element that is mounted on an observer's trunk and configured to capture light emitted by the casing light emitting element. When,
A head-mounted display, comprising: a head tracking unit that analyzes a light image acquired by the image acquisition unit and tracks a relative movement of the housing light-emitting element with respect to the trunk image sensor. The housing further includes a housing imaging unit that images light emitted by the light emitting element that moves in conjunction with the movement of the observer's hand,
The case light emitting element is installed forward in the observer's line of sight and vertically above the installation position of the case imaging unit when the case is mounted on the head of the observer. ,
The head mounted display according to claim 1, wherein the casing is configured to shield between the casing imaging unit and the casing light emitting element.   The head mounted display according to claim 1, wherein the head tracking unit detects a yaw of the housing light emitting element with respect to the trunk image sensor. The head tracking unit includes:
An area setting unit for setting a presence area of the casing light emitting element defined as an area in which the casing light emitting element is imaged in an image captured by the trunk imaging element;
4. The method according to claim 1, further comprising a bright spot analyzing section that analyzes a movement of a bright spot existing in an area set by the area setting section and detects a movement of the casing light emitting element. The head-mounted display according to crab. The bright spot analysis unit
A locus detection unit for detecting a locus of a bright spot of light in an image captured by the trunk image sensor;
A fitting unit that acquires an arc that fits the locus detected by the locus detector;
A vector acquisition unit for acquiring a radial vector starting from a center of a circle including the arc acquired by the fitting unit and having a bright spot on the image as an end point;
The head mounted display according to claim 4, further comprising a yaw acquisition unit that acquires a declination of the radial vector acquired by the vector acquisition unit.   The bright spot analysis unit, when there are a plurality of bright spots in the region set by the region setting unit, at least one of the difference in the size of the bright spot in the image and the difference in the brightness value of the bright spot The head-mounted display according to claim 5, further comprising: a bright spot selection unit that identifies a bright spot due to the housing light emitting element.   The head mounted display according to claim 6, wherein the bright spot selection unit compares a plurality of bright spots in an image and selects a smaller bright spot. The wavelength of light emitted by the housing light emitting element is different from the wavelength of light emitted by the light emitting element that moves in conjunction with the movement of the observer's hand,
The trunk imaging element transmits an optical filter in which the transmittance of light emitted from the housing light-emitting element is larger than the transmittance of light emitted from the light-emitting element moving in conjunction with the movement of the observer's hand. The captured light,
The head mounted display according to claim 6, wherein the bright spot selection unit compares brightness values of a plurality of bright spots in an image and selects a bright spot having a larger brightness value.   9. The area setting unit sets a fan-shaped area centering on a center of a circle including an arc acquired by the fitting unit as an existence area of the housing light emitting element. The head mounted display as described in.   A hand tracking unit that analyzes an image captured by the housing imaging unit and tracks a relative movement of the light emitting element that moves in conjunction with the movement of the observer's hand relative to the housing imaging unit; The head mounted display according to claim 2. The torso imaging device has an element support portion configured to be freely mounted on an observer's torso,
The head-mounted display according to claim 1, wherein the element support part includes an adjustment part that adjusts an inclination of the body image pickup element.   The head mounted display according to claim 1, wherein the housing light emitting element emits light in an invisible light region. A head-mounted display; a torso imaging device configured to be communicable with the head-mounted display; and mounted on an observer's torso; and a light-emitting element that moves in conjunction with movement of the observer's hand of the head-mounted display; A motion detection method in a video presentation system including:
The processor of the head mounted display is
Imaging light emitted by a light emitting element that moves in conjunction with the movement of the observer's hand;
Analyzing the imaged light image of the light emitting element to detect relative movement of the light emitting element with respect to the head mounted display;
Irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on the head of an observer;
Obtaining an image obtained by imaging the light irradiated in the vertical downward direction by the torso imaging device;
Analyzing the acquired image by the torso image sensor and detecting a relative movement of the head mounted display with respect to the torso image sensor, and a motion detection method in a video presentation system. A head-mounted display; a torso imaging device configured to be communicable with the head-mounted display; and mounted on an observer's torso; and a light-emitting element that moves in conjunction with movement of the observer's hand of the head-mounted display; A program for realizing a motion detection function in a computer of a video presentation system including the motion detection function,
A function of irradiating light vertically downward with respect to the head mounted display when the head mounted display is mounted on an observer's head;
A function of imaging light emitted by a light emitting element that moves in conjunction with the movement of the observer's hand;
A function of analyzing the image of the light emitted by the light emitting element that has been imaged and detecting the relative movement of the light emitting element with respect to the head mounted display;
A function of acquiring an image captured by the torso image sensor with respect to the light irradiated vertically downward;
A program for analyzing the acquired image of the torso image sensor and detecting a relative movement of the head mounted display with respect to the torso image sensor.

Images