WO2021005655A1

WO2021005655A1 - Head-mounted display

Info

Publication number: WO2021005655A1
Application number: PCT/JP2019/026857
Authority: WO
Inventors: 塩川　淳司; 橋本　康宣
Original assignee: マクセル株式会社
Priority date: 2019-07-05
Filing date: 2019-07-05
Publication date: 2021-01-14

Abstract

In order to assist with taking address toward a correct target direction on a real-world golf course, this head-mounted display for golf comprises: a camera which captures a target position in response to a predetermined action of a user; a storage device which stores the target position by means of an internal coordinate system managed by the head-mounted display; a sensor which detects movements of the user; a controller which integrates the output of the sensor; and a transparent display which, on the basis of the integration result from the sensor, displays a first virtual object indicating a direction for the target position.

Description

Head mounted display

The present invention relates to a head-mounted display. For example, it relates to the technology of a head-mounted display for golf.

Conventionally, various golf simulation devices have been devised, and it has become possible to enjoy near-pin and match play as if it were an actual course, as well as practice for skill improvement such as confirmation of shot data and repetition of approach. ing.

The golf simulation device analyzes the ball hit by the player and calculates the flight distance and drop point of the ball, so that the player feels as if he or she is actually playing on the course simulated on the display. Can be made to.

Patent Document 1 is a technique for analyzing a ball hit by a player and calculating the flight distance and falling point of the ball.

Japanese Unexamined Patent Publication No. 2003-24493

In the technique described in Patent Document 1, various data including a trigger sensor for detecting that a golf ball has been hit, a CCD camera for capturing a golf ball after hitting, and aerodynamic coefficient data under arbitrary conditions are stored. Then, a PC that calculates the trajectory and flight distance of the golf ball using data such as the speed of the golf ball obtained from the image data captured by the CCD camera and various data including aerodynamic coefficient data, and a calculation by the PC. It is equipped with a monitor that displays the trajectory and flight distance of the golf ball based on the results. In particular, in Patent Document 1, the ballistic trajectory and flight distance of a golf ball are calculated by measuring the aerodynamic coefficient using a golf ball rotating device that actually rotates the ball.

However, many golf simulation devices, including Patent Document 1, are not premised on being used on an actual golf course. For example, there is no mention of assisting the player, such as addressing in the right direction on an actual golf course.

Therefore, an object of the present invention is to provide a head-mounted display for golf that assists in taking an address toward a correct target direction in an actual golf course.

Another object of the present invention is to provide a head-mounted display for golf, which can shorten the time for searching for a ball and, as a result, reduce the number of lost balls.

In order to achieve the above object, one aspect of the head-mounted display for golf of the present invention is a camera that captures a target position in response to a predetermined operation of the user and an internal coordinate system in which the target position is managed by the head-mounted display. A storage device for storing, a sensor for detecting the movement of the user, a controller for integrating the output of the sensor, and a transmissive display for displaying a first virtual object indicating the direction of the target position based on the integration result of the controller. Have.

Further, in order to achieve the above object, one aspect of the head mount display for golf of the present invention is to analyze a camera that captures a state of hitting a golf ball by a club at high speed and an image captured by the camera to analyze the image of the golf ball. It has a controller that calculates a ball speed and a spin amount, and calculates a golf ball fall prediction point based on the ball speed and the spin amount, and a transmissive display that displays a second virtual object at the fall prediction point.

According to the present invention, it is possible to support taking an address in the correct target direction in an actual golf course.

Also, the time to search for the ball can be shortened, and as a result, the lost ball can be reduced.

It is a figure which shows an example of the appearance of the HMD of Example 1. It is explanatory drawing of the operation which sets the launch direction of a ball in the HMD of Example 1. FIG. It is a figure which shows an example of the display at the time of address in the HMD of Example 1. It is a figure which showed an example of the display of the HMD of Example 1. It is a figure which showed an example of the hardware composition of the HMD of Example 1. FIG. It is a figure which showed an example of the functional block of the HMD of Example 1. FIG. It is a figure which showed an example of the flowchart which showed the processing operation of the HMD of Example 1. FIG. It is a figure which showed the application which automatically registers the score by the cooperation of the HMD and the smartphone of Example 2. It is a figure which showed an example of the display of the HMD of Example 2. It is a figure which shows the absolute coordinate system of Example 3. It is a figure which showed the deviation of the internal coordinate system with respect to the absolute coordinate system of Example 3. It is a figure which showed an example of the problem by the deviation of the internal coordinate system with respect to the absolute coordinate system of Example 3. It is a figure which showed an example of the correction of the coordinate system of Example 3. It is a figure which showed an example of the flowchart which showed the processing operation of the HMD of Example 3.

The embodiment will be described with reference to the drawings. It should be noted that the embodiments described below do not limit the invention according to the claims, and all of the elements and combinations thereof described in the embodiments are indispensable for the means for solving the invention. Is not always.

In the following explanation, information may be explained by the expression of [AAA table], but the information may be expressed by any data structure. That is, since the information does not depend on the data structure, the [AAA table] can be described as [AAA information].

Further, in the following description, the [CPU] is a processing unit and is a Central Processing Unit including one or more processors. The processor may include hardware circuits that perform some or all of the processing.

Further, in the following description, the process may be described with the [program] as the main body of operation, but the program is executed by the CPU to appropriately perform the specified process as a storage resource (for example, memory) or the like. The main body of the actual processing is the CPU because it is performed while using. Therefore, in the process described with the program as the subject of operation, the processor may be described as the subject of operation. In addition, a hardware circuit such as an Application Specific Integrated Circuit (ASIC) or a Field-Programmable Gate Array (FPGA) may perform a part or all of the processing performed by the processor.

<System configuration>
First, a system according to an embodiment of the present invention will be described.

FIG. 1 is a diagram showing an example of the appearance of the golf head-mounted display of Example 1 (hereinafter, may be referred to as a head-mounted display or HMD).

The head-mounted display (HMD) 100 has a transmissive display 101, a controller 102, a camera 103, a frame 104 for supporting each part, and a button 105 for the user to wear. The transmissive display 101 is also a kind of head-up display, and the display device is made of a half mirror so that the outside can be seen. Although FIG. 1 shows a display for both eyes, a display device may be attached to only one eye. Further, a display using a holographic element may be used. By using a half mirror of an optical multilayer film, it is possible to see through the outside while displaying only necessary information on the surface of the display plate.

The camera 103 includes a color image camera 103a and a distance image camera 103b. The distance image camera 103b is a camera capable of measuring three-dimensional information using the flight time of light, such as a TOF camera (Time-of-Flight Camera) or the like.
<How to use HMD>
FIG. 2 is an explanatory diagram of an operation of setting the launch direction of the ball in the HMD of the first embodiment. In order to help the understanding of the invention, a method of using the HMD when playing golf will be described, and then how to realize it with the HMD of the embodiment will be described.

First, the player marks the tee ground with a predetermined movement, for example, the movement of the tip of the hand 201, with the launch direction as the target position via the display 101 of the HMD 100. The marked target position is captured by the camera 103 and stored in the internal coordinate system managed by the HMD. The player can recognize the green 203 and the flag 204 through the transmissive display 101 of the HMD 100. The depth coordinate of the point pointed by the finger at this time is a predetermined value, for example, 200 yards ahead. Reference numeral 202 denotes an edge of the display 101 of the HDM 100.

Next, when the HMD 100 recognizes the image of the club and the ball and recognizes that the player has taken the posture of the address, the camera 103 starts high-speed imaging and performs imaging at a high frame rate. The HMD 100 calculates the marked direction with a 3D sensor in the internal coordinate system of the HMD 100, and displays an arrow toward the marked direction coordinate. The start of high-speed imaging may be manually instructed by using the button 105 of the HMD or the like.

FIG. 3 is a diagram showing an example of a display at the time of addressing on the head-mounted display of the first embodiment. The player can recognize the club 301 and the ball 302 through the transmissive display 101 of the HMD 100. The HMD 100 displays an arrow 311 indicating the marked direction described in FIG. When the player faces the marked direction, highlight it by changing the color of the arrow or the density, brightness, solid line / dotted line of the display. If the player is addressing to the right of the marked direction, the arrow 313 is displayed, and if the player is addressing to the left of the marked direction, the arrow 312 is displayed. At this time, the color image camera 103a of the HMD 100 photographs the ball 302 and the club 301, and the HMD 100 detects that the ball 302 and the club 301 are in the address state by image recognition. If the system is registered by the user in advance, the club to be used can be accurately determined by comparing with the image of the registered club. The determination of the club to be used is the information necessary for more accurate calculation of the flight distance of the ball.

At the time of shot, the state where the club head descends from the right to the left in the figure and hits the ball is recognized by image processing. At this time, the HMD analyzes the ball launch speed, the head speed, the meet rate, and the spin amount from the image. As for the analysis method, the existing simulation technique may be applied.

Normally, the player confirms the launch direction of the ball from the front before entering the address, and rotates about 90 degrees to enter the address. At the stage of entering the address after confirming this launch direction, the player loses sight of the correct launch direction and takes the address in the wrong direction, which leads to a miss shot. The HMD 100 of the first embodiment stores the launch direction (direction of the target position) designated by the player on the tee ground, the fairway, or the like by using the internal coordinate system of the HMD 100. Even if you rotate your body when entering the address, you can display the direction marked at the time of address as an AR (Augmented Reality) as a virtual object based on the coordinates of the memorized launch direction, and take the address in the correct direction. be able to.

FIG. 4 is a diagram showing an example of the display of the HMD of the first embodiment. It is the figure which image-processed the state which hit the ball and displayed the result of having analyzed the ball launch speed, the head speed, the meet rate, and the spin amount from the image in the display area 401 of the display 101.

Further, FIG. 4 shows that the image-processed ball launch speed, head speed, meet rate, and spin amount indicate that the ball fall prediction point 402 is AR-displayed. As a result, the player can identify the drop point of the ball on the actual course, so that the time for searching for the ball can be shortened, and as a result, the lost ball can be reduced.
<Hardware configuration of HMD>
FIG. 5 is a diagram showing an example of the hardware configuration of the head-mounted display of the first embodiment.

The HMD 100 of the first embodiment basically has the same configuration as a general-purpose computer (information processing device). That is, as shown in FIG. 5, the HMD 100 includes a controller 102, a camera 103, a display 101, a voice interface (I / F) 540, a communication I / F 520, a sensor 530, a line-of-sight detection device 510, and the like. It includes a button 550 and a bus 560 that electrically connects each part.

The controller 102 performs various processes according to a predetermined program such as the image processing unit 601 shown in FIG. In the first embodiment, the controller 102 AR-displays virtual objects such as the arrow 311 in FIG. 3, the display area 401 in FIG. 4, and the fall prediction point 402 at a predetermined position on the display 101, for example.

The controller 102 of the first embodiment includes a CPU 501, a RAM 502, and a ROM 503. The CPU 501 realizes various functions by loading a program previously stored in the ROM 503 into the RAM 512 and executing the program. The RAM 502 and the ROM 503 are collectively referred to as a storage device 630 (see FIG. 6) when it is not necessary to distinguish them. The controller 102 is arranged on the frame 104, for example.

The camera 103 includes a color image camera 103a and a distance image camera 103b. The color image camera 103a captures a shooting range including the user's field of view and acquires a color image. Further, the distance image camera 103b acquires a distance image in a shooting range substantially the same as that of the color image camera 103a. The camera 103 (color image camera 103a and distance image camera 103b) captures the marked target position shown in FIG. 2 and stores it in the storage device 630 in the internal coordinate system managed by the HMD by the controller 102. For example, in the foremost part of the frame 104, the shooting range is arranged at a position where shooting is possible. Note that both / or either of the color image camera 103a and the distance image camera 103b may have a memory (not shown) for temporarily storing an image at the time of imaging at a high frame rate.

The display 101 is a transmissive device that displays the image acquired by the camera 103 and the display data generated in the HMD 100. The display 101 is composed of, for example, a transmissive liquid crystal device, an organic EL device, an optical scanning type device using MEMS (micro electronics mechanical systems), or the like. However, the device is not limited to this, and any device can be used as long as it can realize a transmissive display structure in which the other side of the display 101 can be seen through while displaying an image on the display 101.

In the HMD 100 of Example 1, the transmissive display 101 is supported in front of one or both eyes of the user. The display 101 can take an arbitrary shape. The display 101 may include right and left display panels, and the display 101 may display one or more UI objects of the graphical user I / F.

The voice I / F540 is, for example, a voice output device such as a microphone, a speaker, and a buzzer. The voice I / F 540 inputs a sound from the outside world and outputs a sound created in the HMD 100 or a sound such as a voice or music transmitted via the communication I / F 520. In the first embodiment, the voice I / F 520 may not be provided.

The communication I / F520 is provided with a code circuit, a decoding circuit, an antenna, etc., and transmits / receives data (data communication) to / from another device which is an external device via a network. In the first embodiment, the communication I / F 520 is an I / F that connects to the network via an access point or the like (not shown) or a base station or the like of a mobile telephone communication network (not shown). .. The HMD 100 transmits / receives data to / from each server connected to the network via the communication I / F 520.

The button 550 is for turning on / off the power of the HMD 100, switching the operation mode, and inputting the start timing of a predetermined operation to the HMD 100 by the user. Button 550 corresponds to button 105 in FIG.

The connection between the HMD100 and the access point is performed by, for example, a wireless communication method such as Wi-Fi (registered trademark) or Bluetooth (registered trademark) or a communication method such as others. The connection between the HMD100 and the base station of the mobile telephone communication network is, for example, W-CDMA (registered trademark) (Wideband Code Division Multiple Access) method, GSM (registered trademark) (Global System for Mobile communications) method, LTE (LTE). It is performed by the Term Evolution) method or other communication methods such as 5G.

The sensor 530 detects the current position, tilt, speed, user operation, etc. of the HMD100. As the sensor 530, the HMD 100 includes, for example, a position information acquisition sensor such as a GPS receiver 531a, a gyro sensor 531b, an acceleration sensor 531c, a geomagnetic sensor 531d, a touch sensor 531e, and the like. The sensor 530 does not have to include all of them.

The line-of-sight detection device 510 detects the user's line-of-sight direction. The line-of-sight detection device 510 is realized by, for example, a line-of-sight detection camera that detects the line-of-sight direction of the user. The line-of-sight detection camera is attached so as to include the iris, pupil, etc. of the user's eye in the imaging range.

The frame 104 supports components of the HMD 100 such as the display 101, the camera 103, and the controller 102.
<Functional block of HMD>
Next, the functions related to the virtual object display realized by the controller 102 of the first embodiment will be described. FIG. 6 is a functional block diagram of the function related to the virtual object display processing of the HMD 100 of the first embodiment. As shown in FIG. 6, the controller 102 realizes the functions of the image processing unit 601, the display control unit 610, and the audio output control unit 625. Further, the display control unit 610 includes a space recognition unit 611, an instruction reception unit 622, a display data generation unit 623, and a display correction unit 624.

Each function is realized by the CPU 501 loading the program stored in the ROM 503 into the RAM 502 and executing it.

Further, the storage device 630 stores color image data 631, distance image data 632, spatial recognition data 633, virtual object data (virtual OJT data) 634, audio data 636, and map data 638. The map data 638 is not always necessary in the first embodiment.

The color image data 631 is an image acquired by the color image camera 103a. The distance image data 632 is an image acquired by the distance image camera 103b.

The image processing unit 601 stores the color image and the distance image acquired by the color image camera 103a and the distance image camera 103b in the storage device 630 as the color image data 631 and the distance image data 632, respectively. In the first embodiment, the color image and the distance image are acquired substantially synchronously.

The space recognition unit 611 is an internal coordinate system generated based on the output from the sensor 530 inside the HMD, recognizes the surrounding real space, and stores the result as space recognition data 633 in the storage device 630. To do. The recognition is performed using the color image data 631 and the distance image data 632 acquired substantially at the same time.

The space recognition unit 611 generates space recognition data 633, which is three-dimensional data (three-dimensional map) of the structure in the shooting range, from each image data at predetermined time intervals according to the scanning operation of the user, and stores the storage device. Store in 630. The surrounding scan is performed, for example, immediately after startup, periodically as an initial setting, or by operating a button of the user.

The map data 638 is absolute coordinate system data, and stores the map information of the golf course downloaded from the communication I / F 520 via the Internet in the storage device 630 by GNSS (Global Navigation Satellite System / Global Positioning Satellite System). The map data includes the course layout of each hole of the golf course. The course layout also includes information on the distance from the teeing ground to the tee and the location and size of obstacles such as bunkers, ponds and OBs.

The space recognition data 633 is created, for example, in the world coordinate system (coordinate system including absolute coordinates) that defines the entire three-dimensional space. As an example, the local coordinate system (internal coordinate system) of the HMD100, which is specified by the position and orientation (initial posture) of the HMD100 main body when the instruction to start spatial recognition is received as the origin and each axial direction of this world coordinate system. Includes) origin and each axial direction. In this coordinate system, for example, in the initial posture of the HMD100 main body, the origin is a predetermined position on the display 101 of the HMD100, the inside of the display 101 is the xy plane, and the z-axis direction is the direction perpendicular to the xy plane (display 101 plane). And.

The amount of displacement and the amount of rotation of the local coordinate system of the HMD100 by the user's scanning operation with respect to the world coordinate system are calculated using the data obtained by various sensors 530.

It should be noted that spatial recognition is performed using, for example, existing technology such as Spatial Mapping. That is, the HMD 100 of the first embodiment scans the surroundings by the color image camera 103a and the distance image camera 103b. Then, using the result, the space recognition unit 611 generates three-dimensional data by using an application such as Spatial Mapping. The spatial recognition data 633 is held as, for example, mesh data.

Further, at this time, not only the three-dimensional data but also the physical understanding that recognizes the type of the structure may be performed at the same time. The space recognition unit 611 can recognize the material and type of the constituents within the photographing range by the spatial understanding. That is, it is possible to recognize whether the structure is, for example, a "wall", a "floor", or a "ceiling". The space recognition unit 611 stores these recognition results in the storage device 630 as attribute data of the space recognition data 633.

The instruction receiving unit 622 receives display instructions and operation instructions for the virtual object displayed on the display 101. The display instruction and the operation instruction include, for example, those by the line of sight (gaze) and those by the movement of the fingers (gesture).

The line-of-sight direction information used for gaze is detected using, for example, the line-of-sight detection device 510. Gestures include, for example, click events (air taps), tap and hold, bloom, etc. for operation points of virtual objects. The instruction receiving unit 622 detects the movement of the fingers in the gesture frame provided within the shooting range of the color image camera 103a and the distance image camera 103b, and detects display instructions, operation instructions, and the like.

For example, when a display instruction is received, the instruction reception unit 622 extracts the data of the instructed virtual object from the virtual object data 634, and causes the display data generation unit 623 described later to generate display data. When an operation instruction is received, the instruction reception unit 622 detects the operation and notifies the display data generation unit 623 described later.

The display data generation unit 623 generates display data from the virtual object data 634 to display the instructed virtual object at a predetermined position on the display 101 in a predetermined shape according to the user's instruction via the instruction reception unit 622. For example, in addition to the arrow 311 in FIG. 3 indicating the marking in the direction supported by the fingertip 201 in FIG. 2, virtual objects such as the display area 401 and the predicted fall point 402 in FIG. 4 calculated by the HMD are displayed on the display 101. Generated to do.

By displaying the display data generated according to the instruction of the instruction receiving unit 622 on the display 101, the virtual object is displayed to move, rotate, and deform according to the instruction of the user.
<HMD processing operation>
FIG. 7 is a flowchart showing the processing operation of the HMD 100.

First, in step S71, the image acquired by the image processing unit 601 from the camera 103 is shown in FIG. 2 by the instruction receiving unit 622 with respect to the three-dimensional data of the structure in the photographing range generated by the space recognition unit 611. The user acquires the position (marking position) specified by the fingertip 201. The marking position may be simply stored by detecting the operation for marking the target position via the camera 103.

Next, in step S72, the display data generation unit 623 displays a target arrow on the display 101 toward the internal coordinate direction marked by the integrated value of the output from each sensor 530. Based on the position and orientation of the internal coordinate system of the HMD200, which is an integral calculation of the output from the sensor 530 (this process is always executed by interrupt processing to the CPU 501 at predetermined time intervals, etc.), the marked launch direction is determined. calculate. An arrow pointing in the marked direction is read from the virtual object data 634, processed in 3D, and displayed on the display 101 of the HMD 100.

In step S73, the image processing unit 601 of the controller 102 detects the address state from the positional relationship between the ball and the club head from the color image from the color image camera 103a. Alternatively, the output of each sensor 530 detects that the player has entered the address state. For example, it can be detected that the address state is entered because the output of the gyro sensor 531b or the acceleration sensor 531c is reduced. Alternatively, it may be performed by pressing the button 105. That is, the controller 102 marks the target position in step S71, integrates the output of the sensor 530 from the time when the target position is captured to the time when the player enters the address state in step S73, and thereby indicates the target position of the virtual object indicating the target position. The display 101 is virtually displayed in the direction of.

In step S74, the controller 102 controls the camera 103 to shoot at a high frame rate, and the camera 103 starts shooting at a high frame rate.

In step S75, the image processing unit 601 of the controller 102 detects the shot state by image processing, and calculates the fall prediction point from the detected ball speed and spin amount.

In step S76, the image processing unit 601 of the controller 102 calculates the fall prediction point from the calculated ball speed and spin amount.

In step S77, the display data generation unit 623 reads the display data for displaying the calculated fall prediction point from the virtual object data 634 and displays it as a virtual object on the display 101 under the control of the display control unit 610 of the controller 102.

According to the first embodiment, since the arrow is displayed on the HMD with the target direction as a virtual object in the actual golf course, the player can take an address toward the correct target direction.

Further, according to the first embodiment, since the predicted drop point of the shot ball can be displayed as a virtual object on the actual golf course, the time for searching for the ball can be shortened, and as a result, the lost ball can be reduced.

Example 2 is an application using the HMD map data 638. FIG. 8 is an application for automatically registering a score by linking the HMD 100 and the smartphone 801.

As shown in FIG. 8, the HMD 100 shown in FIG. 1 and the smartphone 801 are connected by wireless communication such as Bluetooth. The controller 102 of the HMD 100 identifies the hall information corresponding to the current position of the player from the map data 638 and the GPS receiver 531a. The GPS information from the GPS receiver 531a includes information indicating the current position. In the example of FIG. 8, it is displayed that the player is in the second hole 810. The screen 810 of the smartphone 801 acquires and displays the hole number and the scorecard information indicating the number of pars and the distance (number of yards) of the hole from the map data 638. The scorecard information does not have to include information about distance.

Further, the name 811 of the player registered in advance can be displayed, and the score of each player can be input to the stroke number 821 and the putter number 822 by the input unit 830 of the touch panel. The controller 102 transmits the scorecard information corresponding to the specified hall information to an external device such as a smartphone 801 via the communication interface 520. The smartphone 801 displays the scorecard information transmitted from the HMD 100 as shown in FIG.

Since the HMD 100 shown in FIG. 8 can recognize the action of hitting a ball by image processing as described in the first embodiment, if the hole number is specified from the position of the player by the GPS receiver 531a, the number of shots of each hole can be determined. It can be entered automatically. Since the shot and putt operations can be distinguished based on the shape of the club photographed by the camera 103 and the head speed calculated by image processing, the number of shots and the number of putts in each hole can also be automatically input. That is, the controller 102 identifies the hole number to be played by the user based on the GPS information from the GPS receiver 531a and the map data 638, and analyzes the image taken by the camera 103 for the specified hole number. As a result, the number of shots and the number of putts in each hole can be calculated and stored in the storage device 630, or the score can be displayed on the display 101.

FIG. 9 is a diagram showing an example of the display of the HMD of the second embodiment. FIG. 9 shows an example in which the results of analyzing the ball launch speed, head speed, meet rate, and spin amount from the image are displayed on the display 101.

By providing the HMD100 with the GPS receiver 531a and the map data 638, it is possible to determine which point in which hole of the golf course the player is. Based on the GPS receiver 531a, the map data 638, and the ball fall prediction point described in the first embodiment, when the ball fall prediction point corresponds to a bunker, a pond, an OB, a one-penalty area, etc., the display control of the controller 102 is performed. The unit 610 controls the display 101 to change the color of the fall prediction point display or display characters.

The controller 102 can also calculate the distance to the green edge before the shot of the player from the GPS information from the GPS receiver 531a and the map data 638 and display it on the display 101.

According to the second embodiment, the score can be registered by linking the HMD and the smartphone. Further, according to the second embodiment, the number of shots and the number of putts can be automatically registered from the GPS information, the map data, and the operation of the shots and putts that can be grasped by the HMD.

The calibration that matches the absolute coordinates of GPS and the internal coordinates of HMD will be explained. First, the definition of each coordinate will be clarified.

The absolute coordinate system is a coordinate system obtained from latitude and longitude information obtained from satellites. In FIG. 10A, the absolute coordinate system is shown by x', y', and z'.

The internal coordinate system is the internal coordinate system of the HMD100, which always performs inertial calculation from the accelerometer, gyro, and compass to calculate its own position and orientation. FIG. 10B shows the internal coordinate system with x, y, and z. This internal coordinate system gradually deviates from the absolute coordinate system obtained from GPS information due to errors in the gyro, compass, accelerometer, and integration error. In particular, the error of the compass is large, and therefore, as shown in FIG. 10B, the integration of the deviation in the rotation direction is large.

Therefore, as shown in FIG. 10C, the position of the bunker at the position of 101b in the absolute coordinate system is grasped as 101a in the internal coordinate system. As a result, although the fall prediction point 402 is near the bunker in the absolute coordinate system, it is predicted that the ball will fall to a place other than the bunker, so that it is not possible to determine or display that the ball may have entered the bunker. This is because the bunker 101, the map data of the hole, and the ball fall prediction point 402 are calculated in the internal coordinate system. It is necessary to correct the error with the absolute coordinate system accumulated in this internal coordinate system.

FIG. 11 is a diagram showing correction of the coordinate system. It is a figure explaining the method of adjusting the internal coordinate system of HMD100 to the absolute coordinate system.

FIG. 11 shows the positions P1 (x'1, y'1), P2 (x2, y2), and P3 (x3, y3) where the player hit the shot on the map data of the golf course. The locus 1 and the locus 2 represent a locus that linearly connects the positions where the player hits the shot. The alternate long and short dash line indicates that it is parallel to the x-axis of the absolute coordinate system. The locus 1 between the position P1 and the position P2 can be represented by an angle α and a length L with respect to the x-axis of the absolute coordinate system. The internal coordinate system is corrected (calibrated) by replacing the angle with respect to the x-axis when moving from the P1 point to the P2 point obtained by integrating the output of the internal sensor 530 with α and the distance with L. .. The calibration timing may be at regular time intervals, shot points, and the like.

The calibration may be performed by another method. For example, the player manually identifies the bunker position and the pin position of the hole, which are easy to identify on the course, so that the bunker position and the pin position virtually displayed based on the map data are adjusted to the positions specified by the user. You may go by that.

FIG. 12 is a flowchart showing the processing operation of the HMD 100. The steps that perform the same processing as in FIG. 7 are designated by the same reference numerals. That is, steps S71 to S75, and steps S76 and S77 correspond to the processes of steps S71 to S77 described with reference to FIG. 7. In the third embodiment, the process of step S121 is performed between steps S75 and S76.

Step S121 is a step executed by the display correction unit 624 in order to correct the internal coordinates described with reference to FIG. 11, and stores the shot point (absolute coordinates) this time. This is a process of recording the absolute coordinates of the position P2 in FIG. Calculate the slope with the previous shot point (absolute coordinates). The slope of the position P1 in FIG. 11 with respect to the shot point (absolute coordinates) is calculated. The slope obtained by the integral calculation of the sensor 530 is corrected as described above. By correcting the internal coordinate system of the HMD with respect to the absolute coordinate system in this way, the player can correctly grasp the distance to the bunker or pond obtained from the map data and the distance to the green.

According to the third embodiment, the player can grasp the correct situation of the ball fall prediction point by matching the internal coordinate system of the HMD with the absolute coordinate system. In addition, the player can grasp the correct shot target from the map data before the shot.

100: Head-mounted display,
101: Display,
102: Controller,
103: Camera,
104: Frame,
105: Button,
501: CPU,
502: RAM,
503: ROM,
510: Line-of-sight detector,
520: Communication I / F,
530: Sensor,
540: Voice I / F.

Claims

In head-mounted displays for golf
A camera that captures the target position in response to the user's predetermined action,
A storage device that stores the target position in the internal coordinate system managed by the head-mounted display, and
The sensor that detects the movement of the user and
A controller that integrates the output of the sensor and
A head-mounted display for golf, which comprises a transmissive display that displays a first virtual object indicating the direction of the target position based on the integration result of the controller.
In the head-mounted display for golf according to claim 1.
The camera captures a target position for launching a golf ball in response to the movement of the user's hand.
The controller is a head-mounted display for golf, which integrates the output of the sensor from the time when the target position of the camera is captured until the user hits the ball.
In head-mounted displays for golf
A camera that shoots a golf ball hit by a club at high speed,
A controller that analyzes an image taken by the camera, calculates the ball speed and spin amount of the golf ball, and calculates a predicted fall point of the golf ball based on the ball speed and the spin amount.
A head-mounted display for golf, which comprises a transmissive display that displays a second virtual object at the predicted fall point.
In the head-mounted display for golf according to claim 3.
The controller is a head-mounted display for golf, which starts high-speed shooting of the camera when it detects that the user has entered the address state from the images of the club and the ball of the camera.
In the head-mounted display for golf according to claim 3.
It has a sensor that detects the movement of the user
The controller is a head-mounted display for golf, which starts high-speed shooting of the camera based on the output of the sensor.
In the head-mounted display for golf according to claim 3.
It has a storage device that stores the second virtual object that indicates the point where the golf ball falls.
The controller reads the second virtual object from the storage device, and virtually displays the read second virtual object on the transmissive display at the fall prediction point. Head-mounted display for.
In the head-mounted display for golf according to claim 3.
The head-mounted display for golf further includes a GPS receiver, a storage device for storing map data indicating a course layout of a golf course, and a communication interface for wirelessly communicating with an external device.
The controller identifies a hole number based on GPS information from the GPS receiver and the map data, and transmits scorecard information of the specified hole number to an external device via the communication interface. Head-mounted display for golf.
In the head-mounted display for golf according to claim 3.
The golf head-mounted display further includes a GPS receiver and a storage device for storing map data indicating the course layout of the golf course.
The controller identifies the hole number based on the GPS information from the GPS receiver and the map data, and analyzes the image taken by the camera for the specified hole number to calculate the number of shots. A head-mounted display for golf, which is stored in the storage device and displayed on the transmissive display.
In the head-mounted display for golf according to claim 3.
The head-mounted display for golf has a sensor that detects a user's movement, and has a sensor.
The controller manages an internal coordinate system generated based on the output from the sensor and an absolute coordinate system which is a coordinate system obtained from latitude and longitude information obtained from the satellite.
A head-mounted display for golf, characterized in that the internal coordinate system is corrected with respect to the absolute coordinate system at a predetermined timing.
In the head-mounted display for golf according to claim 9.
The golf head-mounted display further includes a GPS receiver and a storage device for storing map data indicating the course layout of the golf course.
The controller is a head-mounted display for golf, characterized in that a third virtual object indicating a course layout of the map data is displayed on the transmissive display according to the corrected internal coordinate system.
In the head-mounted display according to claim 8.
The controller is a head-mounted display for golf, characterized in that the number of shots is stored in the storage device and notified to another mobile terminal device.