JP5907136B2 - Head mounted display and control program - Google Patents

Description

  The present invention relates to a head mounted display and a control program for controlling the head mounted display.

  A head mounted display (HMD) is used by being mounted on a user's head. Therefore, the HMD is required to be small in order to secure the user's field of view and reduce the burden at the time of wearing. When the HMD is downsized, a place where a mechanical user interface (User Interface UI) such as a switch is provided is limited. Patent Literature 1 discloses a technique for identifying corresponding operation information by detecting the movement of a user's hand taken by a camera. By applying this technology to the HMD, the HMD can specify the operation content corresponding to the movement of the object photographed by the camera, and can execute an operation according to the specified operation content. In this case, the HMD can secure a sufficient UI even when the HMD is downsized.

JP 2007-172577 A

  When a camera is provided in the HMD, a method of arranging it together with a display unit on the user's head can be considered. In this case, the shooting area of the camera changes as the head of the user wearing the HMD moves. In this case, the relative position of the object with respect to the imaging region changes as the user's head moves. Usually, the HMD detects the movement of the object based on the relative position of the object with respect to the imaging region. Therefore, the HMD may not be able to accurately detect the movement of the object when the user's head moves. In this case, the HMD has a problem that it cannot execute the operation desired by the user.

  An object of the present invention is to provide a head mounted display capable of accurately detecting the movement of an object and executing a corresponding operation, and a control program for controlling the head mounted display.

  The head-mounted display according to the first aspect of the present invention images a predetermined area including at least a display unit capable of emitting image light in a first direction and a second direction area opposite to the first direction. An imaging unit capable of outputting image data, first determination means for repeatedly determining whether there is a predetermined object in the predetermined area based on the image data output from the imaging unit, and the first determination When it is determined by the means that the object is within the predetermined area, first information indicating a first position that is a position corresponding to the photographing direction of the photographing unit is acquired based on the output of the detecting unit. Acquired by the first acquisition means, the second acquisition means for acquiring the second information indicating the second position, which is the relative position of the object with respect to the predetermined area, based on the image data, and the first acquisition means. Said first Determination means for determining third information based on the information and the second information acquired by the second acquisition means each time the first determination means determines that the object is within the predetermined area. Storage means for storing the third information determined by the determination means in association with order information indicating the determined order in a first storage unit, and a plurality of third information stored in the first storage unit Based on the third information of one of the information and the other third information associated with the order information indicating the order after the order indicated by the order information associated with the first third information, From the time when the first information corresponding to the third information is acquired by the first acquisition means until the time when the first information corresponding to the other third information is acquired by the first acquisition means, Between the first position and the second position. A second determination unit that determines whether the positional relationship has changed, and a position determined based on at least the other third information when the second determination unit determines that the positional relationship has changed. Corresponding to the second position corresponding to each third information based on the first display means for repeatedly displaying the trajectory information on the display unit and the plurality of third information stored in the first storage unit It is determined that the predetermined condition is satisfied by the third determination means for determining whether the tendency of the change in the plurality of correction positions corrected by the first position to satisfy the predetermined condition and the third determination means In this case, an execution unit that executes a corresponding process is provided, and the third information is information corresponding to the positional relationship.

  According to the first aspect, the head mounted display determines the third information based on the first information and the second information, and stores the third information in the first storage unit. The third information is information indicating the positional relationship between the first position (position corresponding to the photographing direction of the photographing unit) and the second position (relative position of the object with respect to the predetermined area). The first position changes according to a change in the shooting direction of the shooting unit. For this reason, the temporal change in the positional relationship between the first position and the second position indicates the actual movement of the object even when the shooting direction of the shooting unit changes. The head mounted display performs a corresponding process when the tendency of the correction position with the second position corrected by the first position over time satisfies a predetermined condition. Therefore, the head mounted display can accurately detect the movement of the object and execute the corresponding process even when the shooting direction of the shooting unit changes.

  In the first aspect, the third information may be information indicating a relative position of the second position with respect to the first position. In this case, the head mounted display can easily specify the positional relationship between the first position and the second position based on the third information.

  In the first aspect, the apparatus includes second display means for displaying n-th (n is a natural number of 2 or more) trajectory information at a second position corresponding to the third information determined by the determination means. The first display means corresponds to the first erasing means for erasing the 1st to (n-1) th trajectory information displayed on the display unit, and the 1st to (n-1) th determined third information. Correction means for correcting the second position to be corrected based on the first position corresponding to the n-th determined third information, and displaying the trajectory information at the corrected second position. In this case, the head mounted display can make the user recognize the trajectory information reflecting the change in the shooting direction of the shooting unit. For this reason, the user can easily move the object so as to satisfy the predetermined condition.

  In the first aspect, the first display means has a third position in which the second position corresponding to the third information determined nth (n is a natural number of 2 or more) determined by the determination means is determined first. Correction may be made based on the first position corresponding to the information, and the locus information may be displayed at the corrected second position. In this case, the head mounted display can make the user recognize the trajectory information reflecting the change in the shooting direction of the shooting unit. For this reason, the user can easily move the object so as to satisfy the predetermined condition.

  In the first aspect, the locus information is displayed on the display unit when the state in which the first determination unit determines that the object is not present in the predetermined region continues for a predetermined first time or longer. A second erasing unit for erasing the trajectory information. In this case, the head mounted display can erase the trajectory information unnecessary for the user from the display unit.

  In the first aspect, when the state in which the positional relationship is determined not to move by the second determination unit continues for a predetermined second time or more, and the trajectory information is displayed on the display unit A third erasing unit for erasing the trajectory information may be provided. In this case, the head mounted display can erase the trajectory information unnecessary for the user from the display unit.

  In the first aspect, there is provided fourth erasing means for erasing the trajectory information when the third judgment means judges that the tendency of change of the plurality of correction positions satisfies a predetermined condition. Good. In this case, the head mounted display can erase the trajectory information unnecessary for the user from the display unit.

  In the first aspect, the third determining means determines whether the specific shape indicated by the template data stored in the second storage unit matches the tendency of change in the plurality of correction positions, It may be determined whether the tendency of change of the plurality of correction positions satisfies a predetermined condition. In this case, the head mounted display can appropriately determine whether the tendency of the change in the plurality of correction positions satisfies a predetermined condition.

  In the first aspect, the execution means may execute a specific process associated with template data indicating a specific shape that is determined to match the tendency of change in the plurality of correction positions. In this case, the head mounted display can easily determine the process corresponding to the movement of the object and execute the process.

  In the first aspect, the first acquisition unit may acquire, as the first information, information indicating a position of a comparison object included in the predetermined area based on the image data. In this case, the head mounted display can appropriately acquire the position corresponding to the shooting direction of the shooting unit as the first information.

  In the first aspect, the first acquisition unit may acquire a plurality of first information based on the image data. In this case, the head mounted display can accurately specify the shooting direction of the shooting unit based on the plurality of pieces of first information.

  In the first aspect, the plurality of pieces of first information may indicate positions near the four corners of the captured image indicated by the image data. In this case, the head mounted display can more accurately specify the shooting direction of the shooting unit based on a plurality of pieces of first information indicating the positions of the four corners of the shot image.

  The control program according to the second aspect of the present invention is such that the imaging unit captures a predetermined area including at least a region in the second direction that is opposite to the first direction in which the display unit can emit image light. A first determination step of repeatedly determining whether or not there is a predetermined object in the predetermined area based on image data output from the imaging unit; and the object is within the predetermined area by the first determination step. A first acquisition step of acquiring first information indicating a first position, which is a position corresponding to the shooting direction of the shooting unit, based on the output of the detection unit; A second acquisition step of acquiring second information indicating a second position, which is a relative position of the object, based on the image data; the first information acquired by the first acquisition step; A determination step of determining third information based on the second information acquired by the acquisition step every time the first determination step determines that the object is in the predetermined area; and A step of storing the determined third information in the first storage unit in association with the sequence information indicating the determined order; and a first one of the plurality of third information stored in the first storage unit 3 information and the other third information associated with the order information indicating the order after the order indicated by the order information associated with the first third information. Before the first information corresponding to the other third information is acquired by the first acquisition step after the corresponding first information is acquired by the first acquisition step, the first position and the previous position A second determination step for determining whether or not the positional relationship with the second position has changed; and when the second determination step determines that the positional relationship has changed, the determination is based on at least the other third information. The second display corresponding to each of the third information based on the first display step of repeatedly displaying the trajectory information indicating the position to be displayed on the display unit and the plurality of third information stored in the first storage unit. If the predetermined condition is satisfied by the third determination step of determining whether the tendency of the change of the plurality of correction positions corrected by the corresponding first position satisfies the predetermined condition and the third determination step If it is determined, the execution step of executing the corresponding process is executed by the computer of the head mounted display. According to the 2nd aspect, there can exist an effect similar to a 1st aspect.

It is a perspective view of HMD1. It is a block diagram which shows the electric constitution of HMD1. It is a figure which shows the scene 25 (25A-25J). It is a flowchart of a main process. It is a flowchart of a correction process. It is a figure which shows the picked-up image 31. FIG. It is a figure which shows the table 541. FIG. It is a figure which shows the table 521. It is a figure which shows the locus | trajectory image 26 (26F-26J, 261-270). It is a flowchart of the correction process in a modification. It is a figure which shows the table 521 in a modification. It is a figure which shows the locus | trajectory image 26 (26F-26J, 281-284) in a modification.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Embodiments of the invention will be described below with reference to the drawings. An outline of a head mounted display (hereinafter referred to as HMD) 1 will be described with reference to FIG. The HMD 1 includes a projection device (hereinafter referred to as a head display or HD) 10 and a control device (hereinafter referred to as a control box or CB) 50. In the following description, the upper, lower, right diagonally downward, left diagonally upward, right diagonally upward and left diagonally downward of FIG. 1 are the upper, lower, forward, backward, right and left sides of the HMD 1, respectively. In the embodiment, in order to help understanding the positional relationship and the directional relationship in various configurations, in the related drawings, the upper, lower, front, rear, right, and left sides of the HMD 1 refer to the axes of the three-dimensional Cartesian coordinate system. Explained.

  An outline of the HD 10 will be described. The HD 10 is attached to the spectacles 5 which is a dedicated wearing tool. The user wears the HD 10 on the head by using the glasses 5 to which the HD 10 is attached. The HD 10 can irradiate image light backward. The image light is incident on the left eye of the user wearing the HD 10. The HD 10 is detachably connected to the CB 50 via the harness 7. The HD 10 includes a housing 2, a half mirror 3, an image display unit 14 (see FIG. 2), an eyepiece optical system (not shown), a camera 20, and the like as main components.

  The housing 2 is provided at the right end of the glasses 5. A half mirror 3 is provided on the left end side of the housing 2. The half mirror 3 is disposed in front of the user's left eye with the user wearing the HD 10 on the head. The image display unit 14 and the eyepiece optical system are provided inside the housing 2. Details of the image display unit 14 and the eyepiece optical system will be described later. The camera 20 is provided on the front surface of the housing 2. The camera 20 can capture an area of a predetermined size (hereinafter referred to as a predetermined area) in the external scenery in front of the housing 2.

  The image display unit 14 includes a liquid crystal element and a light source. The image display unit 14 can display a content image. The content image is a still image or a moving image. Further, the image display unit 14 can display the captured image superimposed on the content image. The captured image is an image captured by the camera 20. The image display unit 14 displays an image based on the video signal transmitted from the CB 50 via the harness 7. The eyepiece optical system condenses image light indicating the image displayed on the image display unit 14 and emits it to the half mirror 3. The image light emitted from the eyepiece optical system is reflected by the half mirror 3. The image light reflected by the half mirror 3 is emitted to the rear of the housing 2 (the direction opposite to the shooting direction of the camera 20). With the HMD 1 being worn by the user, the image light is incident on the left eyeball of the user. The half mirror 3 transmits light from the real image of the outside world toward the left eye of the user. The HMD 1 can allow a user to visually recognize a scene in which an image displayed on the image display unit 14 is superimposed on a real image (for example, an external scene or the like) within the visual field range of the user.

  The glasses 5 hold the HD 10 on the user's head. The glasses 5 include a frame 6 and a support portion 4. Since the shape of the frame 6 is substantially the same as that of normal glasses, detailed description is omitted. A support 4 is provided at the right end of the upper surface of the rim that supports the left-eye lens in the frame 6. The support unit 4 holds the housing 2 of the HD 10. The support part 4 can move the holding position of the housing 2 in the vertical direction and the horizontal direction. The user can adjust the position so that the position of the eyeball of the left eye and the position of the half mirror 3 are aligned in the front-rear direction by moving the housing 2 in the vertical direction and the horizontal direction.

  The HD 10 may be attached to other wearing tools such as glasses, helmets, and headphones that are used on a daily basis by the user. The image display unit 14 may be another spatial modulation element. For example, the image display unit 14 may be a retinal scanning display unit that performs two-dimensional scanning with laser light having an intensity corresponding to an image signal, and an organic EL (Organic Electro-luminescence) display. In FIG. 1, the support unit 4 and the HD 10 are provided on the right side of the frame 6, but the support unit 4 and the HD 10 may be provided on the right side of the frame 6. In this case, the image light is incident on the right eyeball of the user while the HMD 1 is attached to the user.

  In the present embodiment, the HD 10 is a so-called optical see-through head display having a function of allowing a user to visually recognize a scene in which an image displayed on the image display unit 14 is superimposed on a real image of the outside world. However, the present invention may be a head display having another function. For example, the HD 10 is a so-called video that has a function of allowing a user to visually recognize an image displayed on the image display unit 14 and an image of the outside world captured by the camera without causing light from the real image of the outside world to enter the user's eyes. A see-through type head display may be used.

  In this embodiment, the camera 20 is provided on the front surface of the housing 2, but the camera 20 may be detachable from the housing 2. For example, the camera 20 may be directly attached to the frame 6 of the glasses 5 in a state where the front direction of the HD 10 can be photographed.

  An outline of the CB 50 will be described. CB50 is attached to a user's waist belt, an arm, etc., for example. The CB 50 controls the HD 10. The CB 50 is detachably connected to the HD 10 via the harness 7. The CB 50 includes a casing 60, an operation switch 61, a power switch 62, and the like as main components. The shape of the housing 60 is a substantially rectangular parallelepiped with rounded edges. The operation switch 61 is a switch for performing various settings in the HD 10 and various operations during use. The power switch 62 is a switch for turning on or off the power of the HMD 1. The power switch 62 incorporates a power lamp 63.

  The electrical configuration of the HMD 1 will be described with reference to FIG. First, the electrical configuration of the HD 10 will be described. The HD 10 includes a CPU 11 that controls the entire HD 10. The CPU 11 is electrically connected to the RAM 12, the program ROM 13, the image display unit 14, the interface 15, and the connection controller 19. The CPU 11 is electrically connected to the camera 20 via the interface 15. The RAM 12 temporarily stores various data. The program ROM 13 stores a program executed by the CPU 11. The program ROM 13 stores the OS. The program is executed on the OS. The program and the OS are stored in the program ROM 13 when the HMD 1 is shipped. The image display unit 14 can display at least one of a content image and a captured image. The interface 15 is connected to the camera 20 and controls input / output of signals. The connection controller 19 is connected to the connection controller 58 of the CB 50 via the harness 7 and performs wired communication.

  The HD 10 may further include a flash ROM. The flash ROM may store a program executed by the CPU 11. The CPU 11 may execute a program stored in the flash ROM. The program executed by the CPU 51 of the CB 50 may be stored in the program ROM 13 and the flash ROM. The CPU 11 may execute the same process as the process executed by the CPU 51 of the CB 50 instead of the CPU 51.

  The electrical configuration of the CB 50 will be described. The CB 50 includes a CPU 51 that controls the entire CB 50. The CPU 51 is electrically connected to the RAM 52, program ROM 53, flash ROM 54, interface 55, video RAM 56, image processing unit 57, connection controller 58, and wireless communication unit 59. The RAM 52 temporarily stores various data. The RAM 52 stores a table 521 (see FIG. 8). The program ROM 53 stores a program executed by the CPU 51. The program ROM 53 stores the OS. The program is executed on the OS. The program and OS are stored in the program ROM 53 when the HMD 1 is shipped. The flash ROM 54 stores a table 541 (see FIG. 7).

  The interface 55 is connected to the operation switch 61 and the power switch 62 and controls signal input / output. The image processing unit 57 forms an image to be displayed on the image display unit 14 of the HD 10 and stores it in the video RAM 56. The video RAM 56 temporarily stores the image formed by the image processing unit 57. The connection controller 58 is connected to the connection controller 19 of the HD 10 via the harness 7 and performs wired communication. The wireless communication unit 59 communicates with peripheral devices connected to the Internet, LAN, and the like via an access point (not shown) connected to the Internet, LAN, and the like. Examples of peripheral devices include personal computers, smartphones, tablet portable terminals, servers, and the like.

  Note that a program executed by the CPU 51 may be stored in the flash ROM 54 of the CB 50. The CPU 51 may execute a program stored in the flash ROM 54. The program executed by the CPU 11 of the HD 10 may be stored in the program ROM 53 and the flash ROM 54. The CPU 51 may execute the same process as the process executed by the CPU 11 of the HD 10 instead of the CPU 11.

  The HMD 1 may download and install the program and the OS from the program download server via the wireless communication unit 59. For example, the program and the OS may be transmitted from the server to the HMD 1 as a computer-readable temporary storage medium (for example, a transmission signal). The program may be stored in a computer-readable storage device provided in the HMD 1, for example, a flash ROM. The CPU 11 may execute a program based on a program stored in the flash ROM. The CPU 51 may execute a program based on the program stored in the flash ROM 54. However, the storage device may be a storage medium other than a temporary storage medium such as a ROM, a flash ROM, an HDD, or a RAM. Further, the storage device may be a non-temporary storage medium. The non-transitory storage medium may be capable of retaining data regardless of the length of time for storing the data.

  The configuration of the HMD 1 is not limited to the above embodiment, and may be a configuration in which the HD 10 and the CB 50 are integrated, for example. The CPU 11 of the HD 10 and the CPU 51 of the CB 50 may communicate wirelessly instead of the harness 7.

  When the HMD 1 determines that the tendency of the fingertip position over time when the user's fingertip moves within a predetermined area captured by the camera 20 satisfies a predetermined condition, the HMD 1 executes a corresponding process. Specific examples of the corresponding processing include various setting processing, switching of content images displayed on the image display unit 14, scrolling of content images, power control, and the like. The user needs to move the fingertip so as to satisfy a predetermined condition in order to cause the HMD 1 to execute a desired process. The HMD 1 repeatedly displays a trajectory image indicating the position of the fingertip on the image display unit 14 at the timing when the position of the fingertip is specified so that the user can easily recognize a change with time of the position of the fingertip. In the present embodiment, it is assumed that the image displayed on the image display unit 14 is only a trajectory image, and the captured image captured by the camera 20 is not displayed on the image display unit 14.

  This will be specifically described with reference to FIG. In the following, for ease of explanation, the processing executed by the CPU 11 of the HD 10 and the processing executed by the CPU 51 of the CB 51 are not distinguished from each other and will be described as processing executed by the HMD 1. Actually, the following processing is assigned to each of the CPUs 11 and 51 and executed by each of them. Details will be described later with reference to the flowcharts of FIGS.

  FIG. 3 shows a scene 25 visually recognized by a user wearing the HD 10. The scene 25 includes a real image 22 (22A, 22F to 22J). The real image 22 includes a user's hand 23. The real image 22 is an image visually recognized by the user when light from the real image within the visual field range of the user passes through the half mirror 3 and reaches the left eye of the user. When an image is displayed on the image display unit 14, the image is superimposed on the real image 22. Thereby, the user can visually recognize both the real image 22 and the image within the visual field range. An area surrounded by a rectangular dotted line indicates a predetermined area 21 (21A to 21J) photographed by the camera 20.

  An example will be given in which the HMD 1 executes a corresponding process (for example, a setting process of the HMD 1) when it is determined that the position of the fingertip moves vertically from top to bottom as time passes. The HMD 1 acquires a captured image captured by the camera 20 and determines whether the fingertip is included in the captured image. Details of the method for determining whether a fingertip is included will be described later. When the HMD 1 determines that the fingertip is included in the captured image, the HMD 1 specifies the position of the fingertip based on the captured image. Specifically, the specified fingertip position is coordinate information when a specific position in a predetermined area photographed by the camera 20 is used as a reference of the coordinate system.

  The HMD 1 displays the trajectory image 26 (26A to 26J) on the image display unit 14 based on the specified fingertip position. The trajectory image 26 is circular. The HMD 1 adjusts the position of the trajectory image 26 displayed on the image display unit 14 so that the trajectory image 26 overlaps the position of the fingertip included in the real image 22. In a state before the fingertip starts to move, the user can visually recognize the trajectory image 26A superimposed on the position of the fingertip of the hand 23 in the real image 22A, as shown by the scene 25A.

  The HMD 1 repeatedly acquires a captured image captured by the camera 20 and repeatedly specifies the position of the fingertip included in the captured image. The HMD 1 repeatedly displays the trajectory image 26 on the image display unit 14. When the user moves the fingertip from the top to the bottom in the vertical direction, the HMD 1 specifies a plurality of different fingertip positions. The HMD 1 causes the image display unit 14 to display the trajectory images 26B to 26E in order based on the plurality of specified positions. In this case, the HMD 1 does not erase the trajectory image 26 once displayed on the image display unit 14. For this reason, the user visually recognizes the trajectory images 26A to 26E that gradually increase as the fingertip is moved in the vertical direction from the top to the bottom as shown in each of the scenes 25A to 25E. Therefore, the user can easily grasp that the fingertip is moving vertically from top to bottom by visually recognizing the trajectory images 26A to 26E.

  In addition, HMD1 is the timing which displays each of locus image 26B, 26C, 26D, 26E on the image display part 14, between locus image 26A-26B, between locus image 26B-26C, between locus image 26C-26D, and locus | trajectory. A dotted line extending between the images 26 </ b> D- 26 </ b> E may be displayed on the image display unit 14. As a result, the user can more easily grasp the movement of the fingertip over time.

  Further, the HMD 1 determines that the position of the fingertip is moving vertically from top to bottom with the passage of time based on the positions of the plurality of fingertips identified repeatedly. In this case, the HMD 1 identifies the corresponding process and executes the process. Therefore, the user can cause the HMD 1 to execute processing by moving the fingertip vertically from the top to the bottom within a predetermined area photographed by the camera 20.

  The camera 20 is provided on the front surface of the housing 2 of the HD 10 and photographs the outside world in the front direction of the HD 10. Accordingly, the user moves the fingertip in front of the eyes so that the fingertip is photographed by the camera 20. Here, when the user wearing the HD 10 changes the orientation of the head, the predetermined area 21 photographed by the camera 20 also changes. A case where the user wearing the HD 10 changes the direction of the head in the horizontal direction will be described as an example. The predetermined area photographed by the camera 20 gradually changes like the predetermined areas 21F to 21J. At this time, as in the case of the scenes 25A to 25E, a case where the user moves the fingertip vertically from the top to the bottom will be described as an example. In this case, as indicated by the scenes 25F to 25J, the positional relationship between the real images 22F to 22J and the fingertip does not change even when the shooting direction of the camera 20 changes, and matches the case of the scenes 25A to 25E. On the other hand, the positional relationship between the predetermined areas 21F to 21J and the real images 22F to 22J changes in the horizontal direction in response to the shooting direction of the camera 20 changing in the horizontal direction. Accordingly, the relative position of the fingertip with respect to the predetermined areas 21F to 21J changes not only in the vertical direction but also in the horizontal direction as the shooting direction of the camera 20 changes in the horizontal direction.

  The HMD 1 specifies the coordinate information when a specific position in a predetermined area photographed by the camera 20 is used as a reference of the coordinate system as the position of the fingertip. When the user wearing the HD 10 changes the head direction in the horizontal direction, the shooting direction of the camera 20 also changes, so the reference position of the coordinate system corresponding to the coordinate information changes in the horizontal direction. For this reason, even when the user moves the fingertip vertically from the top to the bottom, the HMD 1 cannot determine that the position of the fingertip is moving vertically from top to bottom as time passes. For this reason, the HMD 1 cannot identify the corresponding process.

  Further, as shown by the scenes 25F to 25J, the trajectory images 26F to 22J superimposed on the real images 22F to 22J are vertical when the user changes their fingertips from top to bottom when the shooting direction of the camera 20 changes in the horizontal direction. In spite of being moved to, it is in a state of being arranged in a straight line in an oblique direction. Therefore, it becomes difficult for the user to grasp that the fingertip is moving vertically from top to bottom.

  For this reason, the HMD 1 performs the following processing to identify an appropriate position of the fingertip and execute the corresponding processing even when the shooting direction of the camera 20 changes. Further, the HMD 1 adjusts the position of the trajectory image 26 superimposed on the real image 22 so that the user can appropriately grasp the movement of the fingertip by performing the following processing. Below, the outline | summary of the process performed by CPU11 of HD10 is demonstrated first. Next, processing executed by the CPU 51 will be described in detail with reference to the flowcharts of FIGS.

  An outline of processing executed by the CPU 11 of the HD 1 will be described. When the operation of turning on the power of the HMD 1 is performed via the power switch 62 of the CB 50, the CPU 11 starts the next process. The CPU 11 starts shooting with the camera 20. When shooting starts, the camera 20 continuously outputs image data while shooting in a predetermined area. Hereinafter, the image data output from the camera 20 is referred to as photographed image data. The image format of the captured image data is a Motion JPEG format, an Audio Video Interleave format, a QuickTime format, or the like. The CPU 11 receives captured image data output from the camera 20 via the interface 15. The CPU 11 transmits the captured image data to the CB 50.

  Further, the CPU 11 receives a video signal of the trajectory image 26 (see FIG. 3) from the CB 50. The CPU 11 displays the locus image 26 on the image display unit 14 by controlling the image display unit 14 based on the received video signal. A user wearing the HD1 visually recognizes the trajectory image 26 when light from the image displayed on the image display unit 14 reflects the half mirror 3 and enters the left eye. Since the light from the real image 22 within the visual field range is incident on the left eye of the user wearing the HD1 through the half mirror 3, the user can view the trajectory image 26 displayed on the image display unit 14 from the outside world. The scene superimposed on the real image 22 is visually recognized.

  In the above description, the camera 20 may directly transmit the captured image data to the CB 50. Further, the image display unit 14 may directly receive a video signal from the CB 50.

  Details of processing executed by the CPU 51 of the CB 50 will be described with reference to FIGS. 4 and 5. The main process is started when the CPU 51 executes a program stored in the program ROM 53 when an operation for turning on the power of the HMD 1 is performed via the power switch 62. The CPU 51 initializes the variable n stored in the RAM 52 by substituting 0 (S10). n is associated as an index indicating the stored order when information is stored in a table 541 (see FIG. 7) described later. The CPU 51 receives the captured image data transmitted from the HD 10 via the connection controller 58 (S11). The CPU 51 stores the received captured image data in the RAM 52.

  The CPU 51 performs analysis based on the captured image data stored in the RAM 52 (S13), and specifies the fingertip from the captured image. Specifically, it is as follows. The CPU 51 detects a skin color region from the captured image corresponding to the captured image data. The skin color area can be detected by, for example, detecting an area where pixels having predetermined pixel values (for example, R = 241, G = 187, B = 147) corresponding to the skin color are continuous. Note that the predetermined pixel value may be another value or may have a predetermined width. The CPU 51 identifies a polygon that surrounds the entire detected skin color area. The CPU 51 performs a well-known pattern matching process based on the specified polygonal shape and the fingertip specifying image template stored in advance in the program ROM 53. When the cross-correlation coefficient calculated by the pattern matching process is a value within a predetermined range, the CPU 51 determines that the pattern matching between the specified polygonal shape and the image template is successful. In this case, the CPU 51 specifies the vertex of the polygonal shape as the fingertip. Note that the method for determining whether the captured image includes the fingertip is not limited to the above method.

  The CPU 51 determines whether the fingertip can be specified by the above method (S15). If the fingertip can be identified, the CPU 51 determines that the fingertip is included in the captured image (S15: YES). The CPU 51 adds 1 to n (S16). The CPU 51 starts updating a timer variable for measuring the elapsed time after determining that the fingertip is included in the captured image (S17). The timer variable is incremented by 1 at a constant cycle (for example, 1 ms) by the OS. Based on the value of the timer variable, the CPU 51 can specify the elapsed time since it is determined that the fingertip is included in the captured image.

  The CPU 51 specifies coordinate information indicating the position of the specified fingertip (S18). The reference point of the coordinate information is the lower left corner in the predetermined area 21 (see FIG. 3) photographed by the camera 20. The left-right direction of the predetermined region 21 is the X-axis direction, and the vertical direction is the Y-axis direction. The right direction and the upward direction are positive directions, and the left direction and the downward direction are negative directions. The CPU 51 stores the specified coordinate information in the RAM 52.

  The CPU 51 performs the following process to display the trajectory image 26 (see FIG. 3) on the image display unit 14 based on the coordinate information specified in S18. The CPU 51 specifies a position (fingertip position) on the real image 22 (see FIG. 3) corresponding to the position in the predetermined area 21 indicated by the coordinate information. The CPU 51 drives the image processing unit 57 to form a trajectory image 26 to be displayed at a position on the identified real image 22 and store it in the video RAM 56. The CPU 51 generates a video signal based on the trajectory image 26 stored in the video RAM 56. The format of the video signal is Transition Minimized Differential Signaling (TMDS), Low voltage differential signaling (LVDS), or the like. The CPU 51 transmits a video signal to the HD 10 via the connection controller 58 (S19). Note that the video signal is not limited to being generated by the CPU 51. For example, the connection controller 19 may generate the video signal based on the content image stored in the video RAM 56.

  As described above, the CPU 11 of the HD 10 receives the video signal of the trajectory image 26 transmitted from the CB 50 via the connection controller 19. The CPU 11 displays the locus image 26 on the image display unit 14 by controlling the image display unit 14 based on the received video signal. Therefore, the CPU 51 displays a trajectory image on the image display unit 14 by transmitting a video signal to the HD 10. In the following, the CPU 51 drives the image processing unit 57 to form the trajectory image 26, stores the formed trajectory image 26 in the video RAM 56, and generates a video signal based on the trajectory image 26 stored in the video RAM 56. When the video signal is transmitted to the HD 10 to display the trajectory image 26 on the image display unit 14, the CPU 51 displays the trajectory image 26 on the image display unit 14.

  Further, as will be described later, until the CPU 51 determines that the tendency of the fingertip position over time satisfies a predetermined condition in S35, or after specifying the fingertip in S41, S43, and S45. S19 is repeatedly executed until it is determined that the elapsed time is greater than the predetermined value, and the trajectory image 26 is displayed on the image display unit 14. Further, as will be described later, the CPU 51 causes the image display unit 14 to display the corrected trajectory image 26 by the correction processing in S27. When the CPU 51 displays the trajectory image 26 on the image display unit 14 for the nth time in S19, the trajectory image 26 corresponding to the 1st to (n-1) th time displayed on the image display unit 14 in S19 and S27 is also simultaneously displayed. Display. Therefore, when the trajectory image 26 is displayed on the image display unit 14 for the nth time in S <b> 19, n trajectory images 26 are displayed on the image display unit 14.

  Based on the photographed image data acquired in S11, the CPU 51 identifies characteristic points arranged in the vicinity of the four corners of the predetermined area 21 in the photographed image as points corresponding to the photographing direction of the camera 20 (S21). . Hereinafter, the specified characteristic point is referred to as a reference point. This will be specifically described with reference to the captured image 31 of FIG. The captured image 31 indicates a captured image corresponding to the captured image data received from the HD 10. A frame line surrounding the photographed image 31 indicates a predetermined area 21 photographed by the camera 20. The shape of the predetermined area 21 is a rectangle. The CPU 51 defines arcs 33A, 33B, 33C, and 33D that are centered on vertices 32A (upper left), 32B (upper right), 32C (lower left), and 32D (lower right) of the four corners of the predetermined area 21, respectively. Hereinafter, the arcs 33A to 33D are collectively referred to as an arc 33. The radius of the arc 33 is shorter than the half length of the short side of the predetermined region 21. The CPU 51 includes a fan-shaped inner portion 34A (upper left) including the arc 33A in the photographed image 31, a fan-shaped inner portion 34B (upper right) including the arc 33B, a fan-shaped inner portion 34C (lower left) including the arc 33C, And each part of 34D (lower right) of the fan-shaped inside containing arc 33D is extracted. The portions 34 </ b> A to 34 </ b> D respectively correspond to portions in the vicinity of the four corners of the predetermined region 21 in the captured image 31. For each of the extracted portions 34A to 34D, the CPU 51 uses an algorithm that uses a gradient of the first derivative such as a technique using the Sobe1 operator or a Canny method, or a zero crossing point of a second derivative to detect a local maximum of the gradient. The boundary point is detected by a well-known edge detection method such as an algorithm using. The CPU 51 identifies a point having the largest density difference among the detected plurality of boundary points as a characteristic point in each part. Thereby, the CPU 51 specifies the reference points 35A to 35D from the portions 34A to 34D, respectively. Hereinafter, the reference points 35A to 35D are collectively referred to as a reference point 35.

  Note that the CPU 51 can identify a common characteristic point in the captured image 31 as the reference point 35 from each portion even when S21 (see FIG. 4) is repeatedly executed. The reason is that various parameters (gradient value and density difference) of the edge detection method when specifying the reference point 35 and an algorithm for selecting a reference point from a plurality of boundary points detected by edge detection are: This is because it is common to the cases where S21 is repeatedly executed. Specifically, it is as follows. Each of the house roof (reference point 35A), the roof of the tenement house (reference point 35B), the entrance of the house (reference point 35C), and the specific position of the side wall of the tenement house (reference point 35D) included in the captured image 31 A case where the reference point 35 is specified will be described as an example. The CPU 51 stores the coordinate information indicating the positions of the reference points 35 </ b> A to 35 </ b> D in the RAM 52 in association with the parameters when the reference points 35 </ b> A to 35 </ b> D are specified by the edge detection method. When the CPU 51 repeatedly executes S21, the CPU 51 refers to the parameters stored in the RAM 52. Specifically, the CPU 51 identifies the boundary point corresponding to the parameter closest to the parameter stored in the RAM 52 among the plurality of parameters when the plurality of boundary points are detected by the edge detection method as the reference point 35. . As a result, the CPU 51 changes the shooting direction of the camera 20 by changing the orientation of the user's head, and even if the landscape included in the captured image 31 changes, the roof or the long house of the house included in the captured image 31 is changed. Each of the specific positions of the roof, the entrance of the house, and the side wall of the tenement can be identified repeatedly as reference points 35A-35D.

  The reason for specifying the four reference points 35 is that the reference point 35 may deviate from the inside of the predetermined area 21 photographed by the camera 20 due to the change of the head direction by the user wearing the HD 10. It is. Moreover, it is because the reference point 35 in the predetermined area | region 21 image | photographed with the camera 20 may be hidden by a user's hand. The CPU 51 specifies a total of four reference points 35 from the portions 34 </ b> A to 34 </ b> D near the four corners of the predetermined area 21, so that even when the orientation of the user's head changes or when the position of the user's hand changes. Any reference point 35 can always be specified.

  As shown in FIG. 4, the CPU 51 subtracts each X component of the coordinate information indicating the positions of the four reference points 35 specified in S21 from the X component of the coordinate information indicating the position of the fingertip specified in S18. The calculation results of the four X components are specified. CPU51 subtracts each Y component of the coordinate information which shows the position of the four reference points 35 specified by S21 from the Y component of the coordinate information which shows the position of the fingertip specified by S18, and the calculation result of four Y components Is identified. The CPU 51 associates each of the specified four X component calculation results and the four Y component calculation results, and specifies four difference information (S23). Each of the four pieces of difference information indicates a relative position of the fingertip position when the positions of the four reference points 35 are used as a reference.

  Note that if the CPU 51 cannot identify any of the four reference points 35 in S21, for example, if the user wearing the HD 10 changes the orientation of the head, any one of the reference points 35 is set to the predetermined area. When it deviates from the inner side of 21, the difference information is not calculated based on any reference point 35. However, as described above, the reference point 35 is specified from each of the vicinity of the four corners of the predetermined area 21, so that the CPU 51 always differs from any reference point 35 even when the orientation of the user's head changes. Information can be calculated.

  The CPU 51 associates the four coordinate information indicating the position of the reference point 35 specified in S21, the coordinate information indicating the position of the fingertip specified in S18, and the four pieces of difference information specified in S23, as shown in FIG. Store in the table 541 (S25). The CPU 51 further associates the value of n as an index and stores it in the table 541. Hereinafter, coordinate information indicating the position of the reference point is referred to as first information. The coordinate information indicating the position of the fingertip is referred to as second information. The difference information is referred to as third information.

  The table 541 will be described with reference to FIG. The table 541 stores an index, four pieces of first information, second information, and four pieces of third information in association with each other. The index is the order in which the four first information, the second information, and the four third information are stored in the table 541. In other words, the position of the fingertip is specified in S18, and the position of the reference point 35 is specified in S21. The order in which the third information is calculated in S23 is shown.

  Note that, in FIG. 7, the shooting direction of the camera 20 changes in the horizontal direction and the position of the fingertip moves in the vertical direction from top to bottom as shown by the scenes 25 </ b> F to 25 </ b> J in FIG. 3. The table 541 created in this case is specifically shown. In the scenes 25F to 25J of FIG. 3, the position of the fingertip changes in an oblique direction from the upper right to the lower left in the predetermined area 21 as the shooting direction of the camera 20 changes in the horizontal direction. This tendency of the position change is indicated by the fact that each of the X component and the Y component of the second information decreases by 1 as the index increases. On the other hand, for each of the four pieces of third information, only the Y component decreases as the index increases, and the X component does not change. The reason is that each of the four pieces of third information indicates the relative position of the fingertip position when the position of the reference point 35 is used as a reference. Since the position of the reference point 35 changes according to the shooting direction of the camera 20, the relative position of the fingertip position with respect to the position of the reference point 35 is not affected by the movement of the user's head. As described above, each of the four pieces of third information indicates a tendency of the position change corresponding to the actual movement of the fingertip.

  As illustrated in FIG. 4, the CPU 51 refers to the table 541 and obtains the first information corresponding to the index of n−1 in S21 until the first information corresponding to the index of n is obtained in S21. In the meantime, it is determined whether the relative position of the fingertip with respect to the position of the reference point 35 has changed (S26). Specifically, it is as follows. The CPU 51 acquires four pieces of third information associated with the n-1 index in the table 541 and four pieces of third information associated with the n index. Hereinafter, the four pieces of third information associated with the n−1 index are referred to as four pieces of third information (n−1). The four pieces of third information associated with the n indexes are referred to as four pieces of third information (n). CPU51 compares the 3rd information corresponding to the same reference point 35 about four acquired 3rd information (n-1) and four 3rd information (n). When any of the four pieces of third information is different, the relative position of the fingertip with respect to the reference point 35 is changed. If the CPU 51 determines that the relative position of the fingertip has changed when the reference point 35 is used as a reference (S26: YES), the CPU 51 corrects the trajectory image 26 displayed on the image display unit 14 (correction processing, (See FIG. 5) (S27).

  The correction process will be described with reference to FIG. The CPU 51 determines whether n is greater than 1 (S50). If the CPU 51 determines that n is not greater than 1 (S50: NO), it ends the correction process and returns the process to the main process (see FIG. 4). When the CPU 51 determines that n is larger than 1 (S50: YES), the CPU 51 stores 1 in the variable i stored in the RAM 52 (S53). The CPU 51 acquires four pieces of first information and second information corresponding to the index i in the table 541 (S55). The CPU 51 acquires four pieces of first information corresponding to the n indexes in the table 541 (S57).

  Based on the four pieces of first information (index: i) acquired in S55 and the four pieces of first information (index: n) acquired in S57, the CPU 51 determines whether the third information corresponding to the same reference point 35 is the same. By subtracting each of the X component and the Y component, a difference between the X component and the Y component corresponding to each of the four reference points 35 is calculated. Hereinafter, the difference between the X components is referred to as an X difference, and the difference between the Y components is referred to as a Y difference. The CPU 51 selects an X difference and a Y difference corresponding to one of the reference points 35 among the X difference and the Y difference calculated for the same reference point 35. For example, the CPU 51 in the captured image 31 (see FIG. 6) has an upper left portion 34A (see FIG. 6), an upper right portion 34B (see FIG. 6), a lower right portion 34C (see FIG. 6), and a lower right portion. The order of 34D (see FIG. 6) may be set as the priority. The CPU 51 may preferentially select the X difference and the Y difference corresponding to the reference point 35 specified from the high priority part. The CPU 51 adds the selected X difference and Y difference. Note that the value obtained by adding the X difference and the Y difference is from the time when the first information corresponding to the i index in the table 541 is acquired in S11 to the time when the first information corresponding to the n index is acquired in S11. This corresponds to the amount of movement of the position of the reference point 35 in between. That is, the CPU 51 calculates a value corresponding to the amount of movement of the position of the reference point 35 by adding the X difference and the Y difference (S59). Hereinafter, a value calculated by adding the X difference and the Y difference is referred to as a change amount. The CPU 51 may calculate the amount of change by squaring each of the X difference and the Y difference, adding each squared value, and calculating the square root of the added value.

  The CPU 51 determines whether the change amount of the position of the reference point 35 calculated in S59 is smaller than a predetermined value (for example, 1) (S61). If the CPU 51 determines that the amount of change in the position of the reference point 35 is smaller than the predetermined value (S61: YES), the CPU 51 advances the process to S67. When the amount of change in the position of the reference point 35 is not smaller than the predetermined value (S61: NO), the CPU 51 corrects the second information corresponding to the index i in the table 541 by performing the following process (S61: NO). S63). The CPU 51 subtracts the X difference calculated in S59 from the X component of the second information corresponding to the index i in the table 541. The CPU 51 subtracts the Y difference calculated in S59 from the Y component of the second information corresponding to the index i in the table 541. Hereinafter, the calculation results of the X component and the Y component are referred to as correction information, and the position indicated by the correction information is referred to as a correction position. The CPU 51 associates i and n with the correction information and stores them in the table 521 shown in FIG. 8 (S65).

  The table 521 will be described with reference to FIG. The table 521 stores i, n and correction information in association with each other. In FIG. 8, in order to facilitate understanding, the second information before correction and an arrow are additionally shown before each correction information. As will be described later, correction information is not calculated when i = n (S69, see FIG. 5), but for the sake of easy understanding, the trajectory image displayed nth in S19 (see FIG. 4). Coordinate information indicating the position of 26 is shown in association with i = n. Details of the table 521 will be described later.

  As shown in FIG. 5, after storing the correction information in the table 521 in S65, the CPU 51 adds 1 to i (S67). The CPU 51 compares i and n obtained by adding 1 in S67 (S69). When the CPU 51 determines that i and n do not match (S69: NO), it determines that i is smaller than n. In this case, the CPU 51 returns the process to S55. CPU51 repeats the process of S55-S67 based on i which added 1 by S67. As a result, the CPU 51 corrects the second information stored in the table 541 in the order of the indexes, and stores i, n and the correction information in the table 521 in association with each other.

  When i and n match each other, the second information corresponding to each of the indexes 1 to n−1 in the table 541 has been corrected. When the CPU 51 determines that i and n match (S69: YES), the trajectory image 26 corresponding to the 1st to (n-1) th of the trajectory images 26 displayed on the image display unit 14 is deleted. (S71). The trajectory image 26 corresponding to the 1st to (n-1) th is a trajectory image in which each position of the trajectory image 26 displayed in 1st to nth in S19 is corrected in S63 and displayed in a corrected position in S73 described later. 26 is shown.

  Specifically, it is as follows. The CPU 51 specifies a position in the predetermined area 21 indicated by the second information corresponding to the index n in the table 541. Next, the CPU 51 causes the image display unit 14 to display a trajectory image 26 to be superimposed on a position on the real image 22 corresponding to the specified position. As a result, only the trajectory image 26 displayed in the step S19 is displayed, and the trajectory images 26 corresponding to the 1st to (n-1) th are not displayed. Next, the CPU 51 specifies a correction position in the predetermined area 21 indicated by the correction information stored in the table 521. The CPU 51 causes the image display unit 14 to display the trajectory image 26 superimposed on the position on the real image 22 corresponding to the specified correction position (S73). As a result, instead of each of the trajectory images 26 corresponding to the 1st to (n-1) th deleted in S71, n-1 trajectory images 26 whose positions are corrected are displayed as the nth trajectory image 26. It is added and displayed. The CPU 51 ends the correction process and returns the process to the main process (see FIG. 4).

  Hereinafter, the nth trajectory image 26 is referred to as the nth trajectory image 26. Further, the trajectory image 26 when the position of the 1st to (n-1) th trajectory image 26 is corrected to the correction position by the correction process and displayed on the image display unit 14 is referred to as the 1st to (n-1) th trajectory image. .

  As shown in FIG. 4, after the correction process (S27) is completed, the CPU 51 determines whether or not a change with time of the fingertip position satisfies a predetermined condition based on the plurality of pieces of third information stored in the table 541. (S29, S35).

  The specific method of judgment is as follows. The CPU 51 selects third information corresponding to any reference point 35 from among the four pieces of third information stored in association with each of the plurality of indexes in the table 541. For example, the CPU 51 in the captured image 31 (see FIG. 6) has an upper left portion 34A (see FIG. 6), an upper right portion 34B (see FIG. 6), a lower right portion 34C (see FIG. 6), and a lower right portion. The order of 34D (see FIG. 6) may be set as the priority. CPU51 may select the 3rd information corresponding to reference point 35 specified from the part with high priority. Next, the CPU 41 connects the positions indicated by each of the plurality of pieces of third information selected for each index by a line in the order of the index, and specifies the line segment indicating the locus of movement of the fingertip. As a result, the CPU 51 identifies the change of the fingertip position with time, that is, the movement of the fingertip (S29).

  The CPU 51 performs a well-known pattern matching process based on the specified line segment and a plurality of image templates for specifying a locus stored in advance in the program ROM 53. The plurality of image templates for specifying the trajectory are templates of images showing specific shapes. The specific shape is, for example, a straight line extending in the vertical direction, a straight line extending in the left-right direction, a triangle, or a circle. The CPU 51 calculates a plurality of cross-correlation coefficients corresponding to each of the plurality of image templates by pattern matching processing, and determines whether the cross-correlation coefficient closest to 1 is a value within a predetermined range. If the cross-correlation coefficient closest to 1 is a value within a predetermined range, the CPU 51 determines that the pattern matching has succeeded. In this case, the CPU 51 determines that the change tendency of the fingertip position matches the specific shape indicated by the image template corresponding to the cross-correlation coefficient closest to 1 (S35: YES). In addition, since several 3rd information has shown the relative position with respect to the reference point 35, it shows the position of an actual fingertip. For this reason, the CPU 51 can appropriately determine whether the tendency of the change of the fingertip position satisfies a predetermined condition even when the shooting direction of the camera 20 changes. CPU51 advances a process to S37.

  Note that a method for determining whether a change in the position of the fingertip over time satisfies a predetermined condition is not limited to the above method. For example, when the tendency of change of the plurality of third information selected for each index satisfies any of a plurality of predetermined conditional expressions, the CPU 51 corresponds to any of the predetermined conditional expressions. It may be determined that the condition is satisfied.

  The CPU 51 deletes all the trajectory images 26 displayed on the image display unit 14 (S37). The reason is that since it is already determined that the change in the fingertip position with time satisfies the predetermined condition, the user does not need to move the fingertip any more and the trajectory image 26 is unnecessary for the user. Therefore, the CPU 51 can erase the locus image 26 unnecessary for the user by the above processing. Further, the CPU 51 can notify the user that the tendency of the change of the fingertip position is determined to satisfy a predetermined condition by deleting the trajectory image 26.

  The CPU 51 refers to the processing content DB stored in the program ROM 53. In the processing content DB, information indicating the types of a plurality of image templates and information indicating the processing content of the HMD 1 are stored in association with each other. The CPU 51 specifies the processing content corresponding to the image template corresponding to the cross-correlation coefficient closest to 1 calculated when determining the fingertip movement in S35 by referring to the processing content DB. The CPU 51 executes processing based on the specified processing content (S39). By referring to the processing content DB, the CPU 51 can easily specify the processing corresponding to the change tendency of the fingertip position. The CPU 51 stops updating the timer variable started in S17 (S40). The CPU 51 returns the process to S11.

  When the CPU 51 determines that the fingertip cannot be specified from the captured image corresponding to the captured image data received in S11 while the above processing is repeated (S15: NO), the value of the timer variable that has started updating in S17 Based on the above, the elapsed time when the fingertip is not included in the captured image is specified. The CPU 51 determines whether the specified elapsed time is longer than a predetermined value (hereinafter referred to as the first time) and whether the trajectory image 26 is displayed on the image display unit 14 (S41). If the CPU 51 determines that the specified elapsed time is not longer than the first time or that the trajectory image 26 is not displayed on the image display unit 14 (S41: NO), the process returns to S11. When the CPU 51 determines that the specified elapsed time is longer than the first time and the trajectory image 26 is displayed on the image display unit 14 (S41: YES), the trajectory image displayed on the image display unit 14 is displayed. All 26 are erased (S42). CPU51 advances a process to S40.

  When it is first determined that the fingertip is included in the predetermined area 21 photographed by the camera 20 (S15), and the fingertip is not included in the predetermined area 21 continuously for the first time or more. (S41: YES), the user may not have performed a fingertip movement operation for causing the HMD 1 to execute a specific process. In such a case, in S35, it is impossible to determine whether or not the temporal change in the fingertip position satisfies a predetermined condition. In such a case, the trajectory image 26 becomes unnecessary for the user. Therefore, the CPU 51 can erase the locus image 26 unnecessary for the user by the above processing.

  While the above processing is repeated, the CPU 51 uses the third information corresponding to the same reference point 35 for the four third information (n−1) and the four third information (n) in the processing of S26. Are compared, and it is determined that the relative position of the fingertip has not changed when the reference point 35 is used as a reference (S26: NO). The CPU 51 specifies the elapsed time in which the relative position of the fingertip does not change based on the reference point 35 based on the value of the timer variable that has been updated in S17. The CPU 51 determines whether the specified elapsed time is longer than a predetermined value (hereinafter referred to as the second time) and the trajectory image 26 is displayed on the image display unit 14 (S43). If the CPU 51 determines that the specified elapsed time is not longer than the second time or that the trajectory image is not displayed on the image display unit 14 (S43: NO), the process returns to S11. When the CPU 51 determines that the specified elapsed time is longer than the second time and the trajectory image is displayed on the image display unit 14 (S43: YES), the trajectory image 26 displayed on the image display unit 14 is displayed. Are all deleted (S44). CPU51 advances a process to S40.

  After it is first determined that the fingertip is included in the predetermined area 21 photographed by the camera 20 (S15), a state in which the relative position of the fingertip with respect to the reference point 35 does not change is a second time or more. When continuing (S43: YES), possibility that a user is not performing the movement operation | movement of the fingertip for making HMD1 perform a specific process is high. In such a case, the trajectory image 26 becomes unnecessary for the user. Therefore, the CPU 51 can erase the locus image 26 unnecessary for the user by the above processing.

  In the process of S35, when the cross-correlation coefficient closest to 1 is a value outside the predetermined range in the process of S35, the CPU 51 satisfies the condition that the fingertip movement is indicated by a plurality of image templates. It is determined that the condition is not satisfied (S35: NO). CPU51 specifies the elapsed time of the state where the operation | movement of a fingertip does not satisfy | fill predetermined conditions based on the value of the timer variable which started the update by S17. The CPU 51 determines whether the specified elapsed time is longer than a predetermined value (hereinafter referred to as the third time) (S45). If the CPU 51 determines that the specified elapsed time is not greater than the third time (S45: NO), the process returns to S11. When the CPU 51 determines that the specified elapsed time is longer than the third time (S45: YES), the CPU 51 deletes all the trajectory images 26 displayed on the image display unit 14 (S46). CPU51 advances a process to S40.

  First, it is determined that the fingertip is included in the predetermined area 21 photographed by the camera 20 (S15), and after the trajectory image 26 is displayed (S19), the state where the operation of the fingertip does not satisfy the predetermined condition. When it continues for 3rd time or more (S45: YES), it is highly likely that the user has not performed the fingertip movement operation for causing the HMD 1 to execute a specific process. In such a case, the trajectory image 26 becomes unnecessary for the user. Therefore, the CPU 51 can erase the locus image 26 unnecessary for the user by the above processing.

  The case where the trajectory image 26 is displayed on the image display unit 14 will be specifically described with reference to FIGS. 4, 5, and 7 to 9. When the CPU 51 executes S17 and subsequent steps of the main process (see FIG. 4) in a state where n is “1”, the second information (13, 8) corresponding to the index “1” in the table 541 (see FIG. 7). ), The trajectory image 26F (scenery 25F, see FIG. 9) is displayed as the first trajectory image at the corresponding position of the image display unit 14 (S19, see FIG. 4). When the CPU 51 executes the correction process (S27, see FIG. 4), since n is not larger than 1 (S50: NO, see FIG. 5), the second information is not corrected (S63, see FIG. 5). The correction information is not stored in the table 521 (see FIG. 8) (however, in the table 521, as described above, in order to facilitate understanding, the second information (13, 13) indicating the position corresponding to the locus image 26F is used. 8) is shown in association with i = n = 1.)

  Next, in the case where n is “2” and the main process is executed after S17 of the main process, the CPU 51 displays the image display unit based on the second information (12, 7) corresponding to the index “2” in the table 541. The trajectory image 26G (scenery 25G, see FIG. 9) is displayed as the second trajectory image at the corresponding position 14 (S19, see FIG. 4). The CPU 51 also displays the first trajectory image 26F on the image display unit 14 (S19). In the table 521, the second information (12, 7) indicating the position corresponding to the trajectory image 26G is shown in association with i = n = 2. When executing the correction process (S27, see FIG. 4), the CPU 51 corrects the second information as follows because n is larger than “1” (S50: YES, see FIG. 5).

  The CPU 51 stores the first information (5, 9) (15, 9) (5, 1) (15, 1) and the second information (13, 8) corresponding to the index of “1” (= i) in the table. 541 (see FIG. 7) (S55, see FIG. 5). The CPU 51 acquires the first information (4, 9) (14, 9) (4, 1) (14, 1) corresponding to the index “2” (= n) from the table 541 (see FIG. 7). (See S57, FIG. 5). The CPU 51 calculates the X difference and the Y difference based on the first information acquired in S55 and S57. The calculated X differences are all “1”, and the Y differences are all “0”. The CPU 51 adds the X difference and the Y difference, and calculates a change amount “1” of the position of the reference point 35 (S59, see FIG. 5). Since the calculated change amount “1” of the position of the reference point 35 is not smaller than the predetermined value “1” (S61: NO, see FIG. 5), the CPU 51 sets “1” (= i) in the table 541. The X difference “1” calculated in S59 is subtracted from the X component “13” of the second information corresponding to the index to obtain a corrected X component “12”. The CPU 51 subtracts the Y difference “0” calculated in S59 from the Y component “8” of the second information corresponding to the index of “1” (= i) in the table 541 to obtain the corrected Y component “8”. 8 ”is obtained. The CPU 51 identifies the correction information (12, 8) and stores it in the table 521 (see FIG. 8) in association with i = 1 and n = 2 (S65, see FIG. 5). As a result, the second information is corrected from (13, 8) to (12, 8).

  The CPU 51 adds “1” to i (= 1) and the result “2” is the same as n (= 2) (S69: YES, see FIG. 5), so “1” (= n) in the table 541 The first trajectory image 26F displayed on the image display unit 14 is erased based on the second information corresponding to the index of -1) (S71, see FIG. 5) (see the scene 25G, FIG. 9). The CPU 51 uses the trajectory image 261 (the scene 25G, see FIG. 9) as a new first trajectory image at the correction position indicated by the correction information (12, 8) stored in the table 521 in the image display unit 14. It is displayed (see S73, FIG. 5). As shown in FIG. 9, the locus image 261 (correction information: (12, 8)) is arranged on the left side with respect to the deleted locus image 26F (coordinate information: (13, 8)). In addition, the trajectory image 261 is arranged vertically above the trajectory image 26G (coordinate information: (12, 7)) displayed in S19 (see FIG. 4).

  When the CPU 51 executes S17 and subsequent steps of the main process in a state where n is “3”, the correspondence of the image display unit 14 is based on the second information (11, 6) corresponding to the index “3” in the table 541. The trajectory image 26H (scenery 25H, see FIG. 9) is displayed as the third trajectory image at the position (S19, see FIG. 4). The CPU 51 also displays the first trajectory image 261 and the second trajectory image 26G on the image display unit 14 (S19). When executing the correction process (S27, see FIG. 4), the CPU 51 corrects the second information as follows because n is larger than “1” (S50: YES, see FIG. 5).

  The CPU 51 acquires the first information and the second information corresponding to the index “1” (= i) from the table 541 (see FIG. 7) (S55, see FIG. 5). The CPU 51 acquires the first information (3, 9) (13, 9) (3, 1) (13, 1) corresponding to the index “3” (= n) from the table 541 (see FIG. 7). (See S57, FIG. 5). The CPU 51 calculates the X difference and the Y difference based on the first information acquired in S55 and S57. The calculated X differences are all “2”, and the Y differences are all “0”. The CPU 51 adds the X difference and the Y difference to calculate the change amount “2” of the position of the reference point 35 (see S59, FIG. 5). Since the calculated change amount “2” of the reference point 35 is not smaller than the predetermined value “1” (S61: NO, see FIG. 5), the CPU 51 sets “1” (= i) in the table 541. The X difference “2” calculated in S59 is subtracted from the X component “13” of the second information corresponding to the index to obtain a corrected X component “11”. The CPU 51 subtracts the Y difference “0” calculated in S59 from the Y component “8” of the second information corresponding to the index of “1” (= i) in the table 541 to obtain the corrected Y component “8”. 8 ”is obtained. The CPU 51 identifies the correction information (11, 8) and stores it in the table 521 (see FIG. 8) in association with i = 1 and n = 3 (S65, see FIG. 5). As a result, the second information is corrected from (13, 8) to (11, 8). The CPU 51 adds “1” to i (= 1), so that “2” is smaller than n (= 3) (S69: NO, see FIG. 5). Based on i (= 2) with 1 added. Repeat the process.

  The CPU 51 stores the first information (4, 9) (14, 9) (4, 1) (14, 1) and the second information (12, 7) corresponding to the index of “2” (= i) in the table. 541 (see FIG. 7) (S55, see FIG. 5). CPU51 acquires the 1st information corresponding to the index of "3" (= n) from table 541 (refer to Drawing 7) (S57). The CPU 51 calculates the X difference and the Y difference based on the first information acquired in S55 and S57. The calculated X differences are all “1”, and the Y differences are all “0”. The CPU 51 adds the X difference and the Y difference, and calculates a change amount “1” of the position of the reference point 35 (S59, see FIG. 5). Since the calculated change amount “1” of the position of the reference point 35 is not smaller than the predetermined value “1” (S61: NO, see FIG. 5), the CPU 51 sets “2” (= i) in the table 541. The X difference “1” calculated in S59 is subtracted from the X component “12” of the second information corresponding to the index to obtain a corrected X component “11”. The CPU 51 subtracts the Y difference “0” calculated in S59 from the Y component “7” of the second information corresponding to the index “2” (= i) in the table 541, and corrects the corrected Y component “7”. 7 ”is obtained. The CPU 51 identifies the correction information (11, 7) and stores it in the RAM 52 in association with i = 2 and n = 3 (S65, see FIG. 5). As a result, the second information is corrected from (12, 7) to (11, 7).

  The CPU 51 adds “1” to i (= 2) and the result “3” is the same as n (= 3) (S69: YES, see FIG. 5), so “1” “2” in the table 541 The first trajectory image 261 and the second trajectory image 26G displayed based on the second information corresponding to the index (= n−1) are deleted (S71, see FIG. 5) (Scene 25H, FIG. 9). The CPU 51 uses the trajectory image 262 (the scene 25H, see FIG. 9) as a new first trajectory image at the correction position indicated by the correction information (11, 8) stored in the table 521 in the image display unit 14. Display. In addition, the locus image 263 (scenery 25H, see FIG. 9) is displayed as a new second locus image at the correction position indicated by the correction information (11, 7) (S73, see FIG. 5). As shown in FIG. 9, the trajectory image 262 (correction information: (11, 8)) is arranged on the left side with respect to the deleted trajectory image 261 (correction information: (12, 8)). Further, the locus image 263 (correction information: (11, 7)) is arranged on the left side with respect to the deleted locus image 26G (coordinate information: (12, 7)). Further, the trajectory images 262 and 263 are arranged vertically above the trajectory image 26H (coordinate information: (11, 6)) displayed in S19 (see FIG. 4).

  The above processing is repeated until it is determined in S35 of the main processing that a change with time of the fingertip position satisfies a predetermined condition. As a result, as shown by scenes 25I and 25J in FIG. 9, the (n-1) th trajectory image 26 is corrected each time the correction process (see S27) is repeated. Therefore, as shown by the trajectory images 264 to 266 and 26I of the scene 25I and the trajectory images 267 to 270 and 26J of the scene 25J, the trajectory images 26 are arranged vertically from top to bottom. This arrangement shows the tendency of the actual fingertip position to change over time.

  As described above, the CPU 51 determines the third information based on the first information (information indicating the position of the reference point 35) and the second information (information indicating the position of the fingertip), and stores the third information in the table 541. To do. The third information is information that can specify the relative position of the fingertip with respect to the reference point 35. The position of the reference point 35 changes as the shooting direction of the camera 20 changes. For this reason, the change with time of the relative position of the fingertip with respect to the reference point 35 shows the actual movement of the fingertip even when the shooting direction of the camera 20 changes. The CPU 51 executes a corresponding process when the tendency of the relative change of the fingertip with time satisfies a predetermined condition. Thus, even when the shooting direction of the camera 20 changes, the CPU 51 can accurately identify the actual fingertip movement and execute the corresponding processing.

  Further, the CPU 51 changes the actual fingertip movement of the trajectory image 26 excluding the change in the predetermined area 21 even when the orientation of the head of the user wearing the HD 10 changes and the shooting direction of the camera 20 changes. It can be displayed as a trajectory image 26 showing the state. Therefore, the user can appropriately move the fingertip so as to meet a predetermined condition. Further, the user can easily confirm whether the fingertip is moving so as to meet a predetermined condition. The trajectory image 26 immediately after being displayed on the image display unit 14, that is, the trajectory image 26 displayed on the nth image display unit 14 coincides with the position of the fingertip of the hand 23 included in the real image 22. Therefore, the user can recognize without a sense of incongruity that the trajectory image 26 indicates the position of the fingertip.

  The present invention is not limited to the above embodiment, and various modifications can be made. With reference to FIGS. 7, 10, and 11, correction processing in the modification will be described. The correction process of FIG. 10 is executed instead of the correction process of FIG. 5 in S27 of the main process of FIG. Since the main process is the same as the above embodiment, the process is omitted.

  With reference to FIG. 10, the correction process in a modification is demonstrated. The CPU 51 determines whether n stored in the RAM 52 is greater than 1 (S70). If the CPU 51 determines that n is not greater than 1 (S70: NO), it ends the correction process and returns the process to the main process (see FIG. 4). If the CPU 51 determines that n is greater than 1 (S70: YES), it sets n as the value of i (S71). The CPU 51 acquires four pieces of first information corresponding to the index “1” in the table 541 (see FIG. 7) (S75). CPU51 acquires four 1st information and 2nd information corresponding to the index of i (= n) among tables 541 (S77).

  The CPU 51 determines whether the third information corresponding to the same reference point 35 is based on the four pieces of first information (index: 1) acquired in S75 and the four pieces of first information (index: i) acquired in S77. By subtracting the X component and the Y component, the difference between the X component and the Y component (X difference and Y difference) corresponding to each of the four reference points 35 is calculated. The CPU 51 selects an X difference and a Y difference corresponding to one of the reference points 35 among the X difference and the Y difference calculated for the same reference point 35. The CPU 51 adds the selected X difference and Y difference, and calculates the amount of change in the position of the reference point 35 (S79).

  The CPU 51 determines whether the amount of change in the position of the reference point 35 calculated in S79 is smaller than a predetermined value (for example, “1”) (S81). When the CPU 51 determines that the amount of change in the position of the reference point 35 is smaller than the predetermined value (S81: YES), the CPU 51 ends the correction process and returns the process to the main process (see FIG. 4). When the CPU 51 determines that the amount of change in the position of the reference point 35 is not smaller than the predetermined value (S81: NO), the CPU 51 corresponds to the index i (= n) in the table 541 by performing the following process. The second information is corrected (S83). Specifically, it is as follows. The CPU 51 adds the X difference calculated in S79 to the X component of the second information corresponding to the index i in the table 541. The CPU 51 adds the Y difference calculated in S79 to the Y component of the second information corresponding to the index i in the table 541. The CPU 51 stores the correction information including the calculated X component and Y component in the table 521 (see FIG. 11) in association with i (= n), n and the correction information (S85).

  As shown in FIG. 11, in the table 521, the correction information is stored as described above, so that the correction information is stored in association with i = n. In FIG. 11, in order to facilitate understanding, second information before correction and an arrow are additionally shown before each correction information. In addition, information corresponding to other than i = n is not stored, but for easy understanding, coordinate information indicating the position of the trajectory image 26 continuously displayed on the image display unit 14 is shown based on the correction information. ing.

  As shown in FIG. 10, the CPU 51 deletes the nth trajectory image 26 displayed in S19 (see FIG. 4) from the trajectory image 26 displayed on the image display unit 14 (S87). As a result, the trajectory images 26 corresponding to the 1st to (n-1) th are displayed on the image display unit 14. The trajectory image 26 corresponding to the 1st to (n-1) th is a correction process executed when the positions of the trajectory images 26 displayed in the 1st to (n-1) th in S19 are i = 1 to n-1. The trajectory image 26 displayed at the correction position is shown in S89 (described later) of the correction process corrected in S83 of FIG.

  Next, the CPU 51 specifies a correction position in the predetermined area 21 indicated by the correction information stored in the table 521. The CPU 51 causes the image display unit 14 to display the trajectory image 26 superimposed on the position on the real image 22 corresponding to the specified position (S89). Thereby, instead of the trajectory image 26 corresponding to the nth deleted in S87, the trajectory image 26 corresponding to the nth corrected position is added to the trajectory image 26 corresponding to the 1st to (n-1) th. Displayed. The CPU 51 ends the correction process and returns the process to the main process (see FIG. 4).

  The case where the trajectory image 26 is displayed on the image display unit 14 will be specifically described with reference to FIGS. 4 and 10 to 12. When the CPU 51 executes S17 and subsequent steps of the main process (see FIG. 4) in a state where n is “1”, the second information (13, 8) corresponding to the index “1” in the table 541 (see FIG. 7). ), A trajectory image 26F (scenery 25F, see FIG. 12) is displayed at a corresponding position on the image display unit 14 (S19, see FIG. 4). When the CPU 51 executes the correction process (S27, see FIG. 4), since n is not larger than 1 (S70: NO, see FIG. 10), the second information is not corrected (S83, see FIG. 10). The correction information is not stored in the table 521 (see FIG. 11) (however, in the table 521, as described above, for the sake of easy understanding, the second information (13, 13) indicating the position corresponding to the trajectory image 26F is used. 8) is shown in association with i = n = 1.)

  Next, in the case where n is “2” and the main process is executed after S17 of the main process, the CPU 51 displays the image display unit based on the second information (12, 7) corresponding to the index “2” in the table 541. The trajectory image 26G (scenery 25G, see FIG. 12) is displayed at the corresponding position 14 (S19, see FIG. 4). When the correction process (S27, see FIG. 4) is executed, the CPU 51 corrects the second information as follows because n is larger than “1” (S70: YES, see FIG. 10).

  The CPU 51 acquires the first information (5, 9) (15, 9) (5, 1) (15, 1) corresponding to the index “1” from the table 541 (see FIG. 7) (S75, FIG. 10). The CPU 51 stores the first information (4, 9) (14, 9) (4, 1) (14, 1) and the second information (12, 7) corresponding to the index of “2” (= i = n). , From the table 541 (see FIG. 7) (S77, see FIG. 10). The CPU 51 calculates the X difference and the Y difference based on the first information acquired in S75 and S77. The calculated X differences are all “1”, and the Y differences are all “0”. The CPU 51 adds the X difference and the Y difference to calculate the amount of change “1” of the position of the reference point 35 (S79, see FIG. 10). Since the calculated change amount “1” of the reference point 35 is not smaller than the predetermined value “1” (S81: NO, see FIG. 10), the CPU 51 sets “2” (= i = n) in the table 541. The X difference “1” calculated in S79 is added to the X component “12” of the second information corresponding to the index of) to obtain the corrected X component “13”. The CPU 51 adds the Y difference “0” calculated in S59 to the Y component “7” of the second information corresponding to the index “2” in the table 541 to obtain the corrected Y component “7”. . The CPU 51 identifies the correction information (13, 7) and stores it in the table 521 (see FIG. 11) in association with i = 2 and n = 2 (S85, see FIG. 10). As a result, the second information is corrected from (12, 7) to (13, 7).

  The CPU 51 deletes the trajectory image 26G displayed on the image display unit 14 based on the second information corresponding to the index “2” (= i = n) in the table 541 (S87) (Scene 25G, FIG. 12). reference). The CPU 51 displays the trajectory image 281 (the scene 25G, see FIG. 12) at the correction position indicated by the correction information (13, 7) stored in the table 521 in the image display unit 14 (see S89, FIG. 10). . As shown in the scene 25G in FIG. 12, the trajectory image 281 (correction information: (13, 7)) is arranged on the right side with respect to the deleted trajectory image 26F (coordinate information: (12, 7)). In addition, the trajectory image 281 is arranged vertically below the trajectory image 26F (coordinate information: (13, 8)).

  When the CPU 51 executes S17 and subsequent steps of the main process in a state where n is “3”, the correspondence of the image display unit 14 is based on the second information (11, 6) corresponding to the index “3” in the table 541. A trajectory image 26H (scene 25H, see FIG. 9) is displayed at the position to be played (S19, see FIG. 4). When the correction process (S27, see FIG. 4) is executed, the CPU 51 corrects the second information as follows because n is larger than “1” (S70: YES, see FIG. 10).

  The CPU 51 acquires first information corresponding to the index “1” from the table 541 (see FIG. 7) (S75, see FIG. 10). The CPU 51 stores the first information (3, 9) (13, 9) (3, 1) (13, 1) and the second information (11, 6) corresponding to the index of “3” (= i = n). , From the table 541 (see FIG. 7) (S77, see FIG. 10). The CPU 51 calculates the X difference and the Y difference based on the first information acquired in S75 and S77. The calculated X differences are all “2”, and the Y differences are all “0”. The CPU 51 adds the X difference and the Y difference to calculate the amount of change “2” of the position of the reference point 35 (S79, see FIG. 10). Since the calculated change amount “2” of the position of the reference point 35 is not smaller than the predetermined value “1” (S81: NO, see FIG. 10), the CPU 51 sets “3” (= i = n) in the table 541. The X difference “2” calculated in S79 is added to the X component “11” of the second information corresponding to the index of) to obtain the corrected X component “13”. The CPU 51 adds the Y difference “0” calculated in S59 to the Y component “6” of the second information corresponding to the index “3” in the table 541 to obtain the corrected Y component “6”. . The CPU 51 identifies the correction information (13, 6) and stores it in the table 521 (see FIG. 11) in association with i = 3 and n = 3 (S85, see FIG. 10). As a result, the second information is corrected from (11, 6) to (13, 6).

  The CPU 51 deletes the trajectory image 26H displayed on the image display unit 14 based on the second information corresponding to the index “3” (= i = n) in the table 541 (S87) (Scene 25H, FIG. 12). reference). The CPU 51 displays the trajectory image 282 (scene 25H, see FIG. 12) at the correction position indicated by the correction information (12, 6) stored in the table 521 in the image display unit 14 (see S89, FIG. 10). . As shown in the scene 25H of FIG. 12, the trajectory image 282 (correction information: (13, 6)) is arranged on the right side with respect to the deleted trajectory image 26H (coordinate information: (11, 6)). In addition, the trajectory image 282 is arranged vertically below the trajectory images 26F (coordinate information: (13, 8)) and 281 (coordinate information: (13, 7)).

  The above processing is repeated until it is determined in S35 of the main processing that a change with time of the fingertip position satisfies a predetermined condition. As a result, as shown by the scenes 25I and 25J in FIG. 12, the trajectory image 26 displayed on the nth image display unit 14 is corrected by the correction process (see S27). Accordingly, as shown by the trajectory images 26F and 281 to 283 of the scene 25I and the trajectory images 26F and 281 to 284 of the scene 25J, the trajectory images 26 are arranged vertically from top to bottom. This arrangement shows the tendency of the actual fingertip position to change over time.

  In this way, the CPU 51 moves the actual fingertip of the trajectory image 26 excluding the change in the predetermined area 21 even when the orientation of the head of the user wearing the HD 10 changes and the shooting direction of the camera 20 changes. Can be displayed as a trajectory image 26 showing the state of Therefore, the user can appropriately move the fingertip so as to meet a predetermined condition. Further, the user can easily confirm whether the fingertip is moving so as to meet a predetermined condition. Unlike the case of the above embodiment, the trajectory image 26 once displayed on the image display unit 14 does not move. For this reason, the user can visually recognize the trajectory image 26 without a sense of incongruity.

  In the said embodiment, although CPU51 specified the fingertip by S15, the target object specified is not limited to a fingertip. For example, the CPU 51 may specify a pen tip of a predetermined color (for example, red) as an object. In this case, the CPU 51 can detect a region including the target object by detecting a region where pixels having a predetermined pixel value corresponding to a predetermined color are continuous. Then, a well-known pattern matching process is performed on the basis of the polygonal shape surrounding the entire detected area and the image template having a shape corresponding to the object stored in advance in the program ROM 53. When the cross-correlation coefficient calculated by the pattern matching process is a value within a predetermined range, the CPU 51 determines that the pattern matching between the specified polygonal shape and the image template is successful. In this case, the CPU 51 identifies the polygonal area as the pen tip. The camera 20 may directly transmit the captured image data to the CB 50.

  CPU51 specified the reference point 35 as information which shows the imaging direction of the camera 20 by S21. The CPU 451 may specify information indicating the shooting direction of the camera 20 by another method. For example, the HD 10 may be further provided with an acceleration sensor or an orientation sensor. The acceleration sensor may detect a change in the direction of the user's head as acceleration information. The direction sensor may detect a change in the direction of the user's head as direction information. The CPU 51 may specify a relative direction with respect to the direction of the user's head when the HMD 1 is activated as information indicating the shooting direction of the camera 20 based on information detected by the acceleration sensor or the orientation sensor. The CPU 51 may specify the change amount of the shooting region shot by the camera 20 from the information indicating the shooting direction of the camera 20, and may specify the reference point 35 based on the specified change amount. For example, the CPU 51 may correct the information (first information) indicating the position of the reference point 35 specified in S21 based on the acceleration information acquired from the acceleration sensor, and specify the third information in S23. . As a result, even when the shooting direction of the camera 20 changes, the CPU 51 can continuously specify the common reference point 35 and specify the third information.

  The number of reference points 35 specified by the CPU 51 and the location of the captured image 31 from which the reference points 35 are extracted can be changed. For example, the CPU 41 may specify the reference point 35 by a number larger than 1 to 3 or 4. For example, the CPU 51 may set the position where the reference point 35 is specified as the approximate center of the predetermined area 21.

  In S25, the CPU 51 stores the index, the four pieces of first information, the second information, and the four pieces of third information in the table 541 in association with each other. On the other hand, instead of the index, the time information when S25 is executed may be stored in the table 541 in association with the four pieces of first information, the second information, and the four pieces of third information. Alternatively, the CPU 51 associates the four pieces of first information, the second information, and the four pieces of third information and stores them in the table 541, and the index may not be stored. For example, the CPU 51 may specify the order in which the four pieces of first information, the second information, and the four pieces of third information are stored in the table 541 based on the address information of the flash ROM 54 in which the table 541 is stored.

  In the above embodiment, the CPU 51 corrects the position of the trajectory image 26 by executing the correction process (S27). The CPU 51 may not perform the correction process. The CPU 51 may determine based on the third information whether the temporal change in the finger position satisfies a predetermined condition while maintaining the position of the trajectory image 26 displayed in S19 as it is. The CPU 51 deletes the trajectory image 26 in each of S42, S44, and S46 when the elapsed time is larger than the predetermined value and the trajectory image 26 is displayed on the image display unit 14 in each of S41, S43, and S45. did. On the other hand, the CPU 51 may determine whether or not the number of trajectory images 26 displayed on the image display unit 14 is larger than a predetermined value in each of S41, S43, and S45. The CPU 51 may delete the trajectory image 26 when the number of trajectory images 26 is smaller than a predetermined value.

  The image display unit 14, the eyepiece optical system, and the half mirror 3 are examples of the “display unit” of the present invention. The camera 20 is an example of the “photographing unit” in the present invention. The fingertip is an example of the “object” in the present invention. The CPU 51 that performs the process of S15 is an example of the “first determination unit” in the present invention. The camera 20 is an example of the “detection unit” in the present invention. The captured image data is an example of the “output of the detection unit” in the present invention. The position of the reference point 35 is an example of the “first position” in the present invention. The CPU 51 that performs the process of S21 is an example of the “first acquisition unit” in the present invention. The CPU 51 that performs the process of S18 is an example of the “second acquisition unit” in the present invention. The position of the fingertip with respect to the predetermined area 21 is an example of the “second position” in the present invention. The CPU 51 that performs the process of S23 is an example of the “determination unit” in the present invention. The CPU 51 that performs the process of S25 is an example of the “storage unit” in the present invention. The flash ROM 54 that stores the table 541 is an example of the “first storage unit” in the present invention. The program ROM 53 that stores the image template is an example of the “second storage unit” in the present invention. The CPU 51 that performs the process of S26 is an example of the “second determination unit” in the present invention. The CPU 51 that performs the process of S27 is an example of the “first display means” in the present invention. The CPU 51 that performs the process of S19 is an example of the “second display unit” in the present invention. The CPU 51 that performs the process of S35 is an example of the “third determination unit” in the present invention. The CPU 51 that performs the process of S39 is an example of the “execution unit” in the present invention. The CPU 51 that performs the process of S71 is an example of the “first erasing unit” in the present invention. The CPU 51 that performs the process of S73 is an example of the “correction unit” in the present invention. The CPU 51 that performs the process of S42 is an example of the “second erasing unit” in the present invention. The CPU 51 that performs the process of S44 is an example of the “third erasing unit” in the present invention. The CPU 51 that performs the process of S46 is an example of the “fourth erasing unit” in the present invention.

  The process of S15 is an example of the “first determination step” in the present invention. The process of S21 is an example of the “first acquisition step” in the present invention. The process of S18 is an example of the “second acquisition step” in the present invention. The process of S23 is an example of the “decision step” in the present invention. The process of S25 is an example of the “storage step” in the present invention. The process of S26 is an example of the “second determination step” in the present invention. The process of S27 is an example of the “first display step” in the present invention. The process of S35 is an example of the “third determination step” in the present invention. The process of S39 is an example of the “execution step” in the present invention.

3 Half mirror 14 Image display unit 20 Camera 21 Predetermined area 23 Hand 26 Trajectory image 35 Reference point 51 CPU 51
52 RAM
54 Flash ROM
521, 541 Table

Claims (13)

A display unit capable of emitting image light in a first direction;
A photographing unit capable of photographing a predetermined area including at least a second direction area opposite to the first direction and outputting image data;
First determination means for repeatedly determining whether there is a predetermined object in the predetermined area based on the image data output from the imaging unit;
A first position indicating a first position that is a position corresponding to the shooting direction of the shooting unit based on the output of the detection unit when the first determination unit determines that the object is within the predetermined area. First acquisition means for acquiring information;
Second acquisition means for acquiring second information indicating a second position, which is a relative position of the object with respect to the predetermined area, based on the image data;
The third information based on the first information acquired by the first acquisition unit and the second information acquired by the second acquisition unit is converted into the predetermined area by the first determination unit. A determination means for determining each time it is determined that
Storage means for storing the third information determined by the determination means in the first storage unit in association with the order information indicating the determined order;
Associating with one third information among a plurality of third information stored in the first storage unit and order information indicating an order after the order indicated by the order information associated with the first third information The first information corresponding to the first third information is acquired by the first acquisition means, and the first information corresponding to the other third information is Second determination means for determining whether the positional relationship between the first position and the second position has changed before being acquired by the first acquisition means;
First display means for repeatedly displaying, on the display unit, trajectory information indicating a position determined based on at least the other third information when the second determination means determines that the positional relationship has changed; ,
Based on the plurality of third information stored in the first storage unit, there is a tendency of a change in a plurality of correction positions obtained by correcting the second position corresponding to each third information by the corresponding first position. Third judging means for judging whether or not a predetermined condition is satisfied;
An execution unit that executes a corresponding process when the third determination unit determines that the predetermined condition is satisfied, wherein the third information is information corresponding to the positional relationship. Head mounted display.   The head mounted display according to claim 1, wherein the third information is information indicating a relative position of the second position with respect to the first position. a second display means for displaying n-th (n is a natural number of 2 or more) trajectory information at a second position corresponding to the third information determined by the determination means;
The first display means includes
First erasing means for erasing the 1st to (n-1) th trajectory information displayed on the display unit;
The second position corresponding to the third information determined for each of the 1st to (n-1) th is corrected based on the first position corresponding to the third information determined for the nth, and the corrected second position is set. The head mounted display according to claim 1, further comprising correction means for displaying the trajectory information. The first display means includes
The second position corresponding to the third information determined n-th (n is a natural number of 2 or more) by the determining means is corrected based on the first position corresponding to the first information determined first. 3. The head mounted display according to claim 1, wherein the trajectory information is displayed at the corrected second position.   When the state where the first determination means determines that the object is not present in the predetermined area continues for a predetermined first time or longer, and the trajectory information is displayed on the display unit, The head mounted display according to any one of claims 1 to 4, further comprising second erasing means for erasing the trajectory information.   When the state in which the positional relationship is determined not to change by the second determination unit continues for a predetermined second time or more and the locus information is displayed on the display unit, the locus information is 6. The head mounted display according to claim 1, further comprising third erasing means for erasing.   4. A fourth erasure unit that erases the trajectory information when the third determination unit determines that the tendency of change in the plurality of correction positions satisfies a predetermined condition. Item 7. The head mounted display according to any one of Items 1 to 6.   The third determination means determines whether the specific shape indicated by the template data stored in the second storage unit matches the change tendency of the plurality of correction positions, thereby determining the plurality of correction positions. The head-mounted display according to claim 1, wherein it is determined whether or not the tendency of the change in the condition satisfies a predetermined condition.   9. The head mount according to claim 8, wherein the execution unit executes a specific process associated with template data indicating a specific shape that is determined to match a tendency of change in the plurality of correction positions. display.   2. The head mounted display according to claim 1, wherein the first acquisition unit acquires, as the first information, information indicating a position of a comparison object included in the predetermined area based on the image data. .   The head-mounted display according to claim 10, wherein the first acquisition unit acquires a plurality of pieces of first information based on the image data.   The head mounted display according to claim 11, wherein the plurality of first information indicate positions in the vicinity of each of the four corners of the captured image indicated by the image data. Based on image data output from the imaging unit when the imaging unit captures a predetermined area including at least a region in the second direction that is opposite to the first direction in which the display unit can emit image light. A first determination step of repeatedly determining whether or not there is a predetermined object in the predetermined area;
When the first determination step determines that the object is in the predetermined area, the first position indicating a first position that is a position corresponding to the shooting direction of the shooting unit is based on the output of the detection unit. A first acquisition step of acquiring information;
A second acquisition step of acquiring second information indicating a second position, which is a relative position of the object with respect to the predetermined region, based on the image data;
The third information based on the first information acquired by the first acquisition step and the second information acquired by the second acquisition step is converted into the predetermined area by the first determination step. A determination step that is determined each time it is determined that
A storage step of storing the third information determined by the determination step in the first storage unit in association with the order information indicating the determined order;
Associating with one third information among a plurality of third information stored in the first storage unit and order information indicating an order after the order indicated by the order information associated with the first third information The first information corresponding to the first third information is acquired by the first acquisition step, and the first information corresponding to the other third information is A second determination step for determining whether the positional relationship between the first position and the second position has changed before being acquired by the first acquisition step;
A first display step of repeatedly displaying trajectory information indicating a position determined based on at least the other third information on the display unit when it is determined by the second determination step that the positional relationship has changed;
Based on the plurality of third information stored in the first storage unit, there is a tendency of a change in a plurality of correction positions obtained by correcting the second position corresponding to each third information by the corresponding first position. A third determination step for determining whether a predetermined condition is satisfied;
A control program for causing a computer of a head mounted display to execute an execution step of executing a corresponding process when it is determined by the third determination step that a predetermined condition is satisfied.

Images