JP5979110B2 - Head mounted display and control program - Google Patents

Description

  The present invention relates to a head mounted display and a control program for controlling the head mounted display.

  A head mounted display (HMD) is used by being mounted on a user's head. Therefore, the HMD is required to be small in order to secure the user's field of view and reduce the burden at the time of wearing. When the HMD is downsized, a place where a mechanical user interface (User Interface UI) such as a switch is provided is limited. Patent Literature 1 discloses a technique for identifying corresponding operation information by detecting the movement of a user's hand taken by a camera. By applying this technology to the HMD, the HMD can specify the operation content corresponding to the movement of the object photographed by the camera, and can execute an operation according to the specified operation content. In this case, the HMD can secure a sufficient UI even when the HMD is downsized.

JP 2007-172577 A

  When a camera is provided in the HMD, a method of arranging it together with a display unit on the user's head can be considered. In this case, the shooting area of the camera changes as the head of the user wearing the HMD moves. Accordingly, there is a case where the object is out of the shooting area during shooting by the camera. In this case, since the HMD cannot accurately detect the movement of the object, there is a problem that the corresponding operation content cannot be specified and operated.

  An object of the present invention is to provide a head-mounted display capable of accurately detecting the movement of an object, specifying the corresponding operation content, and operating, and a control program for controlling the head-mounted display. It is.

The head mounted display according to the first aspect of the present invention is capable of displaying at least a content image and capable of emitting image light in a first direction, and an area in a second direction opposite to the first direction. Based on the image data acquired by the acquisition unit, an acquisition unit that acquires image data output from the imaging unit, First judgment means for judging whether or not there is a predetermined object in a predetermined area; and the first judgment means determines that the object is in the predetermined area , and the image data is obtained by the acquisition means. If the acquired time difference between the time that the predetermined region is captured by the imaging unit is smaller than a predetermined value, said display unit and said content image and the object image When the first display means for displaying and the first determination means determine that the object is in the predetermined area, the position of the object in the predetermined area is determined based on the image data. When the second determination means for determining whether the change tendency satisfies a predetermined condition and the second determination means determine that the change tendency of the position of the object satisfies the predetermined condition And execution means for executing processing corresponding to the predetermined condition, wherein the object image corresponds to the image data determined by the first determination means to be within the predetermined area. It is an image.

  According to the first aspect, the head mounted display displays both the content image and the target object image on the display unit when the target object is within the shooting range of the shooting unit. Therefore, the user can recognize whether the object is being photographed by the photographing unit. The head mounted display executes a corresponding process when a change with time of the position of the object satisfies a predetermined condition. The user can cause the head mounted display to execute the process by moving the object such that the tendency of the position of the object to change over time satisfies a predetermined condition. Here, since the user can easily move the object within the imaging range of the imaging unit, the head-mounted display can accurately detect the object and appropriately execute the process.

  In the first aspect, the image processing apparatus further includes a generating unit that generates a superimposed image in which the target image is superimposed on the content image based on the content image and the target image, and the first display unit includes the superimposing unit. An image may be displayed on the display unit. In this case, the head mounted display can display the superimposed image in a small area of the display unit.

  In the first aspect, the generation unit includes a first α value that is an α value of each pixel of the content image, and a second α value that is an α value of each pixel of the object image and is smaller than the first α value. The superimposed image may be generated using. In this case, the content image and the object image are superimposed in a transparent state. Note that the color of the content image can be made darker and clearer by making the α value (second α value) of each pixel of the object image smaller than the α value (first α value) of each pixel of the content image. it can. For this reason, even if the object image is displayed so as to overlap the content image, the object image does not hinder the visibility of the content image. For this reason, the user can recognize a content image more appropriately.

  1st aspect WHEREIN: The 3rd judgment means which judges whether the tendency of the change of the position of the said object satisfy | fills a part of said predetermined conditions is provided, The said production | generation means is the said judgment object by the said 3rd judgment means. When it is determined that the position change tendency satisfies a part of the predetermined condition, the α value of each pixel of the object image is set to a third α value larger than the second α value, and the first α The superimposed image may be generated using the value and the third α value. In this case, the transmittance of the object image is reduced, and the object image is displayed with a deeper color and clearer. Therefore, the user can easily recognize from the change in the transmittance of the object image that the tendency of the change in the position of the object satisfies a part of the predetermined condition. Further, the head mounted display can improve the visibility of the object image by the user.

  In the first aspect, the third α value may be smaller than the first α value. In this case, even if the α value of the object image changes from the second α value to the third α value, the object image does not hinder the visibility of the content image. For this reason, the user can recognize a content image more appropriately.

  In the first aspect, when the second determination unit determines that the tendency of the change in the position of the object satisfies the predetermined condition, a trajectory image indicating the position of the object is superimposed on the content image. Second display means for displaying on the display unit may be provided. In this case, the user can easily recognize from the trajectory image that the tendency of the change in the position of the object is determined by the head mounted display to satisfy the predetermined condition.

  In the first aspect, the second display unit may display the trajectory image on the display unit before the corresponding process is executed by the execution unit. In this case, when the trajectory image is displayed on the display unit, the user can recognize that the corresponding process has not been executed yet.

  In the first aspect, the object image displayed by the first display unit is deleted when the second determining unit determines that the tendency of the change in the position of the object satisfies the predetermined condition. An erasing unit may be provided. In this case, since only the content image is displayed on the display unit, the user can more appropriately recognize the content image.

  In the first aspect, the erasing unit may erase the object image after the execution of the corresponding process by the execution unit. In this case, the user can recognize that the process has been successfully performed on the head mounted display by deleting the target part image.

  In the first aspect, the acquisition unit further acquires time information indicating a time when the inside of the predetermined area was captured by the imaging unit from the imaging unit, and the first display unit is configured to acquire the time information by the acquisition unit. Only when the difference between the time when the time information is acquired and the time indicated by the acquired time information is smaller than a predetermined value, the content image and the object image are displayed on the display unit, When the difference is not smaller than a predetermined value, the object image may not be displayed on the display unit. The head-mounted display does not display the object image on the display unit when a time difference occurs between the movement of the object indicated by the object image and the actual movement of the object. Accordingly, the head mounted display can prevent the user from feeling uncomfortable by delaying the movement of the object indicated by the object image displayed on the display unit as compared with the actual movement of the object.

  In the first aspect, the second determination means determines whether the specific shape indicated by the template data stored in the storage unit matches the tendency of the change in the position of the object, thereby determining the object. It may be determined whether the tendency of the change in the position of the object satisfies a predetermined condition. In this case, the head mounted display can appropriately determine whether the tendency of the change in the plurality of correction positions satisfies a predetermined condition.

  In the first aspect, the execution means may execute a specific process associated with template data indicating a specific shape that is determined to coincide with a change tendency of the position of the object. In this case, the head mounted display can easily determine the process corresponding to the movement of the object and execute the process.

A control program according to a second aspect of the present invention provides a head-mounted display computer including a display unit capable of displaying at least a content image in a direction opposite to the first direction in which the display unit can emit image light. An acquisition unit that acquires image data output from the imaging unit when a predetermined region including at least a region in two directions is captured by the imaging unit, and the image data acquired by the acquisition unit, First judgment means for judging whether or not there is a predetermined object in a predetermined area; and the first judgment means determines that the object is in the predetermined area , and the image data is obtained by the acquisition means. If the acquired time difference between the time that the predetermined region is captured by the imaging unit is smaller than a predetermined value, the content image and the target When the first display means for displaying an image on the display unit and the first determination means determine that the object is within the predetermined area, the first display means displays the image within the predetermined area based on the image data. The second determination means for determining whether the change tendency of the position of the object satisfies a predetermined condition and the second determination means, the change tendency of the position of the object satisfies the predetermined condition. A control program for functioning as execution means for executing processing corresponding to the predetermined condition when the object image is determined by the first determination means. It is an image corresponding to the image data determined to be in the area. According to the 2nd aspect, there can exist an effect similar to a 1st aspect.

It is a perspective view of HMD1. It is a block diagram which shows the electric constitution of HMD1. It is a figure which shows the scene 21 (21A-21E). It is a flowchart of a 1st main process. It is a flowchart of the 2nd main process. It is a figure which shows DB531. It is a figure for demonstrating the judgment method of the tendency of the change of the position of a fingertip. It is a figure for demonstrating the judgment method of the tendency of the change of the position of a fingertip.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Embodiments of the invention will be described below with reference to the drawings. An outline of a head mounted display (hereinafter referred to as HMD) 1 will be described with reference to FIG. The HMD 1 includes a projection device (hereinafter referred to as a head display or HD) 10 and a control device (hereinafter referred to as a control box or CB) 50. In the following description, the upper, lower, right diagonally downward, left diagonally upward, right diagonally upward and left diagonally downward of FIG. 1 are the upper, lower, forward, backward, right and left sides of the HMD 1, respectively. In the embodiment, in order to help understanding the positional relationship and the directional relationship in various configurations, in the related drawings, the upper, lower, front, rear, right, and left sides of the HMD 1 refer to the axes of the three-dimensional Cartesian coordinate system. Explained.

  An outline of the HD 10 will be described. The HD 10 is attached to the spectacles 5 which is a dedicated wearing tool. The user wears the HD 10 on the head by using the glasses 5 to which the HD 10 is attached. The HD 10 can irradiate image light backward. The image light is incident on the left eye of the user wearing the HD 10. The HD 10 is detachably connected to the CB 50 via the harness 7. The HD 10 includes a housing 2, a half mirror 3, an image display unit 14 (see FIG. 2), an eyepiece optical system (not shown), a camera 20, and the like as main components.

  The housing 2 is provided at the right end of the glasses 5. A half mirror 3 is provided on the left end side of the housing 2. The half mirror 3 is disposed in front of the user's left eye with the user wearing the HD 10 on the head. The image display unit 14 and the eyepiece optical system are provided inside the housing 2. Details of the image display unit 14 and the eyepiece optical system will be described later. The camera 20 is provided on the front surface of the housing 2. The camera 20 can capture an area of a predetermined size (hereinafter referred to as a predetermined area) in an external scene in the front direction of the HD 10.

  The image display unit 14 includes a liquid crystal element and a light source. The image display unit 14 can display a content image. The content image is a still image or a moving image. Further, the image display unit 14 can display the captured image superimposed on the content image. The captured image is an image captured by the camera 20. The image display unit 14 displays an image based on the video signal transmitted from the CB 50 via the harness 7. The eyepiece optical system condenses image light indicating the image displayed on the image display unit 14 and emits it to the half mirror 3. The image light emitted from the eyepiece optical system is reflected by the half mirror 3. The image light reflected by the half mirror 3 is emitted to the rear of the housing 2 (the direction opposite to the shooting direction of the camera 20). With the HMD 1 being worn by the user, the image light is incident on the left eyeball of the user. The half mirror 3 transmits light from the real image of the outside world toward the left eye of the user. The HMD 1 can allow a user to visually recognize a scene in which an image displayed on the image display unit 14 is superimposed on a real image (for example, an external scene or the like) within the visual field range of the user.

  The glasses 5 hold the HD 10 on the user's head. The glasses 5 include a frame 6 and a support portion 4. Since the shape of the frame 6 is substantially the same as that of normal glasses, detailed description is omitted. A support 4 is provided at the right end of the upper surface of the rim that supports the left-eye lens in the frame 6. The support unit 4 holds the housing 2 of the HD 10. The support part 4 can move the holding position of the housing 2 in the vertical direction and the horizontal direction. The user can adjust the position so that the position of the eyeball of the left eye and the position of the half mirror 3 are aligned in the front-rear direction by moving the housing 2 in the vertical direction and the horizontal direction.

  The HD 10 may be attached to other wearing tools such as glasses, helmets, and headphones that are used on a daily basis by the user. The image display unit 14 may be another spatial modulation element. For example, the image display unit 14 may be a retinal scanning display unit that performs two-dimensional scanning with laser light having an intensity corresponding to an image signal, and an organic EL (Organic Electro-luminescence) display. In FIG. 1, the support portion 4 and the HD 10 are provided on the right side of the frame 6, but the support portion 4 and the HD 10 may be provided on the right side of the frame 6. In this case, the image light is incident on the right eyeball of the user while the HMD 1 is attached to the user.

  In the present embodiment, the HD 10 is a so-called optical see-through head display having a function of allowing a user to visually recognize a scene in which an image displayed on the image display unit 14 is superimposed on a real image of the outside world. However, the present invention may be a head display having another function. For example, the HD 10 is a so-called video that has a function of allowing a user to visually recognize an image displayed on the image display unit 14 and an image of the outside world captured by the camera without causing light from the real image of the outside world to enter the user's eyes. A see-through type head display may be used.

  In this embodiment, the camera 20 is provided on the front surface of the housing 2, but the camera 20 may be detachable from the housing 2. For example, the camera 20 may be directly attached to the frame 6 of the glasses 5 in a state where the front direction of the HD 10 can be photographed.

  An outline of the CB 50 will be described. CB50 is attached to a user's waist belt, an arm, etc., for example. The CB 50 controls the HD 10. The CB 50 is detachably connected to the HD 10 via the harness 7. The CB 50 includes a casing 60, an operation switch 61, a power switch 62, and the like as main components. The shape of the housing 60 is a substantially rectangular parallelepiped with rounded edges. The operation switch 61 is a switch for performing various settings in the HD 10 and various operations during use. The power switch 62 is a switch for turning on or off the power of the HMD 1. The power switch 62 incorporates a power lamp 63.

  The electrical configuration of the HMD 1 will be described with reference to FIG. First, the electrical configuration of the HD 10 will be described. The HD 10 includes a CPU 11 that controls the entire HD 10. The CPU 11 is electrically connected to the RAM 12, the program ROM 13, the image display unit 14, the interface 15, and the connection controller 19. The CPU 11 is electrically connected to the camera 20 via the interface 15. The RAM 12 temporarily stores various data. The program ROM 13 stores a program executed by the CPU 11. The program ROM 13 stores the OS. The program is executed on the OS. The program and the OS are stored in the program ROM 13 when the HMD 1 is shipped. The image display unit 14 can display at least one of a content image and a captured image. The interface 15 is connected to the camera 20 and controls input / output of signals. The connection controller 19 is connected to the connection controller 58 of the CB 50 via the harness 7 and performs wired communication.

  The HD 10 may further include a flash ROM. The flash ROM may store a program executed by the CPU 11. The CPU 11 may execute a program stored in the flash ROM. The program executed by the CPU 51 of the CB 50 may be stored in the program ROM 13 and the flash ROM. The CPU 11 may execute the same process as the process executed by the CPU 51 of the CB 50 instead of the CPU 51.

  The electrical configuration of the CB 50 will be described. The CB 50 includes a CPU 51 that controls the entire CB 50. The CPU 51 is electrically connected to the RAM 52, program ROM 53, flash ROM 54, interface 55, video RAM 56, image processing unit 57, connection controller 58, and wireless communication unit 59. The RAM 52 temporarily stores various data. The program ROM 53 stores a program executed by the CPU 51. The program ROM 53 stores the OS. The program is executed on the OS. The program ROM 53 stores a database (hereinafter referred to as DB) 531 (see FIG. 6, which will be described later). The program, OS, and DB 531 are stored in the program ROM 53 when the HMD 1 is shipped.

  The interface 55 is connected to the operation switch 61 and the power switch 62 and controls signal input / output. The image processing unit 57 forms an image to be displayed on the image display unit 14 of the HD 10 and stores it in the video RAM 56. The video RAM 56 temporarily stores the image formed by the image processing unit 57. The connection controller 58 is connected to the connection controller 19 of the HD 10 via the harness 7 and performs wired communication. The wireless communication unit 59 communicates with peripheral devices connected to the Internet, LAN, and the like via an access point (not shown) connected to the Internet, LAN, and the like. Examples of peripheral devices include personal computers, smartphones, tablet portable terminals, servers, and the like.

  Note that a program executed by the CPU 51 may be stored in the flash ROM 54 of the CB 50. The CPU 51 may execute a program stored in the flash ROM 54. The program executed by the CPU 11 of the HD 10 may be stored in the program ROM 53 and the flash ROM 54. The CPU 51 may execute the same process as the process executed by the CPU 11 of the HD 10 instead of the CPU 11.

  The HMD 1 may download and install the program and the OS from the program download server via the wireless communication unit 59. For example, the program and the OS may be transmitted from the server to the HMD 1 as a computer-readable temporary storage medium (for example, a transmission signal). The program may be stored in a computer-readable storage device provided in the HMD 1, for example, a flash ROM. The CPU 11 may execute a program based on a program stored in the flash ROM. The CPU 51 may execute a program based on the program stored in the flash ROM 54. However, the storage device may be a storage medium other than a temporary storage medium such as a ROM, a flash ROM, an HDD, or a RAM. Further, the storage device may be a non-temporary storage medium. The non-transitory storage medium may be capable of retaining data regardless of the length of time for storing the data.

  The configuration of the HMD 1 is not limited to the above embodiment, and may be a configuration in which the HD 10 and the CB 50 are integrated, for example. The CPU 11 of the HD 10 and the CPU 51 of the CB 50 may communicate wirelessly instead of the harness 7.

  The HMD 1 executes a corresponding process when the tendency of the change of the fingertip position when the user's fingertip moves within a predetermined area captured by the camera 20 of the HD 10 satisfies a predetermined condition. As an example of the corresponding process, there is a process of scrolling the content image displayed on the image display unit 14. In the present embodiment, the HMD 1 executes a process of scrolling the content image in one of the right, left, up, and down directions by moving the fingertip within a predetermined area captured by the camera 20. It is possible to make it possible. In the present invention, the processing that can be executed by the HMD 1 by moving the fingertip is not limited to the above processing. Below, the outline | summary of the process performed by CPU11 of HD10 is demonstrated with reference to FIG. Next, an outline of processing executed by the CPU 51 of the CB 50 will be described with reference to FIG. Next, details of the processing of the CPU 51 will be described with reference to the flowcharts of FIGS.

  An outline of processing executed by the CPU 11 of the HD 1 will be described. When the operation of turning on the power of the HMD 1 is performed via the power switch 62 of the CB 50, the CPU 11 starts the next process. The CPU 11 starts shooting with the camera 20. When shooting starts, the camera 20 continuously outputs image data while shooting in a predetermined area. Hereinafter, the image data output from the camera 20 is referred to as photographed image data. The image format of the captured image data is a Motion JPEG format, an Audio Video Interleave format, a QuickTime format, or the like. The CPU 11 receives captured image data output from the camera 20 via the interface 15. The CPU 11 transmits the captured image data to the CB 50.

  The CPU 11 acquires the time when the captured image data is received from the camera 20 from the OS. Since the camera 20 outputs the captured image data while capturing, the time when the CPU 11 receives the captured image data from the camera 20 is substantially the same as the time when the camera 20 captures the captured image corresponding to the captured image data. Can be considered. CPU11 transmits the time information which shows the acquired time with respect to CB50 with picked-up image data. Hereinafter, the time indicated by the time information transmitted to the CB together with the photographic image data is referred to as the photographic time. The time information may be included in the header of the captured image data.

  Further, the CPU 11 receives a video signal (described later) transmitted from the CB 50. The CPU 11 displays the image on the image display unit 14 by controlling the image display unit 14 based on the received video signal. The user wearing the HD1 visually recognizes the image by the light from the image displayed on the image display unit 14 reflecting the half mirror 3 and entering the left eye. Note that since the light from the real image within the field of view enters the left eye of the user wearing the HD1 through the half mirror 3, the user displays the image displayed on the image display unit 14 as a real image in the outside world. See the superimposed scenes.

  FIG. 3 shows a scene 21 (21A to 21E) visually recognized by a user wearing the HD 10. The scene 21 includes a real image 211 and images 212 (212A to 212E). The real image 211 is an image visually recognized by the user when light from the real image within the visual field range of the user passes through the half mirror 3 and reaches the left eye of the user. The image 212 is an image visually recognized by the user by being displayed on the image display unit 14 of the HD 10. The image 212 is superimposed on the lower right portion of the area of the real image 211. Thus, the user can visually recognize both the real image 211 and the image 212 within the visual field range.

  An outline of processing executed by the CPU 51 of the CB 50 will be described. The CPU 51 receives content image data from the peripheral device via the wireless communication unit 59. The content image data is data that can form a content image. When the content image is a still image, the image format of the content image data is BMP format, GIF format, JPEG format, etc., and when the content image is a movie, it is Moving Picture Experts Group format, Audio Video Interleave format, QuickTime format, etc. is there. The CPU 51 generates a video signal of a content image corresponding to the received content image data. The CPU 51 transmits the video signal of the content image to the HD 10. As a result, the content image is displayed on the image display unit 14 of the HD 10.

  A scene 21A in FIG. 3 shows a scene visually recognized by a user wearing the HD 10 when the CPU 51 transmits a video signal of a content image to the HD 10. In the scene 21A, the content image 212A is superimposed on the real image 211.

  As an example, the user moves his / her fingertip within a predetermined area photographed by the HD 10 camera 20 in order to scroll the content image 212A to the right. The CPU 51 receives captured image data from the HD 1. The CPU 51 determines whether or not a fingertip is included in the captured image corresponding to the received captured image data. Details of the method for determining whether a fingertip is included will be described later. When the CPU 51 determines that the captured image includes a fingertip, the CPU 51 generates a video signal of an image in which the captured image is superimposed on the content image. Hereinafter, an image in which a captured image is superimposed on a content image is referred to as a superimposed image. The CPU 51 transmits the video signal of the superimposed image to the HD 10. As a result, a superimposed image is displayed on the image display unit 14 of the HD 10.

  A scene 21B in FIG. 3 shows a scene visually recognized by a user wearing the HD 10 when the CPU 51 transmits a video signal of a superimposed image to the HD 10. In the scene 21B, the superimposed image 212B is superimposed on the real image 211. The real image 211 includes the user's hand 22. In the superimposed image 212B, the captured image 24 is superimposed on the content image 23. Since the HD1 camera 20 captures a predetermined area in the user's visual field direction, the captured image 24 shows a part of the real image 211. The captured image 24 includes a fingertip 25A of the user's hand 25. Note that the content image 23 and the captured image 24 are transparent, and both are seen through each other. The transmittance of the content image 23 is smaller than the transmittance of the captured image 24. Therefore, the content image 23 is darker and clearer than the captured image 24.

  The CPU 51 specifies the position of the fingertip included in the captured image. The CPU 51 determines whether the tendency of the fingertip position over time satisfies any of a plurality of predetermined conditions. Each of the plurality of predetermined conditions corresponds to each of the processes for scrolling the content image displayed on the image display unit 14 in any of the right, left, upper, and lower directions. Details of the determination method will be described later.

  If the CPU 51 determines that the tendency of the fingertip position over time to satisfy any one of a plurality of predetermined conditions, the CPU 51 generates a video signal of a superimposed image in which the transmittance of the captured image 24 is further reduced. To do. The CPU 51 transmits the video signal of the superimposed image to the HD 10. The transmittance of the captured image 24 included in the superimposed image displayed on the image display unit 14 of the HD 10 is reduced. For this reason, the photographed image 24 becomes darker and clearer.

  Next, the CPU 51 determines whether the tendency of the fingertip position included in the captured image to change with time satisfies any of a plurality of predetermined conditions. Details of the determination method will be described later. When the CPU 51 determines that the change tendency of the fingertip position with time satisfies any of a plurality of predetermined conditions, the CPU 51 generates a video signal of a superimposed image including a trajectory image indicating the fingertip position. The trajectory image has a line segment that connects each of the fingertip positions at different timings, and an arrow head that indicates the direction of movement. The CPU 51 transmits a video signal to the HD 10. As a result, the superimposed image including the trajectory image is displayed on the image display unit 14 of the HD 10.

  A scene 21C in FIG. 3 shows a scene visually recognized by the user wearing the HD 10 when the CPU 51 transmits a video signal of a superimposed image including a trajectory image to the HD 10. In the scene 21C, the superimposed image 212C is superimposed on the real image 211. In the superimposed image 212C, the captured image 24 is superimposed on the content image 23. Since the transmittance of the captured image 24 is smaller than that of the scene 21B, the captured image 24 is darker and clearer than the scene 21B. In addition, the superimposed image 212C includes a trajectory image 26 that indicates the tendency of the position of the fingertip 25A to change.

  When the CPU 51 determines that the tendency of the fingertip position to change with time satisfies any of a plurality of predetermined conditions, the CPU 51 specifies a process corresponding to any of the predetermined conditions. An example is given in which the processing corresponding to the tendency of the fingertip position over time indicated by the trajectory image 26 of the scene 21C in FIG. 3 is processing to scroll the content image in the right direction. In this case, the CPU 51 generates a video signal of a superimposed image in which the captured image is superimposed on the content image scrolled in the right direction and the locus image 26 is included. The CPU 51 transmits a video signal to the HD 10.

  A sight 21D in FIG. 3 shows a sight visually recognized by a user wearing the HD 10 when the CPU 51 transmits a video signal of a superimposed image including a trajectory image to the HD 10. In the scene 21D, the superimposed image 212D is superimposed on the real image 211. In the superimposed image 212D, the captured image 24 is superimposed on the content image 23 scrolled in the right direction. Further, the superimposed image 212D includes a trajectory image 26 that shows a tendency of the position of the fingertip 25A to change over time.

  After scrolling the content image to the right by a predetermined amount, the CPU 51 generates a video signal of only the content image after scrolling in order to delete the captured image 24 and the trajectory image 26. The CPU 51 transmits a video signal to the HD 10. A scene 21E in FIG. 3 shows a scene visually recognized by the user wearing the HD 10 when the CPU 51 transmits the video signal of the content image after scrolling. In the scene 21E, the content image 212E is superimposed on the real image 211, and the captured image 24 and the locus image 26 are deleted. As a result, the user can visually recognize the content image 212E scrolled rightward.

  Details of processing executed by the CPU 51 of the CB 50 will be described with reference to FIGS. 4 and 5. In the first main process (see FIG. 4) and the second main process (see FIG. 5), the CPU 51 stores the program stored in the program ROM 53 when an operation of turning on the power of the HMD 1 is performed via the power switch 62. Started by executing. The first main process and the second main process are executed in parallel.

  The first main process will be described with reference to FIG. The CPU 51 determines whether content image data has been received from the peripheral device via the wireless communication unit 59 (S1). When the CPU 51 determines that the content image data has been received (S1: YES), the CPU 51 stores the received content image data in the flash ROM 54 (S3). The CPU 51 returns the process to S1. When determining that the content image data has not been received (S1: NO), the CPU 51 determines whether an operation for displaying the content image has been detected via the operation switch 61 (S5). When the CPU 51 determines that an operation for displaying a content image has been detected (S5: YES), the CPU 51 drives the image processing unit 57 to form a content image based on the content image data stored in the flash ROM 54, It is stored in the video RAM 56. The CPU 51 generates a video signal based on the content image stored in the video RAM 56. The format of the video signal is Transition Minimized Differential Signaling (TMDS), Low voltage differential signaling (LVDS), or the like. The CPU 51 transmits a video signal to the HD 10 via the connection controller 58 (S7). The CPU 51 returns the process to S1. If the CPU 51 determines that an operation for displaying a content image has not been detected (S5: NO), the process returns to S1. Note that the video signal is not limited to being generated by the CPU 51. For example, the connection controller 19 may generate the video signal based on the content image stored in the video RAM 56.

  As described above, the CPU 11 of the HD 10 receives the video signal of the content image transmitted from the CB 50 via the connection controller 19. The CPU 11 displays the content image on the image display unit 14 by controlling the image display unit 14 based on the received video signal. Therefore, the CPU 51 displays an image on the image display unit 14 by transmitting a video signal to the HD 10 in S7. In the following, the CPU 51 drives the image processing unit 57 to form an image (content image or photographed image) from image data (content image data or photographed image data), and the formed image is stored in the video RAM 56. The generation of the video signal based on the image stored in the CPU is simply referred to as the CPU 51 generating the video signal based on the image data. The CPU 51 transmits a video signal to the HD 10 so that the CPU 11 of the HD 10 controls the image display unit 14 to display an image corresponding to the video signal. The CPU 51 causes the image display unit 14 to display an image. .

  As a result of the processing of S7, the user wearing the HD1 visually recognizes the content image 212A superimposed on the real image 211 as shown in the sight 21A of FIG.

  The second main process will be described with reference to FIG. The CPU 51 receives the captured image data and time information transmitted from the HD 10 via the connection controller 58 (S11). The CPU 51 stores the received captured image data and time information in the RAM 52 in association with each other. The CPU 51 acquires time information indicating the time when the captured image data and the time information are received (hereinafter referred to as reception time) from the OS and stores the time information in the RAM 52.

  The CPU 51 determines whether the fingertip is included in the captured image based on the captured image data stored in the RAM 52 (S13). First, the CPU 51 specifies the fingertip from the captured image as follows. The CPU 51 detects a skin color region from the captured image corresponding to the captured image data. The skin color area can be detected by, for example, detecting an area where pixels having predetermined pixel values (for example, R = 241, G = 187, B = 147) corresponding to the skin color are continuous. Note that the predetermined pixel value may be another value or may have a predetermined width. The CPU 51 identifies a polygon that surrounds the entire detected skin color area. The CPU 51 performs a well-known pattern matching process based on the specified polygonal shape and the fingertip specifying image template stored in advance in the program ROM 53. When the cross-correlation coefficient calculated by the pattern matching process is a value within a predetermined range, the CPU 51 determines that the pattern matching between the specified polygonal shape and the image template is successful. In this case, the CPU 51 specifies the vertex of the polygonal shape as the fingertip. Note that the method for determining whether the captured image includes the fingertip is not limited to the above method.

  The CPU 51 determines whether the fingertip can be specified by the above method. When it is determined that the fingertip cannot be specified, the CPU 51 determines that the fingertip is not included in the photographed image (S13: NO). The CPU 51 returns the process to S11. When it is determined that the fingertip has been specified, the CPU 51 determines that the fingertip is included in the captured image (S13: YES).

  The CPU 51 specifies the reception time when the captured image data and the time information are received based on the time information stored in the RAM 52. The CPU 51 specifies the shooting time indicated by the time information received from the HD 10. The CPU 51 calculates a delay time required from when the camera 20 captures a captured image until the CPU 51 receives captured image data based on the capture time and the reception time. Note that communication for synchronizing the times of the OSs is executed between the CPUs 11 and 51 at a constant cycle. Therefore, the CPU 51 can calculate the time from the shooting time to the reception time as the delay time. The CPU 51 determines whether the calculated delay time is smaller than a predetermined value (for example, 1 second) (S15). When it is determined that the calculated delay time is not smaller than the predetermined value (S15: NO), the CPU 51 returns the process to S11. If the CPU 51 determines that the calculated delay time is smaller than the predetermined value (S15: YES), the process proceeds to S17.

  As described above, when the delay time is not smaller than the predetermined value, the CPU 51 does not advance the process to S <b> 17 so that the video signal is not transmitted to the HD 10 and the captured image is not displayed on the image display unit 14. The reason is that when the delay time is large, there is a time difference between the fingertip movement when the captured image corresponding to the captured image data includes the fingertip and the fingertip movement when the real image includes the fingertip. This is because of this. As a result, the CPU 51 can prevent the user from feeling uncomfortable by delaying the movement of the fingertip included in the captured image displayed on the image display unit 14 with respect to the movement of the fingertip included in the real image.

  The CPU 51 performs the following process to superimpose the captured image corresponding to the captured image data received in S11 on the content image displayed on the image display unit 14 in S7 (see FIG. 4). The CPU 51, based on the content image data of the content image displayed on the image display unit 14 in S7 and the captured image data received in S11, the video signal of the superimposed image in which the captured image is superimposed on the content image. Is generated (S16). Here, the CPU 51 superimposes the content image and the captured image included in the superimposed image in a transparent state. An example is as follows. The CPU 51 performs α blending of the content image and the captured image in order to superimpose the content image and the captured image. The CPU 51 sets the α value of each pixel of the captured image when α blending is performed to a value smaller than the α value of each pixel of the content image. Hereinafter, the α value of each pixel of the content image is referred to as a first α value, and the α value of each pixel of the captured image is referred to as a second α value. The second α value is smaller than the first α value. In this case, the transmittance of the photographed image is larger than the transmittance of the content image, and the color is lighter and blurred than the content image.

  The CPU 51 transmits the generated video signal to the HD 10 and causes the image display unit 14 to display a superimposed image (S17). The user wearing the HD1 visually recognizes the superimposed image 212B superimposed on the real image 211, for example, as shown by the scene 21B in FIG. The superimposed image 212B includes a content image 23 and a captured image 24. The CPU 51 can make the user visually recognize both the content image 23 and the captured image 24.

  In the scene 21B, the superimposed image 212B includes the user's hand 25 and fingertip 25A. Since the user who visually recognizes the scene 21B can move his / her hand while confirming the position of his / her fingertip, the fingertip can be easily moved to satisfy a predetermined condition. In particular, even when the hand 22 is hidden in the superimposed image 212B superimposed on the real image 211 and the user cannot visually recognize the hand based on the real image 211, the hand 25 included in the superimposed image 212B can be visually recognized. Therefore, the user can always move the fingertip so as to satisfy a predetermined condition regardless of the position of the hand in the real image 211. The CPU 51 causes the superimposed image 212B to be formed by superimposing the captured image 24 on the content image 23. For this reason, since the CPU 51 can reduce the size of the area of the real image 211 that is hidden by the superimposed image 212B, the visibility of the real image 211 by the user can be maintained well.

  In the scene 21B, the transmittance of the captured image 24 included in the superimposed image 212B is larger than the transmittance of the content image 23, and the color is lighter and blurred than the content image 23. Therefore, the user can recognize the content image 23 more clearly than the captured image 24. Thus, the CPU 51 can prevent the visibility of the content image 23 from being hindered when the captured image 24 is superimposed on the content image 23. Therefore, even when the captured image 24 is superimposed on the content image 23, the CPU 51 can allow the user to continue to view the content image 23 satisfactorily.

  The CPU 51 specifies coordinate information indicating the position of the fingertip specified in S <b> 13 and stores it in the RAM 52. By repeating the processes of S11, S13, and S15, a plurality of coordinate information is stored in the RAM 52. The CPU 51 refers to the DB 531 in FIG. 6 based on the plurality of coordinate information stored in the RAM 52, so that the tendency of the fingertip position to change with time satisfies any one of a plurality of predetermined conditions. (S19).

  The DB 531 will be described with reference to FIG. A plurality of determination data and processing content data are stored in the DB 531 in association with each other. The determination data is data indicating a predetermined condition that can identify the tendency of the change of the fingertip position. Details of the determination data will be described later. The processing content data indicates the specific processing content of the HMD 1. The DB 531 stores processing content data indicating processing for scrolling the content image in the right, left, upper, and lower directions.

  With reference to FIG. 7, the details of the determination method will be specifically described. By repeating the processes of S11, S13, and S15 (see FIG. 5), coordinate information (X1, Y1) (X2, Y2) (X2) indicating each of the plurality of positions P1, P2, P3, P4, and P5 is stored in the RAM 52. Assume that X3, Y3) (X4, Y4) (X5, Y5) are stored in order. Note that the horizontal direction of the captured image corresponds to the X-axis direction, and the vertical direction corresponds to the Y-axis direction. Further, the right direction and the upward direction are positive directions, and the left direction and the downward direction are negative directions. FIG. 7 shows a moving path of the fingertip corresponding to the process of scrolling the content image in the right and left directions, and a plurality of coordinate information 31 and 32.

A case where determination is made based on a plurality of coordinate information 31 will be described. When the fingertip moves from the position P1 to the position P4 and the coordinate information of the positions P1 to P4 is stored in the RAM 52, the CPU 51 determines that each of the X component and the Y component of the coordinate information of the positions P1 to P4 is expressed by the following formula ( It is determined that the condition of 1) is satisfied.
X1≈X2≈X3≈X4, Y1>Y2>Y3> Y4 (1)
In this case, the CPU 51 determines that the position of the fingertip is moving substantially vertically from top to bottom.

Next, when the fingertip moves to the position P5 and the coordinate information of the position P5 is stored in the RAM 52, the CPU 51 determines that each of the X component and the Y component of the coordinate information of the positions P4 and P5 is expressed by the following equation (2). It is determined that the condition of
X4 <X5, Y4 <Y5 (2)
In this case, the CPU 51 determines that the moving direction of the fingertip position has changed from a movement from the top to the bottom to a movement from the bottom left to the top right. When the conditions of the above formulas (1) and (2) are satisfied, the CPU 51 determines that the tendency of the fingertip position to change with time satisfies a part of the predetermined condition corresponding to the processing content of the right scroll.

Next, a case where determination is made based on a plurality of coordinate information 32 will be described. When the fingertip moves from the position P1 to the position P4 and the coordinate information of the positions P1 to P4 is stored in the RAM 52, the CPU 51 determines that the condition of the expression (1) is satisfied as described above. When the fingertip moves to the position P5 and the coordinate information of the position P5 is stored in the RAM 52, the CPU 51 determines that each of the X component and the Y component of the coordinate information of the positions P4 and P5 satisfies the condition of the following expression (3). Judging to meet.
X4> X5, Y4 <Y5 (3)
In this case, the CPU 51 determines that the moving direction of the fingertip position has changed from a movement from top to bottom to a movement from bottom right to top left. When the conditions of the above formulas (1) and (3) are satisfied, the CPU 51 determines that the tendency of the fingertip position to change with time satisfies a part of the predetermined condition corresponding to the processing content of the left scroll.

  Although details are omitted, it is determined whether the tendency of the fingertip position to change over time satisfies a part of a predetermined condition corresponding to the processing content for scrolling the content image in the upward or downward direction. Is also performed in the same manner as described above. In the above specific example, the coordinate information initially stored in the RAM 52 is the specific position P1 (coordinate information (X1, Y1)), and the determination is made based on the respective coordinate information of the positions P2, P3. The case where this is done has been described as an example. However, the determination method is not limited to the above specific example. For example, the CPU 51 determines that the tendency of the fingertip position over time is based on coordinate information indicating a plurality of fingertip positions starting from any of the positions P2 to P9 (see FIG. 8). Alternatively, it may be determined whether a part of a predetermined condition corresponding to the processing content for scrolling the content image in each of the lower directions is satisfied.

  As shown in FIG. 5, the CPU 51 refers to the plurality of determination data stored in the DB 531, and the tendency of the change of the fingertip position with time is changed to the right, left, upper, and lower, respectively, by the above method. It is determined whether any one of a plurality of predetermined conditions corresponding to scrolling in the direction is satisfied (S19). If the CPU 51 determines that none of some of the plurality of predetermined conditions is satisfied (S19: NO), the process proceeds to S31.

  When the CPU 51 determines that any one of a plurality of predetermined conditions is satisfied (S19: YES), the CPU 51 reduces the transmittance of the captured image among the superimposed images displayed on the image display unit 14 in S17. Next, the following processing is performed. The CPU 51 superimposes the captured image on the content image based on the content image data of the content image displayed on the image display unit 14 in S7 (see FIG. 4) and the captured image data received in S11. A video signal of the superimposed image is generated (S20). Here, the CPU 51 superimposes the content image and the captured image included in the superimposed image in a transparent state. More specifically, it is as follows. The CPU 51 sets the α value of each pixel of the captured image when performing α blending of the content image and the captured image as the α value of the captured image included in the superimposed image displayed on the image display unit 14 in S17. The third α value is larger than the value and smaller than the first α value that is the α value of the content image. In this case, the transmittance of the captured image is smaller than the transmittance of the captured image included in the superimposed image displayed on the image display unit 14 in S17, and the captured image becomes darker and clearer.

  CPU51 transmits the produced | generated video signal to HD10, and displays a superimposed image on the image display part 14 (S21). As a result, the transmittance of the captured image in the superimposed image displayed on the image display unit 14 becomes small, and the color becomes darker and clearer than before the change in transmittance. As compared with the case, the captured image 24 can be visually recognized. As described above, the CPU 51 changes the transmittance of the captured image, so that the tendency of the fingertip position to change is one of predetermined conditions corresponding to scrolling in the right, left, up, and down directions. It is possible to notify the user that it is determined that any one of the sections is satisfied. Further, since the color of the captured image 24 becomes darker and clearer, the user can easily grasp the moving method when the fingertip is continuously moved.

  Further, since the α value (third value) of the photographed image is smaller than the α value (first α value) of the content image, the photographed image 24 is lighter and unclear than the content image 23. This is maintained even after the transmittance of the liquid crystal becomes smaller. Therefore, the user can clearly recognize the content image 23 even after the transmittance of the photographed image is reduced. Thus, the CPU 51 can prevent the captured image 24 from hindering the visibility of the content image 23 even after the transmittance of the captured image 24 is changed. Therefore, the CPU 51 can allow the user to continue to view the content image 23 satisfactorily.

  The CPU 51 refers to the plurality of determination data stored in the DB 531 in FIG. 6 based on the plurality of coordinate information stored in the RAM 52, so that the tendency of the fingertip position over time is changed to the right, left, and upper. It is determined whether any of a plurality of predetermined conditions corresponding to scrolling in the respective downward direction is satisfied (S23).

  Details of the determination method will be described with reference to FIG. By repeating the processes of S11, S13, and S15 (see FIG. 5), coordinate information (X1, Y1) to (X11, Y11) indicating each of the plurality of positions P1 to P11 is sequentially stored in the RAM 52. To do. FIG. 8 shows a predetermined movement path and a plurality of coordinate information 33 corresponding to the process of scrolling the content image in the right direction.

  When the fingertip moves from the position P1 to the position P4 and the coordinate information of the positions P1 to P4 is stored in the RAM 52, the CPU 51 determines that the condition of the above formula (1) is satisfied. In this case, the CPU 51 determines that the position of the fingertip is moving substantially vertically from top to bottom. Next, when the fingertip moves to the position P5 and the coordinate information of the position P5 is stored in the RAM 52, the CPU 51 determines that the coordinate information corresponding to each of the positions P4 and P5 satisfies the condition of the above formula (2). To do. In this case, the CPU 51 determines that the moving direction of the fingertip position has changed from a movement from the top to the bottom to a movement from the bottom left to the top right.

Next, when the fingertip moves from the position P5 to P7 and the coordinate information of the positions P6 and P7 is stored in the RAM 52, the CPU 51 stores the X component and the Y component of the coordinate information of the positions P4 to P7 stored in the RAM 52. It is determined that each satisfies the condition of the following formula (4).
X4 <X5 <X6 <X7, Y4 <Y5 <Y6 <Y7 (4)
In this case, the CPU 51 determines that the position of the fingertip is moving in an oblique direction from the lower left to the upper right.

Next, when the fingertip moves to the position P8 and the coordinate information of the position P8 is stored in the RAM 52, the CPU 51 determines that each of the X component and the Y component of the coordinate information of the positions P7 and P8 is expressed by the following equation (5). It is determined that the condition of
X7> X8, Y7 <Y8 (5)
In this case, the CPU 51 determines that the moving direction of the fingertip position has changed from a movement from the lower left to the upper right to a movement from the lower right to the upper left.

Next, when the fingertip moves from the position P8 to P10 and the coordinate information of the positions P9 and P10 is stored in the RAM 52, the CPU 51 determines that the X component and the Y component of the coordinate information of the positions P7 to P10 are the following. It is determined that the condition of Expression (6) is satisfied.
X7>X8>X9> X10, Y7 <Y8 <Y9 <Y10 (6)
In this case, the CPU 51 determines that the position of the fingertip is moving in an oblique direction from the lower right to the upper left.

Next, when the fingertip moves from the position P10 to P11 and the coordinate information of the position P11 is stored in the RAM 52, the CPU 51 determines that each of the X component and the Y component of the coordinate information of the positions P10 and P11 is expressed by the following formula ( It is determined that the condition of 7) is satisfied.
X10≈X11, Y10> Y11 (7)
In this case, the CPU 51 determines that the moving direction of the fingertip position has changed from a movement from the lower right to the upper left to a movement from the upper to the lower. When satisfying the conditions of the above formulas (1), (2), (4) to (7), the CPU 51 satisfies the predetermined condition corresponding to the right scroll processing content of the tendency of the fingertip position to change over time. Judge.

  Of the determination data stored in the DB 531 in FIG. 6, the determination data associated with the processing content data indicating the rightward scroll is expressed by the above formulas (1), (2), (4) to (7). It is the data shown. As described above, the determination data indicates a relational expression applied when determining whether the tendency of the change in the fingertip position satisfies a predetermined condition.

  Although details are omitted, it is determined whether the tendency of the fingertip position to change over time satisfies a predetermined condition corresponding to the processing content for scrolling the content image in the left, up, and down directions. Is also performed in the same manner as described above. In the above specific example, the coordinate information initially stored in the RAM 52 is the specific position P1 (coordinate information (X1, Y1)), and the determination is made based on the respective coordinate information of the positions P2, P3. The case where this is done has been described as an example. However, the determination method is not limited to the above specific example. For example, the CPU 51 determines that the tendency of the fingertip position over time based on coordinate information indicating a plurality of fingertip positions starting from any one of the positions P2 to P9 is right, left, top, or bottom, respectively. It may be determined whether a predetermined condition corresponding to the processing content for scrolling the content image in the direction is satisfied. A relational expression applied when each determination is performed is stored in the DB 531 as determination data in association with each processing content.

  The CPU 51 may determine whether the tendency of the fingertip position over time satisfies a predetermined condition by another method. For example, the CPU 51 may connect the positions indicated by each of the plurality of coordinate information stored in the RAM 52 with a line in order, and specify it as a line segment indicating the locus of movement of the fingertip. The CPU 51 may perform a well-known pattern matching process based on the specified line segment and a plurality of image templates for specifying a locus stored in advance in the program ROM 53. Note that the plurality of image templates for specifying the locus are templates of images showing a specific shape. The specific shape is, for example, a triangle shown as a movement path in FIG. The specific shape is not limited to a triangle. For example, the specific shape may be a straight line extending in the up-down direction, a straight line extending in the left-right direction, a circle, or the like. Process content data may be associated with each of the plurality of image templates. The CPU 51 may calculate a plurality of cross-correlation coefficients corresponding to each of the plurality of image templates by pattern matching processing, and determine whether the cross-correlation coefficient closest to 1 is a value within a predetermined range. The CPU 51 may determine that the pattern matching is successful when the cross-correlation coefficient closest to 1 is a value within a predetermined range. In this case, the CPU 51 may determine that the change tendency of the fingertip position matches the specific shape indicated by the image template corresponding to the cross-correlation coefficient closest to 1. As a result, the CPU 51 can more appropriately determine whether the tendency of the fingertip position change satisfies a predetermined condition.

  As shown in FIG. 5, the CPU 51 refers to the plurality of determination data stored in the DB 531, and the tendency of the change in the fingertip position over time is determined in the right direction, the left direction, the upward direction by the above determination method. Then, it is determined whether any one of the predetermined conditions corresponding to the processing content for scrolling the content image in the downward direction is satisfied (S23). If the CPU 51 determines that none of the predetermined conditions is met (S23: NO), it returns the process to S17. When the CPU 51 determines that any one of a plurality of predetermined conditions is satisfied (S23: YES), the CPU 51 performs the following process to include the trajectory image 26 in the superimposed image displayed on the image display unit 14 in S21. The CPU 51 specifies a line segment indicating the movement path of the fingertip position by connecting a plurality of positions indicated by the plurality of coordinate information stored in the RAM 52 with lines in the order in which the coordinate information is stored in the RAM 52. The CPU 51 attaches an arrow head to the end on the coordinate information side stored last in the RAM 52 among the specified line segments, and creates a trajectory image 26. The CPU 51 generates a video signal by including the trajectory image in the superimposed image displayed on the image display unit 14 in S21.

  The CPU 51 transmits a video signal to the HD 10 and causes the image display unit 14 to display a superimposed image including a trajectory image (S25). The user wearing the HD1 visually recognizes the superimposed image 212C including the trajectory image 26, for example, as shown by the scene 21C in FIG. As a result, the user can check the tendency of the fingertip position over time. In addition, the CPU 51 determines that the tendency of the fingertip position to change over time satisfies one of the predetermined conditions corresponding to the scrolling of the content image in any of the right, left, up, and down directions. In this case, the superimposed image including the trajectory image is displayed on the image display unit 14. For this reason, the user can grasp that the tendency of the fingertip position corresponding to the processing content to be executed by the HMD 1 over time has been recognized by the HMD 1. Further, the CPU 51 causes the image display unit 14 to display a superimposed image including a trajectory image before the content image is actually scrolled (see S27, which will be described later). Thereby, the user can recognize that the process corresponding to the movement of the finger has not been executed yet.

  The CPU 51 determines the processing content corresponding to the determination data referred to when the predetermined condition is determined in S23 among the plurality of determination data stored in the DB 531, that is, the direction in which the content image is scrolled. Identify. The CPU 51 executes a process of scrolling the content image based on the specified processing content (S27).

  A specific example will be described. For example, the CPU 51 determines in S23 that the tendency of the fingertip position over time satisfies the predetermined condition corresponding to the determination data “Right.file” in the DB 531. In this case, the CPU 51 specifies that the processing content associated with the determination data “Right.file” indicates processing for scrolling the content image in the right direction. The CPU 51 performs the following process to scroll the content image included in the superimposed image including the trajectory image displayed on the image display unit 14 in S25 in the right direction. The CPU 51 scrolls the content image in the right direction among the superimposed images displayed on the image display unit 14 in S21. The CPU 51 generates a video signal of a superimposed image in which the captured image is superimposed on the content image scrolled in the right direction and includes a trajectory image. The CPU 51 transmits the generated video signal to the HD 10 and causes the image display unit 14 to display a superimposed image. The user wearing the HD1 visually recognizes the superimposed image 212D including the content image 23 scrolled in the right direction, for example, as shown by a scene 21D in FIG. As described above, the CPU 51 can easily specify the process corresponding to the change tendency of the fingertip position by referring to the DB 531.

  Note that when a plurality of image templates stored in the program ROM 53 are used to determine whether the tendency of the fingertip position change satisfies a predetermined condition, the CPU 51 determines the cross-correlation coefficient closest to 1. The processing content data associated with the corresponding image template may be specified by the same method as described above, and the corresponding processing may be executed.

  After the process executed in S27 is completed, the CPU 51 transmits only the content image and the video signal of the captured image to the HD 10, thereby deleting the trajectory image from the superimposed image displayed on the image display unit 14 (S29). ). Next, the CPU 51 deletes the trajectory image from the superimposed image in S29, and then transmits only the video signal of the content image to the HD 10, thereby further erasing the captured image from the superimposed image displayed on the image display unit 14. (S31). Only the content image is displayed on the image display unit 14. The CPU 51 returns the process to S11. As a result, the user can confirm that the HMD 1 has executed a process corresponding to the temporal change in the position of the fingertip. In addition, after the processing is executed in S27, the CPU 51 deletes the captured image and the trajectory image from the superimposed image displayed on the image display unit 14. As a result, the CPU 51 does not allow the user to visually recognize the captured image and the trajectory image that are no longer necessary for the user, and thus allows the user to visually recognize the content image.

  As an example, the CPU 51 executes a process of scrolling the content image in the right direction in S27. In this case, the user wearing the HD 10 visually recognizes the scene in which only the content image 212E scrolled in the right direction is superimposed on the real image 211, as indicated by the scene 21E in FIG. Therefore, the user can visually recognize the content image after the scroll process according to the movement of the fingertip is executed.

  As described above, when the CPU 51 determines that there is a fingertip within a predetermined area photographed by the camera 20 (S13: YES), the CPU 51 causes the image display unit 14 to display a superimposed image including the content image and the photographed image. (S17). Therefore, the user can recognize whether the fingertip is captured by the camera 20 from the superimposed image. CPU51 performs the process corresponding to a predetermined condition, when the tendency of the change of the position of a fingertip satisfies predetermined conditions (S23: YES) (S27). The user can cause the HMD 1 to execute a desired process by moving the fingertip within a predetermined region so that a change with time of the fingertip position satisfies a predetermined condition. For example, when the user confirms only the position of the fingertip included in the real image 211 and moves the fingertip outside the predetermined area photographed by the camera 20, the CPU 51 cannot accurately identify the temporal change in the position of the fingertip. The corresponding process cannot be executed. On the other hand, the CPU 51 can cause the user to confirm that the fingertip is included in the predetermined area captured by the camera 20 by displaying the superimposed image. Since the user can easily move the fingertip within a predetermined area photographed by the camera 20, the HMD1 can appropriately recognize the fingertip. For this reason, the user can cause the HMD 1 to appropriately execute a desired process.

  In addition, this invention is not limited to the said embodiment, A various change is possible. In the said embodiment, although CPU51 specified the fingertip by S15, the target object specified is not limited to a fingertip. For example, the CPU 51 may specify a pen tip of a predetermined color (for example, red) as an object. In this case, the CPU 51 can detect a region including the target object by detecting a region where pixels having a predetermined pixel value corresponding to a predetermined color are continuous. Then, a well-known pattern matching process is performed on the basis of the polygonal shape surrounding the entire detected area and the image template having a shape corresponding to the object stored in advance in the program ROM 53. When the cross-correlation coefficient calculated by the pattern matching process is a value within a predetermined range, the CPU 51 determines that the pattern matching between the specified polygonal shape and the image template is successful. In this case, the CPU 51 identifies the polygonal area as the pen tip.

  The camera 20 may directly transmit the captured image data to the CB 50. In the above embodiment, the CPU 51 causes the image display unit 14 to display a superimposed image in which a captured image is superimposed on a content image. The CPU 51 may display both in a different area of the image display unit 14 without overlapping the content image and the captured image. In this case, the CPU 51 can maintain better visibility of the content image while allowing the user to visually recognize the captured image.

  In the above embodiment, when the superimposed image is displayed on the image display unit 14 in S17, the CPU 51 determines the α value (second α value) of each pixel of the captured image when α blending is performed, as the α value of each pixel of the content image. The value was smaller than the value (first α value). On the other hand, the CPU 51 may make the first α value and the second α value substantially the same. Accordingly, the CPU 51 can display the content image and the captured image on the image display unit 14 with the same transmittance. In this case, the user can recognize both the content image and the captured image well.

  In the above-described embodiment, the CPU 51 changes the α value corresponding to the captured image from the second α value to the third α value larger than the second α value in S21 and causes the image display unit 14 to display the α value. On the other hand, the CPU 51 does not change the α value corresponding to the photographed image and displays the image in S17 even when it is determined in S19 that the change tendency of the fingertip position satisfies a part of the predetermined condition. The superimposed image displayed on the unit 14 may be displayed as it is in S21. Further, the CPU 51 may change the α value corresponding to the photographed image from the second α value to a fourth α value smaller than the second α value in S21 and display the changed value on the image display unit 14. Thus, the CPU 51 can make the user visually recognize a clear captured image that is darker in color than the content image.

  In the above embodiment, when it is determined that the tendency of the fingertip position over time to satisfy a predetermined condition (S23: YES), the CPU 51 causes the image display unit 14 to display a superimposed image including a trajectory image. (S25). On the other hand, if the CPU 51 determines in S19 that the tendency of the fingertip position over time satisfies a part of the predetermined condition (S19: YES), the CPU 51 displays the trajectory image on the image display unit 14. May be. The CPU 51 may display the trajectory image on the image display unit 14 when it is determined in S11 that the fingertip is included in the captured image (S11: YES). In these cases, the user can easily move the fingertip so that the tendency of the change in the position of the fingertip satisfies a predetermined condition by visually recognizing the trajectory image.

  In the above embodiment, the CPU 51 deletes the trajectory image from the image display unit 14 after executing the process corresponding to the predetermined condition in S27. On the other hand, the CPU 51 may delete the trajectory image from the image display unit 14 before executing the process.

  In the above embodiment, the camera 20 and the interface 15 may communicate wirelessly. The HD 10 and the CB 50 may communicate wirelessly. In these cases, there is a possibility that the time from when the photographing by the camera 20 is performed until the CPU 51 receives the photographed image data is longer than in the case of the above embodiment. However, in the above embodiment, when the delay time is not smaller than the predetermined value (S15: NO), the CPU 51 does not display the superimposed image including the captured image on the image display unit 14. For this reason, the movement of the fingertip indicated by the captured image displayed on the image display unit 14 is not significantly delayed compared to the movement of the fingertip included in the real image. Therefore, it is possible to prevent the user from feeling uncomfortable due to the delay.

  The image display unit 14 is an example of the “display unit” in the present invention. The camera 20 is an example of the “photographing unit” in the present invention. The fingertip is an example of the “object” in the present invention. The captured image is an example of the “object image” in the present invention. The CPU 51 that performs the process of S13 is an example of the “first determination unit” in the present invention. The CPU 51 that performs the processes of S17 and S21 is an example of the “first display means” in the present invention. The CPU 51 that performs the process of S23 is an example of the “second determination unit” in the present invention. The CPU 51 that performs the process of S19 is an example of the “third determination unit” in the present invention. The CPU 51 that performs the process of S27 is an example of the “execution unit” in the present invention. The CPU 51 that performs the processes of S16 and S20 is an example of the “generating means” in the present invention. The CPU 51 that performs the process of S25 is an example of the “second display unit” in the present invention. The CPU 51 that performs the process of S31 is an example of the “erasing unit” in the present invention. The program ROM 53 that stores the image template is an example of the “storage unit” in the present invention.

1 HMD
3 Half mirror 10 HD
14 Image display unit 20 Camera 21 Scene 23 Content image 24 Captured image 26 Trail image 50 CB
53 Program ROM
54 Flash ROM

Claims (13)

A display unit capable of displaying at least a content image and emitting image light in a first direction;
An imaging unit capable of imaging a predetermined area including at least an area in a second direction opposite to the first direction and outputting image data;
Obtaining means for obtaining image data output from the photographing unit;
First determination means for determining whether there is a predetermined object in the predetermined region based on the image data acquired by the acquisition means;
The time when the first determination means determines that the object is in the predetermined area , and the time when the image data is acquired by the acquisition means, and the time when the inside of the predetermined area is imaged by the imaging unit A first display means for displaying the content image and the object image on the display unit when the difference between the first image and the target image is smaller than a predetermined value ;
When the first determination unit determines that the object is in the predetermined area, a tendency of a change in the position of the object in the predetermined area based on the image data is determined according to a predetermined condition. A second judging means for judging whether or not
An execution means for executing a process corresponding to the predetermined condition when the second determination means determines that the tendency of the change in the position of the object satisfies the predetermined condition;
The head-mounted display, wherein the object image is an image corresponding to the image data for which the object is determined to be within the predetermined area by the first determination unit. Based on the content image and the object image, further comprising a generating means for generating a superimposed image in which the object image is superimposed on the content image,
The head-mounted display according to claim 1, wherein the first display unit displays the superimposed image on the display unit.   The generating means uses a first α value that is an α value of each pixel of the content image and a second α value that is an α value of each pixel of the object image and is smaller than the first α value, and The head mounted display according to claim 2, wherein a superimposed image is generated. A third determining means for determining whether a tendency of a change in the position of the object satisfies a part of the predetermined condition;
The generation unit determines the α value of each pixel of the object image when the third determination unit determines that the tendency of the change in the position of the object satisfies a part of the predetermined condition, 4. The head mounted display according to claim 3, wherein the superimposed image is generated by using a third α value larger than the second α value and using the first α value and the third α value. 5.   The head mounted display according to claim 4, wherein the third α value is smaller than the first α value.   When the second determination means determines that the tendency of the change in the position of the object satisfies the predetermined condition, a trajectory image indicating the position of the object is superimposed on the content image on the display unit. 6. The head mounted display according to claim 1, further comprising second display means for displaying.   The head mounted display according to claim 6, wherein the second display unit displays the trajectory image on the display unit before the corresponding process is executed by the execution unit.   And an erasing unit that erases the object image displayed by the first display unit when the second determination unit determines that the tendency of the change in the position of the object satisfies the predetermined condition. The head mounted display according to any one of claims 1 to 7, wherein the head mounted display is provided.   9. The head mounted display according to claim 8, wherein the erasing unit erases the object image after the corresponding processing is executed by the execution unit. The acquisition means further acquires time information indicating a time when the inside of the predetermined area is imaged by the imaging unit from the imaging unit,
The first display unit is configured to display the content image and the target only when a difference between a time when the time information is acquired by the acquisition unit and a time indicated by the acquired time information is smaller than a predetermined value. 10. The head according to claim 1, wherein an object image is displayed on the display unit, and the object image is not displayed on the display unit when the difference is not smaller than a predetermined value. Mount display.   The second determination means determines whether the specific shape indicated by the template data stored in the storage unit matches the change tendency of the position of the target object, thereby changing the position of the target object. The head-mounted display according to claim 1, wherein it is determined whether or not the tendency satisfies a predetermined condition.   The head mount according to claim 11, wherein the execution unit executes a specific process associated with template data indicating a specific shape that is determined to match a tendency of a change in the position of the object. display. A head-mounted display computer having a display unit capable of displaying at least a content image,
Image data output from the photographing unit is acquired when a predetermined region including at least a region in a second direction opposite to the first direction in which the display unit can emit image light is photographed by the photographing unit. Acquisition means;
First determination means for determining whether there is a predetermined object in the predetermined region based on the image data acquired by the acquisition means;
The time when the first determination means determines that the object is in the predetermined area , and the time when the image data is acquired by the acquisition means, and the time when the inside of the predetermined area is imaged by the imaging unit A first display means for displaying the content image and the object image on the display unit when the difference between the first image and the target image is smaller than a predetermined value ;
When the first determination unit determines that the object is in the predetermined area, a tendency of a change in the position of the object in the predetermined area based on the image data is determined according to a predetermined condition. A second judging means for judging whether or not
Control for functioning as execution means for executing processing corresponding to the predetermined condition when the second determination means determines that the tendency of the change in the position of the object satisfies the predetermined condition A program,
The control program according to claim 1, wherein the object image is an image corresponding to the image data determined by the first determination means that the object is within the predetermined area.

Images