WO2013147147A1

WO2013147147A1 - Head-mounted display and computer program

Info

Publication number: WO2013147147A1
Application number: PCT/JP2013/059503
Authority: WO
Inventors: 井上　浩; 邦宏伊藤
Original assignee: ブラザー工業株式会社
Priority date: 2012-03-29
Filing date: 2013-03-29
Publication date: 2013-10-03
Also published as: JP2013206411A

Abstract

The purpose of the present invention is to provide a head-mounted display and a computer program for a head-mounted display capable of obtaining advantageous operability. A first movement of a part of the body of a user is acquired in accordance with a photographed image of the outside world, the first movement being set as an instruction for a first processing executed by the head-mounted display in which the light of an image corresponding to the image displayed to the user is formed. The first processing corresponding to the acquired first movement is controlled. A first characteristic value for expressing a characteristic of the first movement registered in relation to identification information for identifying a user (S120). The user is specified (S100). The first characteristic value related to the specified identification information is updated (S118) in accordance with a second characteristic value for expressing a characteristic of the newly acquired first movement when the first characteristic value has been registered in relation to the specified identification information for identifying a specified user.

Description

Head mounted display and computer program

The present disclosure relates to a head mounted display and a computer program for the head mounted display.

A technology related to a head-mounted display that is mounted on the user's head and displays an image to the user has been proposed. Patent Document 1 discloses a display device that can easily give an operation instruction by an operator and can easily give a complicated operation instruction. When the control unit that controls the head-mounted display detects that a part of the operator's body such as a hand or a finger has shown a predetermined movement, the process corresponding to the virtual icon in the space is activated. To do.

JP-A-8-6708

When the control unit controls the operation of the head-mounted display according to the movement of the body part of the user of the head-mounted display, it is desirable that the control unit suitably controls according to the movement of the body part of the user. However, the predetermined one movement has a feature for each user. For example, each user moves with the user's habit. For example, it is assumed that the instruction to set the image displayed on the head mounted display as the image of the next page is set to a movement of moving the right hand from the right side to the left side in the left-right direction of the user. For the first user, the amount of movement from the right side to the left side is the first amount. For the first user, the second user may be a second quantity greater than the first quantity. Even the same user may move differently. For example, the movement may change between wearing a garment that easily moves the body part and wearing a garment that hardly moves the body part.

This disclosure is intended to provide a head-mounted display and a computer program for the head-mounted display that can obtain suitable operability.

According to one aspect of the present disclosure, according to the first acquisition unit that acquires the external field image data indicating the external field image including the predetermined portion of the target imaged by the imaging unit, the external image data acquired by the first acquisition unit, Second acquisition means for acquiring a first movement of a predetermined part corresponding to the first process; and processing means for controlling the first process corresponding to the first movement acquired by the acquisition means; Registration means for registering the identification information for identifying the object having the predetermined part and the first feature quantity representing the feature of the first movement in association with the storage unit, and accepting input of identification information Receiving means, wherein the registration means is configured to register the first feature amount registered in association with the identification information received by the receiving means after the receiving means has received the identification information. Second acquisition means According to a second feature amount representing the feature of the acquired first motion, updating the first feature amount that is associated with the already identified the identification information of a head-mounted display. According to this, the first feature amount representing the first movement feature for the first process is registered in the storage unit in association with the identification information for identifying the user who uses the head mounted display. A user who uses the head-mounted display is identified, and a first feature amount associated with the identified identification information for identifying the identified user is represented by the second feature representing the newly acquired first motion feature. The feature amount can be updated. As described above, the first movement for the first process is not always performed in a constant state. According to this head mounted display, the first feature amount is updated with respect to the first movement, so that the operability for instructing the first process can be improved for each user.

This head mounted display may be as follows. First determination means for determining whether the second feature value acquired by the second acquisition means corresponds to the first feature value registered in the storage unit in association with the specified identification information. And when the first determination means determines that the second feature quantity does not correspond to the first feature quantity, the second feature quantity is compared with the first feature quantity. Second determining means for determining whether the second feature value is included in a predetermined range with respect to the first feature value. The first process corresponding to the first movement is controlled in response to the determination that the second feature value is included in the range of the first feature value. In response to the first movement in response to determining that the predetermined range is not included. Does not control the first process, it may be.

Also, this head mounted display may be as follows. In response to determining that the second feature amount is included in a predetermined range with respect to the first feature amount, the registration unit is configured to follow the second feature amount according to the second feature amount, The first feature value associated with the specified identification information is updated, and the second determination unit does not include the second feature value within a predetermined range with respect to the first feature value. In response to the determination, the first feature amount associated with the identified identification information may not be updated according to the second feature amount. According to this, even if it is not determined that the first movement corresponds to the first feature amount, whether or not the first movement is within a predetermined range on the basis of the first feature amount. Accordingly, the first process and the update of the first feature amount can be controlled.

The second acquisition means corresponds to a second process that is executed by the head mounted display in accordance with the external image acquired by the first acquisition means and has a fixed relationship with the first process. The second movement of the predetermined part to be acquired, the processing means controls the second processing corresponding to the second movement acquired by the second acquisition means, the registration means, After the second determination unit determines that the second feature amount is included in a predetermined range with respect to the first feature amount, the acquisition unit performs the first movement and the second feature amount. And the registration of the first feature amount updated in accordance with the second feature amount in the storage unit in response to the movement of The first feature value before being updated with the second feature value. , It may be so. According to this, the update of the first feature amount can be suitably performed based on continuous instructions. As an instruction for the second process having a certain relationship with the controlled first process, when the second movement is acquired continuously within the predetermined period, It is considered that the first process was not a process intended by the user. According to the head mounted display described above, it is possible to avoid the update of the first feature amount based on the new first movement for the first process unintended by the user.

The registration unit may reset the first feature amount registered in the storage unit when an instruction to reset registration in the storage unit is input to the head mounted display. According to this, the first feature amount registered in the storage unit can be reset.

The first feature acquired by the second acquisition unit after the reception unit has received the identification information, and the first feature amount registered in the storage unit in association with the identified identification information And a third judging means for judging the correspondence between the processing means and the processing means when the judging means judges that the first movement corresponds to the first feature amount by the obtaining means. The first process corresponding to the newly acquired first movement may be controlled. According to this, the first process is controlled, and the operation corresponding to the first process can be suitably executed on the head mounted display.

Another aspect of the present disclosure is a computer program readable by a control unit that controls a head-mounted display, wherein the control unit displays an external world image including a predetermined portion of a target imaged by the imaging unit. First acquisition means for acquiring image data, second acquisition means for acquiring a first movement of a predetermined part corresponding to a first process according to external image data acquired by the first acquisition means, Processing means for controlling the first process corresponding to the first movement acquired by the acquisition means; identification information for identifying an object having the predetermined part; and a first characteristic representing the characteristics of the first movement. And a registration unit that registers the one feature value in association with the storage unit, and a reception unit that receives the input of the identification information. The first feature amount registered in association with the identification information is a second feature representing the first movement feature acquired by the second acquisition unit after the reception unit has received the identification information. A computer program including a function of updating the first feature amount associated with the identified identification information according to a feature amount. According to this, as described above, the first feature amount associated with the identified identification information that identifies the identified user and identifies the identified user is represented by the newly acquired first motion feature. A head-mounted display to be updated can be realized with the second feature amount. This computer program may be specified as a computer program for a head mounted display that further includes the above-described configuration. According to such a computer program, a head-mounted display that further includes the above-described configuration can be realized.

According to the present disclosure, it is possible to obtain a head-mounted display and a computer program for the head-mounted display that can obtain suitable operability.

It is a figure which shows an example of a head mounted display. It is a top view of a display apparatus. It is sectional drawing which cut | disconnected the display apparatus in the center of the up-down direction shown in FIG. It is a block diagram which shows the electric constitution of a head mounted display. It is a figure explaining the feature-value showing the characteristic of the motion of the user's body part set as the instruction | indication for the process performed with a head mounted display. It is a flowchart of a main process. It is a flowchart of a routine process.

Embodiments for carrying out the present disclosure will be described with reference to the drawings. The present disclosure is not limited to the configurations described below, and various configurations can be employed in the same technical idea. For example, some of the configurations shown below may be omitted or replaced with other configurations. Other configurations may be included.

<Head mounted display>
An outline of the head mounted display 1 will be described with reference to FIGS. 1 and 2. The head mounted display is hereinafter referred to as HMD. The front-rear direction and the left-right direction in FIG. 1 and FIG. 2 and the up-down direction in FIG. . The HMD 1 includes a system box 2 and a display device 3. As shown in FIG. 1, the system box 2 and the display device 3 are connected, for example, via a transmission cable 4. The system box 2 transmits an image signal and supplies power to the display device 3.

The display device 3 is detachably attached to the spectacle frame 5. The spectacle frame 5 is mounted on the user's head. The spectacle frame 5 is an example for mounting the display device 3 on the user's head. The display device 3 may be mounted on the user's head by a mounting unit different from the spectacle frame 5. The display device 3 includes a housing 30. The housing 30 is a rectangular tubular resin member, and is formed in an L shape in plan view. A half mirror 31 as a deflection member is provided at the right end of the housing 30. A camera 32 is provided on the upper surface of the housing 30. The camera 32 captures an image around the user. In the present embodiment, the camera 32 is provided on the upper surface of the housing 30 so as to capture an external image in a direction corresponding to the direction in which the user's face faces.

As shown in FIG. 1, the spectacle frame 5 includes a left frame portion 52, a right frame portion 53, a central frame portion 54, and a support portion 56. The left frame portion 52 extending in the front-rear direction is hung on the user's left ear. The right frame portion 53 extending in the front-rear direction is hung on the user's right ear. The central frame portion 54 extending in the left-right direction connects the front end portion of the left frame portion 52 and the front end portion of the right frame portion 53 and is disposed on the user's face portion. A pair of nose pads 55 are provided at the center in the longitudinal direction of the center frame 54. The support portion 56 is provided on the upper left end side of the central frame portion 54. The support part 56 includes a downward extension part 58. The downward extension 58 extends in the up-down direction at the front left of the user's face. The downward extending portion 58 is slidably engaged with a groove 57 formed in the support portion 56 and extending in the left-right direction. The position of the display device 3 in the left-right direction is adjusted by sliding the lower extension portion 58 in the left-right direction.

The display device 3 will be described with reference to FIGS. 2A and 2B. As shown in FIG. 2A, the housing 30 is provided with an attachment portion 33. The attachment portion 33 is provided at a portion of the housing 30 that faces the spectacle frame 5. The attachment portion 33 has a U-shaped groove along the vertical direction. A downward extending portion 58 provided on the support portion 56 of the spectacle frame 5 is slidably engaged with the U-shaped groove of the attachment portion 33. The position of the display device 3 in the vertical direction is adjusted by sliding the casing 30 attached to the downward extending portion 58 in the vertical direction. As illustrated in FIG. 2B, the housing 30 includes an image light forming unit 34 and an eyepiece optical unit 35. In FIG. 2B, the housing 30 is omitted for simplification. The image light Lim emitted from the image light forming unit 34 is collected by the eyepiece optical unit 35. A part of the condensed image light Lim is reflected by the half mirror 31 and guided to the user's eye EB.

The image light forming unit 34 is provided at the left end inside the housing 30. The image light forming unit 34 forms the image light Lim based on the image signal from the system box 2. The image light forming unit 34 is configured by a known spatial light modulation element. The spatial light modulation element is, for example, a liquid crystal display composed of a liquid crystal display element and a light source, or an organic EL (Electro-Luminescence). The image light forming unit 34 may be a known retinal scanning display that projects an image on the retina by mechanically two-dimensionally scanning light from a light source such as a laser instead of the spatial light modulator. .

The eyepiece optical unit 35 includes a lens 36 and a lens holder 37. The left end of the lens holder 37 is in contact with the right end of the image light forming unit 34. A lens 36 is held inside the right side of the lens holder 37. That is, the lens 36 and the image light forming unit 34 are separated by the lens holder 37 by a distance corresponding to the display distance of the virtual image displayed to the user. The lens 36 is a plurality of lenses arranged in the left-right direction. In the present embodiment, the lens 36 is composed of a plurality of lenses in order to achieve desired optical characteristics. However, the lens 36 may be composed of a single lens. The eyepiece optical unit 35 condenses the image light Lim and guides it to the half mirror 31. Since the user visually recognizes a virtual image by the display device 3, the image light Lim collected by the lens 36 is diffused light or parallel light. That is, “collection” is an action on the incident light flux of a lens having a positive power as a whole, and is not limited to that the outgoing light flux is convergent light.

The plate-shaped half mirror 31 is connected to the right end of the housing 30. Specifically, the half mirror 31 is sandwiched from above and below by a predetermined portion of the housing 30 at the right end of the housing 30. The half mirror 31 is formed by vapor-depositing a metal such as aluminum so that the transmittance is 50% on the surface of a plate-like transparent member such as glass or light-transmitting resin. The light transmissive resin is, for example, acrylic, polyacetal or the like. The transmittance of the half mirror 31 may not be 50%. In other words, “translucent” in the present embodiment is such that a part of the image light Lim is reflected and a part of the external light is transmitted, so that the user can substantially visually recognize the image (virtual image) and the external environment. It is a concept including the state which is.

<Electrical configuration>
An outline of the electrical configuration of the system box 2 and the display device 3 will be described with reference to FIG. The system box 2 includes a CPU 20, a program ROM 21, a flash ROM 22, a RAM 23, a communication circuit 24, a video RAM 25, an image processing unit 26, and a peripheral I / F 27. The CPU 20 controls various processes executed in the system box 2. The processes controlled by the CPU 20 are, for example, a main process shown in FIG. 5 and a routine process shown in FIG. The CPU 20 instructs the image processing unit 26 to execute image processing. The program ROM 21 stores computer programs for various processes executed in the system box 2.

The flash ROM 22 stores various data. Data stored in the flash ROM 22 is, for example, image data and a database. The image data is data corresponding to the image displayed on the display device 3. The image data includes data corresponding to images for a plurality of pages. In the present embodiment, image data corresponding to images for a plurality of pages will be described as an example. The user visually recognizes the image of each page corresponding to the image data displayed on the display device 3. The database is a storage unit in which identification information and a first feature amount are registered in association with each other. The identification information is information for identifying a user who uses the HMD 1. The first feature amount is information representing a motion feature of a predetermined part of the target, for example, a part of the user's body, set as an instruction for processing executed by the HMD 1.

The processing executed by the HMD 1 is, for example, page turning processing and page returning processing. The page turning process is a process in which the image displayed on the display device 3 is used as the image of the next page. The page return process is a process in which the image displayed on the display device 3 is the image of the previous page. The page feed process and the page return process are processes having a relation related to S214 of the routine process shown in FIG. The user's body part is, for example, a hand. The movement for the page turning process is a movement of moving one hand from one side to the other side in the left-right direction of the user when the body part of the user is a hand. Specifically, this is a movement of moving the right hand from the right side to the left side. The movement for the page return process is a movement of moving the hand on one side from the other side to the one side in the left-right direction of the user. Specifically, this is a movement of moving the right hand from the left side to the right side. Based on the specific example described above, as shown in FIG. 4, the feature amount representing the feature of the right hand, the feature amount representing the feature of the hand movement direction, and the feature amount representing the feature of the hand movement amount are Are registered in the database in association with the identification information. As is clear from FIG. 4, the first feature quantity that is associated with the identification information and registered in the database is managed for each process.

The RAM 23 serves as a work area when the CPU 20 executes the computer program stored in the program ROM 21. The communication circuit 24 controls communication with the display device 3 and the like. The transmission cable 4 is electrically connected to the communication circuit 24. The communication circuit 24 transmits an image signal to the display device 3 via the transmission cable 4. The communication circuit 24 supplies power from a battery or the like to the display device 3 via the transmission cable 4. The communication circuit 24 receives an external image signal transmitted from the display device 3 via the transmission cable 4. The external image signal is a signal that represents external image data corresponding to the external image captured by the camera 32. The video RAM 25 stores image data transmitted to the display device 3 as an image signal. The video RAM 25 stores external image data based on an external image signal received by the communication circuit 24. The image processing unit 26 reads image data from the flash ROM 22 to the video RAM 25, executes image processing on the image data stored in the video RAM 25, and generates an image signal. The image processing unit 26 generates external field image data from the received external field image signal. The image processing unit 26 performs image processing on the external image data in accordance with a command from the CPU 20. The image processing unit 26 is provided to execute various image processes in order to reduce the processing load on the CPU 20.

The peripheral I / F 27 is an interface to which predetermined parts are electrically connected. For example, a power switch 271, a power lamp 272, and an operation unit 273 are connected to the peripheral I / F 27. The power switch 271 is a switch for switching on and off the power to the HMD 1. When the power switch 271 is turned on, the HMD 1 is activated. Specifically, when the power switch 271 is turned on, power is supplied from the battery to the system box 2 and power is supplied to the display device 3 via the transmission cable 4. The power lamp 272 is a lamp indicating that the power is on. The power lamp 272 is lit when the power switch 271 is turned on. The operation unit 273 is an interface for inputting a predetermined instruction to the system box 2. The operation unit 273 includes a plurality of operation buttons. The predetermined instruction is input by appropriately operating the operation buttons of the operation unit 273.

The display device 3 includes a CPU 38, a program ROM 39, a RAM 40, a communication circuit 41, and a peripheral I / F 43 in addition to the image light forming unit 34. Each unit included in the display device 3 is built in the housing 30 together with the image light forming unit 34 and the like. The CPU 38 controls various processes executed on the display device 3. For example, the CPU 38 drives the image light forming unit 34 to form the image light Lim corresponding to the image signal, and controls the image to be displayed to the user. The program ROM 39 stores computer programs for various processes executed by the display device 3. The process executed by the display device 3 is, for example, a process related to the formation of the image light Lim by the image light forming unit 34. The RAM 40 serves as a work area when the CPU 38 executes the computer program stored in the program ROM 39.

The communication circuit 41 controls communication with the system box 2 and the like. The transmission cable 4 is electrically connected to the communication circuit 41. The transmission cable 4 extends rearward from the housing 30 and is connected to the system box 2. The communication circuit 41 transmits an external image signal to the system box 2 via the transmission cable 4. The communication circuit 41 receives an image signal transmitted from the system box 2 via the transmission cable 4. The communication circuit 41 is supplied with power from the system box 2 via the transmission cable 4. The supplied power is supplied to each part of the display device 3 and the camera 32. The peripheral I / F 43 is an interface to which the camera 32 is connected. An external image signal representing external image data corresponding to the external image captured by the camera 32 is transmitted from the communication circuit 41 to the system box 2 via the peripheral I / F 43.

The HMD 1 is controlled by the CPU 20 of the system box 2 and the CPU 38 of the display device 3. That is, various functions are realized in the system box 2 by the CPU 20 executing the computer program stored in the program ROM 21. The CPU 20 can also be specified as a control unit serving as various functional units included in the HMD 1. Similarly, various functions are realized in the display device 3 by the CPU 38 executing the computer program stored in the program ROM 39. The CPU 38 can also be specified as a control unit serving as various functional units included in the HMD 1. The computer program is written in the program ROM 21 and the program ROM 39 when the HMD 1 is shipped from the factory. The program ROM 39 is an example of a computer-readable storage device. Instead of the program ROM 39, for example, ROM, HDD, RAM, etc. may be used as the storage device. In this case, the storage device is a storage medium excluding a temporary storage medium. The storage device may be a non-transitory storage medium. A non-transitory storage medium can retain data regardless of the length of time to store the data. The computer program may be stored in a storage medium such as an external server. When the computer program is stored in the server, the computer program is downloaded from an external server or the like via the connection interface and stored in the program ROM 21 and the program ROM 39 as appropriate. In this case, the computer program is transmitted to the HMD 1 from an external server or the like as a computer-readable temporary storage medium (for example, a transmission signal).

<Main processing>
The main process executed by the HMD 1 will be described with reference to FIG. The main process is executed by the CPU 20 of the system box 2. The main process is started when the user operates the power switch 271 to turn on the power. The CPU 20 uses the RAM 23 to execute a computer program for main processing stored in the program ROM 21. The computer program includes a computer program module for routine processing.

In S100, the CPU 20 that started the main process executes personal authentication. Identification information for identifying a user is used for personal authentication. For example, the user operates the operation unit 273 and inputs his / her identification information. The identification information may be input by another method. For example, identification information according to the read information may be input by reading a wireless tag including information that can identify an individual with a reading unit. Alternatively, identification information based on the imaged body part may be input by capturing an image of a body part having a characteristic amount that can identify an individual such as a hand with the camera 32. The CPU 20 acquires the input identification information and stores it in the RAM 23. The user using the HMD 1 is specified by the input identification information. In the present embodiment, the identification information acquired in S100 and capable of identifying the user using the HMD 1 is referred to as “specified identification information”. In S102, the CPU 20 performs control to display a user menu. Thereby, the display device 3 displays the user menu, and the user visually recognizes the user menu.

The user menu displayed in S102 includes a plurality of options. For example, the following three options are included. The first option is an option for resetting the learning result of the gesture. The learning result is obtained by the learning process shown in S118 and S210 in FIG. The second option is an option for performing gesture learning. When gesture learning is selected and gesture learning is performed, the first feature value registered in the database in association with the identified identification information is updated. The third option is an option for displaying an image on the display device 3. When an option for displaying an image is selected, the display device 3 displays an image corresponding to the image data stored in the flash ROM 22. The user operates the operation unit 273 to select any option.

In S104, the CPU 20 determines whether or not a learning result reset is input according to the display of the user menu. When reset of the learning result is input (S104: Yes), the CPU 20 resets the first feature amount registered in association with the identified identification information in the database stored in the flash ROM 22 (S106). The first feature value registered in association with the identified identification information is a predetermined initial value. After executing S106, the CPU 20 controls to display the user menu again. In the user menu displayed after resetting, the reset of the learning result may not be input. This is because the first feature amount has already been reset. In the present embodiment, it is assumed that the reset of the learning result cannot be input in the user menu after executing S106.

If the learning result reset is not input (S104: No), or after executing S106, the CPU 20 determines whether or not gesture learning is input according to the display of the user menu (S108). When the image display is input and the gesture learning is not input (S108: No), the CPU 20 shifts the process to S124. When gesture learning is input (S108: Yes), the CPU 20 controls the camera 32 to be activated (S110). As a result, the camera 32 is activated. Thereafter, the CPU 20 controls to display a selection screen for inputting a learning target. Thereby, the display device 3 displays the selection screen, and the user visually recognizes the selection screen. The selection screen includes a plurality of options for specifying each process executed by the HMD 1 or the movement of the body part. For example, the selection screen includes options related to page feed processing and options related to page return processing. The user operates the operation unit 273 and inputs a desired option. For example, the user inputs options related to the page turning process. In S112, the CPU 20 specifies a learning target in accordance with the input option. When an option related to the page turning process is input, the CPU 20 specifies, as a learning target, a first feature amount representing a movement feature for the page turning process associated with the identified identification information in the database. To do.

In S114, the CPU 20 determines whether or not the user is ready to perform the movement represented by the first feature amount of the learning target specified in S112. The user who is ready presses down the operation key associated with the preparation completion instruction on the operation unit 273. The CPU 20 determines S114 according to whether or not the operation key is pressed in the operation unit 273. When the operation key is not pressed, the CPU 20 denies S114 (S114: No) and repeats the process of S114. When the operation key is pressed, the CPU 20 affirms S114 (S114: Yes), and controls the camera 32 to start capturing an external image (S116). When the learning target specified in S112 is related to the page turning process, the user moves the right hand from the right side to the left side, for example, in front of the body. At this time, the user turns his / her face toward the moving right hand. The camera 32 captures an external image including the right hand of the moving user. The imaging of the outside world image ends, for example, at a timing performed for a predetermined period.

An external image signal representing external image data corresponding to the external image captured by the camera 32 is transmitted and received via the transmission cable 4 between the communication circuit 41 of the display device 3 and the communication circuit 24 of the system box 2. The The external image signal is converted into external image data by the image processing unit 26, and the generated external image data is stored in the video RAM 25. In S 118, the CPU 20 executes a learning process on the outside world image data stored in the video RAM 25. In the learning process in S118, the CPU 20 acquires the motion of the body part included in the external image from the external image data stored in the video RAM 25. For example, the CPU 20 acquires the movement of the right hand that moves from the right side to the left side. When acquiring the movement of the right hand, the CPU 20 performs image processing for specifying the movement of the body part such as the right hand included in the external image in the image processing unit 26 with respect to the external image data stored in the video RAM 25. Command to run. The CPU 20 acquires the movement of the body part such as the right hand from the result of the image processing by the image processing unit 26. Thereafter, the CPU 20 obtains a feature amount representing the acquired feature of the movement. The image processing for obtaining the feature amount executed by the image processing unit 26 is performed according to an already developed image processing technique. The feature amount of movement is, for example, the moving distance of the body part in the external image, the moving direction, or both. In the present embodiment, the feature quantity for updating the first feature quantity registered in the database shown in S 118 and S 210 in FIG. 6, like the acquired motion feature quantity, is referred to as a second feature quantity. . The CPU 20 updates the first feature amount of the learning target associated with the identified identification information in the database according to the second feature amount. When S104 is denied (S104: No), the first feature amount to be learned is the first feature amount associated with the identified identification information in the database at the start of the main process. When S106 is executed, the first feature amount to be learned is the first feature amount after reset. The update in the learning process is performed using a neural network (SOM). Neural network (SOM) is a known technique. A specific description of the update by the neural network (SOM) is omitted.

In S120, after executing S118, the CPU 20 registers, in the database, the first feature amount updated in S118 as the first feature amount to be learned associated with the identified identification information. The CPU 20 controls to display the selection screen again. The user operates the operation unit 273 and inputs a desired option. The selection screen includes options for instructing completion of gesture learning. In S122, the CPU 20 determines whether or not the input option is an instruction to complete the gesture learning according to the display of the selection screen. When an option related to a process other than the page turning process is input (S122: No), the process returns to S110, and S110 to S120 are executed. On the selection screen displayed at S112, a plurality of inputs may be made possible. When a plurality of inputs are made, if S112 to S120 are not executed for all input, S122 may be denied (S122: No), and the unprocessed portions may be executed sequentially.

When an option for instructing completion of gesture learning is input (S122: Yes), the CPU 20 shifts the process to S124. In the case where a plurality of inputs can be made on the selection screen displayed at S112, when S112 to S120 are executed for all input, S122 is affirmed (S122: Yes). In S124, the CPU 20 executes a routine process. The CPU 20 ends the main process with the end of the routine process.

<Routine processing>
The routine process executed in S124 of the main process shown in FIG. 5 will be described with reference to FIG. At the start of the routine processing, the HMD 1 starts displaying an image corresponding to the image data stored in the flash ROM 22. Specifically, the image processing unit 26 performs image processing on a predetermined page portion of the image data stored in the flash ROM 22, and an image signal generated by the image processing is transmitted from the communication circuit 24 to the display device 3. The In the display device 3, the image light forming unit 34 forms the image light Lim based on the image signal received by the communication circuit 41, and an image corresponding to the image signal is displayed. The user visually recognizes the displayed image.

The CPU 20 that has started the routine process controls the camera 32 to start capturing an external image (S200). For example, the user moves the right hand from the right side to the left side in front of the body. When moving the right hand from the right side to the left side, the user turns the face toward the moving right hand. The camera 32 captures an external image including the right hand of the moving user. The imaging of the outside world image is continued during the routine processing. An external image signal representing external image data corresponding to the external image captured by the camera 32 is transmitted and received via the transmission cable 4 between the communication circuit 41 of the display device 3 and the communication circuit 24 of the system box 2. The The external image signal is converted into external image data by the image processing unit 26, and the generated external image data is stored in the video RAM 25. The CPU 20 acquires the movement of the body part included in the external image from the external image data stored in the video RAM 25. For example, the CPU 20 acquires the movement of the right hand that moves from the right side to the left side. The movement of the body part is acquired in the same manner as described in S118. Hereinafter, a motion that may be a motion for inputting an instruction for a predetermined process executed by the HMD 1, such as the acquired motion of a body part, is referred to as an “operation motion”.

In S202, the CPU 20 determines the correspondence between the operation action and the first feature amount associated with the identified identification information. Specifically, the CPU 20 determines whether or not the feature quantity representing the feature of the operation action corresponds to the first feature quantity registered in the database in association with the identified identification information. With respect to S202, the “moving right hand from the right side to the left side” which is an operation for instructing the page turning process will be described as an example. In S 202, the CPU 20 includes a “right hand”, a “hand movement direction”, and a “hand movement direction” that are included in the first feature amount indicated in the record “No. 1” in FIG. 4. It is individually determined whether or not it is the same as each feature amount of “movement amount”. First, the CPU 20 determines whether or not the feature quantity representing the feature of the body part in the operation operation matches the feature quantity representing the feature of the right hand. Next, the CPU 20 determines whether or not the feature amount representing the movement direction of the body part in the operation operation matches the feature amount representing the feature of the hand movement direction from the right side to the left side. Furthermore, the CPU 20 determines whether or not the feature amount that represents the movement amount of the body part in the operation operation matches the feature amount that represents the movement amount of the hand.

The determination of the feature values of “right hand”, “hand movement direction”, and “hand movement amount” all match, and the feature amount representing the feature of the operation is registered in the database in association with the identified identification information. If it is determined that it is the same as the first feature amount that has been made, S202 is affirmed (S202: Yes). Also in the case where the difference between the two is in a range set in advance as an error, S202 may be affirmed (S202: Yes), as in the same case. The error may be appropriately changed by an operation via the operation unit 273. S202 is performed for all first feature values registered in the database in association with the identified identification information. For all of the first feature values, at least one of the determination of each feature value of “right hand”, “hand movement direction”, and “hand movement amount” is not the same, and the difference between the two is determined in advance. If it is out of the range, S202 is denied (S202: No). When S202 is affirmed (S202: Yes), the CPU 20 controls processing corresponding to the first feature amount (S204). As shown in the record “No. 1” in FIG. 4, the CPU 20 controls the page turning process based on the example of “moving the right hand from the right side to the left side”. Corresponding to S 204, the image processing unit 26 performs image processing on the next page portion of the image data, and transmits an image signal generated by the image processing to the display device 3 from the communication circuit 24. In the display device 3, the image light Lim is formed by the image light forming unit 34 based on the image signal received by the communication circuit 41, and the next page image corresponding to the image signal is displayed. Thereafter, the CPU 20 shifts the processing to S220.

When S202 is denied (S202: No), the CPU 20 determines whether or not the difference between the feature amount representing the feature of the operation action and the first feature amount is within a predetermined threshold (S206). . As the threshold, a predetermined range is set in advance for the first feature amount. As the predetermined range set as the threshold, a range wider than the above-described error is set. The threshold is appropriately set based on a range in which the operation can be determined to be similar to the movement corresponding to the first feature amount. When the operation operation is outside the threshold, the CPU 20 denies S206 (S206: No), and moves the process to S220. When the operation action is outside the threshold value, it is determined that the operation action is not a movement related to an instruction input to the HMD 1. When the operation operation is within the threshold, the CPU 20 affirms S206 (S206: No). When the operation action is within the threshold value, in S208, the CPU 20 controls the process corresponding to the first feature amount used as the determination criterion in S202, as in S204. For example, the CPU 20 controls page turning processing, and the HMD 1 executes page turning of the image displayed on the display device 3. After executing S208, in S210, the CPU 20 executes a learning process. The learning process is executed in the same manner as S118 of the main process for the operation operation. That is, the CPU 20 obtains a feature amount representing the feature of the operation operation as the second feature amount from the acquired operation operation. The CPU 20 updates the first feature amount associated with the identified identification information in the database according to the obtained second feature amount. The updated first feature value is a feature value for the process controlled in S208. In S212, the CPU 20 registers the first feature quantity updated in S210 in the database as the first feature quantity associated with the identified identification information and related to the process executed in S208.

In S214, the CPU 20 acquires operation operations for processing having a certain relationship with the processing controlled in S208 in succession to the operation operations acquired for processing in S208 within a predetermined period. It is judged whether it was done. The CPU 20 causes the image processing unit 26 to continuously generate image processing for specifying the movement of the body part included in the external image by continuous imaging with the camera 32 and store the external environment stored in the video RAM 25. Command to run on image data. When an operation operation for a process having a certain relationship with the process controlled in S208 has been performed, the CPU 20 determines a certain relationship with the process controlled in S208 from the result of the image processing by the image processing unit 26. The operation action for the process to have is acquired. In S214, the image processing executed by the image processing unit 26 is performed according to an already developed image processing technique.

The process having a certain relationship with the process controlled in S208 is, for example, a process that is paired with the process controlled in S208. Specifically, when the process controlled in S208 is a page turning process, the process paired with the page turning process is a page return process. Prior to the control of S208, the CPU 20 acquires a movement of moving the right hand from the right side to the left side by image processing in the image processing unit 26, and then continuously moves the right hand from the left side to the right side within a predetermined period. It is determined whether or not a movement to perform is acquired. If the process controlled in S208 and executed as a result of S208 is not intended by the user, it is assumed that the user often performs a movement to cancel the process of S208. For example, when the displayed image becomes the image of the next page against the user's intention, it is considered that the user instructs the page return process to display the image that was originally displayed again. The predetermined period is set in consideration of how long the user's action for instructing the page return process is performed.

Regarding S214, for example, whether or not the movement of moving the right hand from the left side to the right side is acquired is indicated by the operation operation acquired by the image processing in the image processing unit 26 in the record “No. 2” in FIG. This is performed according to whether or not it matches the first feature amount of the movement of the body part. Specifically, the CPU 20 includes a “right hand”, a “hand moving direction”, and a “hand movement” that are included in the first feature amount shown in the record “No. 2” in FIG. It is individually determined whether or not it is the same as each feature amount of “movement amount”. First, the CPU 20 determines whether or not the feature quantity representing the feature of the body part in the operation operation matches the feature quantity representing the feature of the right hand. Next, the CPU 20 determines whether or not the feature amount representing the movement direction of the body part in the operation operation matches the feature amount representing the feature of the hand movement direction from the left side to the right side. Furthermore, the CPU 20 determines whether or not the feature amount that represents the movement amount of the body part in the operation operation matches the feature amount that represents the movement amount of the hand. The determination of the feature values of “right hand”, “hand movement direction”, and “hand movement amount” all match, and the feature amount representing the feature of the operation is registered in the database in association with the identified identification information. If it is determined that it is the same as the first feature amount that has been made, S214 is affirmed (S214: Yes). As for the case where the difference between each feature amount of the “right hand”, “hand movement direction” and “hand movement amount” and the first feature amount is a preset range as an error, S214 may be affirmed (S214: Yes). The error may be appropriately changed by an operation via the operation unit 273. At least one of the determination of each feature amount of “right hand”, “hand movement direction”, and “hand movement amount” is not the same, and “right hand”, “hand movement direction”, and “hand movement” When the difference between each feature amount of the “quantity” and the first feature amount is outside the predetermined range, S214 is negative (S214: No).

If the operation operation for the process having a certain relationship with the process in S208 is not acquired within the predetermined period (S214: No), the CPU 20 shifts the process to S220. When the operation operation described above is acquired within the predetermined period (S214: Yes), the CPU 20 executes the first feature amount registered in association with the identification information specified in the database in S212 before executing S212. (S216). Subsequently, the CPU 20 controls processing corresponding to the paired movement determined to have been acquired in S214 (S218). For example, when the paired movement is a movement in which the right hand is moved from the left side to the right side, the CPU 20 controls the page return process shown in the record “No. 2” in FIG. Corresponding to the control of the page return processing, the image processing unit 26 performs image processing on the previous page portion of the image data, and transmits an image signal generated by the image processing from the communication circuit 24 to the display device 3. . In the display device 3, image light Lim is formed by the image light forming unit 34 based on the image signal received by the communication circuit 41, and the previous page image corresponding to the image signal is displayed. Thereafter, the CPU 20 shifts the processing to S220.

In S220, the CPU 20 determines whether or not the user has operated the power switch 271 to turn off the power. When the power is not turned off (S220: No), the CPU 20 returns the process to S202. When the power is turned off (S220: Yes), the CPU 20 ends the routine process. CPU20 which complete | finished routine processing returns a process to a main process, and complete | finishes a main process.

<Effect of this embodiment>
According to this embodiment, the following effects can be obtained.

(1) After the main process is started, personal authentication is performed in S100. After that, when gesture learning is input (S108: Yes), the specified identification information for identifying the user of HMD1 specified by the personal authentication is used. The first feature amount that is associated and is the learning target in S112 is updated in S118 with the second feature amount that represents the motion feature of the user's body image captured in S116 by the learning process, and is updated. The determined first feature amount is registered in S120 in association with the identified identification information. Therefore, the first feature amount related to a specific process can be updated for each user. Since the first feature value is updated by the learning process of S118, the operability for instructing the process corresponding to the motion represented by the first feature value can be improved for each user. Note that the producer who produces the HMD1 executes each processing of S110 to S120 independently when producing the HMD1. By executing each processing of S110 to S120, the first feature value after the reset in S106 as an initial value is registered in the database.

In S104, it is determined whether or not a reset of the learning result is input in the main process. If the reset of the learning result is input (S104: Yes), the CPU 20 specifies the database stored in the flash ROM 22. The first feature amount registered in association with the completed identification information is reset (S106). In order to reset, the first feature amount registered in the database can be returned to the initial value.

(2) In S202, the correspondence between the operation operation acquired by the image processing on the external image data and the first feature amount associated with the identified identification information is determined in a routine process, and the operation operation feature is determined. Is the same as the first feature value registered in the database in association with the identified identification information (S202: Yes), the CPU 20 controls processing corresponding to the operation operation (S204). ) Therefore, the HMD 1 can execute processing corresponding to the operation operation. As described above, in S202, the determination in S202 is also affirmed (S202: Yes) even in the case where the difference between the feature amount representing the feature of the operation action and the first feature amount is within a preset range as an error. It may be.

Even if S202 is negative (S202: No), it is determined whether the correspondence between the operation action and the first feature amount is within the threshold (S206), and if it is within the threshold (S206: Yes), in S208, the CPU 20 controls the process corresponding to the operation. After that, the first feature value related to the processing controlled in S208 is updated in S210, and the updated first feature value is registered in S212. did. However, when an operation operation for processing that is paired with the processing controlled in S208 is acquired within a predetermined period, following the operation operation acquired for the processing in S208 (S214: Yes). ), Canceling the registration in S212, and returning to the setting before the execution of S212 in S216. Since the setting is returned to the setting before the execution of S212, it is possible to realize the control of the process according to the situation and the registration of the first feature amount based on the controlled process. In the case where the operation operations for the two processes that are paired are continuously acquired within the predetermined period, it is considered that the previously controlled process in S208 was not the process intended by the user. Accordingly, it is possible to avoid the update of the first feature amount based on the movement related to the process not intended by the user.

<Modification>
This embodiment can also be performed as follows.

(1) In the routine process, when S202 is denied (S202: No), the process after S206 is executed, and when the power is not turned off (S220: No), the process is returned to S202. . In addition, the processing after S206 may be omitted, and when S202 is denied (S202: No), the processing may be shifted to S220. By adopting a configuration that shifts the processing to S220, S206 to S218 can be omitted, and the routine processing procedure can be simplified.

(2) In the routine processing, when S202 is denied and S206 is affirmed (S202: No, S206: Yes), when S214 is affirmed (S214: Yes), the updated first feature value The registration is canceled (S216). Such processing may be performed as follows. That is, when S206 is affirmed (S206: Yes), the CPU 20 subsequently executes S214. If S214 is negative (S214: No), the CPU 20 executes S208 to S212, and then proceeds to S220. When S214 is affirmed (S214: Yes), the CPU 20 proceeds to S220 without executing S208 to S212. By adopting a configuration that shifts the processing to S220, S216 and S218 can be omitted, and the routine processing procedure can be simplified.

(3) The HMD 1 has a system box 2 and a display device 3 as separate bodies. It is good also considering the predetermined each part of each part with which the system box 2 is provided as integral HMD1 incorporated in the housing | casing 30. FIG. Specifically, the flash ROM 22, the video RAM 25, and the image processing unit 26 are built in the housing 30. In addition to the camera 32, a power switch 271, a power lamp 272, and an operation unit 273 are connected to the peripheral I / F 43 of the display device 3. The operation unit 273 is operated when a predetermined instruction is input to the integrated HMD 1. The battery may be built in the housing 30. When the battery is built in the housing 30, the communication circuit 41 may be omitted. However, when the battery is not built in, the communication circuit 41 is supplied with power from the external battery via the transmission cable 4. The CPU 38 of the display device 3 similarly performs the main process executed by the CPU 20 using the RAM 40. When the CPU 38 executes the main process, the CPU 38 similarly executes a routine process that is executed along with the execution of the main process. The program ROM 39 stores a computer program for main processing including routine processing.

Each process executed by the image processing unit 26 may be executed by the CPU 20 when the CPU 20 or the integrated HMD 1 is used. In the case of the RAM 23 or the integrated HMD 1, a part of the RAM 40 may be allocated as a video RAM. When allocating as video RAM, the image processing unit 26 and the video RAM 25 may be omitted.

1 HMD
2 System Box 3 Display 4 Transmission Cable 5 Eyeglass Frame 20 CPU
21 Program ROM
22 Flash ROM
23 RAM
24 communication circuit 25 video RAM
26 Image processing unit 27 Peripheral I / F
DESCRIPTION OF SYMBOLS 30 Housing | casing 31 Half mirror 32 Camera 33 Mounting part 34 Image light formation part 35 Eyepiece optical part 36 Lens 37 Lens holder 38 CPU
39 Program ROM
40 RAM
41 Communication circuit 43 Peripheral I / F
52 left frame portion 53 right frame portion 54 central frame portion 55 nose pad portion 56 support portion 57 groove 58 downward extension portion 271 power switch 272 power lamp 273 operation portion EB eye Lim image light

Claims

First acquisition means for acquiring external image data indicating an external image including a predetermined portion of a target imaged by the imaging unit;
Second acquisition means for acquiring a first movement of a predetermined part corresponding to the first processing according to the external image data acquired by the first acquisition means;
Processing means for controlling the first processing corresponding to the first movement acquired by the acquisition means;
Registration means for registering identification information for identifying a target having the predetermined part and a first feature amount representing a feature of the first movement in association with a storage unit;
Receiving means for receiving input of identification information,
The registration means acquires the first feature value registered in association with the identification information received by the reception means by the second acquisition means after the reception means receives the identification information. A head mounted display that updates the first feature amount associated with the identified identification information according to a second feature amount representing the feature of the first movement.
First determination means for determining whether the second feature value acquired by the second acquisition means corresponds to the first feature value registered in the storage unit in association with the specified identification information. When,
In response to the first determination unit determining that the second feature value does not correspond to the first feature value, the second feature value is predetermined with respect to the first feature value. And a second judging means for judging whether or not it is included in the range of
The processing means includes
When the second determination unit determines that the second feature amount is included in a predetermined range with respect to the first feature amount, the first feature corresponding to the first movement is determined. Control the process,
In response to determining that the second feature amount is not included in the predetermined range with respect to the first feature amount, the second determination unit corresponds to the first movement. Do not control the first process,
The head mounted display according to claim 1.
The registration means includes
In response to determining that the second feature value is included in a predetermined range with respect to the first feature value, the second determination unit determines the identified identification according to the second feature value. Updating the first feature amount associated with the information;
In response to determining that the second feature amount is not included in a predetermined range with respect to the first feature amount, the second determination unit determines the specified feature according to the second feature amount. The head mounted display according to claim 2, wherein the first feature amount associated with the identification information is not updated.
The second acquisition means corresponds to a second process that is executed by the head mounted display in accordance with the external image acquired by the first acquisition means and has a fixed relationship with the first process. To obtain a second movement of the predetermined part,
The processing means controls the second processing corresponding to the second movement acquired by the second acquisition means,
The registering unit is configured such that after the second determining unit determines that the second feature amount is included in a predetermined range with respect to the first feature amount, the acquiring unit performs the first movement. And the second movement, the storage unit of the first feature amount updated according to the second feature amount in response to the new sequential acquisition within a predetermined period. The head-mounted display according to claim 2, wherein the registration is the first feature amount before being updated with the second feature amount.
2. The registration unit according to claim 1, wherein the registration unit resets the first feature amount registered in the storage unit when an instruction to reset registration in the storage unit is input to the head mounted display. 3. Head mounted display.
The first feature acquired by the second acquisition unit after the reception unit has received the identification information, and the first feature amount registered in the storage unit in association with the identified identification information And a third determination means for determining the correspondence between
When the determination unit determines that the first movement corresponds to the first feature amount, the processing unit corresponds to the first movement newly acquired by the acquisition unit. The head mounted display of Claim 1 which controls a 1st process.
A computer program that can be read by a controller that controls the head-mounted display,
The control unit
Corresponding to the first processing according to the first acquisition means for acquiring the external image data indicating the external image including the predetermined part of the target imaged by the imaging unit, and the external image data acquired by the first acquisition means Second acquisition means for acquiring a first movement of a predetermined part to be performed;
Processing means for controlling the first processing corresponding to the first movement acquired by the acquisition means;
Registration means for registering identification information for identifying a target having the predetermined part and a first feature amount representing a feature of the first movement in association with a storage unit;
Function as a receiving means for receiving the input of the identification information,
The registration means acquires the first feature value registered in association with the identification information received by the reception means by the second acquisition means after the reception means receives the identification information. A computer program comprising a function of updating the first feature quantity associated with the identified identification information according to a second feature quantity representing the feature of the first movement.