WO2021140575A1

WO2021140575A1 - Head-mounted display for displaying ar object

Info

Publication number: WO2021140575A1
Application number: PCT/JP2020/000198
Authority: WO
Inventors: 川前　治; 伊藤　保
Original assignee: マクセル株式会社
Priority date: 2020-01-07
Filing date: 2020-01-07
Publication date: 2021-07-15

Abstract

The purpose of the present invention is to construct a handwriting drawing support system that can easily create a handwriting image conforming to a model image, by using an HMD.　In order to achieve the purpose, provided is a head-mounted display of which displaying is controlled by a control unit and which is provided with a three-dimensional sensor. The control unit is configured to recognize the shape of a subject on the basis of information from the three-dimensional sensor and display, on a display unit, an AR object according to the shape of the subject.

Description

[Name of invention determined by ISA based on Rule 37.2.] Head-mounted display that displays AR objects

The present invention relates to a handwriting drawing support system.

In recent years, various products have been on the market for portable information terminals such as smartphones. Among them, in a head-mounted display (hereinafter referred to as "HMD"), a real-space image and an augmented reality (AR: Augmented Reality) generated image (AR object such as an avatar) are displayed on a glasses-type display screen. ) Can be superimposed and displayed. Further, the user can obtain various information by the sensor mounted on the HMD. As a result, the information from the three-dimensional sensor built into the HMD makes it possible to present the augmented reality AR object by the computer in the real space as if it exists in the real space.

On the other hand, conventionally, by using a model image as a sketch and faithfully tracing the model image by handwriting, it is possible to create a handwritten image conforming to the model image, but the model image is enlarged / reduced, rotated / deformed. It was difficult to do the handwriting.

Patent Document 1 is a prior art document in this technical field. Patent Document 1 describes a method of projecting digital information (image) on a piece of paper on a work surface by a projector to support handwriting drawing.

International Publication No. 2014/0733346

Patent Document 1 discloses that a model image is displayed to support handwriting drawing, but consideration is given to the model image for enlargement / reduction, rotation / deformation, and for a non-planar work surface. It wasn't done. Further, according to Patent Document 1, there is a description that the HMD may be used instead of the projector, but there is no description regarding its realization, and the feasibility of the HMD is unknown.

In view of the above problems, an object of the present invention is to construct a handwriting drawing support system that can easily create a handwritten image conforming to a model image by using an HMD.

The present invention is, for example, a head-mounted display whose display is controlled by a control unit, provided with a three-dimensional sensor, and the control unit grasps the shape of an object based on information from the three-dimensional sensor. Then, the AR object is displayed on the display unit according to the shape of the object.

According to the present invention, it is possible to provide a handwriting drawing support system that enables easy creation of a handwritten image based on a model image.

It is an external view of the HMD in Example 1. FIG. It is a hardware block diagram of the HMD in Example 1. FIG. It is a functional block block diagram of HMD in Example 1. FIG. It is a schematic diagram for demonstrating the operation state in Example 1. FIG. It is a schematic diagram which showed the correspondence between the real space and the AR object on the display screen in Example 1. FIG. It is a flowchart which shows the outline of the processing procedure in Example 1. FIG. It is a flowchart which showed the detail of the model image selection process in Example 1. FIG. It is a schematic diagram which shows the model image presentation state in Example 1. FIG. It is a schematic diagram which shows the model image selection state in Example 1. FIG. It is a schematic diagram which shows the erasing state of the non-selected model image in Example 1. FIG. It is a flowchart which shows the detail of the model image transformation / movement processing in Example 1. FIG. It is a flowchart which shows the detail of the model image handwriting processing in Example 1. FIG. It is a schematic diagram which shows the state before the start of handwriting in Example 1. FIG. It is a schematic diagram which shows the intermediate state of the handwriting in Example 1. It is a schematic diagram which shows the handwritten image which finished the handwriting in Example 1. FIG. It is a schematic diagram which shows the mismatch example of the model image and the handwritten image in Example 1. FIG. It is a gesture operation correspondence table which showed the gesture operation in Example 1 and the processing content of the model image for the operation. It is a voice command correspondence table which showed the voice command in Example 2 and the processing content of the model image for it. It is a schematic diagram which shows the example which applied the handwriting drawing support system in Example 3 to a penmanship. It is a schematic diagram which displays the model image and the handwritten image side by side on the display screen of the HMD in Example 5. 6 is a schematic diagram showing an example in which the handwriting drawing support system in the sixth embodiment is applied to a non-planar surface.

Hereinafter, examples of the present invention will be described with reference to the drawings.

FIG. 1 is an external view showing the HMD in this embodiment. In FIG. 1, the HMD1 has a transmissive display screen 75 at the lens position of the spectacles, and the situation in the real space is observed through the display screen 75. In addition, an augmented reality AR object is displayed on the display screen 75. Therefore, the wearer of the HMD1 can simultaneously visually recognize both the augmented reality AR object displayed on the display screen 75 and the situation in the real space.

FIG. 2 is a hardware configuration diagram of the HMD in this embodiment. In FIG. 2, the HMD 1 is composed of a main control device 2, a system bus 3, a storage device 4, a sensor device 5, a communication processing device 6, a video processing device 7, a voice processing device 8, and an operation input device 9.

The main control device 2 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. That is, each function or the like is realized by software by interpreting and executing an operation program in which the microprocessor unit realizes each function or the like.

The system bus 3 is a data communication path for transmitting and receiving various commands and data between the main control device 2 and each constituent block in the HMD1.

The storage device 4 stores various data such as a program unit 41 that stores an operation program for controlling the operation of the HMD 1, an operation setting value, a detection value from the sensor unit, an object including contents, and library information downloaded from the library. It is composed of various data units 42 to be stored and a rewritable program function unit 43 such as a work area used for various program operations.

Further, the storage device 4 can store an operation program downloaded from the network and various data created by the operation program. In addition, it is possible to store contents such as moving images, still images, and sounds downloaded from the network. In addition, it is possible to store data such as moving images and still images taken by using the camera function. Further, the storage device 4 needs to hold the stored information even when the HMD 1 is not supplied with power from the outside. Therefore, for example, devices such as a semiconductor element memory such as a flash ROM or SSD (Solid State Drive), a magnetic disk drive such as an HDD (Hard Disc Drive), and the like are used. The operation program may be stored in the program unit 41 or the like of the HMD1 in advance at the time of product shipment. Further, it may be acquired from various server devices or the like on the Internet after the product is shipped. Further, the operation program provided by the memory card, the optical disk, or the like may be acquired. Further, each operation program stored in the program unit 41 can be updated and its function can be expanded by a download process from each server device on the network.

The sensor device 5 is a sensor group of various sensors for detecting the state of the HMD1. The sensor device 5 includes a GPS (Global Positioning System) receiving unit 51, a geomagnetic sensor unit 52, a three-dimensional sensor unit 53, an acceleration sensor unit 54, a gyro sensor unit 55, and the like. With these sensor groups, it is possible to detect the position, tilt, direction, movement, etc. of the HMD1. Further, the HMD1 may further include other sensors such as an illuminance sensor, an altitude sensor, and a proximity sensor.

The three-dimensional sensor unit 53 of this embodiment describes the phase difference method (phase shift method) as an example, but is not limited to this method. The phase difference method is a method of irradiating an object with a plurality of modulated laser beams and measuring the distance to the object by the phase difference of the returning diffuse reflection component. The three-dimensional sensor unit 53 can grasp the distance to each point of the object.

The communication processing device 6 is composed of a LAN (Local Area Network) communication unit 61 and a telephone network communication unit 62. The LAN communication unit 61 is connected to a network network such as the Internet via an access point or the like, and transmits / receives data to / from each server device on the network. The connection with the access point or the like may be made by a wireless connection such as Wi-Fi (registered trademark). The telephone network communication unit 62 performs telephone communication (call) and data transmission / reception by wireless communication with a base station or the like of a mobile telephone communication network. Communication with base stations, etc. is W-CDMA (Wideband Code Division Multiple Access) (registered trademark) method, GSM (Global System for Mobile communications) (registered trademark) method, LTE (Long Term Evolution) method, or other communication methods. May be done by. The LAN communication unit 61 and the telephone network communication unit 62 each include a coding circuit, a decoding circuit, an antenna, and the like. Further, the communication processing device 6 may further include other communication units such as a Bluetooth (registered trademark) communication unit and an infrared communication unit.

The video processing device 7 is composed of an imaging unit 71 and a display unit 72. The image pickup unit 71 inputs image data of the surroundings and an object by converting the light input from the lens into an electric signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. It is a camera unit that does. The display unit 72 is a display device of a transmissive display using, for example, a laser projector and a half mirror, constitutes a display screen 75, and provides image data to a user of HMD1.

The voice processing device 8 is composed of a voice input / output unit 81, a voice recognition unit 82, and a voice decoding unit 83. The voice input of the voice input / output unit 81 is a microphone, and the user's voice or the like is converted into voice data and input. Further, the voice output of the voice input / output unit 81 is a speaker, and outputs voice information and the like necessary for the user. The voice recognition unit 82 analyzes the input voice information and extracts instruction commands and the like. The voice decoding unit 83 has a function of performing decoding processing (speech synthesis processing) of the coded voice signal and the like, if necessary.

The operation input device 9 is an instruction input unit for inputting an operation instruction to the HMD1. The operation input device 9 is composed of operation keys and the like in which button switches and the like are arranged. Other operating devices may be further provided. The communication processing device 6 may be used to operate the HMD 1 by using a separate mobile terminal device connected by wired communication or wireless communication. Further, the voice recognition unit 82 of the voice processing device 8 may be used to operate the HMD 1 by a voice command of an operation instruction. Further, the HMD1 may be operated by analyzing the captured image of the imaging unit 71 of the image processing device 7 and performing an operation such as a gesture.

Although the configuration example of HMD1 shown in FIG. 2 includes many configurations that are not essential to this embodiment, the effect of this embodiment is not impaired even if these configurations are not provided. Further, configurations (not shown) such as a digital broadcast reception function and an electronic money payment function may be further added.

FIG. 3 is a functional block configuration diagram of the HMD in this embodiment. In FIG. 3, the control unit 11 is mainly composed of the main control device 2, the program unit 41 of the storage device 4, and the program function unit 43, and constitutes a handwriting drawing support system.

The three-dimensional sensor information acquisition unit 12 is a function of acquiring information from the three-dimensional sensor unit 53 of the sensor device 5. The information from the three-dimensional sensor unit 53 includes distance information from the HMD 1 to each point of the object.

The three-dimensional data processing unit 13 has a function of grasping the shape of the object based on the information from the three-dimensional sensor unit 53 (distance information from the HMD 1 to each point of the object).

The three-dimensional data storage unit 14 has a function of storing the three-dimensional data obtained by the three-dimensional data processing unit 13 in various data units 42 of the storage device 4.

The shooting data acquisition unit 15 is an imaging unit 71 of the video processing device 7, and has a function of shooting a real space and acquiring shooting data. The shooting data acquisition unit 15 can acquire handwritten image data information and information such as gesture operation.

The communication processing unit 16 is composed of a LAN communication unit 61 and a telephone network communication unit 62 of the communication processing device 6, and can be uploaded to an external network server via the Internet network or from an external network server via the Internet network. It has a function to download various information. The image of the AR object such as the model image is also downloaded from the external server or uploaded to the external server by using the communication processing unit 16. By downloading the model image from the external server to the various data units 42 of the storage device 4, various abundant model images can be presented to the user.

The AR image information storage unit 17 is a function of storing the AR image information obtained by the communication processing unit 16 in various data units 42 of the storage device 4.

The AR image generation processing unit 18 is a function of generating an AR object based on the AR object information stored in the AR image information storage unit 17. The AR image generation processing unit 18 can perform processing such as enlargement / reduction, rotation / deformation of the model image.

The AR image display processing unit 19 is a function of displaying the AR object generated by the AR image generation processing unit 18 on the display screen 75 of the HMD1. Further, the model image (virtual image of the AR object) can be displayed along the shape of the handwritten surface based on the shape information of the handwritten surface stored in the three-dimensional data storage unit 14.

The operating status of this embodiment will be described below. FIG. 4 is a schematic diagram for explaining an operating state in this embodiment. In FIG. 4, the user 10 wearing the HMD1 visually recognizes the handwriting surface 28 via the display screen 75 of the HMD1. The AR object displayed on the display screen 75 of the HMD1 can be visually recognized as a virtual image 27 of the AR object existing on the handwriting surface 28 by the user 10 wearing the HMD1.

The three-dimensional information of the handwriting surface 28 stored by the three-dimensional data storage unit 14 is the difference between the distance from the HMD 1 to the front side of the handwriting surface 28 and the distance from the HMD 1 to the back side of the handwriting surface 28 (to the front side). The distance to the back side is longer than the distance of.) Is shown as information. The situation will be described with reference to the schematic diagram of FIG.

FIG. 5 is a schematic diagram showing the correspondence between the real space and the AR object on the display screen, although it is slightly deformed. As shown in FIG. 5, in the user 10 wearing the HMD1, the rectangular AR object 22 displayed on the display screen 75 of the HMD1 has a shorter side length on the back side than the length on the front side. It is displayed (perspective display), and the thickness of the back side is displayed thinner than the thickness of the front side (perspective display), so it is as if it exists on the handwritten surface 28. The virtual image 27 of the AR object can be visually recognized.

Next, the outline of the processing procedure in the HMD 1 executed by the control unit 11 in this embodiment will be described with reference to the flowchart of FIG. In FIG. 6, when the process is started, first, in step S420, a model image selection process is performed to select a model image to be handwritten. The details of the model image selection process S420 will be described later.

Next, in step S430, it is determined whether or not there is an instruction such as deformation / movement for the selected model image.

If there is an instruction to transform / move the selected model image in the process of S430, the model image is transformed / moved in step S440 such as enlargement / reduction / rotation / transformation / position movement. Confirm the model image to be handwritten. The details of the model image transformation / movement process S440 will be described later.

In the process of S430, if there is no instruction to transform / move the selected model image, that is, if the selected model image can be used as it is without being transformed / moved, the process proceeds to step S460.

The process of step S460 is a model image handwriting process that supports handwriting drawing when handwriting on the handwriting paper on the handwriting surface, and supports handwriting drawing by tracing the model image. The details of the model image handwriting process S460 will be described later. Further, even during the execution of the model image handwriting process S460, instructions such as deformation and movement of the model image are effective, and the optimum model image can be presented each time. This completes the outline of the processing procedure in HMD1.

Next, the details of each processing procedure will be described. FIG. 7 is a flowchart showing the details of the model image selection process S420. In FIG. 7, when the model image selection process S420 is started, first, the shape grasping process (S422) of the handwritten surface is performed. In the handwriting surface shape grasping process (S422), the three-dimensional sensor information acquisition unit 12 measures the distance from the HMD 1 to each point of the handwriting surface 28, and the three-dimensional data processing unit 13 grasps the shape of the handwriting surface 28. It is a process. The shape of the handwriting surface 28 obtained here is stored in various data units 42 of the storage device 4 by the three-dimensional data storage unit 14. If the shape of the handwritten surface has already been grasped and stored in various data units 42 of the storage device 4, the stored shape information of the handwritten surface is used.

The shape of the handwriting surface 28 in this embodiment assumes a rectangular plane. Since the handwriting surface 28 is rectangular, the shape of the handwriting surface 28 can be grasped by grasping the positions of the four corners. Further, once the positions of the four points are grasped, even if one point is out of the measurable range for borrowing, the position of the fourth point can be estimated from the positions of the remaining three points. Instead of grasping the positions of the four corners, the shape of the hand-drawn surface 28 may be grasped by grasping the positions of the four sides.

Next, the model image presentation process (S423) for presenting the model image is performed. FIG. 8 is a schematic view showing the first model image 31 and the second model image 32 on the handwritten surface 28. For the sake of simplicity, the number of models to be selected is two, and the first model image 31 is a rectangle and the second model image is a triangle, and illustration line drawings are used.

In this embodiment, the video information from the shooting data acquisition unit 15 is acquired, the gesture operation is analyzed, and the model image to be presented is selected and displayed. In the state where the model image of FIG. 8 is presented, the gesture motion of the finger or the entire hand is targeted. That is, the screen of the model image to be presented can be exchanged by the gesture operation of sliding the entire hand to the left or right. For example, when the gesture operation of sliding the entire hand to the left is performed, the screen of the immediately preceding model image is presented, and when the gesture operation of sliding the entire hand to the right is performed, the screen of the next model image is presented.

Next, the model image selection process (S424) for selecting the model image is performed. FIG. 9 is a schematic view showing a state in which the first model image 31 presented on the handwriting surface 28 is selected. The model image 31 is selected by the finger 39 pointing to the first model image 31 presented on the handwriting surface 28.

In this embodiment, AR that acquires the image information of the finger 39 and the three-dimensional distance information from the shooting data acquisition unit 15 and the three-dimensional sensor information acquisition unit and displays them on the display screen 75 of the HMD1 by the AR image display processing unit 19. The image of the object is processed and displayed. Therefore, in the schematic view of FIG. 9, it is shown that the first model image 31 (virtual image of the AR object) existing under the finger 39 can be visually recognized as if it is hidden.

The selection instruction is performed by the gesture operation of the finger 39. Specifically, when it is detected on the first model image 31 that the distance within the range of the hand-drawn surface 28 and the finger 39 are at the same distance, and the finger 39 indicates the model image 31 , The selection can be instructed by the gesture operation of double-clicking the finger 39.

Next, the non-selection model image erasing process (S425) for erasing the non-selection model image presented on the handwritten surface 28 is performed. Here, an example of selecting the model image 31 on the hand-drawn surface 28 is shown, but it is not always necessary to select the model image 31 on the hand-drawn surface 28, and the model image 31 may be selected on the display screen 75 of the HMD1. ..

FIG. 10 is a schematic view showing a state in which only the selected first model image 31 is presented after being erased from the handwritten surface 28 because the second model image 32 was not selected. This completes the model image selection process S420.

Next, FIG. 11 is a flowchart showing the details of the model image transformation / movement process S440. In this embodiment, the procedure for analyzing the instruction by the gesture operation is performed in the order of enlargement / reduction, rotation / deformation, and position movement, but the procedure is not particularly limited to this order.

In FIG. 11, when the model image deformation / movement process S440 is started, the gesture operation instruction regarding the deformation / movement of the model image is analyzed. First, in step S422, it is determined whether or not the given gesture operation instruction is an enlargement or reduction instruction. If the given instruction is an instruction for enlargement or reduction, the enlargement / reduction process (S443) is performed to enlarge or reduce the model image and present it. When it recognizes the gesture movement that opens between the thumb and index finger, it enlarges and presents the model image. In addition, when it recognizes the gesture motion of closing between the thumb and index finger, the model image is reduced and presented. The enlargement ratio and reduction ratio in this embodiment depend on the magnitude of the gesture operation, but may be any preset enlargement ratio or reduction ratio.

Next, it is determined whether the given gesture operation instruction is a rotation or deformation instruction (process S424). If the given gesture operation instruction is a rotation or transformation instruction, the rotation / transformation process (S445) is performed to rotate or transform the model image and present it. When the gesture motion of sliding the finger in an arc is recognized, the model image is rotated and presented in the sliding direction of the finger. The rotation angle in this embodiment depends on the arc drawn by the finger, but may be any preset rotation angle. In addition, when the gesture motion of pressing and sliding one point on the outer shape of the model image is recognized, the model image is deformed and presented in the sliding direction of the finger.

Next, it is determined whether or not the given instruction is a position movement (process S446). If the given instruction is a position move, the position move process (S447) is performed to move the position of the model image. When the gesture motion of pressing and sliding the inside of the model image is recognized, the model image is moved and presented in the sliding direction of the finger. The moving distance in this embodiment depends on the magnitude of the gesture movement, but may be any preset moving distance. This completes the model image transformation / movement process S440.

Next, FIG. 12 is a flowchart showing the details of the model image handwriting process S460. In FIG. 12, when the model image handwriting process S460 is started, first, the shape grasping process (S462) of the handwriting paper is performed. The handwriting paper shape grasping process (S462) is a process in which the three-dimensional sensor information acquisition unit 12 measures the distance from the HMD 1 to each point of the handwriting paper, and the three-dimensional data processing unit 13 grasps the shape of the handwriting paper. is there. The shape of the handwritten paper obtained here is stored in various data units 42 of the storage device 4 by the three-dimensional data storage unit 14. If the shape of the handwriting paper has already been grasped and stored in various data units 42 of the storage device 4, the shape information of the stored handwriting paper is used.

The shape of the handwriting paper in this embodiment is assumed to be a rectangular plane. Since it is a rectangular plane, the shape of the handwriting paper can be grasped by grasping the positions of the four corners of the handwriting paper. Of course, once the overall shape of the hand-drawn paper and the distance to each point on the paper can be measured, the shape of the hand-drawn paper is not limited to a rectangular plane.

Next, the selected model image presentation process (S463) for presenting the selected model image is performed. FIG. 13 is a schematic view in which the handwriting paper 25 is placed on the handwriting surface 28, and the model image 35 of the rabbit, which is a virtual image of the AR object, is visually recognized. Here, if it is determined that the rabbit model image 35 deviates from the handwriting paper 25 or the balance is inappropriate, the model image transformation / movement process (S440) is executed again to perform the model image. Image 35 is optimized.

Next, the model image handwriting process (S464) is performed by tracing the model image and handwriting. FIG. 14 is a schematic view showing a state in which the outer shape (face) of the model image 35 of the rabbit is handwritten using the pen 38. In this embodiment, video information is acquired from the shooting data acquisition unit 15, and what part and how much is handwritten is analyzed. In the schematic diagram of FIG. 14, the handwritten portion 36 (the face portion of the rabbit) handwritten by tracing the model image 35 of the rabbit is expressed as a slightly thicker line. In the schematic diagram of FIG. 14, the portion of the rabbit model image 35 that overlaps with the handwritten portion 36 is erased and displayed at the stage of the AR object on the display screen 75 of the HMD1. As a result, the corresponding portion of the rabbit model image 35 that overlaps with the handwritten portion 36 is erased from the handwritten paper 25 and presented.

FIG. 15 is a schematic view showing a handwritten image 37 in a state where handwriting is completed by tracing all the model images 35 of the rabbit. In the schematic view of FIG. 15, since the rabbit model image 35 is all handwritten (handwritten image 37), the AR object is not displayed on the display screen 75 of the HMD1. Therefore, the rabbit model image 35 is not presented on the handwriting paper 25. In the schematic view of FIG. 15, the model image 35 of the rabbit and the handwritten image 37 are shown to be in perfect agreement with each other. This completes the model image handwriting process S460.

Although the drawings of this embodiment are schematic drawings that are handwritten only in black, it goes without saying that any color can be used and the rabbit can be colored with any color. It is also possible to match the line thickness of the model image with the line thickness of the handwritten image. It is also possible to add a center line (a line representing the center of the width of the line) to the line of the model image with a dotted line, a broken line, a one-dot difference line, etc., and present it in a color (including white) different from that of the model image.

Further, in this embodiment, the model image 35 and the handwritten image 37 are completely matched, but may not be completely matched. The schematic diagram of FIG. 16 is a schematic diagram showing a state in which the model image 35 of the rabbit and the handwritten image 37 do not completely match. In FIG. 16, the rabbit face portion of the handwritten image 37 does not match the rabbit model image 35, and in order to clarify the state, the rabbit model image that does not match is shown in this schematic diagram. Part 33 of 35 is shown by a broken line. This allows the user after handwriting to clearly recognize which part is different from the model image.

In the schematic view of FIG. 16, the model image 35 and the handwritten image 37 are superimposed and visually recognized, but the model image 35 and the handwritten image 37 can be visually recognized side by side. In addition to the above-mentioned broken line display, there are various methods for distinguishing the handwritten image 37 from the model image 35. For example, the color of the model image 35 may be different from that of the handwritten image 37, the line thickness and style of the model image 35 may be changed, or the model image 35 may be blinked to obtain the handwritten image 37. The difference from the model image 35 can be clarified.

Next, the gesture operation used in this embodiment and the processing content of the model image 35 for the gesture operation will be described. FIG. 17 is a gesture motion correspondence table showing the gesture motion in this embodiment and the processing content of the model image for the gesture motion. In FIG. 17, the gesture operation correspondence table 500 is composed of a gesture operation list 501 used in this embodiment and a processing list 502 showing the processing contents of a model image for the gesture operation.

For example, as a gesture operation, when a gesture operation 503 in which an open hand is placed on a handwritten surface 28 on which a model image is presented and the hand is moved to the right is executed, the processing content related to the model image for the gesture operation 503 is executed. 504 means to present a screen showing the next model image. The description of other gesture operations described in the gesture operation correspondence table 500 and the processing contents of the model image for the gesture operation will be omitted here. It should be noted that each of the presented gesture actions is only an example, and of course, other gesture actions can be enabled.

In this way, in this embodiment, the shape of the handwritten surface such as paper for creating a handwritten image is grasped by the information from the three-dimensional sensor built in the HMD, and the shape of the handwritten surface is as if it were in the real space. It presents an augmented reality AR object (model image) by a computer on the handwritten surface as if it exists, and supports the creation of a handwritten image by superimposing the model image and the handwritten image. In addition, the HMD control process performs processing such as enlargement / reduction and rotation / deformation of the model image.

As described above, according to the present embodiment, the model image can be arranged in the optimum size and the optimum position, and the handwritten image conforming to the model image can be easily created. A support system can be provided.

This embodiment describes a method of instructing the processing content of the model image by a method other than the gesture operation. The basic hardware configuration of this embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

In this embodiment, instead of the gesture operation, a voice command is used to instruct the processing content of the model image. The voice command is extracted as a voice command by analyzing the user's voice input via the voice input / output unit 81 of the voice processing device 8 by the voice recognition unit 82.

Here, the voice command to be used will be explained. FIG. 18 is a correspondence table of voice commands used in this embodiment. In FIG. 18, the voice command correspondence table 520 is composed of a voice command list 521, a meaning list 522 showing the meaning of each voice command, and a processing list 523 showing the processing contents for the model image corresponding to each voice command. ..

For example, when "maegamen" 524 is uttered as a voice command, the processing content 525 regarding the model image for the voice command (meaning the front screen) means that the screen of the previous model image is presented. The other voice commands listed in the voice command correspondence table 520 and the processing contents of the model image for the voice commands will not be described here. Further, each voice command is only an example, and of course, other voice commands can be enabled.

It is also possible to perform a voice response corresponding to a voice command by using the voice synthesis function of the voice decoding unit 83 of the voice processing device 8. As a result, for example, if the user responds with the same voice (echolalia) as the input voice command, or confirms the input voice command, for example, if the user's response to "XX Desune?" Is "high". If the user's response to "XX Wojikkoushimasu" and "XX Desune?" Is "Yeah", a voice response such as "XX Wotrikeshimasu" is possible. In the voice command correspondence table 520 of FIG. 18, voice commands (“high”, “yes”) related to voice response are not posted, but it goes without saying that they are supported.

As described above, in this embodiment, since the processing content of the model image can be instructed by using a voice command, it is possible to select the selection of the model image without performing the gesture operation. Needless to say, there are other methods for instructing the processing content of the model image. For example, it can be realized by mounting a touch sensor on the side of the HMD1 (glasses-shaped vine portion) and performing a slide operation or a click operation up, down, left or right.

In Example 1, for simplification, the model image uses figures such as rectangles and triangles and illustration line drawings such as rabbits, but other images can also be used as model images. In this embodiment, an example in which the handwriting support system described in Examples 1 and 2 is applied to penmanship will be described. The basic hardware configuration of this embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

FIG. 19 is a schematic diagram showing the procedure when the character "Ei", which is said to be the basis of penmanship and is well known as the Eight Principles of Eight, is used as a model for penmanship. FIG. 19 shows a procedure for learning the stroke order of the five-stroke kanji character “Ei”.

In FIG. 19 (a), the first stroke (dot) of the character "eternal" is displayed in white as a model image. The user traces the first image (dot) of the model image displayed in white and handwrites it with a brush.

In FIG. 19B, after the user finishes handwriting the first stroke (dot), the second stroke (horizontal stroke, vertical stroke, splash) of the character "eternal" is displayed in white as a model image. Indicates the state of being.

The model image of FIG. 19 (c) is the third image (horizontal image rising to the right, left side), the model image of FIG. 19 (d) is the fourth image (short left image), and the model image of FIG. 19 (e). In the image, the fifth image (right side) is shown in white. The user performs handwriting with a brush according to the presentation order of the model images displayed in white.

FIG. 19 (f) shows a state in which the handwriting of the character "Ei" is completed and only the handwritten character "Ei" 606 is written.

By this embodiment, the stroke order of kanji can be easily learned visually. Instead of the white display, the black ink may be slightly lighter or the color may be changed.

In this embodiment, the optimum method for learning the stroke order, in which the model image is sequentially divided and presented for each stroke, has been described, but other than the stroke order, the power in the process from the first stroke to the last stroke is explained. It is possible to visually clarify the difference from the model image in terms of the degree of insertion and the balance of characters.

Further, in this embodiment, the model image is sequentially displayed for each screen, but the entire model image can be displayed and used as the model image. Further, in this embodiment, a brush is used, but it goes without saying that another writing tool such as a pen may be used.

Also, as an application of this embodiment, a model image can be used as a photograph. By tracing the outer shape of the subject (building, person, object, etc.) reflected in the photograph, the same processing as the above-mentioned model image can be performed, and the handwriting drawing support system of the present invention can be realized.

In the above-described embodiment, it was assumed that the writing handwriting paper 25 does not move, but in this embodiment, the processing when the writing handwriting paper 25 moves will be described. The basic hardware configuration of this embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

The state of the handwritten paper for writing in this embodiment means, for example, the state of the schematic diagram of FIG. 14 (partially handwritten state).

In this embodiment, when the three-dimensional sensor unit 53 of the sensor device 5 detects the movement (including rotation) of the handwriting paper 25, the model image 35 of the rabbit is displayed in the movement direction (rotation angle in the case of rotation) of the handwriting paper 25. ) And the movement including the rotation according to the movement distance. As a result, the corresponding part of the rabbit model image 35 that overlaps with the handwritten part 36 is moved to a position that overlaps with the handwritten part 36, and is handwritten on the handwriting paper 25. The relative positions of the finished portion 36 and the rabbit model image 35 are in agreement with each other.

If the user's posture changes and the distance or angle between the handwriting paper 25 and the HMD1 changes, there is a problem that the perspective cannot be matched on the handwriting paper 25 if the model image 35 remains as it is.

The following method is applied to such cases as well. That is, when the three-dimensional sensor unit 53 of the sensor device 5 detects that the distance or angle between the handwriting paper 25 and the HMD1 has changed, the three-dimensional sensor unit 53 of the sensor device 5 again determines the shape of the handwriting paper 25. By grasping the model image 35, the model image 35 is corrected and presented as if it exists on the handwriting paper 25.

Further, even when the handwriting paper 25 is tilted vertically from the handwriting surface, it is the same as when the distance or angle between the handwriting paper 25 and the HMD1 changes, as if the model image 35 exists on the handwriting paper 25. , The model image 35 is corrected, and the model image 35 tilted in the vertical direction from the handwriting surface is presented and realized.

As described above, according to the present embodiment, even if the handwriting paper is moved during writing or the distance or angle between the handwriting paper and the HMD is changed, the model image is as if it exists on the handwriting paper. Since it can be displayed, the user can use the handwriting drawing support system without any discomfort.

In the above-described embodiment, by displaying the AR object of the model image on the display screen 75 of the HMD 1, the model image is visually recognized as a virtual image of the AR object on the handwritten surface 28 as if the model image exists on the handwritten surface 28. can do.

Here, in this embodiment, the processing when the user wearing the HMD1 raises his / her face from the handwritten surface and the handwritten paper will be described. The basic hardware configuration of this embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

FIG. 20 shows the display on the display screen 75 of the HMD1 in this embodiment. In FIG. 20, the model image 35 and the handwritten image 37 are displayed side by side on the display screen 75. In the model image handwriting process S464 of FIG. 12, the line-of-sight destination of the user wearing the HMD 1 is detected by the gyro sensor unit 55 of the sensor device 5, a line-of-sight sensor (not shown), or the like, and the user can use the handwriting surface and handwriting paper. The handwritten image on the handwriting paper is imaged by the image pickup unit 71, taken into various data units 42 in the storage device 4, and the captured image is displayed on the display screen 75, triggered by the case where the face is raised. Of course, the handwritten image to be displayed may be an image in the middle of handwriting.

As described above, according to this embodiment, the model image and the handwritten image can be compared even if the user does not always pay attention to the handwritten surface and the handwritten paper.

In the above-described embodiment, the shape of the handwriting surface and the shape of the handwriting paper are described as a rectangular plane, but a non-planar shape other than the rectangular plane is also possible. Therefore, in this embodiment, a case where the handwriting surface and the handwriting paper are non-planar will be described. The basic hardware configuration of this embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

FIG. 21 is a schematic view of the present embodiment in which the handwriting surface 621 has a non-planar cylindrical shape, and the handwriting paper 622 is arranged on the cylinder. The handwriting paper 622 is curved in a curved surface (arc) shape along the surface of the cylinder.

In this embodiment, the three-dimensional sensor unit 53 of the sensor device 5 grasps that the handwriting surface 621 is a cylindrical curved surface (see processing S422 in FIG. 7), and further, the handwriting paper 622 is curved in a curved surface shape. It is understood that this is done (see process S462 in FIG. 12).

Although not shown in FIG. 21, the model image (virtual image of the AR object) is formed on the curved surface of the handwriting paper 622 based on the handwriting paper shape obtained by this handwriting paper shape grasping process (see processing S462 in FIG. 12). Present by curving along.

In this embodiment, the handwriting paper 622 was used, but it goes without saying that the handwriting surface 621 of the cylinder can be directly handwritten with paint or the like. In addition, since the model image can be presented on a non-planar surface such as a plate or a cup along the shape of the non-plane, handwriting is performed by tracing the model image presented on the non-planar surface such as a plate or a cup. Can be done.

Although the examples of the present invention have been described above, the configuration for realizing the technique of the present invention is not limited to the present examples, and various modifications can be considered. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the category of the present invention. In addition, numerical values and messages appearing in sentences and figures are merely examples, and the effects of the present invention may not be impaired even if different ones are used.

Further, the above-mentioned functions and the like of the present invention may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, the software processing described in the examples and the hardware may be used together.

Further, the present invention is not limited to the glasses-type HMD, and may be any information processing device having a display device in front of the eyeball, and may be, for example, a goggle type or a contact lens type.

1: HMD 2: Main control device 3: System bus 4: Storage device 5: Sensor device, 6: Communication processing device, 7: Video processing device, 8: Voice processing device, 9: Operation input device, 11 : Control unit, 12: 3D sensor information acquisition unit, 13: 3D data processing unit, 18: AR image generation processing unit, 19: AR image display processing unit, 22: AR object, 25: Handwritten paper, 28: Handwritten Surface, 31, 32, 35: Model image, 37: Handwritten image, 42: Various data units, 53: Three-dimensional sensor unit, 81: Audio input / output unit, 82: Audio recognition unit.

Claims

A head-mounted display whose display is controlled by a control unit.
With a 3D sensor
The head-mounted display is characterized in that the control unit grasps the shape of an object based on information from the three-dimensional sensor and displays an AR object on the display unit along the shape of the object.
The head-mounted display according to claim 1.
The control unit is a head-mounted display characterized in that the AR object is enlarged, reduced, rotated, or deformed and displayed on the display unit.
The head-mounted display according to claim 1.
The head-mounted display is characterized in that, when there are a plurality of candidates for the AR object, the control unit sequentially displays the candidate AR objects on the display unit.
The head-mounted display according to claim 1.
Has an imaging unit
The head-mounted display is characterized in that the control unit analyzes the shooting data acquired by the imaging unit to acquire information on the gesture operation, and gives a control instruction for the display by the gesture operation.
The head-mounted display according to claim 1.
Has a voice entry,
The head-mounted display is characterized in that the control unit analyzes a user's voice input by the voice input unit by voice recognition, extracts it as a voice command, and gives a control instruction for the display by the voice command.
The head-mounted display according to claim 1.
When the relative positions of the object and the head-mounted display change, the control unit grasps the shape of the object based on the information from the three-dimensional sensor after the change, and the control unit grasps the shape of the object. A head-mounted display characterized by correcting an AR object according to its shape and displaying it on the display unit.
The head-mounted display according to claim 1.
The AR object is a model image, and the object is a handwritten surface.
It has an imaging unit and a storage device.
When the control unit detects that the front direction of the head mount display has moved from the handwriting surface to a predetermined direction based on the image information captured by the image pickup unit or the information from the three-dimensional sensor, the control unit A feature is that a handwritten image on a handwriting sheet arranged on the handwriting surface is captured by the imaging unit, captured in the storage device, and the captured handwritten image is displayed on the display unit together with the model image. Head mount display.
The head-mounted display according to claim 1.
Even when the shape of the object is non-planar, the control unit grasps the shape of the object based on the information from the three-dimensional sensor and corrects the AR object according to the shape of the object. A head-mounted display characterized by displaying on the display unit.
The head-mounted display according to claim 1.
The AR object is a model image, and the object is a handwritten surface.
A head-mounted display characterized in that a virtual image of the model image displayed on the display unit is displayed on the handwritten surface.
The head-mounted display according to claim 9.
A head-mounted display characterized in that it supports the creation of a handwritten image by changing the virtual image of the model image into different displays before and after handwriting on the handwritten surface.