JP2023103050A - Vr head-mounted system - Google Patents

Abstract

To provide a 6DOF-enabled head-mounted system which allows a user to view an entire-celestial-sphere image even if a head-mounted body is attached with a portable device.SOLUTION: There is provided a head-mounted system 100 in which a portable device SP including a camera CA on a first surface and a display unit DIS on a second surface is attached to a head-mounted body 10, and which allows a user to view an image projected onto the display unit DIS. The head-mounted body 10 includes a storage pocket 12 which stores the portable device such that the first surface faces forward, and which is notched such that the camera can image the front side. The portable device 10 comprises: a rotation amount sensor JS which senses a rotation amount in the three rotation directions of roll, pitch, and yaw; a translation movement amount calculation unit 32 which detects a feature point from a photographed image of the camera and calculates a translation movement amount in the three orthogonal axial directions; and an in-visual field image generation unit 36 which generates an in-visual field image from an entire-celestial-sphere image on the basis of the rotation amount and the translation movement amount.SELECTED DRAWING: Figure 3

Description

The present invention relates to a VR head-mounted system that uses a portable device such as a smartphone to display an omnidirectional image or the like in which the image within the field of view changes according to 6DOF (Degree of Freedom).

2. Description of the Related Art In the field of VR (virtual reality) and the like, the use of omnidirectional images captured by a camera capable of capturing 360° surroundings is progressing. A user attaches a portable device such as a smartphone to a head mount, displays an omnidirectional image as a stereo image with parallax that enables stereoscopic viewing on the display unit of the portable device, and can view the omnidirectional image. Patent Literature 1 discloses an invention for attaching a portable device to a head mount.

When the portable device is attached to the head mount, the angular velocity sensor of the portable device can detect three movements (roll, pitch, yaw) around the X, Y, and Z axes. As a result, it is possible to sense the rotation and tilt of the user's head and view 3DOF omnidirectional images.

However, the head mount to which the portable device of Patent Document 1 is attached cannot detect three movements of "translational movement" in the X-axis, Y-axis, and Z-axis directions. Therefore, there is a problem that the user cannot view omnidirectional video corresponding to 6DOF including translational movement. Furthermore, in order to not only view the omnidirectional video, but also to rewind and advance the omnidirectional video, it was necessary to prepare a hand-held controller in addition to a portable device such as a smartphone.

Japanese Patent Publication No. 2019-510328

Therefore, an object of the present embodiment is to make it possible to view a 6DOF omnidirectional video even with a head-mounted portable device. Another object of the present invention is to enable processing (stop processing, rewind processing, etc.) for omnidirectional video without preparing a controller.

The head mount system of the present embodiment is a head mount system in which a portable device including a camera on the first surface and a display unit on the second surface is attached to a head mount body and an image projected on the display unit is viewed. The head mount main body has a storage pocket cut out so that the portable device can be stored with the first surface facing forward and the camera can photograph the front. Further, the portable device has a rotation amount sensor that senses the amount of rotation in the three rotation directions of roll, pitch, and yaw, a translation amount calculation unit that detects feature points from an image captured by the camera and calculates the amount of translational movement in three orthogonal axis directions, and an in-field image generation unit that generates an in-field image from an omnidirectional image based on the amount of rotation and the amount of translational movement.

The portable device preferably has a hand tracking unit that detects and tracks the user's hand from an image captured by the camera, a hand image generation unit that generates a hand image when the hand is in the captured image for longer than a predetermined time, a GUI image generation unit that generates a GUI image related to the video within the field of view when the hand is in the captured image for a longer time than the predetermined time, and a video synthesizing unit that combines the hand image and the GUI image with the video within the field of view.

The portable device may also have a GUI instruction determination unit that determines whether the hand image has been present on the instruction icon of the GUI image for a predetermined time, and the GUI instruction determination unit may instruct execution of the operation of the instruction icon when the hand image is on the instruction icon of the GUI image.
The portable device also has a shopping store determination unit that determines whether or not the image within the field of view is a shopping store where products can be purchased, and a product image storage unit that stores a plurality of product images sold at the shopping store. It is preferable that the product image storage unit supplies the product image to the image synthesizing unit when the GUI instruction determination unit determines that the hand image is on the shopping GUI image.

Moreover, the portable device has a product identification determination unit for determining whether the hand image exists in the product image for a predetermined time, and the product identification determination unit preferably integrates the hand image and the product image, and separates the hand image and the product image in the cart image.
When the product image storage unit supplies the product image to the image synthesizing unit, it is preferable that the image synthesizing unit does not synthesize the in-field image.

The method of projecting an in-field image according to this embodiment is a method in which a portable device including a camera on the first surface and a display unit on the second surface is stored in a housing pocket of the head mount body, and the in-field image is projected on the display unit. And the storage pocket is cut out so that the camera can shoot forward. Then, the portable device senses the amount of rotation in the three rotational directions of roll, pitch, and yaw, detects feature points from the image captured by the camera, calculates the amount of translational movement in the three orthogonal axis directions, and generates an in-field image from the omnidirectional image based on the amount of rotation and translational movement.
Project the image within the field of view.

The VR head-mounted system of this embodiment can view 6DOF omnidirectional video. In addition, it is possible to instruct processing for the omnidirectional video without preparing a hand controller.

1 is a perspective view of a head-mounted system to which the portable device of this embodiment is attached; FIG. (A) is a view of the head mount (excluding the portable device) viewed from four directions. (B) is an explanatory diagram of a mechanism for adjusting the distance between the pupils, and (C) is an explanatory diagram of a mechanism for adjusting the focal length of the lens. 1 is a block diagram of a portable device used in a VR head-mounted system; FIG. 1 is a flow chart 1 for hand tracking; (A) is an example of a hand-tracked hand image. (B) is an example of a GUI image in which five instruction icons are displayed. (C) to (F) are examples of in-field images displayed on the display unit of the portable device. It should be noted that the image or video on the display unit is drawn as one video within the field of view recognized by the human brain. Fig. 2 is flow chart 2 for hand tracking; (A) to (F) are examples of images displayed on the display unit of the portable device. It should be noted that the image or video on the display unit is drawn as one video within the field of view recognized by the human brain.

[Overall configuration of head-mounted system]
FIG. 1 is a perspective view showing a head mount system 100 in which a portable device SP according to this embodiment is attached to a head mount body 10. FIG. As shown in FIG. 1, the head mount system 100 includes a head mount body 10, a strap 20 for securely attaching the head mount body 10 to the user's head, and a portable device SP.

The portable device SP is stored in the storage pocket 12 of the head mount body 10. - 特許庁The portable device SP disclosed in this embodiment has one or more cameras CA on the first surface (−Y axis side) and a display unit DIS (see FIG. 2) on the second surface (+Y axis side). Portable devices SP include, but are not limited to, so-called smart phones, tablet terminals or handheld visual media players.

An exemplary portable device SP includes a central processing unit (CPU) (not shown), a display unit DIS, a camera CA, and a communication unit to enable running applications for use with the system. Portable device SP50 incorporates a rotation angle sensor such as one or more gyro sensors, accelerometers, hydrometers, or magnetometers. In this embodiment, the rotation angle sensor can detect the amount of rotation of the head mounted system 100 in the three directions of roll, pitch, and yaw.

A strap 20 is attached to the head mount body 10, and the user's head is firmly fixed with the strap 20.例文帳に追加note that. For example, head mount body 10 can be incorporated into a helmet-like device that is secured to the crown of the head without straps.

A storage pocket 12 for storing the portable device SP has a structure that bends like a leaf spring, and the portable device SP is sandwiched between the peripheral wall 11 and the storage pocket 12.例文帳に追加The width (X-axis direction) of the storage pocket 12 is formed in a notched shape shorter than the length of the portable device SP, so that the camera CA of the portable device SP can photograph the front. If the width of the storage pocket 12 is long, the periphery corresponding to the camera CA of the portable device SP may be cut out so that the camera CA can photograph the front.

The peripheral wall 11 is provided with a focus adjustment lever 14 for adjusting the focal length of the lens and an interpupillary distance adjustment dial 15 for moving the lens according to the distance between the pupils.

[Composition of the head mount body]
FIG. 2 is a diagram showing the head mount body 10 according to this embodiment. FIG. 2A is a view of the head mount body 10 viewed from four directions. 2B and 2C are diagrams showing the periphery of the lens arranged inside the head mount body 10. FIG.

This is because the peripheral wall 11 and the storage pocket 12 of the head mount body 10 are preferably made of a plastic material such as ethylene vinyl acetate (EVA), polyurethane (PU), or ABS resin. Each of these can be used alone or in various combinations. In a preferred embodiment, the peripheral wall 11 and storage pocket 12 are molded, such as by injection molding. The forward portion of peripheral wall 11 has a generally rectangular or box-like shape.

The distance between the front surface 11a of the peripheral wall 11 and the storage pocket 12 is formed to be slightly shorter than the thickness of the portable device SP, and the storage pocket 12 has an S shape when viewed from the side (X-axis direction). Due to the elasticity and shape of the plastic itself, the storage pocket 12 acts like a leaf spring and can firmly attach the portable device SP (not shown) to the front face 11a of the peripheral wall 11 . A strap pin 19 to which a strap 20 (see FIG. 1) is attached is formed on the top and side surfaces of the peripheral wall 11 .

The width L1 of the storage pocket 12 can vary depending on the type and size of the portable device SP, but is typically intended to grip and hold the portable device SP. The average size of the display unit of the portable device SP is about 5 inches (12.7 cm) to 6.5 inches (16.5 cm), and the position of the camera CA on the first surface of the portable device SP is often arranged on the upper left side when viewed from the Y-axis direction. When the portable device SP is laid horizontally as shown in FIG. The width L1 of the storage pocket 12 is notched so that the camera CA is not hidden.

The face edge portion 17 for contacting the user's forehead is preferably made of a flexible material such as rubber or urethane foam. The peripheral wall 11 and the face edge portion 17 are joined by an adhesive, a fitting, or the like. The face edge 17 prevents light from entering the hollow 13 from the surroundings when it contacts the user's face.

A pair of lenses LZ are arranged in the hollow 13 for viewing the image on the display of the portable device SP. The peripheral wall 11 has a focus adjustment lever 14 and an interpupillary distance adjustment dial 15 for moving the pair of lenses LZ in the Y-axis direction and the X-axis direction. The interpupillary distance adjustment dial 15 is a dial for adjusting the interpupillary distance, which varies depending on the user, according to the user. As shown in FIG. 2B, each of the pair of lens holders 18 holding the lens LZ has a rack 18a. A pinion 15a is formed on the adjustment dial 15, and the rack 18a and the pinion 15a are engaged with each other. By rotating the pinion 15a, the lens LZ held by the lens holder 18 approaches the center side or moves away from the outside. In FIG. 2(B), the pinion 15a and the rack 18a of the interpupillary distance adjustment dial 15 are drawn separately to facilitate understanding.

As shown in FIG. 2C, the focus adjustment lever 14 is moved back and forth (in the Y-axis direction) by the user to adjust the focal length of the lens LZ. A bridge 18b is formed on the lens holder 18, and the clamp 14a of the focus adjustment lever 14 clamps the bridge 18b. The lens LZ is positioned between the user's eye and the display of the portable device SP. As can be seen, the user's eyes are aligned with the lens LZ so that the user can look through the lens LZ to view the display of the portable device SP. The lens LZ can focus the user's field of vision on a left or right discontinuous area of the in-field image projected on the display of the portable device SP. Proper alignment of the user's view through the lens is especially important in virtual reality applications.

[Portable device configuration]
FIG. 3 is a block diagram showing the functional configuration of the portable device SP. Portable devices SP include smartphones, tablet computers, personal computers, and the like. The display device DIS is a display device such as a liquid crystal display or an organic EL display. The display device DIS may have separate display screens for the right and left eyes of the user, or may have only one display screen. An application is downloaded from a store such as a website to the portable device SP. By activating the application, the portable device SP has the functions described below. You can also download multiple spherical image data from the app.

The rotation amount sensor JS detects movement of the head mount body 10 . The rotation amount sensor JS is capable of detecting three-axis rotation (3DOF; Degrees Of Freedom) of yaw, roll, and pitch. Note that the rotation amount sensor JS may be an IMU (inertial measurement unit) or various combinations such as a gyro sensor. If the rotation amount sensor JS is an IMU, it can detect movement in three directions: the Y-axis direction (back and forth), the X-axis direction (left and right), and the Z-axis direction (up and down).

The portable device SP further includes a communication unit WF, a video data storage unit 31, a translation calculation unit 32, a hand tracking unit 33, a hand image generation unit 34, a GUI image generation unit 35, an in-field video generation unit 36, a video synthesis unit 37, and a GUI instruction determination unit 38. The portable device SP also includes a shopping store determination unit 41 , a product image storage unit 42 and a product identification determination unit 43 .

The communication unit WF performs long-distance communication such as 5G or 4G, or short-distance communication such as WiFi (trademark) or bluetooth (trademark). The video data storage unit 31 acquires omnidirectional video data (hereinafter referred to as video data) via the communication unit WF. The omnidirectional image is a 360° omnidirectional image centered on a certain point, and is an image obtained by synthesizing images captured using an omnidirectional camera or images captured by a plurality of cameras. The video data storage unit 31 may acquire video data by reading video data stored in the portable device SP. When the image data storage unit 31 acquires the image data, the image data storage unit 31 decodes the image data and generates an omnidirectional image. The image data storage unit 31 supplies the generated omnidirectional image to the in-field image generation unit 36 .

The translational movement calculation unit 32 periodically acquires images captured by the camera CA, and calculates the amount of translational movement in the XYZ-axis directions using characteristic points existing in the physical space as indices. Specifically, it is disclosed in WA Hoff and K. Nguyen, "Computer vision-based registration techniques for augmented reality", Proc. SPIE, vol.2904, pp. 538-548, Nov. 1996.
In FIG. 3, the translational movement calculation unit 32 is depicted to calculate the translational movement only from the captured image of the camera CA. However, if the rotation amount sensor JS is an IMU (inertial measurement unit), 6DOF can be detected. Calculation of the amount of translational movement using regularly captured images and sensors is specifically disclosed in S. You and U. Neumann, "Fusion of vision and gyro tracking for robust augmented reality registration", Proc. IEEE Virtual Reality 2001, pp.71-78, Mar. 2001. If the accuracy of the translational movement in the XYZ-axis directions can be low, the translational movement detected by an inertial measurement unit (IMU) is used, and the translational movement calculator 32 that uses the image captured by the camera CA may be omitted.

The hand tracking unit 33 recognizes the hand and fingers of the user based on the image captured by the camera CA, and tracks the position of the hand and the movement of the fingers. The hand tracking unit 33 can use a deep learning model to determine whether or not a hand is present in the captured image captured by the camera CA.

The hand image generation unit 34 generates a hand image 51 to be displayed in the video based on the recognition of the user's hand by the hand tracking unit 33 . It is preferable that the hand image generator 34 generates the hand image 51 when the user's hand is in the captured image for longer than a predetermined time. This is to eliminate an event in which the user's hand temporarily enters the visual field while the user is waving and walking. FIG. 5A is an example of a hand image 51 projected on the display unit DIS. The hand image 51 is preferably a translucent image so that viewing of the in-field video is not disturbed even if the hand image 51 is superimposed on the in-field video. The hand image generation unit 34 may prepare a plurality of unique hand images 51 (a hand image pointing, a hand image grasping an object with the thumb and forefinger, etc.) in advance, and select the hand image 51 that is close to the shape of the hand in the image captured by the camera CA. Alternatively, the hand image 51 may be generated by detecting the contour of the photographed hand by image processing. The generated hand image 51 is supplied to the video synthesizing section 37 . When the hand disappears from the captured image captured by the camera CA, the hand image generator 34 stops generating the hand image, and the hand image is erased.

The GUI image generator 35 generates a GUI image 52 related to the video based on the recognition of the hand. The GUI image generator 35 preferably prepares a plurality of GUI images 52 in advance. The GUI image 52 is, for example, instruction icons such as rewind, review, stop, cue, and fast forward. FIG. 5B is an example of a GUI image 52 projected onto the display unit DIS. Alternatively, as will be described later, it may be an instruction icon for purchasing a product. In FIG. 3, the arrow is drawn so that the GUI image 52 is generated after the hand image generating unit 34 generates the hand image. However, the GUI image 52 may be generated simultaneously with the hand image 51 after the hand tracking unit 33 recognizes the hand. In other words, it is preferable that the GUI image generator 35 generates the GUI image 52 when the user's hand is in the captured image for longer than the predetermined time. In the GUI image 52, it is preferable that the number and types of instruction icons change according to the in-field video.

The GUI image generator 35 preferably localizes the generated GUI image 52 in the display space. The generated GUI image 52 is supplied to the video synthesizing section 37 . The GUI image 52 is preferably a translucent image so that viewing of the in-field video is not disturbed even when the GUI image 52 is displayed superimposed on the in-field video. Thus, even if the user wearing the head mount body 10 changes direction or translates in the real space, the GUI image 52 is substantially fixed at a predetermined position based on the viewpoint of the display space, and the GUI image 52 does not move.

The in-field image generation unit 36 generates an in-field image, which is an image to be displayed on the display unit DIS, from the omnidirectional image supplied from the image data storage unit 31 . The in-field image generation unit 36 acquires the amount of rotation and translation from the rotation amount sensor JS and the translation calculation unit 32, and extracts a part of the omnidirectional image according to the orientation of the head mount body 10 and the movement of the user, thereby generating the in-field image. When the in-field image generation unit 36 detects the rotation of the three axes of yaw, roll, and pitch and the amount of movement in the XYZ-axis directions, it moves the range of the in-field image of the omnidirectional image according to the rotation and movement amount.

As a result, when the user moves his/her head and moves forward, backward, left, right, up and down, the range of the omnidirectional video image within the visual field moves following the movement of the head mount body 10, and the user can view the omnidirectional video image as if looking around. That is, the omnidirectional image realized by the in-field image forms a display space in which the position of the viewpoint is substantially independently controlled with respect to the change in the position of the user's head. In-field-of-view images are projected on the display unit DIS for the right eye and the left-eye respectively, but in FIGS. 5 and 7, they are depicted as one in-field image for recognition by the human brain.

The image synthesis unit 37 synthesizes the in-field image supplied from the in-field image generation unit 36, the hand image supplied from the hand image generation unit 34, and the GUI image 52 supplied from the GUI image generation unit 35. The synthesized image is projected on the display unit DIS.

The GUI instruction determination unit 38 determines whether the user's hand image exists in the GUI image 52 projected on the display unit DIS by the video composition unit 37 for a predetermined time (1 to 2 seconds). If the GUI image 52 contains a plurality of instruction icons, it is determined whether one of them has been identified. For example, the user moves his/her hand to move the hand image to the fast-forward button instruction icon of the GUI image 52 and holds his/her hand for one second. The GUI instruction determination unit 38 determines that a fast-forward instruction has been given, and instructs the in-field image generation unit 36 to fast-forward the image.

The shopping store determination unit 41 determines whether or not there is a shopping store in the image generated by the in-field image generation unit 36 . The shopping store determination unit 41 uses a deep learning model to determine whether or not there is a shopping store in the image within the field of view. Without using deep learning, a unique signal indicating the presence of a shopping store may be included in the omnidirectional video in advance, and the shopping store determination unit 41 may detect the unique signal and determine the presence of the shopping store.

The product image storage unit 42 stores images (still images/moving images) of products that can be sold at shopping stores. For example, if the product is a bag, a photograph of a sellable shoulder bag, handbag, or the like is taken in advance, and the product image storage unit 42 stores a plurality of these images. In addition, the product image storage unit 42 sequentially supplies product images of each product to the image synthesizing unit 37 and supplies a plurality of products displayed as thumbnails to the image synthesizing unit 37 .

The product specifying unit 43 determines whether the user's hand image exists for a predetermined time (1 to 2 seconds) in the product image projected on the display unit DIS by the image synthesizing unit 37, and determines whether the product has been specified. Alternatively, when the hand tracking unit 33 tracks the action of the user's hand pinching the product image with the thumb and forefinger, the product identification unit 43 determines that the product has been identified. After that, the process moves to the procedure for purchasing the product.

[Operation of the head-mounted system]
FIG. 4 is a flow chart showing the operation of the head mount system 100. As shown in FIG.
First, the user activates an application that has been downloaded to the portable device SP (S401). Then, the user sets the portable device SP in the storage pocket 12 of the head mount body 10 (S402). A user wears the head mounted system 100 .

The image within the field of view is projected onto the display unit DIS (S403), and the user views the image within the field of view as shown in FIG. 5C, for example. If necessary, the user moves the focus adjustment lever 14 to adjust the focal length of the lens, and turns the interpupillary distance adjustment dial 15 to move the lens according to the distance between the pupils. The image within the field of view changes based on the motion (yaw, roll, and pitch) signals of the head mount body 10 from the rotation amount sensor JS and the translation amount signal based on the camera CA of the portable device SP.

At the same time that the in-field image is projected onto the display unit DIS, hand tracking by the camera CA is started (S404). When the user sticks out his/her hand forward, the user's hand enters the field of view of the camera CA of the portable device SP. A hand tracking unit 33 recognizes the user's hand and fingers, and tracks the position of the hand and the movement of the fingers. When the user's hand is continuously within the field of view for a predetermined time (for example, 1 second) (S405 YES), the hand image 51 generated by the hand image generating unit 34 is projected onto the display unit DIS (S406). As shown in FIG. 5D, a translucent hand image 51 is superimposed on the image within the field of view.

Simultaneously with the projection of the hand image 51 onto the display unit DIS or several seconds later, the GUI image 52 generated by the GUI image generation unit 35 is projected onto the display unit DIS (S407). As shown in FIG. 5(E), a semi-transparent GUI image 52 is superimposed on the image within the field of view and displayed in the center. In one example of the present embodiment, the GUI image 52 displays, from the left, a 10-second rewind instruction icon, a review instruction icon, a stop instruction icon, a cue instruction icon, and a 30-second fast forward instruction icon. When the in-field video is stopped, the GUI image 52 switches from the stop instruction icon to the reproduction instruction icon. In this embodiment, the GUI image 52 is projected at the center of the display section DIS, but it may be projected at the upper end or the lower end of the display section DIS. It is preferable that the GUI image 52 is projected at a fixed position even if the user moves the head mount body 10 by moving the head left or right and the image within the field of view changes.

The GUI image 52 is projected at a fixed position even if the in-field image changes. Therefore, as shown in FIG. 5(E), by moving the user's own hand, the hand tracking unit 33 moves from, for example, the hand image 51 drawn with a dotted line to the hand image 51 drawn with a solid line, and the 10-second rewind instruction icon is maintained for a predetermined time. The GUI instruction determination unit 38 determines whether or not the hand image 51 exists in the instruction icon of the GUI image 52 (S408). In FIG. 5E, the GUI instruction determination unit 38 determines that the 10-second rewind instruction icon has been instructed, and instructs the in-field image generation unit 36 to rewind the image by 10 seconds (S409).

On the other hand, when the user takes his or her hand out of the field of view of the camera CA (S405 NO), the image shown in FIG. 5(E) changes to the image shown in FIG. 5(F). That is, the hand image 51 is erased from the display unit DIS (S410), and the GUI image 52 is erased at the same time or a little later.

Next, a case where the user selects video data that the user wants to view from a plurality of video data and a case where the user purchases a product when the video data is a shopping video will be described with reference to the flowchart of FIG. 6 and FIGS. 7(A) to 7(F).

When the application of the portable device SP is started, a thumbnail display is projected on the display unit DIS so that a plurality of video data categories can be selected (S601). FIG. 7A is a thumbnail display showing an example on the display unit DIS. For example, the category of video data includes a video of a trip to Japan, a video of a trip to Taiwan, an underwater video, a shopping video, etc. By moving the user's own hand and placing the hand video 51 on the horizontal arrow 71 for a predetermined time, it can be moved to a thumbnail display that is not displayed. FIG. 7A shows a state in which the hand image 51 has selected the shopping video category. Then, as shown in FIG. 7(B), a thumbnail display is projected on the display section DIS so that one shopping image can be selected from a plurality of shopping images. If the user wishes to view an underwater image instead of the shopping image, the user can move his or her hand and place the hand image 51 on the upper category arrow 73 for a predetermined period of time. In this embodiment, the user selects the shopping image B by placing the hand image 51 on the shopping image B74 for a predetermined period of time.

The in-field image generator 36 projects the shopping image B (S602). FIG. 7C is an example of a shopping image showing a virtual shopping mall 74 . When the user moves his or her head forward, backward, left, or right, the rotation amount and the translation amount are obtained from the rotation amount sensor JS and the translation calculation unit 32, and the in-field image generation unit 36 projects the image of the shopping mall and the in-field image of the store in the shopping mall. When the user moves and enters a virtual store (for example, a bag store) in the shopping mall, the shopping store determining unit 41 determines whether or not it is a shopping store where products can be purchased. If products can be purchased at all virtual stores in the shopping mall, the shopping store determination unit 41 may be omitted. However, if products can be purchased at some virtual stores in the shopping mall, for example, the shopping store determination unit 41 detects a specific signal included in the omnidirectional image or recognizes it by deep learning in order to indicate to the user whether the virtual store the user entered is a shopping store (S603).

If the virtual store to which the user is moving is a shopping store where products can be purchased (S603 YES), the shopping store determination unit 41 projects a shopping GUI image 52a onto the display unit DIS (S604). FIG. 7(D) is an example in which a shopping GUI image 52a is projected on the lower end of the display section DIS on the video within the field of view of the shopping store 75. FIG. When the user moves the head in the visual field image of the shopping store 75, the visual field image changes, but the shopping GUI image 52a remains projected at a fixed position. When the user moves forward, backward, leftward, and rightward to leave the shopping store 75 and is in the aisle of the shopping mall 74, the shopping GUI image 52a is erased. If the virtual store to which the user is moving is not a shopping store where products can be purchased (S604 NO), the image within the field of view of the shopping mall is continuously projected on the display unit DIS (S403).

Next, the GUI instruction determination unit 38 determines whether or not the hand image 51 exists in the shopping GUI image 52a (S605). FIG. 7D shows a state in which the hand image 51 exists in the shopping GUI image 52a. When the hand image 51 exists in the shopping GUI image 52 a ( S<b>605 YES), the product image storage unit 42 supplies a plurality of product images (still images or moving images) of the shopping store 75 to the video synthesizing unit 37 . If the hand image 51 is not in the shopping GUI image 52a, the hand tracking unit 33 continues to track the hand position and finger motion based on the image captured by the camera CA (S404).

When the product image storage unit 42 supplies a plurality of product images to the image synthesizing unit 37, the product image 76 is projected as shown in FIG. In FIG. 7E, only the product image 76 is projected so that the user can easily see the product image 76 without synthesizing the in-field video of the shopping store 75 . In other words, the image synthesis unit 37 does not synthesize the in-field image supplied from the in-field image generation unit 36 . However, the image within the field of view of the shopping store 75 may be projected semi-transparently.

The user moves his or her hand, and the hand tracking section 33 tracks the hand position and finger movements. The product identification determination unit 43 determines that the product image 76 has been identified when the hand image 51 is maintained on the product image 76 for a predetermined time (S608). Alternatively, when the hand tracking unit 33 tracks the motion of the user's hand pinching the product image 76 with the thumb and forefinger, the product identification unit 43 determines that the product image 76 has been identified (S608). When the product image 76 is specified, the hand image 51 and the product image 76 are integrated and become movable.

Next, the product identification determination unit 43 determines whether the hand image 51 has been integrated with the product image 76 and moved to the cart 77, that is, whether the hand image 51 exists in the cart image 77 (S609). If the hand image 51 exists in the cart image 77 (S609 YES), the product image 76 enters the cart 77 and the hand image 51 and the product image 77 are separated (S610). Then, as shown in FIG. 7F, a product purchase GUI image 52b is projected (S611). A “proceed to purchase” instruction icon, a “return product” instruction icon, a “continue shopping” and “exit shopping” instruction icon, and the like are projected onto the product purchase GUI image 52b. The user moves his/her hand to move the hand image 51 to one of these instruction icons. When the hand image 51 remains on the "proceed to purchase" instruction icon for a predetermined time, the user can proceed to a screen for purchasing the product.

In this embodiment, an example in which a product can be purchased in the case of a shopping image has been described, but for example, when an image of a snorkel or flippers appears in the middle of an underwater image, a screen for purchasing a snorkel or flippers may be projected.

100... Head mount system 10... Head mount main body 11... Peripheral wall 12... Storage pocket 14... Focus adjustment lever 15... Interpupillary distance adjustment dial 17... Face edge 20... Strap 31... Image data storage unit 32... Translational movement calculation unit 33... Hand tracking unit 34... Hand image generation unit 35... GUI image generating unit 36... Visual field image generating unit 37... Video synthesizing unit 38... GUI instruction determination unit 41... Shopping store determination unit 42... Product image storage unit 43... Product specific determination unit 51... Hand image 52... GUI image CA... Camera DIS... Display unit JS... Rotation amount sensor SP... Portable device WF … communication

Claims (10)

A portable device including a camera on a first surface and a display unit on a second surface is attached to a head mount body, and a head mount system for viewing an image projected on the display unit,
The head mount body is
The portable device is stored so that the first surface faces forward, and the camera has a notched storage pocket so that the camera can shoot forward,
The portable device
a rotation amount sensor that senses the amount of rotation in the three directions of roll, pitch, and yaw;
a translational movement amount calculation unit that detects feature points from the image captured by the camera and calculates the amount of translational movement in three orthogonal axis directions;
an in-field image generation unit that generates an in-field image from an omnidirectional image based on the amount of rotation and the amount of translational movement;
a head-mounted system. The portable device
a hand tracking unit that detects and tracks the user's hand from the image captured by the camera;
a hand image generation unit that generates a hand image when the hand is in the captured image for longer than a predetermined time;
a GUI image generation unit that generates a GUI image related to the in-field image when the hand is in the captured image for a period longer than a predetermined time;
a video synthesizing unit that synthesizes the hand image and the GUI image with the in-field video;
has
2. The head mounted system according to claim 1, wherein the in-field image, the hand image and the GUI image are projected on the display unit. The portable device
a GUI instruction determination unit that determines whether the hand image has existed in the instruction icon of the GUI image for a predetermined time;
3. The head-mounted system according to claim 2, wherein said GUI instruction determination unit instructs execution of an operation of said instruction icon when said hand image is on said instruction icon of said GUI image. The portable device
a shopping store determination unit that determines whether or not the in-field image is a shopping store where products can be purchased;
a product image accumulating unit for accumulating images of a plurality of products sold at the shopping store;
has
4. The head-mounted system according to claim 3, wherein said product image storage unit supplies said product image to said image synthesizing unit when said hand image is on a shopping GUI image. The portable device
a product identification determination unit that determines whether the hand image has existed in the product image for a predetermined time;
5. The head-mounted system according to claim 4, wherein said product identification determination unit integrates said hand image and said product image, and separates said hand image and said product image in a cart image. When the product image storage unit supplies the product image to the video synthesis unit,
5. The head-mounted system according to claim 4, wherein said video synthesizing unit does not synthesize said in-field video. A method of storing a portable device including a camera on a first surface and a display unit on a second surface in a storage pocket of a head mount body and projecting an image within the field of view onto the display unit,
The storage pocket is notched so that the camera can shoot forward,
the portable device
Detects the amount of rotation in three directions: roll, pitch, and yaw.
Detecting feature points from the image captured by the camera and calculating the amount of translational movement in three orthogonal axis directions,
generating an in-field image from an omnidirectional image based on the amount of rotation and the amount of translation;
A method of projecting an in-field image. the portable device
detecting and tracking a user's hand from the image captured by the camera;
generating a hand image when the hand is in the captured image for longer than a predetermined time;
generating a GUI image related to the in-field image when the hand is in the captured image for longer than a predetermined time;
synthesizing the hand image and the GUI image with the in-field video;
8. The method of projecting an in-field image according to claim 7, wherein the in-field image, the hand image and the GUI image are projected on the display unit. the portable device
determining whether or not the hand image has existed on the pointing icon of the GUI image for a predetermined time;
9. The method of projecting an in-field image according to claim 8, wherein when the hand image is on the pointing icon of the GUI image, it indicates to perform the action of the pointing icon. the portable device
determining whether the in-field image is a shopping store where products can be purchased;
accumulating a plurality of product images to be sold at the shopping store;
10. The method of projecting an in-field image according to claim 9, wherein the product image accumulator supplies the product image to the video synthesizer when the hand image is on a shopping GUI image.