JP2016154044A - Head mount type augmented reality video presentation device and virtual display object operation method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To operate a virtual display object in a synthesized image without using a touch panel or the like.SOLUTION: A head mount type augmented reality video presentation device for displaying a virtual display object in an object image captures the image of a pointer 7 which is a point light source capable of changing at least one of light-emitting light and a light-emitting pattern as a part of the object image, synthesizes the object image and a virtual display object 1 indicating virtual object data to display a synthesized image, and a required function corresponding to a required operation is exhibited to the virtual object data corresponding to the virtual display object 1 understanding that the required operation has been executed for the virtual display object 1 when a correlation between the image of the pointer 7 in the synthesized image and the virtual display object 1 satisfies a predetermined condition.SELECTED DRAWING: Figure 14

Description

  The present invention relates to a head-mounted augmented reality video presenting apparatus, and more particularly to a head-mounted augmented reality video presenting apparatus that allows a user to easily manipulate a virtual display displayed in front of the user and a method for operating the virtual displayed object. .

  Software that presents augmented reality images has been developed and used. For example, in application software for a mobile phone with a camera, information (such as tourist information around the area) that matches the scene captured by the camera is acquired via the network, and a virtual display object (such as an icon) indicating the presence of this information is captured. And display it on the display screen of the mobile phone. Then, when the user selects the virtual display object on the display screen using a touch panel or the like, the virtual display object is enlarged and detailed information (such as the history of the tourist facility) is browsed or the information is copied to a local recording area. I can do it. In addition, a virtual display object to which information is given by the user himself / herself can be generated.

  There is a problem that it is difficult to view such augmented reality video on a small display screen of a mobile phone. It also lacks realism. Therefore, by connecting a head-mounted display device (HMD) to the mobile phone and displaying the captured video and the virtual display object, the user can see a large and easy-to-see video with a sense of reality.

  However, when the head-mounted display device is used, the user's eyes are blocked by the display screen covering the front of the eyes, and there is a problem that the virtual display object in the display screen cannot be operated with the mobile phone. .

  As a means for solving this problem, for example, as described in Patent Document 1 below, a gyro sensor is attached to the user's hand so that the head-mounted display device grasps the position of the hand and displays it on the display screen. It is conceivable to operate the virtual display object using the hand image by displaying the hand image in the virtual space. Alternatively, it is also possible to attach an infrared camera to the display screen side of the head mounted display device, and issue an operation instruction by sensing the user's line of sight.

  However, when a sensor such as a gyro sensor or a special device such as an infrared camera is provided, there is a problem that the cost of the head-mounted display device increases, resulting in a large-scale device configuration.

Japanese Patent Laid-Open No. 2003-337962

  An object of the present invention is to provide a head-mounted augmented reality video presentation device that can easily operate a virtual display object displayed on a head-mounted display device without using expensive devices such as sensors, and a virtual display object thereof. It is to provide an operation method.

  A head-mounted augmented reality video presentation device according to the present invention includes a pointer that is a point light source capable of changing at least one of a light emission color and a light emission pattern according to a user instruction, an imaging unit that captures an image of the pointer, and virtual object data. A virtual object data acquisition unit that acquires a virtual object data from a virtual object database, an image synthesis unit that generates a synthesized image by synthesizing a pointer image captured by the imaging unit and a virtual display object obtained by iconifying the virtual object data, and the synthesis Head-mounted display means for displaying an image; pointer function identification means for recognizing at least one of a color component and a light emission pattern of the point light source from the pointer image and identifying a function for the virtual object data; and the composite image When the distance relationship between the pointer image in the center and the virtual display object satisfies a predetermined condition, the pointer function identification In which and a control means for expressing a function identified by means.

  According to the virtual display object operating method of the present invention, an image of a pointer that is a point light source capable of changing at least one of a light emission color and a light emission pattern according to a user's instruction is captured by an imaging unit, and the pointer image captured by the imaging unit and a virtual image A virtual display object obtained by iconifying virtual object data acquired from the object database is synthesized to generate a composite image, the composite image is displayed on a head-mounted display unit, and the pointer image and the virtual image in the composite image are displayed. When the distance relationship with the display object satisfies a predetermined condition, a function for the virtual object data identified based on at least one of the color component and the light emission pattern of the point light source recognized from the pointer image is developed.

  According to the present invention, it is possible to intuitively and easily operate a virtual display object displayed on a head-mounted display unit without using expensive devices such as sensors.

1 is a functional block diagram of a head-mounted augmented reality video presentation device according to an embodiment of the present invention. It is a figure which shows the example of an image displayed on the display screen shown in FIG. It is a figure which shows the example of taking in the fingertip image which concerns on another embodiment. It is a figure which shows the operation procedure when making a fingertip image into a pointer image. It is explanatory drawing of the example of operation using a fingertip image. It is a figure explaining another example of operation using a fingertip picture. It is a figure explaining another example of operation using a hand picture. It is a figure which takes in a marker image as a pointer image. It is a figure which shows the operation procedure when making a marker image into a pointer image. It is explanatory drawing of the example of operation using a marker image. It is explanatory drawing of another example of operation using a marker image. It is a figure which takes in a point light source image as a pointer image. It is a figure which shows the operation procedure when making a point light source image into a pointer image. It is explanatory drawing of the example of operation using a point light source image. It is a figure explaining another example of operation using a point light source image.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram of the head-mounted augmented reality video presentation device according to the first embodiment of the present invention. The head-mounted augmented reality video presentation device 10 according to the present embodiment includes a central processing unit (CPU) 11 that performs overall control of the device 10 and a bus 12 connected to the CPU 11.

  The bus 12 includes an imaging control unit 15 connected to the camera 14, a main memory 16, a signal processing unit 17 that digitally processes an image signal captured by the camera 14, a photographic lens 14a, and an imaging sensor 14b. A measurement unit 18 that measures the position and orientation of the camera 14, a GPS (global positioning sensor) 26 that acquires position information, a geomagnetic sensor 27 that acquires direction information, a virtual display object and the like, which will be described later, and an imaging sensor 14b. An image synthesis unit 19 that synthesizes the video that has been shot and signal processed is connected.

  Further, the bus 12 includes a communication control unit 21 including an antenna 20, a pointer detection recognition unit 22 and a pointer monitoring unit 23, which will be described in detail later, and an image display control unit 25 to which a display screen unit (EVF) 24 is connected. Is connected.

  The head-mounted augmented reality video presentation device 10 communicates with an external database through the communication control unit 21 to acquire virtual object data. An external database 30 in which a large number of virtual object data (including various information such as text data and image data) is stored includes a virtual object data search unit 31, a communication control unit 32, and an antenna 33. Yes.

  The head-mounted augmented reality video presentation device 10 calculates the shooting range of the subject image captured by the imaging sensor 14b by the camera position / posture measurement unit 18 based on information acquired by the GPS 26 and the geomagnetic sensor 27. Then, the data of the direction and position corresponding to the shooting range of the subject image is transmitted to the communication control unit 32 of the database 30 through its own communication control unit 21 and antenna 20.

  The virtual object data search unit 31 searches the database 30 for virtual object data corresponding to the shooting range of the subject image based on the received data, and acquires the searched virtual object data from the head mounted augmented reality video presentation device 10. Return to

  The CPU 11 of the head-mounted augmented reality video presentation device 10 converts the received virtual object data into an icon and passes it to the image composition unit 19, and the image composition unit 19 displays this icon on the image captured by the image sensor 14b as a virtual display object. Are combined and displayed on the display screen 24.

  The head-mounted augmented reality video presentation device 10 includes the communication control unit 21 and the antenna 20, but these two functions may be substituted with the functions of the mobile phone.

  Since some mobile phones are equipped with a camera, the imaging control unit 15 and the camera 14 of the head-mounted augmented reality video presentation device 10 can be replaced with a camera mounted on the mobile phone. The GPS sensor 26 and the geomagnetic sensor 27 may be replaced with a GPS or geomagnetic sensor mounted on a mobile phone. When there is an overlapping function (camera function or GPS function in the above example) among the functions of the mobile phone and the head-mounted augmented reality video presentation device 10, the function of the head-mounted augmented reality video presentation device 10 is given priority. The function of the mobile phone is preferably used when a special instruction is input from the user.

  FIG. 2 is a diagram illustrating a display screen example of the head-mounted augmented reality video presentation device 10 illustrated in FIG. A landscape photographed by the camera 14 is displayed on the screen 24. The CPU 11 executes augmented reality (AR) application software and displays the virtual object data icons acquired from the virtual object database 30 as described above as virtual display objects (also referred to as air tags) 1a, 1b,. . Each air tag 1a, 1b,... Is displayed for each place such as a building corresponding to each of the subject images.

  For example, in order for the user to browse the information assigned to the air tag here, it is necessary to select a desired air tag from a plurality of displayed air tags on the display screen and to start application software for browsing. Thus, in order to develop a specific function for a desired air tag, the user needs to interact with the air tag.

  However, when the user wears the head-mounted augmented reality video presentation device 10, the display device (display screen in FIG. 1) 24 covers the entire vicinity of the user's eyes, and thus the user cannot see his / her hand. For this reason, a device such as a touch panel and hardware keys cannot be operated.

  Therefore, in this embodiment, an image of the user's fingertip is captured by the camera 14 or the camera of the mobile phone, and the image of the fingertip is displayed on the screen 24. When photographing the fingertip with the camera 14, the user places his / her fingertip in front of the camera 14 (in front of the head-mounted augmented reality video presentation device 10) so that the fingertip image 2 is captured simultaneously with the landscape. Then, as will be described next, the CPU 11 uses this fingertip image 2 to shift to AR user interface processing.

  FIG. 3 is a diagram illustrating how a user captures an image of his / her fingertip from a camera of a mobile phone. 1 and 2, the image of the fingertip of the landscape image is captured by the camera 14, but in FIG. 3, the image of the landscape and the image of the fingertip are captured by the camera attached to the mobile phone 5. Then, the CPU 11 of the head mounted augmented reality video presentation device 10 connected to the mobile phone 5 by cable captures the image data transmitted from the mobile phone 5 and displays it on the display screen 24.

  FIG. 4 is an explanatory diagram of a user interface processing procedure for operating the air tag using the user's fingertip image as a pointer. First, a captured image is captured (step S1), and the hue is observed to detect a skin color region (hand, fingertip) (step S2). Next, the arrangement coordinate data of the virtual display object (air tag) is acquired (step S3), and the linear distance between the contour of the skin color area and the air tag is calculated (step S4).

  In parallel with this, a difference image between the skin color detection result (step S2a) of the photographed image of the previous frame and the skin color detection result (step S2) of the current frame is generated (step S5), and the motion vector of the skin color region is calculated (step). S6).

  After step S4 and step S6, the process proceeds to step S7, where the operation instruction is recognized from the calculation results so far and the air tag to be operated is recognized. The recognition method will be described later. In the next step S8, a function based on the operation instruction is developed for the air tag to be operated. That is, if the operation instruction is an erasure instruction, the air tag to be operated is erased, and the process returns to step S1.

  In step S7, when the air tag to be operated cannot be recognized or when the operation instruction cannot be recognized, the process returns from step S7 to step S1.

  By performing such user interface processing, the air tag can be operated without using a special device such as a gyro sensor.

  5 to 7 are explanatory diagrams of the above-described operation instruction. FIG. 5 (a) shows a grasping operation, and FIG. 5 (b) shows a releasing operation. The increase area and decrease area of the skin color area are calculated based on the inter-frame difference of the skin color area 2, and the state where the area of the increase area is less than a certain period continues for a certain period of time. When the air tag 1 is in contact with the skin color region (fingertip image) 2 at two or more points, the air tag 1 is regarded as “picked”, and thereafter, the air tag 1 is moved to the fingertip as shown in the right diagram of FIG. The image 2 is moved following the movement. Alternatively, when the skin color region 2 is close to the air tag 1 by a certain amount or more, it may be regarded as “picked”.

  When the skin tag region 2 is moved at a certain speed or more as shown in the right diagram of FIG. 5B from the state in which the air tag 1 is “pinched” (FIG. 5B left diagram), the air tag 1 is “released”. The air tag 1 is not allowed to follow the skin color region 2.

  FIG. 6 is an explanatory diagram of an operation instruction for discarding (erasing) the air tag 1. The air tag 1 is discarded by superimposing the air tag 1 on an icon for expressing the function, in this case, the trash can icon 4 to be discarded, by following the movement of the fingertip image 2 while the air tag 1 is pinched.

  FIG. 7 is an explanatory diagram for performing an operation of removing the air tag 1 by hand. When the entire skin is photographed with a camera, the skin color area 2 moves at a certain speed or higher, and the air tag 1 exists in the route of the movement, the air tag 1 moves according to the moving speed and direction of the skin color area 2. Like to blow away.

  That is, an increase area and a decrease area of the flesh color area are calculated based on the difference between the captured image frames of the flesh color area 2. If the area of the increase area exceeds a certain value and the air tag is in the increase area, an acting force is generated on the air tag Let Alternatively, when the air tag is on the extension of the line segment connecting the center coordinates of the increasing area and the center coordinates of the decreasing area and in the direction from the decreasing area to the increasing area, an acting force is generated on the air tag.

  The acting force depends on the following parameters: the area of the increasing region (the larger the acting force is), the direction of the line segment connecting the center coordinates of the increasing region and the center coordinate of the decreasing region (force vector of the acting force) Let In this process, the user's action of removing the air tag is recognized, and an acting force is applied to the air tag.

  Thus, the user can express various functions to the air tag by operating his / her hand or fingertip like a gesture.

  In the above-described embodiment, the user operates the air tag using an image of his / her hand or fingertip as a pointer, but other than the fingertip or the like, another can be used as a pointer.

  FIG. 8 shows an example in which a camera-captured image of a stick-shaped marker is used as a pointer. The marker 6 is photographed by the camera of the mobile phone 5, and this photographed image is sent to the head mounted augmented reality video presentation device 10 and used as a pointer. The marker 6 describes a two-dimensional geometric pattern. In the example shown in the figure, a pattern meaning “download”, that is, an arrow for taking in a box is described.

  FIG. 9 is an explanatory diagram of a user interface processing procedure for operating the air tag using the marker image 6 shown in FIG. 8 as a pointer. First, the marker image 6 is captured from the camera (step S11). Then, in the next step S12, after the luminance component of the marker image 6 is binarized, the marker image 6 is detected and recognized by template matching with a previously registered marker image, and the recognized marker is detected. Calculate the position and orientation relative to the camera.

  In the next step S13, the arrangement coordinate data of the air tag 1 is acquired, and in step S14, the distance between the center of the marker image 6 and the air tag 1 is calculated.

  In parallel with this, in step S15, the marker position and orientation data of the captured image of the previous frame are held, and in step S16, the marker position and orientation data held in step S15 and the current frame in step S12 are stored. The motion vector of the marker image 6 is calculated from the difference between the marker position and orientation data.

  Then, based on the motion vector of the marker image and the distance between the marker image center and the air tag obtained in step S14, the air tag that is the operation target of the user is recognized and the operation instruction is also recognized ( Step S17).

  In the next step S18, the function is developed for the air tag to be operated based on the recognition result in step S17, and the process returns to step S11. If any of the recognition is disabled in step S17, the process returns to step S11.

  Thereby, it becomes possible to operate an air tag by intuitive operation, without using a special apparatus.

  FIG. 10 is an explanatory diagram of an operation instruction using the marker image 6. When the air tag 1 is to be downloaded, the marker image 6 is brought close to the air tag 1 and is brought close to the air tag 1 for a certain period of time or both are overlapped. Thereby, the data corresponding to the air tag 1 is downloaded.

  Each marker image has a pattern drawn on the front and back of a flat mount, and each has a different function associated with it. A trash can pattern is drawn on the back of the marker image 6. When the trash can pattern is photographed with the trash can pattern side facing the camera side, the state shown in FIG. 11 is obtained. The air tag can be discarded by bringing the trash box pattern close to the air tag and approaching the air tag for a certain period of time.

  In the above example, the two patterns of the marker have been described. However, if other patterns corresponding to various functions are prepared, various functions can be realized. A plurality of markers may be arranged not only on a flat mount but also on a polyhedral mount. Further, a configuration may be adopted in which patterns indicating various functions are switched and displayed by a user instruction on a display device such as a liquid crystal display.

  With such a configuration, various functions can be expressed on the air tag.

  FIG. 12 shows an example in which a light emitting body (LED) attached to the tip of a stick is used as a pointer. The LED 7 is photographed by a camera of a mobile phone and sent to the head mounted augmented reality video presentation device 10.

  FIG. 13 is an explanatory diagram of a user interface processing procedure for operating the air tag using the LED image 7 shown in FIG. 12 as a pointer. First, the LED image 7 is captured from the camera (step S21). In step S22, the luminance component of the LED image 7 is binarized, and then the LED point light source is detected. At this time, the detected color component or light emission pattern of the LED point light source may be recognized. Thereafter, the arrangement coordinate data of the air tag 1 is acquired in step S23, and the distance between the LED point light source 7 and the air tag 1 is calculated in step S24.

  In parallel with this, in step S25, the position of the LED point light source in the captured image of the previous frame is held, and in step S26, the motion vector of the LED point light source 7 is calculated from the difference from the captured image of the current frame. Based on the motion vector and the distance obtained in step S24, the air tag that is the operation target of the user is recognized and the operation instruction is also recognized (step S27).

  In the next step S28, the function is developed for the air tag to be operated based on the recognition result in step S27, and the process returns to step S21. If any of the recognition is disabled in step S27, the process returns to step S21.

  Thereby, an air tag can be operated by intuitive operation.

  14 and 15 are explanatory diagrams of operation instructions using an LED point light source. The LED point light source used in the present embodiment can change the emission color to yellow or red according to a user instruction, and the expression function is changed for each color. The expression function may be changed by changing the light emission pattern (for example, blinking speed) instead of the color. You may change an expression function with the combination of a color and a light emission pattern.

  The light emission color of the LED point light source 7 is changed to yellow, and the LED point light source is brought close to the air tag 1 as shown in FIG. Thereby, application software corresponding to the air tag is executed.

  The emission color of the LED point light source 7 is changed to red, and the LED point light source is brought close to the air tag 1 as shown in FIG. As a result, the air tag is replaced with another air tag.

  According to the present embodiment, various functions can be expressed with respect to the air tag only by moving the LED point light source.

  As described above, the following items are disclosed in this specification.

  The disclosed head-mounted augmented reality video presentation apparatus combines an imaging unit and the imaging unit to capture a pointer image as a part of a subject image and synthesize the subject image with a virtual display object indicating virtual object data. An image composition unit for generating an image, a head-mounted display unit for displaying the composite image, and a virtual display when the distance relationship between the pointer image and the virtual display object in the composite image satisfies a predetermined condition. Control means for causing the virtual object data corresponding to the object to exhibit a function corresponding to the pointer image.

  In the disclosed head-mounted augmented reality video presentation device, the pointer image is a marker image, and a two-dimensional geometric pattern for identifying the content of the function for the virtual object data is displayed on the marker. Is.

  In the disclosed head-mounted augmented reality video presentation device, the pointer image is an image of a point light source, and the different functions are identified by the light emission color or light emission pattern of the point light source.

  In the disclosed head-mounted augmented reality video presenting device, the control means is configured such that when the distance relationship between the pointer image in the composite image and the virtual display object satisfies a predetermined condition for a predetermined time, A function corresponding to the pointer image is developed for the virtual object data corresponding to the display object.

  The disclosed database is communicable with the head-mounted augmented reality video presentation device, and stores the virtual object data.

  According to the disclosed virtual display object operating method, an imaging unit captures a pointer image as a part of a subject image, generates a composite image by combining the subject image and a virtual display object indicating virtual object data, and generates the composite image. Is displayed on a head-mounted display means, and when the distance relationship between the pointer image in the composite image and the virtual display object satisfies a predetermined condition, the virtual object data corresponding to the virtual display object is A function corresponding to the pointer image is developed.

  The marker image may be a polyhedron image, and different two-dimensional geometric patterns may be displayed on each surface of the polyhedron.

  The marker is a display means, and a plurality of types of the two-dimensional geometric patterns to be displayed on the display means may be prepared and switched.

  According to each embodiment described above, even if the user's eyes are covered with a head-mounted display screen and the user cannot visually recognize his / her hand, the virtual display object in the composite image can be displayed without special sensors. It becomes possible to operate.

  The head-mounted augmented reality video presentation device according to the present invention easily manipulates a virtual display object in a video without using an expensive pointer device while displaying the augmented reality video on the head-mounted display device. Therefore, it is useful for spreading the head mounted augmented reality video presentation device.

1, 1a, 1b Virtual display object (air tag)
2 User fingertip image 5 Mobile phone 6 Marker image 7 LED point light source image 10 Head mounted augmented reality video presentation device 11 CPU
14 Built-in camera 19 Image composition unit 21 Built-in communication control unit 22 Pointer detection recognition unit 23 Pointer monitoring unit 24 Display screen 26 GPS
27 Geomagnetic Sensor 30 Virtual Object Database 31 Virtual Object Data Search Unit

Claims (7)

A pointer that is a point light source capable of changing at least one of the light emission color and the light emission pattern according to a user instruction;
Imaging means for capturing an image of the pointer;
Virtual object data acquisition means for acquiring virtual object data from a virtual object database;
Image synthesizing means for generating a synthesized image obtained by synthesizing the pointer image taken by the imaging means and the virtual display object obtained by converting the virtual object data into an icon;
Head-mounted display means for displaying the composite image;
Pointer function identification means for recognizing at least one of a color component and a light emission pattern of the point light source from the pointer image and identifying a function for the virtual object data;
A head-mounted augmented reality video comprising: control means for causing a function identified by the pointer function identifying means when a distance relationship between the pointer image in the composite image and the virtual display object satisfies a predetermined condition; Presentation device. The head-mounted augmented reality video presentation device according to claim 1,
The pointer is a head-mounted augmented reality video presentation device that is a light emitter attached to the tip of a stick. A head-mounted augmented reality video presentation device according to claim 1 or claim 2,
The function which the pointer function identification means identifies is a head-mounted augmented reality video presentation device including a function of replacing the virtual display object with another virtual display object. The head-mounted augmented reality video presentation device according to any one of claims 1 to 3,
The control means is a head mount that causes the function identified by the pointer function identifying means to be manifested when a state in which a distance relationship between the pointer image in the composite image and the virtual display object satisfies a predetermined condition continues for a certain period of time. Type augmented reality video presentation device.   An image of a pointer, which is a point light source capable of changing at least one of the light emission color and the light emission pattern according to a user instruction, is captured by the imaging unit, and the pointer image captured by the imaging unit and the virtual object data acquired from the virtual object database are displayed as icons. The synthesized virtual display object is synthesized to generate a synthesized image, the synthesized image is displayed on the head-mounted display means, and the distance relationship between the pointer image and the virtual display object in the synthesized image satisfies a predetermined condition. When satisfied, a virtual display object operating method for expressing a function for the virtual object data identified based on at least one of a color component and a light emission pattern of the point light source recognized from the pointer image. The virtual display object operating method according to claim 5,
The virtual display object operating method includes a function for replacing the virtual display object with another virtual display object. The virtual display object operating method according to claim 5 or 6,
A virtual display object operating method in which a function for the virtual object data is exhibited when a state in which a distance relationship between the pointer image in the composite image and the virtual display object satisfies a predetermined condition continues for a predetermined time.

Images