JP2011203984A - Navigation device, navigation image generation method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a navigation device, a navigation image generation method and a program, with improved convenience.SOLUTION: A user wears a wearable monitor 1 loaded with a camera 2 and a gyro sensor 4, navigation image data are generated on the basis of three-dimensional information obtained by performing feature point extraction and tracking to photographed image data photographed by the camera 2, positional information acquired by a position information receiver 3, and azimuth information acquired by the gyro sensor 4 and indicating the azimuth of the line of sight of the user, and they are superimposed on the photographed image data to generate display image data. By displaying the generated display image data on the wearable monitor 1, various kinds of navigation are performed.

Description

  The present invention relates to a navigation apparatus, a navigation image generation method, and a program that perform intuitive navigation using AR (Augmented Reality) technology.

In recent years, AR technology has attracted attention.
The AR technology is a technology that superimposes an existing space and various information such as video, music, and characters through information processing by a computer. Specifically, the AR technology specifically includes, for example, an optical glass element (for example, a car) that is superposed on a monitor with a real landscape photographed by a camera, CG (Computer Graphics), characters, etc., or placed in front of the user's eyes. This is realized by projecting and displaying images and characters there.

In the AR technology, there are the following two methods for displaying an image superimposed on an actual landscape.
The first is a method in which markers are prepared in advance in an actual landscape, markers in the actual landscape are extracted by image recognition, and images and characters corresponding to the positions of the markers are displayed. The marker is composed of a specific graphic pattern. The camera recognizes the marker, calculates the direction in which the marker is directed, and the like, and an image prepared in advance is displayed at the position of the marker in the landscape.
The second is a method of extracting and tracking feature points such as edges and corner points in an actual landscape, obtaining three-dimensional information of the feature points, and displaying images and characters at desired positions.

  Attempts have been made to apply such AR technology to navigation devices that perform intuitive navigation for the user.

No. 2004-048895

In the moving body navigation information display method described in Patent Document 1, first, a landscape in front of the vehicle is photographed by a camera having a camera angle that is substantially the same as the line of sight of the driver of the vehicle, and the moving direction of the vehicle by the GPS sensor and INS sensor Get data and vehicle speed data. Then, the image data generation unit names the road shape data and the road shape model, estimates the posture data, and generates image data for accurately combining and displaying the navigation information with the camera image.
With such a method, the driver of the vehicle can intuitively and accurately recognize the correspondence relationship between the navigation information and the photographed image or the actual scenery.

  In the prior art, there is a vehicle-mounted navigation device as in Patent Document 1. However, on the other hand, with regard to an apparatus for navigating a person who moves without boarding a vehicle, it is necessary to carry a special device (for example, PDA (Personal Digital Assistant) etc.). There was the disadvantage that there was only something that could be damaged.

  In order to eliminate such disadvantages, an object of the present invention is to provide a navigation device, a navigation image generation method, and a program with improved convenience.

  In order to achieve the above-described object, a navigation device according to a first aspect of the present invention includes a camera that captures a live-action moving image that is attached to a navigation target person, and a position information acquisition unit that acquires current position information of the navigation target person. And an image generation unit that analyzes the actual moving image captured by the camera and generates a navigation image for the navigation target person based on the analysis result and the current position information acquired by the position information acquisition unit; A monitor device that is mounted on the navigation target person and displays the navigation image generated by the image generation unit so as to overlap the field of view of the navigation target person, and the image generation unit captures an actual image captured by the camera. In a moving image, a small number of elements of the shape, movement, position, orientation, and size of the navigation target person's hand. Both by extracting one performs reception of input in accordance with the element, in response to the input, for generating the navigation image.

  A navigation image generation method of a second invention is a navigation image generation method of a navigation device having a camera, a position information acquisition unit, an image generation unit, and a monitor device, wherein the navigation image generation method is mounted on a navigation target person. A first step in which the camera captures a live-action moving image; a second step in which the position information acquisition unit acquires current position information of the navigation target person; and the image generation unit in the first step. A third step of analyzing the real moving image photographed in step S3, and generating a navigation image for the navigation target person based on the analysis result and the current position information acquired in the second step; The monitor device attached to the navigation target person has the navigation device generated in the third step in the field of view of the navigation target person. And a fourth step of displaying the action image in a superimposed manner, and in the third step, the image generation unit includes a shape of the hand of the navigation target person in the live-action moving image photographed by the camera, By extracting at least one of the elements of movement, position, orientation, and size, an input corresponding to the element is received, and the navigation image is generated according to the input.

  A program of a third invention is a program to be executed by a computer included in a navigation device, and includes a first procedure for capturing a real moving image, a second procedure for acquiring current position information of a navigation target person, A third moving image for analyzing the real moving image photographed in the first procedure and generating a navigation image for the navigation target person based on the analysis result and the current position information acquired in the second procedure. And a fourth procedure for displaying the navigation image generated in the third procedure superimposed on the field of view of the person to be navigated, and the camera executes the third procedure. In the photographed live-action moving image, the shape, movement, position, orientation, and size of the navigation target person's hand By extracting one even without performs reception of input in accordance with the element, in response to the input, to execute a fifth step of generating the navigation image, to the computer.

  ADVANTAGE OF THE INVENTION According to this invention, the navigation apparatus, the navigation image generation method, and program which improved convenience can be provided.

FIG. 1 is a diagram illustrating an example of a configuration of a navigation device. FIG. 2 is a diagram for explaining a specific example of display image data generated by the image processing unit. FIG. 3 is a flowchart illustrating an operation example during operation of the navigation device. FIG. 4 is a flowchart for explaining an operation example of the book navigation apparatus of the first example. FIG. 5 is a diagram illustrating an example of display image data in the book navigation apparatus of the first example. FIG. 6 is a block diagram illustrating an example of the configuration of the remote conference system of the second example. FIG. 7 is a diagram showing a specific example of display image data in the remote conference system of the second example. FIG. 8 is a flowchart for explaining an operation example of the remote conference system of the second example. FIG. 9 is a diagram illustrating an example of a user's field of view during navigation of the destination navigation device of the third example.

Hereinafter, the navigation apparatus 100 of embodiment of this invention is demonstrated.
FIG. 1 is a diagram illustrating an example of the configuration of the navigation device 100.
As shown in FIG. 1, the navigation device 100 includes a wearable monitor 1, a camera 2, a position information receiver 3, a gyro sensor 4, and an image processing unit 5.
The wearable monitor 1 is a device that has a shape like glasses or goggles and can be used as a monitor / display. The wearable monitor 1 has a screen (display) corresponding to the position of the lens in the eyeglasses, and displays image data (display image data to be described later) obtained by synthesizing a real landscape photographed by the camera 2 and a navigation image. It is a device that can display.

The camera 2 is a small camera such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and is mounted on the wearable monitor 1. For example, the camera 2 may be mounted on the vine portion of the wearable monitor 1 or the upper portion of the lens.
The camera 2 is installed so that the optical axis is as close as possible to the user's line of sight. That is, the camera 2 is installed so that the image data (captured image data) captured by the camera 2 is substantially the same as the image of the actual landscape that the user is viewing.
The camera 2 desirably captures moving image data in order to perform more detailed navigation.

The position information receiver 3 is a receiver that acquires position information related to the current position by a positioning method such as GPS (Global Positioning System) or LPS (Local Positioning System). When using GPS, position information such as latitude and longitude is acquired by communicating with a GPS satellite. When using LPS, current position information is obtained by communicating with a local positioning system.
The gyro sensor 4 is a measuring instrument that measures the speed and direction of an object. In the present embodiment, the gyro sensor 4 is used to measure in which direction the user's line of sight is directed and generate azimuth information. In order to measure the direction of the line of sight, it is desirable that the gyro sensor 4 is mounted on the wearable monitor 1 like the camera 2, for example.

The image processing unit 5 displays the image on the wearable monitor 1 based on the captured image data captured by the camera 2, the positional information obtained by the positional information receiver 3, and the user's gaze direction information measured by the gyro sensor 4. Display image data is generated.
The display image data generated by the image processing unit 5 is an image that gives great convenience to the user when the user superimposes it on an actual landscape viewed through the lens portion of the wearable monitor 1. The target for navigating display image data generated by the image processing unit 5 will be described in detail later.

Hereinafter, the image processing of the image processing unit 5 will be described.
The image processing unit 5 extracts feature points from photographed image data of a landscape that is photographed by the camera 2 and that is substantially the same as the actual landscape viewed by the user. The feature points are, for example, edges and corner points in the image. As a method for extracting feature points, for example, an existing corner detection algorithm may be used.
Next, feature point tracking is performed, and three-dimensional information of the captured image data captured by the camera 2 is acquired.
Then, display image data to be displayed on the wearable monitor 1 is generated based on the position information acquired by the position information receiver 3 and the direction information measured by the gyro sensor 4.

The image processing unit 5 creates display image data by superimposing three types of image data: captured image data captured by the camera 2, navigation image data for navigation, and operation image data for operation / input. . However, the navigation image data and the operation image data are not necessarily overlapped at the same time. That is, the display image data is created by superimposing photographed image data and at least one of navigation image data and operation image data.
The operation image data is image data of an operation device that can be operated intuitively, such as icons, operation keys, and buttons, displayed in the display image data. These operation devices extract, for example, three-dimensional information of a human hand by extracting feature points in the captured image data, thereby determining whether or not the hand is touching the operation device. If the input operation is accepted, it is determined that the input operation is not performed.
The determination in the processing is, for example, whether or not the tip of the hand is in a predetermined space determined based on the three-dimensional information of the hand in the display image data, particularly the three-dimensional information of the tip of the hand. To the standard.

A specific example of display image data generated by the image processing unit 5 will be described.
FIG. 2 is a diagram for explaining a specific example of display image data generated by the image processing unit 5.
FIG. 2A is a diagram showing display image data in which only operation image data is superimposed on photographed image data.
A screen 10 shown in FIG. 2A is a screen displayed on the wearable monitor 1 in a state where a user wearing the navigation device 100 intends to use the navigation of the navigation device 100 from now on.

As shown in FIG. 2A, on the screen 10, the icon 11 is displayed superimposed on the captured image data serving as the background.
As shown in FIG. 2A, the icons 11 are, for example, “Search” icons, “Navigation” icons, “Object” icons, “Settings” icons, etc., and operation input means suitable for the purpose for which the navigation device 100 is used. Arranged as. The icon 11 illustrated in FIG. 2A is an example, and the function of each icon displayed as the icon 11 may be set in advance according to the object to be navigated by the navigation device 100.

FIG. 2B is a diagram illustrating an example in which the icon 11 is operated by the user's hand 12.
FIG. 2B shows an example of display screen data when the user's hand 12 presses (operates) the “Settings” icon.
As described above, the image processing unit 5 extracts the shape of the “human hand” based on the captured image data captured by the camera 2 and calculates, for example, three-dimensional coordinate information of the tip portion of the hand. It is determined in advance which space in the virtual three-dimensional space of the display image data the “Settings” icon occupies, and the image processing unit 5 has the calculated tip of the hand in the space occupied by the icon. In this case, it is determined that the icon has been operated.
As shown in FIG. 2B, the image processing unit 5 causes the user to recognize that the operation has been accepted, such as shining an icon determined to have been operated or displaying the icon in a color different from the original color. Perform proper processing.

  The display image data illustrated in FIG. 2 is an example, and the image processing unit 5 may change the icon illustrated in FIG. 2A according to an object that the navigation device 100 navigates.

Next, an operation example of the navigation device 100 will be described.
FIG. 3 is a flowchart illustrating an operation example of the navigation device 100 during operation (when navigation is performed).

Step ST1:
The camera 2 captures an image and generates captured image data.
As described above, the camera 2 is installed on the wearable monitor 1 worn by the user of the navigation device 100 so that the optical axis is as close as possible to the user's line of sight. For this reason, the photographed image data photographed and generated by the camera 2 is almost the same as the scenery viewed by the user.
At the time of this step, the wearable monitor 1 worn by the user of the navigation device 100 displays the operation image data (icons, buttons, etc.) superimposed on the captured image data generated by the camera 2.
Step ST2:
The image processing unit 5 recognizes the three-dimensional position / shape of the user's hand in the captured image data generated by the camera 2 in step ST1.
Recognition of the hand in the captured image data is performed by a method such as first specifying the shape of the hand in the image by a technique such as template matching and then specifying the position of the hand, particularly the position corresponding to the fingertip. Just do it. Template matching is a technique for detecting a target shape and position by extracting preset template images (in the present embodiment, hand images of various shapes, etc.) from image data.

Step ST3:
The image processing unit 5 determines the contact between the hand and the icon based on the position of the user's hand recognized in step ST2 and the respective positions of the preset operation image data.
As described above, this contact determination is used when, for example, the tip part of the hand is in the icon occupation space based on the three-dimensional information of the tip part of the hand and the three-dimensional information of the icon occupation space. It is determined that the user's hand is in contact with the icon, that is, the user is about to operate the icon.
Step ST4:
The image processing unit 5 determines whether or not an operation requesting navigation is performed as a result of the contact determination between the hand and the icon performed in step ST3. The operation requesting navigation refers to, for example, a plurality of icons displayed on the wearable monitor 1 and an icon for starting navigation (for example, “Search” icon, “Navigation” icon, etc. in FIG. 2A). Means that the hand touched.
If it is determined in this step that an operation requesting navigation is performed, the process proceeds to step ST5, and if not, the process returns to step ST3.

Step ST5:
The image processing unit 5 performs feature point extraction of the captured image data.
As a feature point extraction method, as described above, an existing corner detection algorithm or the like may be used.
Step ST6:
The image processing unit 5 tracks the feature points extracted in step ST5.
Tracking is a technique for tracking where a point in the previous image has moved in the next image.
As a tracking method, for example, an existing technique such as a KLT (Kanade Lucas Tomasi) method may be used.

Step ST7:
The image processing unit 5 calculates three-dimensional information in the captured image data from the result of tracking in step ST6.
That is, based on the tracking result, the three-dimensional coordinates of each feature point in the photographed image data are calculated, so that the photographing area of the photographed image data is reconstructed three-dimensionally.
By this processing, it is possible to calculate how each object in the captured image data is positioned three-dimensionally.

Step ST8:
The position information receiver 3 acquires position information (latitude, longitude, etc.) regarding the current position using GPS or LPS.
Step ST9:
The gyro sensor 4 acquires azimuth information indicating the current user's line-of-sight direction.
Note that the position information / orientation information acquisition processing in step ST8 and step ST9 does not necessarily need to be executed after step ST7, and is preferably always performed while the navigation device 100 is being used by the user. Alternatively, in order to save power of the navigation apparatus 100, it is desirable to execute it every predetermined time.
In order to be used in the next step ST10, it is desirable to always obtain the latest position information and direction information.

Step ST10:
The image processing unit 5 performs matching of the three-dimensional information in the captured image data calculated in step ST7, the position information acquired in step ST8, and the orientation information acquired in step ST9.
Step ST11:
The image processing unit 5 uses the image data (navigation image data) for navigating the navigation target requested in step ST4 based on the three-dimensional information, position information, and direction information of the captured image data matched in step ST10. To generate.
That is, the image processing unit 5 grasps where the user requested to navigate in step ST4, and generates navigation image data for navigating the navigation object based on the position information and the direction information.
Note that information related to the object to be navigated (position, shape of the object to be navigated, navigation method up to the object, etc.) is stored in advance in a database or the like not shown in FIG. 1, and the image processing unit 5 is stored in the database. Information regarding the object to be navigated may be obtained by accessing.

Step ST12:
The image processing unit 5 generates display image data to be displayed on the wearable monitor 1 by superimposing the navigation image data generated in step ST11 on the captured image data.
Step ST13:
Wearable monitor 1 displays the display image data generated by image processing unit 5 in step ST13.
As a result, the user can be navigated to the navigation target.

  It is desirable that the above-described processing of steps ST1 to ST13 is always repeatedly executed while the user's navigation device 100 is in use. That is, the above-described processing is repeatedly executed until the user reaches the navigation target. In this way, navigation can be performed reliably.

In the above-described embodiment, the camera 2 and the gyro sensor 4 are described as being mounted on the wearable monitor 1 worn by the user. However, the location information receiver 3, the image processing unit 5, and the installation location of the database are described. There is no particular limitation.
In order to reduce the burden on the user of the navigation device 100, it is preferable that the user wears only the wearable monitor 1. For this reason, for example, the wearable monitor 1 may be provided with a wireless communication function, and the image processing unit 5 and the database may be provided at any place where the wearable monitor 1 can communicate. In this case, the position information receiver 3 needs to be mounted on the wearable monitor 1.
Alternatively, the position information receiver 3, the image processing unit 5, and the database are collected in a small casing and designed to be mounted on, for example, a user's chest, hip pocket, or waist. When using the navigation device 100, the user puts on the wearable monitor 1 and places a housing (including the position information receiver 3, the image processing unit 5, and the database) that can communicate with the wearable monitor 1 in a wired or wireless manner anywhere on the body. The navigation device 100 can be carried around. Even in such a case, if the casing is made small and light, the navigation device 100 can be carried around without burdening the user.

So far, the overall operation example of the navigation device 100 of the present embodiment has been described.
Hereinafter, a specific example of what navigation can be performed using the navigation device 100 will be described.

<First example>
As a first example, a book navigation apparatus 200 will be described.
The book navigation apparatus 200 has the same configuration as the navigation apparatus 100 described above. Therefore, in the following description, an operation example of the book navigation apparatus 200 will be described using the same reference numerals as the components of the navigation apparatus 100.

The book navigation device 200 is a device for navigating where a desired book is located in a facility with a large number of books such as a library.
Hereinafter, an operation example of the book navigation apparatus 200 will be described.

  FIG. 4 is a flowchart for explaining an operation example of the book navigation apparatus 200. In the flowchart shown in FIG. 4 to be described below, processing different from the operation example will be mainly described based on the operation example described in FIG.

Step ST21:
The image processing unit 5 of the book navigation device 200 determines whether or not an operation requesting navigation is performed as a result of the contact determination between the hand and the icon.
An example of the display image data in this specific example 1 is shown in FIG.
FIG. 5A is a diagram illustrating an image 500 that is an example of display image data when an operation requesting navigation is performed.

The image 500 is an image that the user who is using the book navigation apparatus 200 sees via the wearable monitor 1 in a facility where there are a large number of books, for example. As shown in FIG. 5A, the image 500 includes a background image 501, a hand image 502, and an icon 503.
Here, as described above, the background image 501 is an image of a landscape that is substantially the same as the landscape that the user originally sees, taken by the camera 2 having the same line of sight as the user mounted on the wearable monitor 1.
The image 500 includes an image (hand image) 502 of the user's hand. At this time, the image processing unit 5 recognizes the position and shape of the hand, and acquires three-dimensional position information (three-dimensional information) of the tip part of the hand (for example, the fingertip portion 504).
An icon 503 is also displayed in the image 500 at the same time. The image processing unit 5 determines contact with each other based on the three-dimensional information of the tip portion of the hand and the three-dimensional information of the space occupied by the preset icon 503.
FIG. 5A shows a situation where the “Search” icon is selected by the user. That is, in FIG. 5A, it is determined that the tip portion of the user's hand is in contact with the “Search” icon. As a result, the image processing unit 5 determines that the “Search” command has been input, and performs book search processing described below.
In FIG. 5A, only one “Search” icon is displayed as an example, but other types of icons may be displayed as shown in FIG.

In FIG. 5A, the display image data when the user operates the “Serach” icon has been described. However, even when the user is using the book navigation device 200, the book image data is displayed. In a situation where no search is performed, the icon 503 need not be displayed. In such a case, for example, only the background image may be displayed.
For example, if the icon 503 is superimposed on the display image data on the condition that the user's hand is detected in the display image data, and the user does not search for the book, The icon 503 is not displayed, and a situation in which the user's field of view is prevented by the icon does not occur. That is, only when the user wants to search for a book, puts his / her hand in the field of view (in the display image data), and on this condition, the icon 503 is displayed in the display image data and searches for the book. Is possible.

  If it is determined in step ST21 that an operation requesting navigation has been performed, the process proceeds to step ST22. If not, step ST21 is repeated.

Step ST22:
The image processing unit 5 displays display image data for performing a book search on the wearable monitor 1.
The display image data for performing a book search is, for example, an image in which 50 kana input icons are arranged, or an image in which icons for inputting 26 alphabets, numbers, and symbols are arranged.

Step ST23:
The image processing unit 5 determines whether or not a book search condition is input.
The image processing unit 5 accepts input of characters by the user by performing contact determination between the icon and the tip of the user's hand in the display image data for performing the book search displayed in step ST22. And the book which a user requests | requires a search is specified by receiving the input of a some character (character string).
Here, examples of the character string for specifying a book input by the user include a book name, an author name, a book genre, a publisher name, and a book classification code. In other words, the user can narrow down the candidates by inputting the name of the book or the book classification code for which the search is requested directly by inputting characters using the icons described above, or by inputting the author name, genre, or publisher name. The book to be searched is input to the book navigation apparatus 200 by any one of the methods for specifying the book.
If it is determined in step ST23 that a search condition has been input, the process proceeds to step ST24. If not, the process returns to step ST22.

Step ST24:
The image processing unit 5 specifies a search target, that is, a navigation target book, based on the search result input in step ST23.

Step ST25:
The image processing unit 5 accesses a database (not shown), and acquires location information of the navigation target book specified in step ST24, image data of the cover and spine, and the like. The book location information is, for example, information indicating where the search target book exists in the facility, and the information is input and stored in advance in a database by a person in charge of the facility, for example. The location information is composed of, for example, a floor number, a bookcase number, information indicating which level in the bookcase, and the like.

Step ST26:
The image processing unit 5 acquires the information on the navigation target book acquired in step ST25, the three-dimensional information obtained by performing feature point extraction and tracking on the captured image data acquired from the camera 2, and the position information receiver 3 acquires. Navigation image data is generated on the basis of the position information obtained and the direction information indicating the direction of the user's line of sight acquired by the gyro sensor 4.
Specifically, the image processing unit 5 determines where the user goes in the facility based on the location information of the user in the facility obtained by LPS, the location information of the navigation target book, and the direction information, for example. Navigation image data such as an arrow indicating whether it is acceptable is generated.

Step ST27:
The image processing unit 5 generates display image data to be displayed on the wearable monitor 1 by superimposing the navigation image data generated in step ST <b> 26 and the captured image data acquired by the camera 2.
FIGS. 5B and 5C are diagrams illustrating specific examples of display image data generated by the image processing unit 5.

FIG. 5B shows an example of display image data during navigation.
An image 600 shown in FIG. 5B is generated by superimposing the book information image 602 and the navigation image 603 on the background image 601.
The book information image 602 is an image that displays information (book name, author name, location where the book is located, etc.) related to the navigation target book identified in step ST24.
The navigation image 603 is an image for navigating the user, and an arrow shape is used here as an example. When the user moves according to the navigation image 603, the user can approach the navigation target book.

FIG. 5C shows an example of display image data when the navigation target is reached.
An image 700 shown in FIG. 5C is generated by superimposing a book information image 702, a cover image column 703, a cover image 704, and navigation target instruction images 705a and 705b on a background image 701.

As shown in FIG. 5C, the image 700 is an image intended to show the navigation target book to the user in an easily understandable manner.
The book information image 702 is an image that displays information on the navigation target book identified in step ST24.
The cover image column 703 is a space for displaying the cover image 704.
A cover image 704 is an image of a cover of a book to be navigated, and the image processing unit 5 reads and displays it when it is stored in advance in the database.
The navigation target instruction image 705a is an image for showing the navigation target book in an easy-to-understand manner to the user, and an arrow-shaped one is used here as an example.
The navigation target instruction image 705b is one of the variations of the image for clearly showing the navigation target book. Here, as an example, the back cover of the navigation target book is highlighted (it is easy to call the user's attention). Display in a fluorescent color).
As described above, the navigation target instruction images 705a and 705b are images that are arbitrarily displayed in order to easily show the location of the navigation target book to the user.
When the image processing unit 5 generates the navigation target instruction image 705b, for example, the back cover image acquired from the database is collated with the images of the back cover of a plurality of books in the photographed image data. If there is something to do, the book may be highlighted as a navigation target book.

According to the book navigation apparatus 200 that is the first specific example of the navigation apparatus 100 described above, for example, in a facility such as a library that holds a large number of books, position information about the current position of the user is received as position information such as LPS. The direction in which the user's line of sight is acquired is acquired by the gyro sensor 4 as direction information, and feature point extraction and tracking are performed on the captured image data that is almost the same as the user's field of view to obtain a three-dimensional image. Information is acquired, information about the navigation target entered by the user is acquired from the database, and navigation image data for navigating the user to the location of the navigation target book is generated based on this information and taken. Display image data is generated by superimposing the image data. The display image data is displayed on the wearable monitor 1 worn by the user.
For this reason, the user sees an image in which an image for navigation is superimposed on a real landscape via the wearable monitor 1, and it is easy to grasp the contents of navigation instantaneously and intuitively. .
In addition, the user needs to hold the wearable monitor 1 while using the book navigation apparatus 200, but also needs to hold a small casing containing the image processing unit 5, for example. Navigation that does not burden the user is possible. Furthermore, when the wearable monitor 1 is provided with a communication function and communicates with the image processing unit 5 and the database installed in the facility to exchange information about the navigation target, the burden on the user is further reduced. be able to.
Further, after navigating to the navigation target book, the navigation target book is indicated by an arrow, and the spine of the navigation target book is highlighted. For this reason, for example, even when many books are lined up on the bookshelf and it is not obvious where the navigation target is at first glance, the user can instantly grasp the location of the navigation target.

<Second example>
Next, the remote conference system 300 will be described as a second example.
In the remote conference system 300, the wearable monitor 1 is attached to a plurality of users, and images of other users prepared in advance are displayed superimposed on the captured image data, so that the other users are in front of each other. It is a system that can perform a remote conference.
FIG. 6 is a block diagram illustrating an example of the configuration of the remote conference system 300.

FIG. 6 shows a configuration example of the remote conference system 300 when a plurality of users A, B, C, and D conduct a remote conference.
As shown in FIG. 6, wearable monitor 1A used by user A, camera 2A, position information receiver 3A, gyro sensor 4A and wearable monitor 1B used by user B, camera 2B, position information receiver 3B, gyro The sensor 4B is connected to the image processing unit 5-1.
The image processing unit 5-1 is connected to the communication unit 6-1 that communicates with the communication unit 6-2 through the network 301. Here, the network 301 is a communication network such as the Internet or a LAN (Local Area Network).

The communication unit 6-2 is connected to the image processing unit 5-2.
The image processing unit 5-2 includes a wearable monitor 1C used by the user C, a camera 2C, a position information receiver 3C, a gyro sensor 4C, and a wearable monitor 1D used by the user D, a camera 2D, and a position information receiver 3D. Are connected to the gyro sensor 4D.

  In the teleconferencing system 300 shown in FIG. 6, wearable monitors 1 </ b> A to 1 </ b> D worn by the users A to D include users other than themselves (that is, remotes other than themselves) in the captured image data captured by the cameras 2 </ b> A to 2 </ b> D. Display image data generated by superimposing the attendee image data, which is an image indicating a conference attendee), and the object image data, which is an image indicating an object used in the conference, is displayed. The display image data displayed on each of the wearable monitors A to D reflects the position information acquired by the corresponding position information receivers 3A to 3D and the direction information acquired by the corresponding gyro sensors 4A to D, respectively. It is generated according to the situation.

  In FIG. 6, as described above, wearable monitor 1A, camera 2A, position information receiver 3A, gyro sensor 4A used by user A, wearable monitor 1B, camera 2B, position information receiver used by user B are shown. 3B and the gyro sensor 4B are connected to the image processing unit 5-1. This assumes that the users A and B are located in physically close positions (for example, the same room). Similarly, it is assumed that the users C and D are also physically close to each other. The users A and B and the users C and D are physically located far from each other.

An outline of the remote conference system 300 shown in FIG. 6 will be briefly described.
First, at the start of a remote conference, each of users A to D wears wearable monitors 1A to 1D, cameras 2A to 2D, position information receivers 3A to 3D, and gyro sensors 4A to 4D. Then, the photographed image data generated by the cameras 2A to 2D, which is close to the landscape viewed with their own line of sight, is displayed on their wearable monitors 1A to 1D. At this time, images corresponding to the participants of the remote conference, that is, the users A to D, are set in advance and registered in advance in a database not shown in FIG. The images corresponding to the users A to D are, for example, photographs of the users A to D, predetermined illustrations, and the like. Hereinafter, an image corresponding to each of the users A to D is referred to as avatar image data.
And each user's remark is delivered to all the users by the microphone and speaker which are not illustrated, for example. In this way, even if the attendee is at a remote place, it is possible to hold the conference as if it were in front of you.

  As described above, at the start of the teleconference, each wearable monitor 1A to D is generated by superimposing the avatar image data of the user other than the user on the photographed image data that is a real landscape corresponding to the user's line of sight. The displayed display image data is displayed. As described above, in the example shown in FIG. 6, the users A and B and the users C and D exist in the same room. For this reason, for example, there is a high possibility that the appearance of the user B is reflected in the captured image data generated by the camera 2A of the user A. Also in this case, as described above, as the display image data of the user A, the avatar image data corresponding to the user B may be displayed at the position where the user B exists. Similarly, the display image data of the user B only needs to display the avatar image data corresponding to the user A at the position where the user A exists.

  Here, as described above, the display image data of the user A includes the avatar image data of other users, that is, the users B to D. As described above, the user B's avatar image data is superimposed on the position of the user B in the captured image data. The avatar image data of the users C and D who are physically far away are also superimposed in the captured image data, but their positions are arbitrary. For example, a frame for displaying avatar image data is set at a predetermined position of the display image data (for example, the upper right portion and the upper left portion) regardless of what is shown in the captured image data, and the avatar image data is displayed in the frame. That's fine.

FIG. 7 is a diagram illustrating a specific example of display image data in the remote conference system 300.
FIG. 7A shows an image 800 as an example of display image data displayed on the wearable monitor 1A worn by the user A at the start of the remote conference.
As shown in FIG. 7A, the image 800 includes a background image 801, avatar image data 802, 804, 806, and avatar image data display frames 803, 805.
Since the image 800 shown in FIG. 7A is an example of display image data displayed on the wearable monitor 1A worn by the user A, the user is placed at the position of the user B in the same room as the user A. Avatar image data 802 corresponding to B is displayed. Then, the avatar image data 804 and 806 of the users C and D in the remote place are displayed in the avatar image data display frames 803 and 805 at a predetermined position in the image 800.

  Note that FIG. 7A shows display image data displayed on the wearable monitor 1A worn by the user A as an example, but the wearable monitor 1 worn by other users is also shown to each user. Display image data with corresponding contents is displayed. That is, for example, on the wearable monitor 1B worn by the user B, the avatar image data corresponding to the user A superimposed on the position where the user A exists and the avatar image data display frame similar to FIG. The avatar image data corresponding to the users C and D is displayed in the screen. The wearable monitor 1C worn by the user C corresponds to the avatar image data corresponding to the user D superimposed on the position where the user D exists in the same room, and corresponds to the users A and B in the avatar image data display frame. Avatar image data to be displayed is displayed.

  In the remote conference system 300 according to this specific example, with the above-described configuration, when a remote conference with a large number of people is performed, it is as if the attendees (other users) of the conference exist in front of the eyes via the wearable monitor 1. Since the conference can be performed while viewing the displayed image data, the remote conference can be preferably performed.

Next, a case where a specific object is listed on the agenda in the remote conference system 300 will be described.
FIG. 7B shows an image 900 which is an example of display image data displayed on the wearable monitor 1A worn by the user A when a specific object is handled as an agenda item.

As shown in FIG. 7B, the image 900 includes a background image 901, avatar image data 902, 904, 906, avatar image data display frames 903, 905, and an object image 907.
Among these, for the background image 901, avatar image data 902, 904, 906, and avatar image data display frames 903, 905, the background image 801, avatar image data 802, 804, 806, avatar image data shown in FIG. Since it is the same as the display frames 803 and 805, the description is omitted.

The object image 907 is, for example, an image of an object (object) listed on the agenda in a remote conference. It is desirable that the objects listed in the agenda for the remote conference are three-dimensionally analyzed in advance by the hand of the organizer of the remote conference to generate polygon data. Then, the polygon data of the object is stored in advance in a database (not shown in FIG. 6), and the polygon data of the object may be displayed as the object image 907 when the object is listed on the agenda.
Note that, for example, when a participant in the conference (any of users A to D) operates an icon as shown in FIG. 5A, the image processing units 5-1 and 5-2 determine whether the object is on the agenda. Determine whether.

  Here, since the object image 907 is polygon data as described above, it can be freely rotated and moved by the user's operation in the display image data as shown in FIG. 7B. . In this operation, for example, as in the contact determination between the user's hand and the icon described with reference to FIG. 5A, it is determined whether or not the user's hand has come into contact with the preset space occupied by the object. When the user is in contact, the input is made according to how the user's hand has moved. For example, when the user's hand moves to rotate the object, the object also rotates in the display image data, and when the user's hand moves to move the object, the display image data But the object moves.

  Thus, in the remote conference system 300, an object can be displayed as polygon data in display image data, and can be operated freely and intuitively. For example, the remote conference system 300 of this specific example can be used effectively when a certain product is presented in a remote conference.

Hereinafter, an operation example of the remote conference system 300 of this specific example will be described.
FIG. 8 is a flowchart for explaining an operation example of the remote conference system 300.

Step ST31:
The image processing units 5-1 and 5-2 determine whether or not attendees (users) of the remote conference are ready and wearable monitor 1 is attached and preparations for starting the remote conference are completed.
This determination is performed, for example, by determining whether or not an icon corresponding to the start of the remote conference has been operated by any user in his / her display image data.
In step ST31, when it is determined that the conference is ready to start, the process proceeds to step ST31. Otherwise, step ST31 is repeated.

Step ST32:
The image processing units 5-1 and 2 display the display image data in which the avatar image data of other users are superimposed on the captured image data on the wearable monitor 1 worn by each user.

Step ST33:
The image processing units 5-1, 2 determine whether to display an object.
This determination is performed, for example, by determining whether or not an icon for requesting the start of display of an object has been operated by any user in the display image data of the user.
If it is determined in step ST33 that the object is displayed, the process proceeds to step ST34. If not, step ST33 is repeated.

Step ST34:
The image processing units 5-1 and 5-2 display display image data in which polygon data of an object stored in advance in a database (not shown) is superimposed on captured image data on the wearable monitor 1 worn by each user.

Step ST35:
The image processing units 5-1 and 5-2 determine whether or not an operation for the object has been requested.
This determination is performed, for example, by determining whether an input requesting an operation for the object has been made by any user in the display image data of the user.
If it is determined in step ST35 that an operation on the object has been requested, the process proceeds to step ST36, and if not, step ST35 is repeated.

Step ST36:
The image processing units 5-1, 2 display image data obtained by superimposing the object image data obtained by performing the operation determined to be requested in step ST35 on the wearable monitor 1 worn by each user on the captured image data. Is displayed.

As described above, in the remote conference system 300 according to the present specific example, when a remote conference with a large number of people is performed by the above-described configuration, it is as if the attendee of the conference (other users) via the wearable monitor 1. Since the conference can be performed while viewing the display image data that exists in front of the user, the remote conference can be preferably performed.
In the remote conference system 300, an object can be displayed as polygon data in the display image data, and can be operated freely and intuitively. For example, the remote conference system 300 of this specific example can be used effectively when a certain product is presented in a remote conference.

In this specific example, the case where the users A and B and the users C and D are in the same room has been described as an example, but the present invention is not limited to this. For example, the users A to D may exist in different remote locations, and the present invention can be applied even when there are more or fewer users.
In this specific example, the case where there is only one object has been described, but the present invention is not limited to this. The number of objects is arbitrary, and polygon data may be prepared for each object in advance. In addition, when the details of the object are not well understood only by the polygon data, detailed texture image data may be generated in advance and pasted on the polygon data for display.

<Third example>
Hereinafter, the destination navigation apparatus 400 will be described as a third example.
The destination navigation apparatus 400 has the same configuration as the navigation apparatus 100 described above in terms of configuration. Therefore, in the following, when an operation example of the destination navigation apparatus 400 is described, the description will be made using the same reference numerals as the respective components of the navigation apparatus 100.

  The destination navigation device 400 is intended to present to a user wearing the wearable monitor 1 useful information when moving, such as directions to the destination, traffic jam information, estimated arrival time, and direction.

  In the destination navigation apparatus 400, unlike the other examples described above, it is assumed that the user moves on a public place such as a public road while wearing the wearable monitor 1. Therefore, as described above, in the method of displaying the display image data generated by superimposing the photographed image data and the navigation image data on the wearable monitor 1, if the wearable monitor 1 malfunctions, the user should It is possible that the view of the

  Therefore, particularly in the destination navigation apparatus 400 of this specific example, the lens portion of the wearable monitor 1 is configured by an optical glass element that transmits visible light, and navigation image data is configured to be projected onto the optical glass element. It is desirable. In this way, the user sees the navigation image data superimposed on the actual scenery seen through the optical glass element. For this reason, even when the wearable monitor 1 malfunctions, the navigation image data is not displayed, and a situation in which the user's view is blocked can be avoided.

FIG. 9 is a diagram illustrating an example of the user's field of view when the destination navigation apparatus 400 is navigating.
In the example illustrated in FIG. 9, a destination instruction image 1002, a direction image 1003, and information presentation images 1004 and 1005, which are navigation image data, are superimposed on an actual landscape 1001 that the user sees through the wearable monitor 1.

In the example illustrated in FIG. 9, for example, a landscape viewed by a user who has boarded a vehicle traveling on a road is assumed.
For this reason, the actual landscape 1001 in FIG. 9 is a landscape when the road is viewed from above the vehicle.
The navigation image data 1002 is an image for performing navigation to a destination input in advance by the user. In the example of FIG. 9, an arrow is displayed on the road, and it is possible to reach the destination by moving according to the arrow. A plurality of arrows constituting the navigation image data 1002 are displayed so that their sizes change so that the size can be intuitively grasped, and arrows closer to the destination are displayed smaller. Therefore, the user can instantly and intuitively know how to move from the current location to reach the destination.

The orientation image 1003 is an image generated so that the relationship between the user's field of view and the direction can be seen at a glance based on the location information acquired by the location information receiver 3 and the orientation information of the gyro sensor 4. From the azimuth image 1003, the user can grasp the direction corresponding to his / her direction.
Information presentation images 1004 and 1005 are images that display various types of information in the form of characters and symbols. In the example of FIG. 9, the arrival speed obtained from the average speed of the user calculated based on the time-dependent change of the position information acquired by the position information receiver 3 and the distance to the destination acquired by accessing the database. The scheduled time is displayed as an information presentation image 1004. In addition, real-time traffic jam information accessed on a network or the like is displayed as an information presentation image 1005.

As described above, according to the destination navigation apparatus 400 of this specific example, useful information is displayed on the wearable monitor 1 when moving, so that the user can move efficiently. . In particular, since information is projected onto the lens portion (optical glass element) of the wearable monitor 1 that transmits visible light, information regarding movement can be acquired while viewing the movement direction. Therefore, for example, unlike a conventional car navigation device, it is not necessary to look at a device that displays information by turning the line of sight from the front during driving of the vehicle, and safety is high. Even if the wearable monitor 1 malfunctions, the lens portion transmits visible light, so that the user's field of view is not blocked and safety is high.
Furthermore, if an input operation to the destination navigation device 400 such as a destination input or a setting change is accepted by an input operation using an icon as in the example of FIG. However, it is not necessary to shift the line of sight from the front, and safety can be improved.

As described above, according to the navigation device 100 of the present embodiment, the navigation image data up to the navigation target and the like on the wearable monitor 1 worn by the user are superimposed on the actual scenery that the user originally sees, and the like. The image data including useful information can be displayed.
In addition, when the user uses the navigation device 100, the user only has to hold the position information receiver 3, the image processing unit 5, and a small case housing the database in addition to wearing the wearable monitor 1. There is little burden on it. In particular, the camera 2, the position information receiver 3, and the gyro sensor 4 are mounted on the wearable monitor 1 and configured to exchange data by performing, for example, wireless communication with the image processing unit 5 installed at a predetermined position. Thus, the burden on the user can be further reduced.
In addition, since the navigation image data is sequentially changed according to the movement of the user's line of sight or the movement of the user's position, the user can be surely navigated to the navigation target.
Furthermore, according to the purpose of use of the navigation device 100, the operation corresponding to various navigation objects can be performed, for example, as in specific examples 1 to 3 described above, by appropriately processing the processing of the image processing unit 5. Is possible.

In addition, this invention is not limited to embodiment mentioned above.
That is, when implementing the present invention, various modifications, combinations, sub-combinations, and alternatives may be made to the components of the above-described embodiments within the technical scope of the present invention or an equivalent scope thereof.

  DESCRIPTION OF SYMBOLS 1 ... Wearable monitor, 2 ... Camera, 3 ... Position information receiver, 4 ... Gyro sensor, 5 ... Image processing part, 6 ... Communication part, 100 ... Navigation apparatus, 200 ... Book navigation apparatus, 300 ... Remote conference system, 301 ... Network, 400 ... Destination navigation device

Claims (5)

A camera that is attached to a navigation target person and that captures a live-action moving image;
A position information acquisition unit for acquiring current position information of the navigation target person;
An image generation unit that analyzes the actual moving image captured by the camera and generates a navigation image for the navigation target person based on the analysis result and the current position information acquired by the position information acquisition unit;
A monitor device that is mounted on the navigation target person and displays the navigation image generated by the image generation unit in a field of view of the navigation target person;
Have
The image generation unit extracts at least one of each element of the shape, movement, position, orientation, and size of the hand of the navigation target person in the live-action moving image photographed by the camera. A navigation device that receives an input according to an element and generates the navigation image according to the input. The navigation device according to claim 1, wherein the image generation unit generates a suitable navigation image by extracting a feature point in a real moving image photographed by the camera and analyzing a motion of the feature point. A gyro sensor for acquiring azimuth information related to a direction in which the navigation target person is facing;
The navigation according to claim 1, wherein the image generation unit generates the navigation image based on the azimuth information acquired by the gyro sensor, the analysis result of the actual moving image, and the position information acquired by the position information acquisition unit. apparatus. A navigation image generation method of a navigation device having a camera, a position information acquisition unit, an image generation unit, and a monitor device,
A first step in which the camera attached to the navigation target person captures a live-action moving image;
A second step in which the position information acquisition unit acquires current position information of the navigation target person;
The image generation unit analyzes the actual moving image photographed in the first step, and based on the analysis result and the current position information acquired in the second step, navigation for the navigation target person A third step of generating an image;
A fourth step in which the monitor device attached to the navigation target person displays the navigation image generated in the third step in a superimposed manner on the field of view of the navigation target person;
Have
In the third step, the image generation unit includes at least one of the shape, movement, position, orientation, and size elements of the navigation target person's hand in the live-action moving image captured by the camera. The navigation image generation method of receiving the input according to the said element by extracting, and generating the said navigation image according to the said input. A program to be executed by a computer included in a navigation device,
A first procedure for shooting a live-action moving image;
A second procedure for acquiring the current position information of the navigation target person;
A third moving image for analyzing the real moving image photographed in the first procedure and generating a navigation image for the navigation target person based on the analysis result and the current position information acquired in the second procedure. And the steps
A fourth procedure for displaying the navigation image generated in the third procedure in a superimposed manner on the field of view of the navigation target person;
To the computer,
In the third procedure, by extracting at least one of the shape, movement, position, orientation, and size elements of the navigation target person's hand in the live-action moving image captured by the camera, A fifth procedure for receiving an input corresponding to the element and generating the navigation image according to the input;
A program for causing the computer to execute.

Images