JP2009284175A - Calibration method and apparatus of display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To adjust an object position of an actual space and a pointer on a see-through display easily without a complicated optical system even when fixing methods of users are varied. <P>SOLUTION: In an apparatus in which a camera is arranged on a glasses part, a calibration marker is displayed on a top edge and a bottom edge of a display arranged on a lens part of the glasses, one image is obtained while each marker and a faraway object are superimposed, a value of a coordinate with a calibration object in an image is calculated by two images, and a calibration parameter in an up-and-down direction is found based on the coordinate value. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a display device calibration method and apparatus, and more particularly to a display device calibration method and apparatus for displaying a real-world image and a virtual space image in an overlapping manner. More specifically, the present invention relates to a vertical calibration method and apparatus for an apparatus in which a camera is arranged on a frame portion of glasses among glasses-type apparatuses represented by a transmissive head-mounted display.

As shown in FIG. 31, a conventional transmissive head-mounted display uses a prism and a half mirror, and has a method of optically matching a camera and a display unit (LCD) with a line of sight (for example, non-patent) Reference 1).
A. Takagi, S. Yamazaki, Y. Saito and N. Taniguchi, "Development of a stereo video see-through HMD for AR systems," Proc. ISAR 2000, pp.68-77, 2000

  However, in the above conventional method, since the line of sight is optically matched, it is necessary to accurately adjust the arrangement of the optical system, and there is no degree of freedom in the wearing method. For example, the wearing position is different from that of normal glasses. It was difficult to adjust the optical system in the case of deviation. In addition, the configuration in which the mirror is installed in front of the prism increases the space for forming the optical system. For example, it is difficult to mount in a size that can be worn daily, such as normal glasses. .

  Some applications that use head-mounted displays have some tolerance, for example, when the camera is attached to the eyeglass frame, the application can tolerate a deviation between the camera and the side of the eye. Conceivable. For an application that can tolerate a slight positional deviation, a simple calibration method is suitable even if the calibration accuracy is rough, rather than strict calibration using an optical system in the prior art.

  The present invention has been made in view of the above points, and does not require a complicated optical system, and even if there are variations in the wearing method of the user, the object position in the real space and the position of the pointer on the see-through display can be easily obtained. An object of the present invention is to provide a display device calibration method and apparatus capable of adjusting the above.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a calibration method for a transmissive display device having a camera using a calibration marker in the real world,
Read out the calibration image display coordinate data having at least two coordinate values of the coordinate value of the upper end of the transmissive display device and the coordinate value of the lower end as the calibration image display coordinate value from the storage means. The calibration image is displayed on the coordinate value specified by the calibration image display coordinate data on the transmissive display device (step 1).
Instruct the wearer of the display device to change the orientation of the face so that the calibration marker in the real world and the calibration image displayed on the display device appear to overlap (step 2),
Receive information indicating that the calibration marker in the real world and the calibration image displayed on the display device appear to overlap (step 3), capture the camera image (step 4),
A calibration marker in the real world is detected from the captured image of the camera image, and the coordinate value of the marker in the camera image is calculated (step 5).
A calibration parameter between the coordinate value in the camera image and the coordinate value of the transmissive display device is calculated using the coordinate value of the marker (step 6).

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 2) is a calibration apparatus for a transmissive display device 3 having a camera 4 using a calibration marker in the real world,
Storage means 170 that stores calibration image display coordinate data having at least two coordinate values of the coordinate value of the upper end of the transmissive display device and the coordinate value of the lower end as the calibration image display coordinate values. When,
Calibration image display for reading calibration image display coordinate data from the storage unit 170 and displaying a calibration image on the coordinate value specified by the calibration image display coordinate data on the transmissive display device 3 Control means 120;
Direction indicating means 161 for instructing the wearer of the display device to change the face direction so that the calibration marker in the real world and the calibration image displayed on the transmissive display device 3 appear to overlap each other. When,
Video capture means 110 for receiving information indicating that the calibration marker in the real world and the calibration image displayed on the display device 3 appear to overlap and capturing a camera image;
Marker coordinate value calculation means 150 for detecting a calibration marker in the real world from a captured image of a camera image and calculating a coordinate value of the marker in the camera image;
Parameter calculating means 160 for calculating a calibration parameter between the coordinate value in the camera image and the coordinate value of the transmissive display device 3 using the coordinate value of the marker;

  As described above, according to the present invention, the camera is installed at substantially the same height as the line-of-sight position, the calibration direction is limited to the vertical direction that most affects the superimposed display, and object selection is performed. It is not necessary to assemble a complicated optical system, and the mounting space can be reduced. In addition, the calibration using the marker makes it possible to easily adjust the object position in the real space and the position of the pointer on the see-through display even when the user's wearing method varies.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 3 shows an apparatus configuration according to the first embodiment of the present invention.

  The apparatus shown in the figure includes a camera 4, a see-through display 3, and a control unit 100.

  The control unit 100 includes a video capture unit 110, a display position control unit 120, an object image extraction unit 130, a pointer coordinate calculation unit 140, a marker coordinate calculation unit 150, a calibration unit 160, and a storage unit 170.

  The storage unit 170 includes a pointing feature amount used to detect a pointing operation when the wearer of the see-through display 3 points an object in the real world from a series of images, and a coordinate value (or image region) where the pointing operation is detected. And the pointer coordinate value (eg, the position of the center of gravity of the image area in which the pointing motion is detected is set as the pointer coordinate value) and the action associated with the pointing motion (eg, the coordinate value is determined, for example: The operation menu of the control unit 100 is displayed on the see-through display 3 in advance. There may be a plurality of types of pointing operations to be detected.

  The video capture unit 110 of the control unit 100 converts the image input from the camera 4 into a digital signal that can be analyzed, and outputs it as a camera image.

  The pointer coordinate calculation unit 140 detects a pointing operation using the pointing feature amount recorded in the storage unit 170 from the camera image captured by the video capture unit 110. The pointer coordinate value is calculated according to the information recorded in advance in the storage unit 170 in accordance with the detected operation and the detected position (coordinate value or image area) in the image.

  The object image extraction unit 130 receives the coordinate information from the pointer coordinate calculation unit 140, and extracts and views the object image from the camera image using the received coordinate value. For example, when only one coordinate is received from the pointer coordinate calculation unit 140, a predetermined area around the coordinate may be set as the extraction target area. When two coordinate values are given from the pointer coordinate calculation unit 140, a rectangular region having the two coordinate values as diagonal vertices may be used as the extraction target region.

  The display position control unit 120 converts the coordinate value received from the pointer coordinate calculation unit 140 into a display coordinate value using a conversion formula obtained by calibration, and displays it on the see-through display 3.

  The marker coordinate calculation unit 150 detects a marker used for calibration from the camera image captured by the video capture unit 110. The barycentric coordinate value in the image of the marker used for the detected calibration is output as the marker coordinate value.

  The calibration unit 160 performs calibration and calculates a correspondence expression between the coordinate position in the camera image and the display coordinate position on the see-through display.

  FIG. 4 is a flowchart of the calibration operation in the first embodiment of the present invention. The operation shown in the figure calculates the correspondence between the coordinate position in the camera image and the coordinate position of the display device.

  The types of markers in the real space used for calibration may be determined in advance. For example, it may be determined in advance that a bright point light source is used as a marker in the screen. Alternatively, an initial setting screen may be displayed on the see-through display 3, and the wearer of the see-through display 3 may select it in a menu format. For example, a plurality of options including “bright point light source” are displayed as marker options, and the wearer of the see-through display 3 selects the marker type from the menu, whereby the marker type is selected.

  The wearer of the see-through display 3 starts calibration by using the menu of the control unit 100 to start calibration. Hereinafter, an operation for calculating a calibration parameter when the wearer of the see-through display 3 designates a type to be used for calibration will be described.

  Step 101) The calibration unit 160 performs initial setting for calibration.

  First, the calibration unit 160 reads an algorithm for detecting a calibration marker designated by a wearer of the see-through display 3 (hereinafter referred to as a user) from the storage unit 170 and sets the algorithm in the marker coordinate calculation unit 150.

  Next, the calibration unit 160 acquires calibration image data and calibration image display coordinate value data on the see-through display from the calibration data stored in the storage unit 170. The calibration image display coordinate value data includes two or more coordinate values including two coordinate values: a coordinate value indicating one of the upper end positions of the see-through display 3 and a coordinate value indicating one of the lower end positions. Consists of. The subsequent processing from step 102 to step 105 is performed for all coordinate values included in the calibration image display coordinate value data.

  Step 102) The calibration unit 160 passes the calibration image data and the calibration image display chart value on the see-through display 3 to the display position control unit 120. The display position control unit 120 displays a calibration image on the see-through display 3.

  Step 103) The calibration unit 160 changes the orientation of the face so that the calibration image displayed on the see-through display 3 and the calibration marker in the real world (for example, a distant point light source) appear to overlap. The user is instructed to take a predetermined action when calibration markers in the real world (for example, distant point light sources) appear to overlap. As an instruction method to the user, for example, a message indicating the instruction content is displayed on the see-through display. In the case of an apparatus including a speaker, an instruction may be given by voice using the speaker.

  Step 104) The calibration unit 160 indicates that the user is in a position where the calibration image displayed on the see-through display 3 and the calibration marker in the real world (for example, a distant point light source) can be seen overlapping each other. A command indicating that a predetermined action has been taken is received, and a camera image at the time of receiving the command is acquired from the video capture unit 110.

  As a predetermined action indicating that the calibration image and the calibration marker in the real world are in a position where they appear to overlap, for example, the calibration image and the calibration marker in the real world overlap in advance. A pointing operation indicating that the camera is in a visible position may be determined, and the user may perform the pointing operation within the camera shooting range. In this case, a pointing feature amount corresponding to a predetermined action indicating that the calibration marker in the real world is at a position where it appears to overlap is stored in advance in the pointing feature amount of the storage unit 170, and the pointer coordinate calculation unit 140 It is detected whether or not this pointing operation has been performed, and the coordinate value at which this pointing operation has been performed. The calibration unit 160 receives whether or not the pointing operation has been performed from the pointer coordinate calculation unit 140 and the coordinate value on which the pointing operation has been performed. Upon receiving these pieces of information, the calibration unit 160 acquires a camera image from the video capture unit 110.

  Step 105) The calibration unit 160 inputs the image acquired in Step 104 to the marker coordinate calculation unit 150, acquires the marker coordinate value from the marker coordinate calculation unit 150, and outputs this value. The marker coordinate calculation unit 150 extracts a marker from the camera image captured in step 104 using the marker feature amount stored in the storage unit 170, and calculates the coordinate value of the marker in the camera image. Output.

  Step 106) It is determined whether or not the processing from Step 102 to Step 105 has been performed for all the coordinate values included in the calibration image display coordinate values. If there is a calibration image display coordinate value that has not been executed, the process returns to step 102. When the processing from step 102 to step 105 has been performed for all the calibration image display coordinate values, the process proceeds to step 107.

  Step 107) When the calibration image is displayed at the upper end of the see-through display 3, the coordinate position on the see-through display 3 where the calibration image is displayed at step 102 with respect to the two cases where the calibration image is displayed at the lower end Then, using the combination of the coordinate positions of the markers in the camera image detected in step 105 (two sets of combinations), a conversion parameter between the coordinates of the camera image and the see-through display 3 is calculated.

  Of all the coordinate values detected in step 106, the image is captured by the video capture unit 110 when the calibration image is displayed at the coordinate value at the upper end of the see-through display 3 and at the coordinate value at the lower end. The coordinate value on the obtained image is calculated.

  In FIG. 5, it is assumed that the upper end of the two extracted coordinate values is (xc1, yc1) and the lower end is (xc2, yc2).

The number of pixels in the vertical direction of the display device of the see-through display 3 is N. At this time, the calibration parameters α and β are
α = yc1
β = | yc1-yc2 | / N
Asking.

  Note that the number N of pixels in the vertical direction of the display device of the see-through display is stored in the storage unit 170 in advance.

At this time, when the pointer is displayed at the coordinate value of (xd, yd) of the see-through display 3, the coordinates where the object that the pointer appears to overlap as seen from the wearer of the see-through display are reflected in the captured image. The position (xc, yc) is
xc = xd
yc = yd × β + α
Can be obtained as

  By obtaining α and β, calibration of the camera coordinate system and the coordinate system of the see-through display is realized.

[Second Embodiment]
The display device is calibrated using the calibration technique of the present invention, and a process of detecting a specific shape of a human finger from a camera image is added to select an object in real space, and the selected object An example applied to an application for retrieving information (hereinafter referred to as real space information retrieval) is shown.

  FIG. 6 shows a configuration of a calibration apparatus according to the second embodiment of the present invention.

  The apparatus shown in FIG. 1 includes a glasses main body 1, a see-through display 3, a camera 4, a control device 10, and a wireless communication device 20.

  The user wears the glasses main body 1 and views the target object 2 through the see-through display 3. The camera 4 is arranged near the user's effect, captures an image close to the image seen by the user's effect, and sends the captured real space information to the control device 10. The control device 10 uses the movement information of the user's finger photographed by the camera 4 or the acceleration sensor information acquired from the wireless communication device 20 to transmit the image information including the object attribute information and the object position selected by the user to the wireless communication device. Send to 20. In addition to the obtained object attribute information and object image information, the wireless communication device 20 uses, as a search key, any one or both of the user position information acquired by GPS or the like and the time information built in the wireless communication device. Search for object information held by a remote server or wireless communication device 20. The search result is transmitted to the control device 10, and the control device 10 combines the display images of the search results and displays them on the see-through display 3. The user can view the search result related to the target object as information overlaid on the actual space in which the user is actually viewing.

  Examples of commercially available display devices (sometimes referred to as “video glasses”) that are glasses-shaped and have a transmissive display device installed on the lens portion of the glasses include the following.

・ Video Eyeglasses (Israel, LUMUS)
(http://www.lumus-optical.com/index.php?optioon=com_content&task=view&id=9&Itemid=15) "
・ Eye-Trek (Olympus)
(http://www.watch.impress.co.jp/av/docs/20011010/olymus.htm)
Next, the arrangement of the camera 4 and the see-through panel (display) 3 will be described.

  FIG. 7 shows an example of the arrangement of the camera and see-through panel of the present invention. As shown in the figure, in order to make the focal position of the user close to the focal position of the camera, the camera 4 is arranged on the effective side, and the focal position of the camera 4 is a depth close to the focal position of the user's eyes. Arrange so that Further, in order to reduce the vertical error in object selection and pointer display, the camera focus position, the center of the see-through display 3 and the focus position of the user's eyes are arranged to match in the vertical direction. In object selection, the vertical position between the see-through display 3 and the plane on which the object exists is used, and the pointer position on the see-through display matches the real-time object position. It is valid.

  FIG. 8 shows a configuration of a control device and a wireless communication device according to the second embodiment of the present invention.

  The control device 10 includes a video capture unit 11, an object image extraction unit 12, a pointer coordinate calculation unit 13, a display position control unit 14, and a display information generation unit 15.

  The wireless communication unit 20 includes a search control unit 21, a GPS function unit 22, a date acquisition unit 23, and a communication control unit 24.

  The video capture unit 11 of the control device 10 converts the image input from the camera 4 into a digital signal that can be analyzed.

  The object image extraction unit 12 uses the coordinate information from the pointer coordinate calculation unit 13 to extract digital information of an object to be searched.

  The pointer coordinate calculation unit 13 has a function of detecting a finger action from the digital information received from the video capture unit 11 and specifying an object position selected by the user or a selected menu position.

  FIG. 9 is a diagram (No. 1) for explaining the finger detection algorithm in the pointer coordinate calculation unit according to the second embodiment of the invention.

  As shown in image A in the figure, after closing the fingertip in front of the camera being photographed, the index finger and thumb are placed so as to sandwich the object without changing the position of the base of the finger as in image B. open. In this action, since the finger is in front of the background and the amount of change in the image is large compared to the background, the movement information between the image A and the image B is displayed as shown in the image C. Can be detected. As shown in image C, the end points (the portions indicated by black circles) where the movement was maximum are extracted as the upper and lower ends of the object.

  In addition, for the detection of finger movement information, for example, the method proposed in “Eijijio Technical Report, VIS2001-103, Vol.25, No. 85, pp.47-52 (2001)” is used. (Http://www.is.aist.go.jp/weavy/paper/distribution/2001/ITE2001-kurata.pdf).

  FIG. 10 is a diagram (No. 2) for explaining the finger detection algorithm in the pointer coordinate calculation unit according to the second embodiment of the invention.

  The difference from the algorithm shown in FIG. 9 is that a marker that can be easily distinguished from the background color is attached to the finger so that the movement of the finger can be easily detected. Like the image A shown in FIG. 9, after closing the fingertip in front of the camera that is shooting the field of view, the index finger and the thumb are sandwiched between the objects without changing the position of the base of the finger as in the image B. open. A marker portion is extracted from the photographed image as image C, and the upper and lower ends of the object can be detected from the upper and lower ends of the marker.

  FIG. 11 is a diagram for explaining a pointer coordinate determination algorithm of the pointer coordinate calculation unit according to the second embodiment of the present invention.

  In the following description, the vertical phrase with respect to the see-through display 3 is taken as the X axis, and the vertical axis parallel to the see-through display 3 is taken as the Y axis.

(X0, Y0) are the focal positions of the eyes and the camera 4. In this description, it is assumed that the focal positions of the eyes and the camera 4 are substantially the same on the X axis and the Y axis. X1 indicates the position on the X-axis of the see-through display 3 included in the glasses 4. X3 indicates a position on the X axis on the captured image. Y0 indicates the position on the Y-axis on the extension of the perpendicular line dropped from the focal position to the see-through display 3. (X3, Y1) and (X3, Y2) indicate the coordinates of the upper and lower ends of the object. (X1, y1) and (X1, y2) are coordinates (X3, yc1) and (X3, yc2) corresponding to the top and bottom edges of the object on the see-through display 3, and are the images captured from the camera 4 The positions on the X3 axis corresponding to the upper end (X1, a1) and the lower end (X1, a2) of the see-through display 3 are shown.
The camera image indicates an image acquired by the camera 4 attached to the glasses main body 1.

  The position of the finger detected by the algorithm shown in FIGS. 9 and 10 corresponds to the position of the black circle on the camera image. The relationship between the position of the black circle on the camera image and the position of the black circle on the see-through display 3 can be determined as follows.

Y1 = y1 × β + α
Y2 = y2 × β + α
As α and β, those calculated in the first embodiment are used. In this example, N = | a ₁ −a ₂ |. Therefore, the pointer position to be displayed can be calculated as follows from the position of the finger in the image.

y1 = (Y1-α) / β
y2 = (Y2-α) / β
The display position control unit 14 adjusts and combines the display position of the image received from the pointer coordinate calculation unit 13.

  The display information generation unit 15 converts the search result information received from the search control unit 21 of the wireless communication device 20 into image information.

  The search control unit 21 of the wireless communication device 20 uses the digital information received from the object image extraction unit 12 of the control device 10, the position information received from the GPS function unit 22, and the date / time information received from the date / time acquisition unit 23 to Information retrieval instruction and result reception are performed. Also, an instruction for processing such as character recognition is given.

  The communication control unit 24 communicates with a server at a remote location according to an instruction from the search control unit 21.

  Next, the operation in the above configuration will be described.

  FIG. 13 is a flowchart of object selection in the second embodiment of the present invention, FIG. 14 is a diagram for explaining the screen image of the operation, and P0 in FIG. The initial position of the pointer when the appearance of the finger is detected is shown. P1 and P2 in FIG. 2B are the positions of the tips of the index finger and the thumb obtained by detecting the action of the finger, and P3 is a position for displaying an image captured by the object image extraction unit 12 in the marked area. Indicates.

  Step 301) When the user holds his / her finger in front of the effect, the pointer coordinate calculation unit 13 detects the appearance of the finger, and the display position control unit 14 displays the pointer at P0.

  Step 302) The user virtually grasps the pointer by moving the index finger and the thumb to the position where the pointer is visible and aligning both fingers.

  Step 303) With the thumb and index finger closed, the pointer is moved to the object position to be searched, and the finger is opened at the object position.

  Step 304) The pointer coordinate calculation unit 13 detects the action of opening the finger, and determines the upper end P1 and the lower end P2 from the positions of the index finger and the tip of the thumb.

  Step 305) The display position control unit 14 displays a mark in a circular or square area surrounded by the upper end P1 and the lower end P2.

  Step 306) The image of the marked area is captured by the object image extraction unit 12.

  Step 307) The captured image is displayed at the position P3.

  FIG. 14 is a flowchart of an object search operation in the second embodiment of the present invention, and FIGS. 15 and 16 are image diagrams of object search. Q0 in FIG. 15A indicates the display position of the target object type list selection screen, and Q1 in FIG. 15B changes and displays the background color of the selected desired object type. Indicates the position. R0 in FIG. 16C indicates the display position of the search result. In FIG. 16D, R1 indicates the position of the selected desired object candidate, and S1 indicates the display position of the object information.

  Step 401) When the user closes the finger again after selecting the object by the processing of FIG. 12, the object image extraction unit 12 detects the action of closing the finger again, and the display position control unit 14 detects the target object. The type list selection screen is displayed at the display position Q0.

  Step 402) The user grasps the pointer with a finger and moves the pointer to Q0.

  Step 403) The user opens the finger and closes it again to select the desired object type Q1.

  Step 404) The search control unit 21 of the wireless communication device 20 acquires the current date and time and location from the date and time acquisition unit 23 and the GPS function unit 22.

  Step 405) The search control unit 21 of the wireless communication apparatus 20 determines the target DB from the object type, and searches for object information from the captured image, date / time, and location.

  Step 406) Upon receiving the search result from the search control unit 21, the display information generation unit 15 of the control device 20 converts it into image information, and outputs candidates in descending order of similarity to the position of R0 of the see-through display 3.

  Step 407) Grasp the pointer with a finger, move the pointer to R1 of the see-through display 3, open the finger, and close it again to display detailed information on S1 of the see-through display 3.

  Next, an operation when a character or a signboard is selected as the object type will be described.

  FIG. 17 is a flowchart of the character recognition operation in the second embodiment of the present invention, and FIGS. 18 and 19 are image diagrams of character recognition. Q0 in FIG. 18A indicates the display position of the target object type list selection screen, and Q1 in FIG. 18B changes and displays the background color of the selected desired object type. Indicates the position. R0 in FIG. 19C indicates the display position of the search result, R1 in FIG. 19D indicates the position of the selected desired object candidate, and S1 indicates the display position of the object information.

  Step 501) An image including the target object is displayed, an action of closing the finger again is detected, and a selection screen of the target object type list is displayed at the display position Q0 (FIG. 18A).

  Step 502) The user pinches the pointer with a finger and moves the pointer to Q0.

  Step 503) A desired object type Q1 (signboard or label (FIG. 18B)) is selected by opening and closing the finger again.

  Step 504) Character recognition is performed on the target object area, and recognition result candidates are displayed in R0 (FIG. 19C) in descending order of reliability.

  Step 505) Display candidates in descending order of similarity in the area of R0.

  Step 506) The pointer is pinched with a finger, and the pointer is moved to R1 (FIG. 19D).

  Step 507) By closing the finger again, a search for merchandise, a store, a station, etc. is performed, and the search result is displayed in S0 (FIG. 19D).

[Third Embodiment]
This embodiment is different from the second embodiment in that an acceleration sensor and an operation unit are provided in the configuration of the wireless communication apparatus 20 in FIG. 8 in the second embodiment described above.

  FIG. 20 shows a configuration of a control device and a wireless communication device according to the third embodiment of the present invention. In this figure, the same components as those in FIG.

  The wireless communication device 20 shown in FIG. 20 has a configuration in which an acceleration sensor 25 and an operation unit 26 are added to the configuration of FIG. As shown in FIG. 21, the acceleration sensor 25 and the operation unit 26 are provided at the bottom of the touch panel 34, a determination button 31 for the user to determine the pointer position, and a cancel button 32 for releasing and resetting the pointer position. The configuration includes a home position button 33 for re-selecting an object from the beginning, and an acceleration sensor 25 and a GPS function unit 22 provided above the touch panel 34.

  The pointer coordinate calculation unit 13 in FIG. 8 detects the action of the finger and specifies the object position selected by the user or the selected menu position. In the present embodiment, the motion information from the acceleration sensor 25 is specified. Thus, the object position selected by the user or the selected menu position is specified.

  The acceleration sensor 25 detects the movement of the wireless communication device 20 and sends it to the pointer coordinate calculation unit 13. The operation unit 26 detects that the user has pressed the determination button 31 and sends operation information to the pointer coordinate calculation unit 13. The pointer coordinate calculation unit 13 projects and converts the movement from the acceleration sensor 25 into a see-through display planar movement.

  Next, the operation of the pointer coordinate calculation unit 13 in the present embodiment will be described.

  FIG. 22 shows an algorithm for determining an image extraction range from pointer coordinates in the pointer coordinate calculation unit according to the third embodiment of the present invention.

  In the case of the configuration shown in FIG. 20, instead of detecting the pointer position from the camera image, the acceleration sensor 25 attached to the wireless communication apparatus 20 moves the pointer displayed on the see-through display 3 to select the object. Therefore, it is necessary to determine the corresponding object position on the camera image from the pointer position. (X0, Y0) are the focal positions of the eyes and the camera 4. In the present invention, it is assumed that the focal positions of the eyes and the camera 4 are substantially the same on the X axis and the Y axis. X1 indicates a position on the X-axis of the see-through display 3 included in the glasses 1. X3 indicates the position on the X axis where the object exists. Y0 indicates a position on the Y-axis on an extension line of a perpendicular line dropped from the focal position to the see-through display. (X3, Y1) and (X3, Y2) indicate the coordinates of the upper and lower ends of the object. (X1, y1) and (X1, y2) indicate coordinates corresponding to the upper end and the lower end of the object on the see-through display 3. (X3, yc1) and (X3, yc2) are positions on the X3 axis corresponding to the upper end (X1, a1) and the lower end (X1, a2) of the see-through display 3 among the images captured by the camera 4. Show. The camera image indicates an image image acquired by the camera 4.

  The coordinates of (X1, a1) and (X1, a2) are determined when the user presses the determination button of the wireless communication device 20. The position of the object on the camera image is calculated as follows.

Y1 = y1 × β + α
Y2 = y2 × β + α
Note that α and β are those calculated in the first embodiment. In this example, N = | a ₁ −a ₂ |.

  FIG. 23 is a flowchart at the time of object selection in the second embodiment of the present invention, FIG. 24 is an image diagram of object selection, and P0 in FIG. Indicates the initial position of the pointer when pressed. P1 and P2 in FIG. 5B are the positions of the upper and lower ends of the object set by moving the pointer in the wireless communication device 20 and pressing the enter button 31, and P3 is an image of the marked area displayed as an object image extraction unit. 12 shows a position for displaying the captured image.

  Step 601) When the user presses the enter button 31 of the wireless communication device 20, a pointer is displayed on P0 (FIG. 24A).

  Step 602) By moving the wireless communication device 20 with the acceleration sensor 25, the pointer is moved left and right and up and down to the object position to be searched.

  Step 603) Based on information from the acceleration sensor 25, the pointer moves the display position to P1 (FIG. 24B).

  Step 604) The determination button 31 of the wireless communication device 20 is pressed at the upper end P1 of the object to determine the upper end of the object.

  Step 605) Based on the information from the acceleration sensor 25, the center position of the pointer image is moved to P2 (FIG. 24B).

  Step 606) Again, the wireless communication device 20 with the acceleration sensor 25 is moved to the lower end P2 of the object, P2 is determined, and a circle or square surrounded by P1 and P2 to indicate the selected area The mark is displayed in the area.

  Step 607) The image of the area surrounded by the determined P1 and P2 is captured as the target object, and the captured image is displayed at the position P3 (FIG. 24B).

  Step 608) A mark indicating selection is displayed in an area surrounded by P1 and P2.

  Next, an example during an object search operation will be described.

  FIG. 25 is a flowchart at the time of object search in the third embodiment of the present invention, and FIGS. 26 and 27 are image diagrams of object search. Q0 in FIG. 26A indicates the display position of the selection screen for the target object type list. Q1 in FIG. 26B indicates a position where the background color of the selected desired object type is changed and displayed. In FIG. 27C, R0 indicates the display position of the search result. In FIG. 4D, R1 indicates the position of the selected desired object candidate, and S1 indicates the display position of the object information.

  Step 701) When an image including the target object is displayed, a selection screen of the target object type list is displayed at the display position Q0 (FIG. 26A).

  Step 702) The wireless communication device 20 with the acceleration sensor 25 is moved, and the pointer is moved to Q0.

  Step 703) The determination button 31 of the wireless communication apparatus 20 is pressed to select a desired object type Q1 (FIG. 26B).

  Step 704) The search control unit 21 acquires date / time and location information from the date / time acquisition unit 23 and the GPS function unit 22.

  Step 705) The search control unit 21 determines the target DB from the object type, and searches for candidates from the captured image, date / time, and location.

  Step 706) The display information generation unit 15 outputs candidates in descending order of similarity at the position of R0 (FIG. 27C).

  Step 707) The wireless communication device 20 with the acceleration sensor 25 is moved, and candidates are output in descending order of similarity at the position of the pointer R0 (FIG. 27C).

  Step 708) When the wireless communication device 20 is moved, the pointer is moved to R1 (FIG. 27D), and the determination button 31 of the wireless communication device 20 is pressed, detailed information is displayed in S0 (FIG. 27D). To do.

  Next, a case where a character or a signboard is selected as the object type will be described.

  FIG. 28 is a flowchart of the character recognition operation according to the third embodiment of the present invention, and FIGS. 29 and 30 show screen images during the character recognition operation. Q0 in FIG. 29A indicates the display position of the target object type list selection screen, Q1 in FIG. 29B indicates the position where the background color of the selected desired object type is changed, and displayed. R0 in FIG. 30C indicates the display position of the search result, R1 in FIG. 30D indicates the position of the selected desired object candidate, and S1 indicates the display position of the object information.

  Step 801) A target object type list selection screen is displayed at the display position Q0 (FIG. 29A).

  Step 802) The wireless communication device 20 with the acceleration sensor 25 is moved, and the pointer is moved to Q0.

  Step 803) The determination button 31 of the wireless communication device 20 is pressed to select a desired object type Q1 (signboard or label).

  Step 804) The object image extraction unit 12 extracts a signboard or a label candidate from the target object area.

  Step 805) Character recognition is performed on the target object region, and recognition result candidates are displayed in R0 (FIG. 30C) in order of reliability.

  Step 806) The wireless communication device 20 with the acceleration sensor 25 is moved, and the pointer is moved to R1 (FIG. 30D).

  Step 807) When the determination button 31 of the wireless communication device 20 is pressed, the search control unit 21 performs translation or searches for products, stores, stations, and the like.

  Step 808) The display information generating unit 15 displays the translation result and the search result on S0 (FIG. 30D).

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a calibration technique.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. 1 is a configuration diagram of a calibration device according to a first embodiment of the present invention. It is a flowchart of the calibration operation | movement in the 1st Embodiment of this invention. It is a figure for demonstrating the coordinate value in the 1st Embodiment of this invention. It is a block diagram of the calibration apparatus in the 2nd Embodiment of this invention. It is an example of the camera and see-through panel arrangement | positioning in the 2nd Embodiment of this invention. It is a block diagram of the control apparatus and radio | wireless communication apparatus in the 2nd Embodiment of this invention. It is FIG. (1) for demonstrating the finger detection algorithm in the pointer calculation part in the 2nd Embodiment of this invention. It is FIG. (2) for demonstrating the finger detection algorithm in the pointer calculation part in the 2nd Embodiment of this invention. It is a figure for demonstrating the pointer display position determination algorithm of the pointer coordinate calculation part in the 2nd Embodiment of this invention. It is a flowchart of the object selection operation | movement in the 2nd Embodiment of this invention. It is a screen image of the object selection operation | movement in the 2nd Embodiment of this invention. It is a flowchart of the object search operation | movement in the 2nd Embodiment of this invention. It is a screen image (1 of s) of the object search operation in the second exemplary embodiment of the present invention. It is the screen image (the 2) of the object search operation | movement in the 2nd Embodiment of this invention. It is a flowchart of the character recognition operation | movement in the 2nd Embodiment of this invention. It is the screen image (the 1) of the character recognition operation | movement in the 2nd Embodiment of this invention. It is the screen image (the 2) of the character recognition operation | movement in the 2nd Embodiment of this invention. It is a block diagram of the control apparatus and radio | wireless communication apparatus in the 3rd Embodiment of this invention. It is an example of arrangement | positioning of the radio | wireless communication apparatus in the 3rd Embodiment of this invention. It is a figure for demonstrating the image extraction process from the pointer position in the 3rd Embodiment of this invention. It is a flowchart at the time of the object selection in the 3rd Embodiment of this invention. It is a screen image of the object selection operation | movement in the 3rd Embodiment of this invention. It is a flowchart at the time of the object search in the 3rd Embodiment of this invention. It is the screen image (the 1) of the object search operation | movement in the 3rd Embodiment of this invention. It is a screen image (the 2) of the object search operation | movement in the 3rd Embodiment of this invention. It is a flowchart of the character recognition operation | movement in the 3rd Embodiment of this invention. It is the screen image (the 1) of the character recognition operation | movement in the 3rd Embodiment of this invention. It is the screen image (the 2) of the character recognition operation | movement in the 3rd Embodiment of this invention. It is a block diagram of the conventional transmission type head mounted display.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Glasses body 2 Target object 3 Transparent display device, see-through display 4 Camera 10 Control apparatus 11 Image | video capture part 12 Object image extraction part 13 Pointer coordinate calculation part 14 Display position control part 15 Display information property western part 20 Wireless communication apparatus 21 Search control Unit 22 GPS function unit 23 date and time acquisition unit 24 communication control unit 25 acceleration sensor 26 operation unit 31 determination button 32 cancel button 33 home position button 34 touch panel 100 control device 110 video capture unit, video capture unit 120 image display control unit for calibration , Display position control unit 130 object image extraction unit 140 pointer coordinate calculation unit 150 marker coordinate value calculation unit, marker coordinate calculation unit 160 parameter calculation unit, calibration unit 161 direction indicator 170 storage unit, the storage unit

Claims (2)

A calibration method for a transmissive display device having a camera using a calibration marker in the real world,
Read out the calibration image display coordinate data having at least two coordinate values of the upper end of the transmissive display device and any of the lower end coordinate values as the calibration image display coordinate values from the storage means. , Displaying a calibration image on the coordinate value specified by the calibration image display coordinate data on the transmission type display device,
Instructing the wearer of the display device to change the orientation of the face so that the calibration marker in the real world and the calibration image displayed on the display device appear to overlap,
Receiving information indicating that the calibration marker in the real world and the calibration image displayed on the display device appear to overlap, capturing a camera image;
Detecting a calibration marker in the real world from the captured image of the camera image to calculate the coordinate value of the marker in the camera image,
A calibration method for a display device, wherein a calibration parameter between a coordinate value in the camera image and a coordinate value of the transmissive display device is calculated using the coordinate value of the marker. A transmissive display device calibration apparatus having a camera using a calibration marker in the real world,
Storage means for storing calibration image display coordinate data having at least two coordinate values of the upper end of the transmissive display device and any of the lower end coordinate values as calibration image display coordinate values; ,
A calibration image for reading the calibration image display coordinate data from the storage means and displaying the calibration image on the coordinate value specified by the calibration image display coordinate data on the transmissive display device. Display control means;
Direction indicating means for instructing the wearer of the display device to change the orientation of the face so that the calibration marker in the real world and the calibration image displayed on the transmissive display device appear to overlap. When,
Video capture means for receiving information indicating that the calibration marker in the real world and the calibration image displayed on the display device appear to overlap and capturing a camera image;
Marker coordinate value calculating means for detecting a calibration marker in the real world from the captured image of the camera image and calculating a coordinate value of the marker in the camera image;
Parameter calculation means for calculating a calibration parameter between the coordinate value in the camera image and the coordinate value of the transmissive display device using the coordinate value of the marker;
A display device calibration apparatus comprising:

Images