JP6505354B2 - Display device - Google Patents

Description

  The present invention relates to a display device provided with a transmissive display.

  In recent years, transmissive displays that can be viewed behind have been realized by using transparent materials. As one of the usage forms of the display device provided with such a transmissive display, for example, there is a form (augmented reality) in which information on an object visually recognized by the user through the transmissive display is displayed on the display. For example, in the display system described in Patent Document 1, a transmissive head mounted display is disclosed. In this display system, the visibility of virtual information is improved by correcting the end point position of the virtual information displayed on the transmissive display according to the depth from the user's point of view.

JP, 2011-47866, A

  By the way, in order to determine on the apparatus side which object the user is viewing through the transmissive display, it is conceivable to use an imaging means such as a camera. However, in order not to impair the visibility of the transmissive display, it is necessary to dispose the imaging means in the area outside the transmissive display. For this reason, a shift occurs between the position (viewing position) of the object viewed by the user through the transmissive display and the position (imaging position) of the object in the image captured by the imaging unit. This shift is particularly noticeable when the distance of the object is short, and therefore, a technique for matching the position of the object viewed through the transmissive display with the position of the information on which the information is displayed is required. .

  The present invention has been made to solve the above problems, and it is an object of the present invention to provide a display device capable of displaying information at an appropriate position in accordance with the position of an object viewed through a transmissive display. Do.

  In order to solve the above problems, a display device according to the present invention is a display device provided with a transmissive display that allows a user to view the back side, and an imaging unit disposed in an outer area of the transmissive display; Imaging data transmitting means for transmitting imaging data of the object acquired by the means to the information providing server; and related information receiving means for receiving related information of the object transmitted from the information providing server in response to transmission of the imaging data; Display position setting means for setting the display position of the related information received by the related information reception means on the transparent display, and display control for displaying related information on the transparent display based on the display position set by the display position setting means And the display position setting unit is based on the distance from the user to the imaging unit, the distance from the imaging unit to the object, and the angle of view of the imaging unit. There are, it is characterized in that for estimating the coordinates of the object in the transmissive display from the coordinates of the object in the imaging surface of the imaging means.

  In this display device, on the transmissive display from the coordinates of the object on the imaging surface of the imaging device, based on the distance from the user to the imaging device, the distance from the imaging device to the object, and the angle of view of the imaging device. The coordinates of the object of are estimated. Thereby, the deviation between the position (viewing position) of the object viewed by the user through the transmissive display and the position (imaging position) of the object in the image captured by the imaging means is corrected, and viewed through the transmissive display The related information can be displayed at an appropriate position in accordance with the position of the target object.

  Moreover, it is preferable that an imaging means has an auto-focus function, and acquires the distance from an imaging means to a target object using the said function. In this case, the distance from the imaging means to the object can be easily obtained.

  Preferably, the apparatus further comprises calibration means for performing in advance calibration of the distance from the user to the imaging means. In this case, the display position setting means can simplify the process of estimating the coordinates of the object on the transmissive display from the coordinates of the object on the imaging surface of the imaging means.

  According to the present invention, information can be displayed at an appropriate position according to the position of an object viewed through the transmissive display.

It is a front view concerning one embodiment of a display concerning the present invention. It is a block diagram which shows the functional component of the display apparatus shown in FIG. It is a figure which shows the positional relationship of a user, a display apparatus, and a target object. FIG. 6 is a diagram illustrating coordinates of an object viewed by a user through a transmissive display. It is a figure which shows the positional relationship of an imaging part and a target object. It is a figure which shows the coordinate of the target object in an imaging surface. It is a figure which shows an example of singularity data. It is a figure which shows an example of the information stored in a related information database. It is a flowchart which shows operation | movement of a display apparatus. It is a flowchart of a setting process of a display position. It is a flowchart which shows operation | movement of an information provision server.

  Hereinafter, preferred embodiments of a display device according to the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a front view showing an embodiment of a display device according to the present invention. The display device 1 shown in the figure is, for example, in the form of a thin plate having a substantially rectangular shape in a front view, and is provided on the transmissive display 2 allowing the user to view the back side and the side (the right side here) of the transmissive display 2 And the main body 3.

  The display device 1 causes the transmissive display 2 to display the related information of the object P, for example, when the user gripping the main body 3 views the object P several tens cm to several m away through the transmissive display 2 It has an information display function. The related information includes, for example, various information on the object P, such as a place name, a building name, a person's name, a product name, and an image commentary.

  Physically, the main unit 3 incorporates a storage device such as a CPU (central processing unit), a memory, and various data communication devices. In addition, the housing portion of the main body portion 3 is a grip portion of the display device 1 by the user K, and the imaging portion 4 is positioned on the back side of the housing portion in the outer region of the transmissive display 2 It is provided.

  FIG. 2 is a block diagram showing functional components of the display shown in FIG. As shown in the figure, the display device 1 is connected to the information providing server 21 so as to be able to communicate information mutually via the network N. The display device 1 includes, as functional components, an imaging data processing unit 101, a processing data transmission unit 102, a related information reception unit 103, a display position setting unit 104, a calibration unit 105, and a display control unit 106. have.

  The imaging unit 4 is a part that captures an image of the object P, and is configured of, for example, a camera having an autofocus function. The imaging unit 4 outputs the acquired imaging data of the target object P to the imaging data processing unit 101. Further, the imaging unit 4 acquires the distance from the imaging unit 4 to the object P by the autofocus function, and outputs the acquired distance information to the display position setting unit 104.

  The imaging data processing unit 101 is a portion that processes imaging data acquired by the imaging unit 4. More specifically, the imaging data processing unit 101 performs format conversion (for example, jpeg, png, etc.) of imaging data so that extraction of singular points can be performed by the related information extracting unit 202 of the information providing server 21 described later. Perform conversion to your own format. The imaging data processing unit 101 outputs the processing data after format conversion to the processing data transmission unit 102. The processing data may include position information indicating a point captured by the imaging unit 4.

  The processing data transmission unit 102 is a part that transmits processing data. The processing data transmits the processing data received from the imaging data processing unit 101 to the information providing server 21 via the network N. Further, the related information receiving unit 103 is a part that receives the related information of the object P transmitted from the information providing server 21 in response to the transmission of the processed data by the processed data transmitting unit 102. The related information receiving unit 103 outputs the received related information to the display control unit 106.

  The display position setting unit 104 is a portion for setting the display position of the related information on the transmissive display 2. The display position setting unit 104 is between the position (viewing position) of the object P viewed by the user through the transmissive display 2 and the position (imaging position) of the object P in the image captured by the imaging unit 4. Target on the imaging surface 30 of the imaging unit 4 based on the distance from the imaging unit 4 to the object P, the distance from the user K to the imaging unit 4, and the angle of view of the imaging unit 4 in order to correct From the coordinates of the object P, the coordinates of the object P on the transmissive display 2 are estimated.

  As shown in FIG. 3, the distance from the imaging unit 4 to the object P is L1, the distance from the user K to the imaging unit 4 is L2, and as shown in FIG. 4, the coordinates of the viewing position of the object P by the user Let A1 (X, Y). At this time, since the position of the imaging unit 4 and the center position of the transmissive display 2 do not match, as shown in FIG. 5, when the angle of view of the imaging unit 4 is θ, as shown in FIG. The coordinates A2 (x, y) of the imaging position of the object P on the imaging surface 30 determined by the angle of view θ do not match A1 (X, Y).

  Therefore, the display position setting unit 104 corrects the coordinate A2 (x, y) of the imaging position to the coordinate A1 (X, Y) of the visual recognition position based on L1, L2, and θ. Calculate This correction value (α, β) is (α, β) = Δ × (1 / L 1) (a, when the coordinates of the imaging unit 4 with respect to the center position of the transmissive display 2 are (a, b). Calculated in b). Δ is a coefficient defined by L 2 and θ, which is obtained in advance by calibration by the calibration unit 105. The display position setting unit 104 outputs information indicating the calculated correction value to the display control unit 106.

The calibration unit 105 is a part that performs calibration of the distance from the user K to the imaging unit 4 in advance, and is a part that obtains Δ used for calculation of the correction value in advance and outputs it to the display position setting unit 104. The calibration unit 105 receives a plurality of inputs from the user K with respect to the viewing position when the object P is positioned in front of the imaging unit 4. Thus, (α1, β1) = Δ × (1 / L1 ₁ ) (a, b), (α2, β2) = Δ × (1 / L1 ₂ ) (a, b). By obtaining an approximate expression of Δ from the data, Δ can be obtained. It is preferable that the input of the visual recognition position be a tap input of the transmissive display 2 by the user K.

  The display control unit 106 is a part that controls an image or the like displayed on the transmissive display 2. When the display control unit 106 receives the related information from the related information receiving unit 103, the display control unit 106 displays the related information on the transmissive display 2 based on the information of the correction value from the display position setting unit 104. As a result, related information is displayed on the transmissive display 2 in the vicinity of (or superimposed on) the object P that the user K is viewing through the transmissive display 2 (see FIG. 1).

  On the other hand, the information providing server 21 is a computer system physically provided with a CPU, a memory, a communication interface, a storage unit such as a hard disk, and a display unit such as a display. As shown in FIG. 2, the information providing server 21 includes a processed data receiving unit 201, a related information extracting unit 202, a related information database 203, and a related information transmitting unit 204.

  The processing data receiving unit 201 is a part that receives processing data from the display device 1. The processed data receiving unit 201 outputs the received processed data to the related information extracting unit 202.

  The related information extraction unit 202 is a part that extracts related information from the related information database 203. When the related information extraction unit 202 receives the processed data from the processed data reception unit 201, first, as shown in FIG. 7, the related information extraction unit 202 extracts a singular point W of the processed data. The singular point is a point having a hue, lightness, saturation, or the like having a value exceeding a certain threshold value, and extraction of the singular point makes it possible to grasp an outline of the object P or the like. The related information extraction unit 202 acquires information (singularity data) including, for example, image data including the extracted singularity W and coordinates of each of the singularity W by extracting the singularity W.

  Next, the related information extraction unit 202 refers to the related information database 203 based on the singular point data. FIG. 8 is a diagram showing an example of information stored in the related information database 203. As shown in FIG. In the example shown in the figure, in the related information database 203, related information (product name A, sales date B, ...) is stored in association with the singular point data (AAA), and is stored in the singular point data (BBB). Related information (place name C, building name D,...) Is stored in association. The related information extraction unit 202 extracts related information based on the singular point data, and outputs the related information to the related information transmission unit 204.

  The related information transmission unit 204 is a part that transmits related information. The related information transmission unit 204 transmits the related information received from the related information extraction unit 202 to the display device 1 via the network N.

  Subsequently, the operation of the display device 1 described above will be described.

  FIG. 9 is a flowchart showing the operation of the display device 1. As shown in the figure, first, imaging data of the object P is acquired (step S01). Next, format conversion of imaging data is performed, and processing data is generated (step S02). Then, the generated processed data is transmitted to the information providing server (step S03).

  After transmission of the processing data, setting processing of the display position is performed, and the display position of the related information on the transmissive display 2 is set based on the correction value (step S04). Thereafter, the related information is received from the information providing server 21 (step S05), and the related information is displayed on the transmissive display 2 (step S06).

  FIG. 10 is a flowchart of the display position setting process. As shown in the figure, in the setting processing of the display position, first, the autofocus of the imaging unit 4 is activated (step S11). Next, it is determined whether the object P is in focus by autofocus (step S12). If the object P is not in focus, the correction value is set to (0, 0) (step S13), and the coordinates are estimated to be the visual position of the object P on the transmissive display 2 (step S16) .

  In step S12, when the object P is in focus, the coordinates of the object P on the imaging surface 30 and the distance between the imaging unit 4 and the object are acquired (step S14). Then, the correction value is calculated together with Δ and the angle of view of the camera previously obtained by calibration (step S15), and the coordinates based on the correction value are estimated to be the viewing position of the object P on the transmissive display 2 (Step S16).

  FIG. 11 is a flowchart showing the operation of the information providing server. As shown in the figure, the information providing server 21 first receives the processed data from the display device 1 (step S21). Next, singular point data is acquired based on the processed data (step S22), and related information is extracted based on the singular point data (step S23). Then, the extracted related information is transmitted to the display device 1 (step S24).

  As described above, in the display device 1, based on the distance L 1 from the user K to the imaging unit 4, the distance L 2 from the imaging unit 4 to the object P, and the angle of view θ of the imaging unit 4, The coordinates of the object P on the transmissive display 2 are estimated from the coordinates of the object P on the imaging surface 30 of FIG. Thereby, the shift between the visual recognition position of the target object P that the user K is visually recognizing through the transmissive display 2 and the imaging position of the target object P on the imaging surface 30 imaged by the imaging unit 4 is corrected. The relevant information can be displayed at an appropriate position in accordance with the position of the object P viewed through the display 2.

  In the display device 1, the distance L 2 from the imaging unit 4 to the object P is acquired using the autofocus function of the imaging unit 4. Thereby, the distance L2 from the imaging unit 4 to the object P can be easily acquired using the existing function. Furthermore, the display device 1 further includes a calibration unit 105 that performs calibration of the distance L1 from the user K to the imaging unit 4 in advance. As a result, in the display position setting unit 104, the process of estimating the visual recognition position from the imaging position can be simplified.

  In the above embodiment, after acquisition of imaging data, processing for setting the display position is performed after transmitting processing data to the information providing server 21. However, processing for setting the display position is performed before generation of processing data. Alternatively, after processing data is generated, it may be performed before transmitting processing data to the information providing server 21. Even in this case, the same effects as those of the above embodiment can be obtained.

  DESCRIPTION OF SYMBOLS 1 ... display apparatus, 2 ... transmission type display, 4 ... imaging part (imaging means), 21 ... information provision server, 102 ... processed data transmission part (imaging data transmission means), 103 ... related information receiving part (related information receiving means 104, display position setting unit (display position setting unit) 105, calibration unit (calibration unit) 106, display control unit (display control unit), K user, P object.

Claims (1)

A display device comprising a transmissive display that allows a user to view the back side,
Imaging means disposed in the outer area of the transmissive display;
Imaging data transmission means for transmitting imaging data of the object acquired by the imaging means to an information providing server;
A related information receiving unit that receives the related information of the object transmitted from the information providing server in response to the transmission of the imaging data;
Display position setting means for setting the display position of the related information received by the related information receiving means on the transmissive display;
Display control means for displaying the related information on the transmissive display based on the display position set by the display position setting means;
Calibration means for performing in advance calibration of the distance from the user to the imaging means, determining a coefficient depending on the distance from the user to the imaging means used for calculation of a correction value, and outputting the coefficient to the display position setting means ; Equipped with
The display position setting means
The coordinates of the imaging position of the object by the imaging means, and calculates the correction value for correcting the coordinates of the viewing position of the object by the user, the on the transmissive display based on the calculated the correction value Set the coordinates of the object to the display position,
The correction value is calculated from the coefficients and the imaging means depending on the distance from the user to the imaging means by using the distance to the object,
It said calibration means prior to the acquisition of the image data of the object by the imaging means prior to calculation of the correction value by the display position setting unit, display device and executes the calibration.

Images