JP2012033073A - Additional information service system and image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an additional information service system capable of delivering additional information to be added to an object without impairing a design property of the object, improving scalability of the design, and improving the easiness of creation and change of a marker and appropriateness of the display thereof.SOLUTION: An additional information visualization control system S includes: a backlight device 100 provided at the back face of a display space 10 where an object 20 is displayed; a liquid crystal panel 200 for making a mark image 30 provided in front of the display space 10, a latent image; and a portable terminal device 300 with an image pickup camera function, while visualizing a latent mark image 30, displaying the additional information 40 related to the object 20 based on the mark image 30.

Description

  The present invention belongs to the technical field of augmented reality (AR), and relates to a technique for displaying additional information added to an object existing in the real space on a display unit such as a display together with the imaged object.

  In recent years, with the advancement of portable terminal devices such as mobile phones, the current location of the terminal device is specified in conjunction with the GPS (Global Positioning System) function, and is called an “AR tag” fixed to the location. A technique for displaying additional information (characters / images / sounds) superimposed on a captured real space image is becoming popular. For example, as a portable terminal device having an AR tag display function, a built-in camera function is used, and when a predetermined direction is imaged at a predetermined location, the current position and imaging direction of the portable terminal device are recognized. The captured moving image is displayed on the display panel, and additional information (that is, the AR tag) is displayed on the displayed moving image. In particular, recently, a system is known in which a physical marker in a real space is provided for an object in the real space, and additional information is generated as an image and displayed on a display device based on the marker (for example, Patent Document 1).

JP-A-11-136706

  However, in the system described in Patent Document 1, when the design of the appearance is important, such as a display of a product or an advertisement, because the marker always exists in the real space, the design is inhibited. It will be. That is, in such a system, while it is possible to virtually present additional information for an object in the real space based on the marker, the marker must always exist in the real space. Product displays or advertisements must be designed with markers. Therefore, when such a system is used in the field of product display or advertising, the design expandability is poor, and the markers themselves often impair their designability.

  The present invention has been made to solve the above-described problems, and its object is to impair the design of an object such as a product or an advertisement while providing additional information to be added to the object to satisfy a user's request. It is another object of the present invention to provide an additional information providing system, an imaging apparatus, and the like that can improve the design extensibility, improve the ease of marker generation and change, and the accuracy of display.

  (1) In order to solve the above-mentioned problem, the additional information providing system of the present invention is formed on the back surface or the front surface of the object with light incident means for entering the polarized light from the back surface of the object existing in the real space. Based on mark image data for recognizing the position coordinates of the object in the real space, a latent image for rotating a part of the incident polarized light to make the mark image a latent image. An imaging device having an imaging means, an imaging element and a display means, and displaying an object image obtained by imaging the object using the polarized light from the front surface of the object and the optical rotation means on the display means; The imaging device visualizes the latent mark image based on the rotated polarized light when imaging the object, and Polarizing plate for superimposing on the object image and displaying on the display means, additional information for adding and displaying the object image to be displayed, and reference data for recognizing the mark image from the object image Based on the stored reference data, the storage means for storing additional information display data including at least the object information is analyzed and the mark image included in the object image is recognized. And a control unit that superimposes the stored additional information on the image of the object and displays it on the display unit based on the recognized mark image.

  With this configuration, the additional information providing system of the present invention causes a mark functioning as a marker (hereinafter also referred to as “mark image”) that cannot be visualized unless the image is picked up by the image pickup device to be latent in the real space. Accordingly, the mark image cannot be viewed without using the imaging device, while the user who requests additional information can be displayed on the display unit of the imaging device. Further, according to the present invention, a marker for displaying additional information can be formed by an image, and the position of displaying additional information and the positional relationship between objects can be accurately recognized using the marker. Therefore, the present invention makes it possible to improve the extensibility of the design without impairing the designability in the real space while making the additional information surely visible, and the ease of generating and changing the marker and its The accuracy of display can be improved.

  (2) Further, in the additional information providing system of the present invention, the imaging device specifies the position coordinates in the image area on the image display area of the display unit in the mark image recognized by the recognition unit, and the identification And calculating means for calculating the position coordinates in the real space of the mark image based on the position coordinates in the image area of the mark image, and the control means includes the calculated position in the space of the mark image. Based on the coordinates, the acquired additional information is superimposed on the object image and displayed in the image display area.

  With this configuration, the additional information providing system of the present invention can display the mark image at an arbitrary position in the display space and accurately set the position in the real space with the object. The additional information can be displayed at an appropriate display position in relation to the object.

  (3) Further, in the additional information providing system according to the present invention, the additional information display data includes the position information in the space of the object in the real space with respect to the position of the mark image in the real space together with the additional information and the reference data. Relative coordinates of the object in the real space for displaying the additional information based on the calculated position coordinates in the space of the mark image and the relative coordinates. The image processing apparatus further includes specifying means for specifying position coordinates, and the control means superimposes the acquired additional information on the object image based on the position coordinates of the object in the real space specified by the specifying means. It has the structure displayed on the said image display area.

  With this configuration, the additional information providing system according to the present invention can specify the position of the mark image in the real space for displaying the additional information based on the position coordinates of the mark image in the real space. Since it is assumed that the positional relationship in the space is precisely defined, the additional information can be displayed at an appropriate display position in the object relationship.

  (4) Further, the additional information providing system according to the present invention includes a configuration in which the imaging apparatus further includes an acquisition unit that acquires the additional information display data from a database via a network and stores the acquired data in the storage unit. Yes.

  With this configuration, the additional information providing system of the present invention can centrally manage the additional information, so that the ease of maintenance of the additional information can be improved and the latest additional information can be easily provided. It is possible to reduce the processing load on the imaging apparatus when providing additional information.

  (5) Moreover, the additional information provision system of this invention has a structure in which the said acquisition means acquires the said additional information from the said database based on the present position information and imaging direction of an imaging device.

  With this configuration, the additional information providing system according to the present invention can acquire additional information for an object existing in the imaging direction when imaging is started in the imaging apparatus, thereby improving ease of user operation. it can.

  (6) Further, in the additional information providing system according to the present invention, the imaging device acquires the additional information from the recognized mark image while recognizing the mark image from the object image formed by imaging. A configuration further includes reading means for storing in the storage means.

  With this configuration, the additional information providing system of the present invention can provide additional information even if it is a terminal device that does not have a function of connecting to a network as long as the reading means is installed. An imaging device can be used as the imaging device of the system.

  (7) Further, the additional information providing system of the present invention has a configuration in which the acquisition unit acquires the additional information by reading the two-dimensional code and recognizing the read two-dimensional code.

  With this configuration, the additional information providing system according to the present invention can provide additional information by using a two-dimensional code that is being generalized, so that many imaging devices can be used as the imaging device of the system.

  (8) Further, the additional information providing system of the present invention has a configuration in which the mark image is a rotationally asymmetric mark.

  With this configuration, the additional information providing system of the present invention can recognize the mark image even if the mark image is captured from all directions.

  (9) In order to solve the above-described problem, an imaging apparatus according to the present invention is a mark for allowing polarized light to enter from the back of an object existing in the real space and recognizing the position coordinates of the object in the real space. An image pickup apparatus that picks up a mark image formed by rotating a part of the incident polarized light into a latent image based on image data together with the object from the front surface of the object using the polarized light. A polarizing plate for visualizing the latent mark image based on the optically polarized light when the object is imaged, and displaying the mark image on the display unit by superimposing the mark image on the object image; , Additional information for adding and displaying the displayed object image, and reference for recognizing the mark image from the object image A storage unit for storing additional information display data including at least data, and a mark image included in the object image by analyzing an object image displayed on the display unit based on the stored reference data And a control unit that superimposes the stored additional information on the image of the object and displays it on the display unit based on the recognized mark image. .

  With this configuration, the additional information providing system of the present invention causes a mark functioning as a marker (hereinafter also referred to as “mark image”) that cannot be visualized unless the image is picked up by the image pickup device to be latent in the real space. Accordingly, the mark image cannot be viewed without using the imaging device, while the user who requests additional information can be displayed on the display unit of the imaging device. Further, according to the present invention, a marker for displaying additional information can be formed by an image, and the position of displaying additional information and the positional relationship between objects can be accurately recognized using the marker. Therefore, the present invention makes it possible to improve the extensibility of the design without impairing the designability in the real space while making the additional information surely visible, and the ease of generating and changing the marker and its The accuracy of display can be improved.

  The additional information providing system and the imaging apparatus according to the present invention are configured such that the mark image cannot be viewed unless the imaging apparatus is interposed. On the other hand, for the user who requests additional information, the display of the imaging apparatus is performed. Since it can be displayed on the means, it is possible to improve the extensibility of the design without impairing the design property in the real space while making the additional information visible, and to easily generate and change the marker. And the accuracy of the display can be improved.

It is a system configuration figure of a 1st embodiment in the additional information visualization control system concerning the present invention, and is a figure containing the division perspective view of a liquid crystal panel and a portable terminal device. It is a figure for demonstrating the relationship between the marker image superimposed on the display space image in 1st Embodiment, 1st polarized light, and 2nd polarized light. It is an example of the image of the display space on which the mark image displayed on the portable terminal device in 1st Embodiment was superimposed. It is a figure for demonstrating the positional relationship of the mark image in the display space in 1st Embodiment, an object, and additional information. It is a figure which shows the example of the polarization direction of the 1st polarized light and 2nd polarized light (1st polarized light in a latent image formation area) in 1st Embodiment. It is a functional block diagram which shows the function of the portable terminal device in 1st Embodiment. It is a flowchart which shows operation | movement of the additional information visualization control process in 1st Embodiment. It is a functional block diagram which shows the function of the portable terminal device of 2nd Embodiment in the additional information visualization control system which concerns on this invention. It is a flowchart which shows operation | movement of the additional information visualization control process in 2nd Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the embodiment described below, a marker image is formed into a latent image in a display space where an object is displayed using polarized light, and additional information is displayed together with the display space imaged based on the marker image formed into the latent image. This is an embodiment in which the additional information providing system and the imaging apparatus of the present invention are applied to an additional information visualization control display system that presents the image as an image to the user. In the description of the present application, light that has passed through polarizing means such as a polarizing plate or a polarizing filter is defined as “polarized light”.

<First Embodiment>
First, the first embodiment of the additional information visualization control system S according to the present invention will be described with reference to FIGS.

[Additional information visualization control system]
First, the configuration of the additional information visualization control system S in the present embodiment will be described with reference to FIGS. FIG. 1 is a system configuration diagram showing the configuration of the additional information visualization control system S of the present embodiment, including a divided perspective view of the liquid crystal panel 200 and the portable terminal device 300, and FIG. It is a figure for demonstrating the relationship of the marker image superimposed on the image (henceforth a "display space image") IM of display space 10 and 1st polarized light and 2nd polarized light in embodiment. 3 is an example of the display space image IM on which the mark image 30 displayed on the portable terminal device 300 is superimposed. FIG. 4 shows the mark image 30, the object 20, and the additional information 40 in the display space 10. It is a figure for demonstrating positional relationship.

  As shown in FIG. 1, the additional information visualization control system S of the present embodiment includes a backlight device 100 provided on the back of a display space 10 (that is, a real space) on which an object 20 such as a product is displayed, and a display. A liquid crystal panel 200 for forming a latent image of the mark image 30 provided on the front surface of the space 10 and additional information 40 relating to the object 20 based on the mark image 30 while visualizing the latent mark image 30 are displayed. And a portable terminal device (hereinafter simply referred to as “portable terminal device”) 300 with an imaging camera function to be displayed. In particular, the display space 10 is a space closed with a transparent case such as glass or acrylic such as a show window, and the liquid crystal panel 200 is provided in front of the display space 10.

  For example, the backlight device 100 of the present embodiment constitutes the light incident means of the present invention, and the liquid crystal panel 200 constitutes the latent image forming means of the present invention. Further, for example, the portable terminal device 300 of the present embodiment constitutes an imaging device of the present invention.

  The portable terminal device 300 is configured to be connected to the network N via the base station BS, and image data for displaying the additional information 40 (hereinafter simply referred to as “image data”), Data for recognizing the mark image 30 (hereinafter referred to as “marker data (reference data)”) and position coordinates indicating the relative positional relationship of the additional information 40 with the position of the mark image 30 in the display space 10. Data (hereinafter simply referred to as “relative coordinate data”) is acquired from the database 400 in advance. Then, when the liquid crystal panel 200 passes through a region (hereinafter also referred to as “polarized light output region”) 50 of the backlight device 100 that irradiates the first polarized light, the change can be recognized by humans. A part of the polarized light that cannot be rotated is rotated, and the mark image 30 is formed into a latent image in the panel display area 60 formed so that the display space 10 can be visually recognized. Then, when the portable terminal device 300 captures the display space 10 in which the mark image 30 is formed into a latent image, the portable terminal device 300 converts the optical rotation of the polarized light rotated by the liquid crystal panel 200 into a light transmittance, and displays the mark image 30. The additional information 40 acquired from the database 400 via the network N based on the visualized mark image 30 is visualized in the image display area 70 formed in the portable terminal device 300 and the mark image 30 and A display space image IM having the object 20 is displayed.

  For example, the backlight device 100 irradiates the display space 10 with polarized light that passes through the polarized light output region 50 and vibrates only in a uniform direction as shown in FIG. The liquid crystal panel 200 in front of the space 10 causes the mark image 30 as shown in FIG. 2B to become a latent image on the front surface of the display space 10 in the panel display region 60 using the incident first polarized light. It is like that. When the display space 10 in which the mark image 30 is converted into a latent image is picked up by the portable terminal device 300, the portable terminal device 300 displays the latent mark image as shown in FIG. 30 is visualized and displayed in the image display area 70. At this time, the portable terminal device 300 recognizes the position on the display space 10 when the displayed mark image 30 is analyzed and the additional information 40 acquired from the database 400 is displayed, as shown in FIG. The additional information 40 is superimposed on the captured display space image IM and displayed at the recognized position.

  Further, the mark image 30 to be latent image defines the positional relationship with the object 20 in the display space image IM (that is, the image display area 70), while the portable terminal device 300 uses the display space 10 Images are taken from every point (imaging point) in front. For this reason, the mark image 30 must determine the positional relationship with the object 20 at a constant angle regardless of the angle or the distance from the imaging point. Therefore, in the present embodiment, the mark image 30 is formed with a rotationally asymmetric shape that allows the vertical and horizontal directions to be individually recognized (that is, the posture of the mark can be recognized). 2 (b) and 2 (c), the rotationally asymmetric shape is a shape formed by a combination of a quadrilateral shape and a star shape incorporated in the quadrangular shape. However, the shape may not be the same shape (not overlapping) even if the shape is rotated with respect to the center point of the shape, such as 45 degrees, 90 degrees, or 180 degrees.

Furthermore, as shown in FIG. 4, the mark image 30 has a positional relationship on the display space 10 with the object 20 displayed in the display space 10, that is, the mark image 30 is the origin (0 in the coordinate system of the display space 10). , 0, 0), the coordinates (x ₁ , y ₁ , z ₁ ) of the object 20 are precisely defined and displayed. Then, the position coordinates of the mark image 30 in the display space 10 (hereinafter, also referred to as “three-dimensional position coordinates”) are specified, and the additional information 40 is displayed on the display space 10 based on the three-dimensional position coordinates of the mark image 30. If the three-dimensional position coordinates (x ₂ , y ₂ , z ₂ ) are determined, it is possible to display the additional information 40 in a state in which the positional relationship with the object 20 is precisely defined in the display space 10. Become. Therefore, in the present embodiment, while specifying the position of the mark image 30 on the display space 10, the position on the display space 10 where the additional information 40 is displayed is provided as relative coordinate data based on the specified position. The additional information 40 can be displayed at a position suitable for the object 20.

  As described above, the additional information visualization control system S has a configuration that allows the user to visually recognize the additional information 40 in the display space 10, and allows the user to visually recognize the additional information 40. In addition, it is possible to improve the extensibility of the design without impairing the appearance design of the object 20 or the display space 10 thereof.

  In the present embodiment, since the mark image 30 is clearly visualized in the portable terminal device 300, the mark image 30 displayed on the portable terminal device 300 and an image of the other part, that is, an image is captured. The liquid crystal panel 200 causes the mark image 30 to become a latent image based on the data of the mark image 30 (hereinafter referred to as “mark image data”) so that the contrast ratio with the display space image IM increases. Yes. In particular, the mark image 30 is optimally displayed in black, but is not limited thereto as long as the user can clearly see the mark image 30 in the portable terminal device 300.

[Backlight device]
Next, the backlight device 100 of this embodiment will be described with reference to FIG.

  The backlight device 100 includes an illumination unit 110 that irradiates the display space 10 from the back surface, and a polarizing plate 120 that polarizes light emitted from the illumination unit 110.

[LCD panel]
Next, the configuration of the liquid crystal panel 200 according to the present embodiment and the operation of each unit will be described in detail with reference to FIGS. 5 shows a part of the polarized light (hereinafter referred to as “first polarized light”) emitted from the backlight device 100 according to this embodiment and the first polarized light rotated in the latent image forming region 62. It is a figure which shows the example of the polarization direction of this polarized light (henceforth "2nd polarized light").

  The liquid crystal panel 200 includes a transmission region 61 that is equal to or greater than the irradiation area of the backlight device 100 including a plurality of pixels (pixels), and rotates a part of the first polarized light that is a part of the transmission region 61. And a latent image forming area 62 for converting the mark image 30 into a latent image. In particular, the liquid crystal panel 200 has a latent image forming region 62 at the left corner of the transmissive region 61 so that the mark image 30 is superimposed on the display space image IM in the portable terminal device 300 as described above. ing. The liquid crystal panel 200 specifically includes a panel unit 210 for rotating a part of the incident polarized light, and a drive control unit 220 for driving the panel unit.

  The panel unit 210 includes a pair of glass substrates (hereinafter referred to as “first glass substrate” and “second glass substrate”) 211 and 212 and a pair of transparent electrodes (hereinafter referred to as “second glass electrodes”). Also referred to as “first transparent electrode” and “second transparent electrode”) 213, 214 and a pair of alignment layers sandwiched between the pair of transparent electrodes (hereinafter referred to as “first alignment layer” and “second alignment”). Also referred to as a “layer”) 215, 216 and a liquid crystal layer 217 sandwiched between a pair of alignment layers.

  In particular, the liquid crystal layer 217 includes liquid crystal molecules that are twisted and aligned at a predetermined angle (specifically, 90 degrees) between the first alignment layer 215 and the second alignment layer 216. The liquid crystal layer 217 rotates the polarized light incident from the backlight device 100 along the twisted and aligned liquid crystal molecules with respect to the polarized light that passes through the region other than the latent image forming region 62. Is passed through to be 90 degrees. On the other hand, the liquid crystal layer 217 is a part of the first polarized light that has entered according to the control of the first transparent electrode 213 and the second transparent electrode 214 based on the mark image data input to the drive control unit. The mark image 30 is converted into a latent image by adjusting the optical rotation of the first polarized light transmitted through the latent image forming region 62 for each pixel. For example, the liquid crystal layer 217 has a range of polarized light from 0 degrees (an angle at which the first polarized light enters the liquid crystal layer 217) to 90 degrees in order to make the mark image 30 a latent image in the display space 10. Thus, the optical rotation is adjusted for each pixel.

  The panel unit 210 has an intersection (that is, a pixel) between m columns of data lines (not shown) extending along the column direction and n rows of scanning lines (not shown) extending along the row direction (not shown). The liquid crystal layer is controlled every time, and the first transparent electrode 213 and the second transparent electrode 214 are controlled based on the signal from the scanning line and the data line selected by the mark image data input to the drive control unit 220. The optical rotation of the first polarized light in each pixel of the latent image forming area 62 is individually controlled.

  For example, as shown in FIG. 2 described above, the latent image forming area 62 is formed at the left corner of the transmissive area 61, and the first polarized light emitted from the backlight device 100 is shown in FIG. In the first polarized light that passes through the latent image forming region 62, the liquid crystal layer 217, as shown in FIG. With reference to the optical rotation, the display space 10 on the back of the mark image 30 is displayed as it is from 0 degree (the optical rotation displayed as black on the portable terminal apparatus 300) to a maximum of 90 degrees (on the portable terminal apparatus 300 ( The optical rotation is adjusted so that it is polarized in the optical display). In this case, as shown in FIGS. 5A and 5B, the liquid crystal layer 217 has the first alignment layer 215 and the second alignment layer 217 in the first polarized light that passes outside the latent image formation region 62. By the action of the alignment layer 216, the light is rotated by 90 degrees and transmitted.

  That is, in the present embodiment, in the first polarized light passing outside the latent image forming region 62, no voltage is applied to the first transparent electrode 213 and the second transparent electrode 214 (that is, the data line and Since the voltage of the first transparent electrode 213 and the second transparent electrode 214 is turned off by the scanning line), the liquid crystal layer 217 cannot be driven and is twisted by 90 degrees as shown in FIGS. The first polarized light is rotated by 90 degrees along the liquid crystal molecules. On the other hand, since the mark image 30 is visualized by the portable terminal device 300, the first polarized light (that is, the second polarized light) that passes through the latent image forming region 62 is transmitted to the drive control unit 220. Since the voltage is applied to the first transparent electrode 213 and the second transparent electrode 214 based on the input mark image data (that is, the voltage of the first transparent electrode 213 and the second transparent electrode 214 by the data line and the scanning line). 5 (a) and 5 (b), the liquid crystal layer 217 is individually driven for each pixel as shown in FIGS. 5 (a) and 5 (b), and the optical rotation is started from 0 degree with reference to the optical rotation of the input first polarized light. Adjustment is made within a range of 90 degrees.

  In FIG. 4B, the mark image 30 is a simple image formed from a portion where the optical rotation is polarized by 90 degrees (transmission) and a portion where the optical rotation is 0 degrees (blocking). A mark image 30 having a gray scale may be formed depending on the gradation.

  In addition, since each pixel of the liquid crystal layer 217 is driven and controlled by the first transparent electrode 213 and the second transparent electrode 214, the latent image forming region 62 that makes the mark image 30 a latent image is fixed at the left corner. However, any region in the liquid crystal layer 217 that allows the incident first polarized light to pass therethrough may be used, and the content of the latent image can be changed. ing.

  As shown in FIG. 1, the drive control unit 220 includes a data line driving circuit 221 that controls the liquid crystal layer 217 corresponding to each pixel via a plurality of data lines, and a pixel that is connected to each pixel via a plurality of scanning lines. A scanning line driving circuit 222 that controls the corresponding liquid crystal layer 217, a memory circuit 223 that temporarily stores mark image data to be displayed on the panel unit 210, and a control circuit 224 that controls each unit are provided.

  The data line driving circuit 221 has a plurality of m data lines under the control of the control circuit 224, and selects a corresponding data line based on the input mark image data. Further, the scanning line driving circuit 222 has a plurality of n scanning lines under the control of the control circuit 224, and selects the corresponding scanning line based on the input mark image data.

  The control circuit 224 is configured to comprehensively control each of the above parts. In particular, the control circuit 224 converts the mark image data into matrix data for controlling the optical rotation in the liquid crystal layer 217 of each pixel, and the data line driving circuit 221 and the scanning line based on the converted matrix data. The drive circuit 222 is controlled. Further, mark image data to be displayed on the panel unit 210 is supplied from the PC 500 via the control circuit 224 to the memory circuit 223, and the supplied mark image data is stored therein.

[Portable terminal device]
Next, the portable terminal device 300 of this embodiment will be described with reference to FIGS. 1 and 6 described above. FIG. 6 is a functional block diagram showing functions of the portable terminal device 300 in the present embodiment.

  The portable terminal device 300 is a portable communication terminal device such as a portable phone, a game machine, a PDA (Personal Digital Assistant), or a tablet portable terminal device. The portable terminal device 300 is configured to be connected to the network N via the base station BS.

  Structurally, as shown in FIG. 1, the portable terminal device 300 includes a housing 310, a polarizing filter 320 provided on the back surface (that is, a surface facing the liquid crystal panel 200), and the polarizing filter 320. The camera unit 330 for imaging the transmission region 61 of the liquid crystal panel 200 and an image of the display space 10 and the like captured by the camera unit 330 are displayed on the front surface (that is, the surface opposite to the surface facing the liquid crystal panel 200). Display panel 340.

  In particular, the camera unit 330 includes an imaging device (hereinafter referred to as “CCD”) that performs photoelectric conversion, a housing that stores the CCD, and a camera mechanism such as an imaging lens. Based on the operation of the operation unit 357, An image of a predetermined subject range formed on the CCD is converted into predetermined image data and displayed on the display panel 340. In addition, the camera unit 330 controls the data processing unit 360 to perform an additional information visualization control process on the captured image, that is, an image of the display space 10 (specifically, a captured image). To output.

  The polarization filter 320 transmits the first polarized light, and the second polarized light is transmitted through the second polarized light based on the optical rotation of the second polarized light rotated for each pixel in the liquid crystal panel 200. The rate is to be controlled. In other words, the polarizing filter 320 is arranged so as to be parallel to the first polarized light, and controls its transmittance in accordance with the optical rotation of the second polarized light based on the latent mark image 30. ing. Note that the polarizing filter 320 may be fixed to the housing 310 or may be detachable from the housing 310.

On the other hand, the portable terminal device 300 is equipped with an application program for displaying the additional information 40 in the display space 10 based on the mark image 30 while visualizing the mark image 30. The application program cooperates with various hardware included in the portable terminal device 300,
(1) Data acquisition processing for acquiring image data, marker data, and relative coordinate data (hereinafter also referred to as “additional information display data”) from the database 400 based on the current position and imaging direction of the portable terminal device 300. ,
(2) Image analysis processing for recognizing a visualized mark image 30 by analyzing a captured image (that is, a display space image IM) while capturing a predetermined area by the camera unit 330;
(3) A coordinate recognition process for specifying the position coordinates of the recognized mark image 30 on the display space 10 and for specifying the position coordinates of the additional information 40 to be added to the object 20 on the display space 10, and
(4) Additional information display processing for displaying the superimposed additional information on the display space image IM while converting the position coordinates of the specified additional information 40 on the display space 10 into the position coordinates on the image display area 70 of the display panel 340 ,
This is a program for executing additional information visualization control processing including various types of processing.

  As shown in FIG. 6, the portable terminal device 300 performs a communication interface 351 that communicates with the database 400 and a data storage unit 352 that stores the acquired additional information display data, as shown in FIG. And a data processing unit 360 that performs each of the above-described data acquisition processing, image analysis processing, coordinate recognition processing, and additional information display processing. The portable terminal device 300 includes a display panel 340 that displays various images, a drawing control unit 354 that controls various drawing operations on the display panel 340, a current position detection unit 355 that detects a current position, and a camera unit. 330 stores a direction detection unit 356 that detects a direction in which an image is captured, an operation unit 357 for inputting a user operation, a terminal management control unit 359 that controls the whole, and stores various programs. ROM / RAM 358 having a memory function used when executed, and each member is connected to each other by a bus B.

  For example, the communication interface 351 of the present embodiment constitutes an acquisition unit of the present invention, and the camera unit 330 constitutes a reading unit of the present invention. The data storage unit 352 of the present invention constitutes a storage unit of the present invention, and the data processing unit 360 constitutes an acquisition unit, a recognition unit, a control unit, a calculation unit, and a specifying unit of the present invention.

  The communication interface 351 has an antenna AT under the control of the data processing unit 360, constructs a communication line with the database 400 via the base station BS connected to the network N, and performs various types of data such as additional information display data. Send and receive data.

  The data storage unit 352 stores additional information display data acquired by the communication interface 351 under the control of the data processing unit 360.

  The data processing unit 360 realizes an additional information visualization control process executed by an application program that executes the additional information visualization control process (hereinafter also referred to as “additional information visualization control program”). For example, the data processing unit 360 controls each unit of the portable terminal device 300 with the additional information visualization control program recorded in the ROM / RAM 358, while the communication interface 351, the camera unit 330, the drawing control unit 354, and the operation unit 357. The various processes described above are executed in conjunction with or by controlling them.

  Specifically, the data processing unit 360 activates the additional information visualization control program to execute the above-described processes, the data acquisition processing unit 361, the image analysis processing unit 362, the coordinate recognition processing unit 363, and the additional information display. A processing unit 364 is constructed. For example, the data acquisition processing unit 361 of the present embodiment constitutes an acquisition unit of the present invention, and the image analysis processing unit 362 constitutes a recognition unit. Further, the coordinate recognition processing unit 363 of the present embodiment constitutes a calculation unit and a specifying unit, and the additional information display processing unit 364 constitutes a control unit of the present invention. Details of the operation of each unit in the data processing unit 360 will be described later.

  The display panel 340 has an image display area 70 of a predetermined size, and is configured by a panel of a liquid crystal element or an EL (Electro Luminescence) element, for example. The display panel 340 displays a predetermined image based on the drawing data generated by the drawing control unit 354. In particular, in the present embodiment, the display panel 340 is linked to the data processing unit 360, the camera control unit, and the operation unit 357 under the control of the drawing control unit 354 when the additional information visualization control program is executed. However, the display space image IM, the mark image 30, and the additional information 40 are displayed.

  The drawing control unit 354 generates drawing data necessary for drawing a predetermined image on the display panel 340 under the control of the terminal management control unit 359 or the control of the data processing unit 360. The generated drawing data is output to the display panel 340.

  The current position detection unit 355 recognizes the position of the GPS satellite via the network N under the control of the data processing unit 360 during the execution of the additional information visualization control program, and transmits the satellite signal ( GPS signal) is detected, and the coordinate value of the current position of the portable terminal device 300 is calculated (ie, detected) based on the detected GPS signal. The current position detection unit 355 provides the calculated coordinate value to the data processing unit 360 as position information. However, when the portable terminal device 300 has a telephone function, the current position of the portable terminal device 300 is calculated based on the direction and the strength of the radio wave received by the telephone base station. May be.

  The direction detection unit 356 is formed by a magnetic sensor that detects weak geomagnetism and an acceleration sensor that detects the direction of gravity, and under the control of the data processing unit 360 during execution of the additional information visualization control program, The imaging direction of the camera unit 330 is detected, and the detected direction is provided to the data processing unit 360 as direction information.

  The operation unit 357 includes various confirmation buttons, operation buttons for inputting operation commands, a number of keys such as a numeric keypad, and a touch sensor provided on the display panel 340, and is used when performing each operation. It has become. Specifically, the operation unit 357 is used when performing an operation for executing imaging or other processing by the camera unit 330 when the additional information visualization control program is started.

  The terminal management control unit 359 is mainly configured by a central processing unit (CPU) and includes various input / output ports such as a key input port and a display control port, and includes general functions and additions of the portable terminal device 300. The overall function for executing the information visualization control program is comprehensively controlled.

  In the ROM / RAM 358, various control programs for functioning as the portable terminal device 300 are recorded, and an additional information visualization control program is recorded. The ROM / RAM 358 is used as a work area when various processes are executed.

2. Additional Information Display Data Next, additional information display data used in the additional information display process of the present embodiment will be described.

  The image data is data for generating an image to be displayed as the additional information 40 on the display panel 340 when the additional information display process is executed, and the image position of the object 20 corresponding to the captured display space image IM. It is comprised from the various data for producing | generating the image displayed matched with this. Specifically, the image data includes, in addition to content information to be displayed as an image, parameter data necessary for superimposing display on the display space image IM such as the size of the image display area 70 of the content. . For example, the image data includes text indicating the product name, price, inventory information for each type, etc. of the display product as the object 20, a format for controlling display of text such as the font, size and color of the text, a photo or Data to be displayed as an image in the display space 10 such as content such as a moving image and the size of the content with respect to the display area is included. The content information includes a moving image such as a description VTR of a usage example in addition to a still image such as a photograph. One image data may be data composed of a single content, or may be data formed of a plurality of contents.

  The marker data is data for recognizing the mark image 30 that is visualized and displayed from the display space image IM captured by the camera unit 330 when the image analysis process is executed. Data of the same image pattern. Specifically, the marker data is data of each pixel that forms the same image as the image when the mark image 30 is normalized, such as binarized data of each pixel.

  The relative coordinate data is position coordinate data on the display space 10 that displays additional information 40 based on the position coordinates of the mark image 30 on the display space 10. Specifically, the relative coordinate data is three-dimensional position coordinate data including an X coordinate, a Y coordinate, and a Z coordinate with the position coordinate of the mark image 30 as the origin, as shown in FIG. 4 described above. Note that, for example, the three-dimensional position coordinates indicate the center when the additional information 40 is displayed as an image, but are not limited thereto, and may indicate a position such as upper left or lower right in the image of the additional information 40 to be displayed. Good.

3. Details of Data Processing Unit Next, the configuration and operation of each unit in the data processing unit 360 of this embodiment will be described with reference to FIGS. 3 and 6.

  Based on the operation of the operation unit 357 by the user, the data acquisition processing unit 361 acquires each data of image data, marker data, and additional information display data of relative coordinate data from the database 400 in conjunction with the communication interface 351. The data is stored in the data storage unit 352. Specifically, when the data acquisition processing unit 361 detects the start of imaging of the display space 10 by the camera unit 330 when the additional information visualization control program is activated, the data acquisition processing unit 361 controls the current position detection unit 355 to control the portable terminal. The coordinate value of the current position of the apparatus 300 is acquired, and direction information indicating the imaging direction of the camera unit 330 detected by controlling the direction detection unit 356 is acquired. The data acquisition processing unit 361 acquires additional information display data specified based on the current position and direction information from the database 400 via the communication interface 351.

  The data acquisition processing unit 361 is a database by short-range wireless communication such as BLUETOOTH (IEEE: the Institute of Electrical and Electronics Engineers 802.15 / registered trademark) or Wi-Fi (IEEE 802.11 series / registered trademark). Each data of the additional information display data may be directly acquired from 400 and stored in the data storage unit 352. In this case, the communication interface 351 has a predetermined communication circuit, and when belonging to a predetermined communication distance range of the database 400 under the control of the data acquisition processing unit 361, predetermined authentication and other connection processing ( For example, pairing) is executed to connect the communication lines. Then, the data acquisition unit 361 may directly acquire each data of the additional information display data via the connected communication circuit and store it in the data storage unit 352.

  The image analysis processing unit 362 analyzes the display space image IM captured by the camera unit 330, and recognizes the mark image 30 from the display space image IM based on the marker data stored in the data storage unit 352. ing. Specifically, the image analysis processing unit 362 performs template matching with a normalized correlation coefficient using a template for the display space image IM that is the recognition target image. Specifically, the image analysis processing unit 362 performs binarization processing on the spatial image, corrects the distortion generated by the camera unit 330, and calculates a transformation matrix calculated in advance for the corresponding region. To perform perspective transformation. This corresponding area refers to an area corresponding to an image used as a template. In the present embodiment, the same image as the mark image 30 is a template, that is, a quadrangle having a rotationally asymmetric shape at the center. Therefore, the image analysis processing unit 362 extracts the rectangular area from the binarized spatial image while correcting the rectangular area, and performs perspective transformation on the extracted rectangular area. The image revision processing unit 362 recognizes a region similar to the template as the mark image 30 from the perspective-transformed region.

  For details of matching using a template in the image analysis processing unit 362, "Kazuhiro Kato, Mark Billinghurst, Koichi Asano, Keihachiro Tachibana" Augmented Reality System Based on Marker Tracking and Its Calibration "The Virtual Reality Society of Japan Journal Vol. 4, no. 4 (1999) and Hirokazu Kato “Development of Augmented Reality System Construction Tool ARTToolKit” IEICE Technical Report PRMU 2001-232 (2002-2).

  The coordinate recognition processing unit 363 specifies a two-dimensional position coordinate at the center of the recognized mark image 30, that is, a position coordinate in the coordinate system of the image display area 70, and displays an image based on a previously calculated conversion matrix. The perspective transformation of the two-dimensional coordinate system on the region 70 into the three-dimensional position coordinates of the center of the mark image 30 in the coordinate system of the display space 10 is performed while taking into account the size difference with the template. The three-dimensional position coordinates are specified. In addition, the coordinate recognition processing unit 363 recognizes the three-dimensional position coordinates on the display space 10 on which the additional information 40 is displayed based on the relative coordinate data stored in the data storage unit 352, and calculates a conversion matrix calculated in advance. The additional information 40 in the coordinate system of the display space 10 is displayed based on the two-dimensional position coordinates on the image display area 70 from the three-dimensional position coordinates, and the additional information 40 is displayed 2. Dimension position coordinates are specified.

  The additional information display processing unit 364 displays the drawing control unit 354 and the drawing control based on the two-dimensional position coordinates for displaying the additional information 40 recognized by the coordinate recognition processing unit 363 and the image data stored in the data storage unit 352. The unit 354 is controlled so that the additional information 40 is superimposed on the display space image IM as an image and displayed on the image display area 70 of the display panel 340. Specifically, the additional information display processing unit 364 controls parameter data such as text and image and content data included in the image data based on the two-dimensional position coordinates for displaying the recognized additional information 40. The additional information 40 is displayed as shown in FIG. 3 while specifying the size displayed as an image.

  Note that the additional information display processing unit 364 specifies the size of the display in the additional information 40 based on the predetermined size of the additional information 40 included in the parameter data and the position in the display space 10. The display size of the additional information 40 is specified by performing perspective transformation of information on the display space image IM (two-dimensional image) system using the transformation matrix. However, instead of this specifying method, the additional information display processing unit 364 has a predetermined reference distance in the display space 10 between the portable terminal device 300 and the mark image 30 included in the parameter data and the size of the additional information 40. If the reference distance included in the parameter data is inversely proportional to the actual distance between the portable terminal device 300 and the mark image 30 in the display space, that is, if the actual distance is shorter than the reference distance, the additional information is referred to. When the actual size is larger than the standard size included in the size parameter data of 40 and longer than the reference distance, the display size of the additional information 40 is specified so as to be smaller than the standard size included in the size parameter data of the additional information 40 May be.

[Additional information visualization control processing]
Next, the additional information visualization control process in the portable terminal device 300 of this embodiment will be described with reference to FIG. FIG. 7 is a flowchart showing the operation of the additional information visualization control process in this embodiment.

  First, when the terminal management control unit 359 detects activation of the additional information visualization control program by a user operation on the operation unit 357 (step S101), the camera unit 330 is activated to start imaging of the display space 10 (step S101). S102). The drawing control unit 354 causes the display panel 340 to display an image captured by the camera unit 330 when imaging by the camera unit 330 is started.

  Next, the data acquisition processing unit 361 acquires current position information indicating the current position of the portable terminal device 300 and direction information indicating the imaging direction of the camera unit 330, and based on the acquired current position information and direction information, Additional information display data including image data, marker data, and relative coordinate data is acquired from the database 400 via the communication interface 351 and stored in the data storage unit 352 (step S103).

  Next, the image analysis processing unit 362 captures the display space image IM (one frame image) captured by the camera unit 330 and displayed on the display panel 340, and based on the marker data stored in the data storage unit 352. Analysis processing of the display space image IM is executed (step S104).

  Next, the image analysis processing unit 362 determines the presence / absence of the mark image 30 in the display space image IM (step S105). For example, the image analysis processing unit 362 calculates the degree of similarity with the template while normalizing the quadrilateral area extracted as described above as the candidate for the mark image 30, and recognizes the quadrilateral area with the highest degree of similarity as the mark image 30. To do. However, the image analysis processing unit 362 determines that the mark image 30 does not exist (that is, cannot be recognized) in the display space image IM when the calculated similarity is equal to or less than a predetermined threshold.

  When it is determined that the mark image 30 is present (recognized) in the display space image IM (step S105: Yes), the process proceeds to step S106, and the mark image 30 is displayed in the display space image IM. If it is determined that it does not exist (cannot be recognized) (step S105: No), the process proceeds to step S109.

  Next, the coordinate recognition processing unit 363 specifies a two-dimensional position coordinate on the image display area 70 at the center of the mark image 30 and uses the predetermined transformation matrix to calculate the two-dimensional position coordinate on the display space 10. A three-dimensional position coordinate is specified (step S106).

  Next, the coordinate recognition processing unit 363 displays additional information 40 in the space of the object 20 based on the three-dimensional position coordinates at the center of the recognized mark image 30 and the relative coordinate data stored in the data storage unit 352. Is determined (step S107).

  Next, the additional information display processing unit 364 converts the three-dimensional position coordinates of the additional information 40 in the object 20 space into two-dimensional coordinates on the image display area 70 using a predetermined conversion matrix, The additional information 40 is displayed on the display panel 340 based on the two-dimensional position coordinates of the specified additional information 40 and the image data stored in the data storage unit 352 (step S108). At this time, the additional information display processing unit 364 controls the drawing control unit 354 to control parameter data such as size, text, and image and content data based on the image data, and adds the additional data as shown in FIG. Information 40 is displayed.

  Finally, the data acquisition processing unit 361 determines whether or not the operation for ending the additional information visualization control program has been executed by the operation unit 357 (step S109), and when the operation for ending the additional information visualization control program has not been executed. (Step S109: No), the process proceeds to Step S103, and when the end operation of the additional information visualization control program is executed (Step S109: Yes), the imaging in the camera unit 330 is stopped (Step S109). S110) This operation is terminated.

[Function and effect]
As described above, the additional information visualization control system S of the present embodiment has a configuration in which the mark image 30 functioning as a marker can be visualized in the display space 10 so that it cannot be visualized unless the camera unit 330 captures an image. On the other hand, the user who requests the additional information 40 can be displayed on the display panel 340 of the camera unit 330. Therefore, the additional information visualization control system S of the present embodiment can improve the extensibility of the design without impairing the designability in the display space 10 while making the additional information 40 surely visible. The ease of generating and changing the marker and the accuracy of the display can be improved.

  The additional information visualization control system S of the present embodiment can form a marker for displaying the additional information 40 by an image and display the marker at an arbitrary position in the display space 10, and the additional information 40 can be displayed using the marker. Since the positional relationship between the position to be displayed and the object 20 can be recognized accurately, by easily setting the positions of the mark image 30 and the object 20 in the display space 10, the ease of marker generation and change and The accuracy of the display can be improved.

  The additional information visualization control system S of the present embodiment can specify the position in the display space 10 for displaying the additional information 40 based on the position coordinates of the mark image 30 on the display space 10, and the mark image 30. Therefore, the additional information 40 can be displayed at an appropriate display position in the relationship of the object 20.

  Since the additional information visualization control system S of the present embodiment provides the additional information 40 from the database 400, the additional information 40 can be centrally managed and the ease of maintenance of the additional information 40 can be improved. Thus, it is possible to easily provide the latest additional information 40 and to reduce the processing load of the imaging apparatus when the additional information 40 is provided.

  When the camera unit 330 starts imaging, the additional information visualization control system S of the present embodiment obtains marker data for recognizing the mark image 30 from the database 400 based on the current position and imaging direction of the portable terminal device 300. Since it can acquire, the ease of user operation can be improved.

  Since the additional information visualization control system S of the present embodiment uses a rotationally asymmetric mark for the mark image 30, the mark image 30 can be recognized even if the mark image 30 is captured from any direction. .

[Modification]
Next, a modified example of the additional information visualization control system S of the present embodiment will be described.

  In the first embodiment, the liquid crystal panel 200 is provided on the front surface of the display space 10, but may be provided on the back surface of the display space 10. However, even in this case, the liquid crystal panel 200 is provided on the front surface of the backlight device 100 so that the first polarized light from the backlight device 100 is received.

  Further, in the first embodiment, the additional information visualization control system S is adapted to be applied to the display space 10 closed with a transparent case such as glass or acrylic such as a show window. The same configuration can be applied to an open space such as a shelf that is displayed.

  In addition, the portable terminal device 300 of the first embodiment described above includes the current position information and direction information of the portable terminal device 300 recognized by activating the camera unit 330 when the additional information visualization control program is activated. Based on this, the additional information display data is acquired from the database 400, but before starting the camera unit 330, the additional information display data to be acquired in accordance with the user operation or the additional information visualization control program is obtained. You may make it acquire from the database 400. FIG. In this case, when the additional information 40 visualization display process is executed by a user operation, the acquired additional information display data is designated.

  In addition, the liquid crystal panel 200 of the first embodiment may include a color filter arranged in correspondence with each pixel between the second glass substrate 212 and the second transparent electrode 214. In this case, since the mark image 30 can be colored, various images or information can be provided as the mark image 30.

  In the additional information visualization control system S of the first embodiment, the first polarized light and the second polarized light for forming the latent image 30 are converted into circularly polarized light, respectively. The first polarized light and the second polarized light converted to circularly polarized light are respectively converted into linearly polarized light on the surface of the polarizing filter 320 of the portable terminal device 300 on the liquid crystal panel 200 side while having the first quarter wavelength plate. You may have the 2nd quarter wave plate to convert. Specifically, in FIG. 1, a quarter wavelength plate is disposed on the surface of the second glass substrate 212 on the portable terminal device 300 side, and the quarter wavelength plate is disposed on the surface of the polarizing filter 320 on the liquid crystal panel 200 side. Will be installed. By doing so, the first polarized light and the second polarized light can be circularly polarized between the portable terminal device 300 and the liquid crystal panel 200, so that the light intensity of the first polarized light and the second polarized light can be reduced. The mark image 30 can be clearly provided while being resistant to changes. Also, in this case, the convenience and convenience are improved because the brightness / darkness of the mark image 30 does not change no matter what angle the liquid crystal panel 200 is imaged using the portable terminal device 300.

  In the additional information visualization control system S of the first embodiment, additional information display data is acquired from the database via the network N and stored in the data storage unit 352. You may make it display additional information using the various data previously stored in the storage medium which can be attached or detached to portable terminal devices 300, such as memory.

  Further, in the additional information visualization control system S of the first embodiment, the portable terminal device 300 further includes another camera unit (polarization) different from the camera unit 330 in addition to the camera unit 330 having the polarization filter 320. In order to distinguish it from the camera unit 330, it may be referred to as a “second camera unit” hereinafter. In this case, the portable terminal device 300 can switch the use of the camera unit 330 and the second camera unit while interlocking with the operation unit 357 and the drawing control unit 354 under the terminal management control unit 359. It is configured. When the display space 10 is imaged by the camera unit 330, the display panel 340 displays the mark image 30 and the additional information 40 as shown in FIG. On the other hand, when the display space 10 is imaged by the second camera unit, since the second polarized light cannot be recognized because there is no polarizing filter, the display panel 340 displays the mark image 30 and the additional information 40. The object 20 cannot be displayed and only the object 20 is displayed.

<Second Embodiment>
Next, a second embodiment of the additional information visualization control system S according to the present invention will be described with reference to FIGS.

  In the present embodiment, the mark image 30 is formed by a two-dimensional code in which additional information display data is incorporated, and the portable terminal device 300 includes a data acquisition processing unit in the data processing unit 360 in the first embodiment. It is characterized in that it has a two-dimensional code analysis unit 371 instead of 361. In addition, the other structure of this embodiment is the same as that of 1st Embodiment, About the same member as 1st Embodiment, the same number is attached | subjected and the description is abbreviate | omitted. The two-dimensional code is a matrix type two-dimensional code, and is also called, for example, a QR (Quick Response) code (registered trademark).

[Data processing section]
When the data processing unit 370 of the portable terminal device 300 according to the present embodiment recognizes a two-dimensional code as the mark image 30 on the display space image IM, the recognized two-dimensional code is analyzed and image data, marker data, and relative data are analyzed. Data for displaying additional information of coordinate data is acquired.

  Specifically, the data processing unit 370 activates the additional information visualization control program, and as shown in FIG. 8, the two-dimensional code analysis unit 371, the image analysis processing unit 362, the coordinate recognition processing unit 363, and the additional information are displayed. A display processing unit 364 is constructed. For example, the two-dimensional code analysis unit 371 of the present embodiment constitutes an acquisition unit of the present invention, and the image analysis processing unit 362 constitutes a recognition unit. Further, the coordinate recognition processing unit 363 of the present embodiment constitutes a calculation unit and a specifying unit, and the additional information display processing unit 364 constitutes a control unit of the present invention.

  When a two-dimensional code is identified as the mark image 30 by the image analysis processing unit 362, the two-dimensional code analysis unit 371 acquires an image of the recognized two-dimensional code and decodes the acquired two-dimensional code as a decode. The additional data display data of the image data, the marker data, and the relative coordinate data is acquired and stored in the data storage unit 352.

  The image analysis processing unit 362 recognizes the position detection pattern formed by a predetermined pattern at the corner (for example, three points) of the two-dimensional code on the quadrilateral from the display space image IM, and A partial image (hereinafter referred to as “two-dimensional code image”) is identified.

[Additional information visualization control processing]
Next, the additional information visualization control process in the portable terminal device 300 of this embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing the operation of the additional information visualization control process in this embodiment.

  First, when the terminal management control unit 359 detects activation of the additional information visualization control program by a user operation on the operation unit 357 (step S201), the camera unit 330 is activated to start imaging the display space 10 (step S201). S202). The drawing control unit 354 causes the display panel 340 to display an image captured by the camera unit 330 when imaging by the camera unit 330 is started.

  Next, the image analysis processing unit 362 captures a display space image IM (one frame image) captured by the camera unit 330 and displayed on the display panel 340, and based on the marker data stored in the data storage unit 352. Analysis processing of the display space image IM is executed (step S203).

  Next, the image analysis processing unit 362 determines whether or not the mark image 30 (that is, a two-dimensional code) is present in the display space image IM (step S204). For example, as described above, the image analysis processing unit 362 recognizes the position detection pattern at the corner of the two-dimensional code image on the quadrilateral and identifies the two-dimensional code image. However, when the position detection pattern cannot be recognized, the image analysis processing unit 362 determines that the mark image 30 does not exist (that is, cannot be recognized) in the display space image IM.

  If it is determined that the mark image 30 is present (recognized) in the display space image IM (step S205: No), the process proceeds to step S205, and the mark image 30 is displayed in the display space image IM. If it is determined that it does not exist (cannot be recognized) (step S204: No), the process proceeds to step S209.

  Next, the two-dimensional code analysis unit 371 acquires the identified two-dimensional code image, decodes the two-dimensional code, and acquires additional information display data of image data, marker data, and relative coordinate data to obtain data. It memorize | stores in the memory | storage part 352 (step S205).

  Next, the coordinate recognition processing unit 363 specifies a two-dimensional position coordinate on the image display area 70 at the center of the mark image 30 as a two-dimensional code, and uses the predetermined transformation matrix to determine the two-dimensional position coordinate. A three-dimensional position coordinate on the display space 10 is specified (step S206).

  Next, the coordinate recognition processing unit 363 displays additional information 40 in the space of the object 20 based on the three-dimensional position coordinates at the center of the recognized two-dimensional code and the relative coordinate data stored in the data storage unit 352. Are identified (step S207).

  Next, the additional information display processing unit 364 converts the three-dimensional position coordinates of the additional information 40 in the object 20 space into two-dimensional coordinates on the image display area 70 using a predetermined conversion matrix, Based on the two-dimensional position coordinates of the specified additional information 40 and the image data stored in the data storage unit 352, the additional information 40 is displayed on the display panel 340 (step S208). At this time, the additional information display processing unit 364 controls the drawing control unit 354 to control parameter data such as size, text, and image and content data based on the image data, and adds the additional data as shown in FIG. Information 40 is displayed.

  Finally, the data acquisition processing unit 361 determines whether or not the operation for ending the additional information visualization control program has been executed by the operation unit 357 (step S209), and when the operation for ending the additional information visualization control program has not been executed. In step S203, when the end operation of the additional information visualization control program is executed, the imaging in the camera unit 330 is stopped (step S210), and this operation is ended.

[Function and effect]
As described above, in addition to the effects of the first embodiment, the additional information visualization control system S of the present embodiment is a terminal device that does not have a function of connecting to the network N as long as the reading unit is installed. Since the additional information 40 can be provided, many portable terminal devices 300 can be used as the portable terminal device 300 of the system.

  In addition, since the additional information visualization control system S of the present embodiment can provide the additional information 40 with a two-dimensional code that is being generalized, many portable terminal devices 300 are used as the portable terminal devices 300 of the system. Can be used.

[Modification]
Next, a modified example of the additional information visualization control system S of the present embodiment will be described.

  Similarly to the modification of the first embodiment, the liquid crystal panel 200 of the second embodiment may be provided on the back surface of the display space 10. However, even in this case, the liquid crystal panel 200 is provided on the front surface of the backlight device 100 so that the first polarized light from the backlight device 100 is received.

  Further, similarly to the modification of the first embodiment, the same configuration can be applied to an open space such as a shelf on which a product is displayed.

  Similarly to the modification of the first embodiment, in the second embodiment, before starting the camera unit 330, additional information display data to be acquired in accordance with a user operation or an additional information visualization control program is stored in the database. You may make it acquire from 400. In this case, when the additional information 40 visualization display process is executed by a user operation, the acquired additional information display data is designated.

  Similarly to the modification of the first embodiment, the liquid crystal panel 200 of the second embodiment has a color filter arranged in color corresponding to each pixel between the second glass substrate 212 and the second transparent electrode 214. May be. In this case, since the mark image 30 can be colored, various images or information can be provided as the mark image 30.

  As in the modification of the first embodiment, in the additional information visualization control system S of the second embodiment, the first polarized light and the second polarized light for forming the latent mark image 30 are formed. And the first polarized light converted into circularly polarized light on the surface of the polarizing filter 320 of the portable terminal device 300 on the liquid crystal panel 200 side. You may have the 2nd quarter wavelength plate which each converts 2nd polarized light into linearly polarized light. Specifically, in FIG. 1, a quarter wavelength plate is arranged on the surface of the second glass substrate 212 on the portable terminal device 300 side, and a quarter wavelength plate is used instead of the polarizing filter. . By doing so, the first polarized light and the second polarized light can be circularly polarized between the portable terminal device 300 and the liquid crystal panel 200, so that the light intensity of the first polarized light and the second polarized light can be reduced. The mark image 30 can be clearly provided while being resistant to changes. Also, in this case, the convenience and convenience are improved because the brightness / darkness of the mark image 30 does not change no matter what angle the liquid crystal panel 200 is imaged using the portable terminal device 300.

  The additional information visualization control system S of the present invention provides a display of a show window in a department store or other store, a product such as an information presentation type amusement facility such as a museum or the like, and a display on which the object 20 is displayed. It can be used in a facility where a balance in the design of the space 10 is required.

S ... Additional information visualization control system 10 ... Object space 20 ... Object 30 ... Mark image 40 ... Additional information 50 ... Polarized light output area 60 ... Panel display area 70 ... Image display area 100 ... Backlight device 110 ... Liquid crystal display panel 120 ... Illumination unit 200 ... Liquid crystal panel 210 ... Panel portion 211 ... First glass substrate 212 ... Second glass substrate 213 ... First transparent electrode 214 ... Second transparent electrode 215 ... First alignment layer 216 ... Second alignment layer 217 ... Liquid crystal layer 220 ... Drive control unit 221 ... Data line drive circuit 222 ... Scan line drive circuit 223 ... Memory circuit 224 ... Control circuit 300 ... Portable terminal device 310 ... Housing 320 ... Polarizing filter 330 ... Camera unit 340 ... Display panel 351 ... Communication Interface 352 ... D Storage unit 354 ... Drawing control unit 355 ... Current position detection unit 356 ... Direction detection unit 357 ... Operation unit 359 ... Terminal management control unit 360, 370 ... Data processing unit 361 ... Data acquisition processing unit 362 ... Image analysis processing unit 363 ... Coordinates Recognition processing unit 364 ... Additional information display processing unit 371 ... Two-dimensional code analysis unit 400 ... Database 500 ... PC

Claims (9)

A light incident means for entering polarized light from the back of an object existing in the real space;
A means formed on the back surface or the front surface of the object, wherein a part of the incident polarized light is rotated based on mark image data for recognizing a position coordinate of the object in the real space. Latent image forming means for converting the mark image into a latent image;
An imaging device having an imaging element and a display unit, and displaying an object image acquired by imaging the object using the polarized light from the front surface of the object and the optical rotation unit on the display unit;
With
The imaging device is
When imaging the object, a polarizing plate for visualizing the latent mark image based on the rotated polarized light and displaying the mark image superimposed on the object image on the display means;
Storage means for storing additional information display data including at least additional information for adding to the displayed object image and reference data for recognizing the mark image from the object image;
Recognizing means for analyzing an object image displayed on the display means based on the stored reference data and recognizing a mark image included in the object image;
Based on the recognized mark image, control means for superimposing the stored additional information on the image of the object and displaying it on the display means;
The additional information provision system characterized by having. In the additional information provision system according to claim 1,
The imaging apparatus specifies the position coordinates in the image area on the image display area of the display means in the mark image recognized by the recognition means, and based on the position coordinates in the image area in the specified mark image A calculation means for calculating the position coordinates in the real space of the mark image;
The additional information providing system in which the control unit superimposes the acquired additional information on the object image based on the calculated position coordinates of the mark image in space and displays the additional information on the image display area. In the additional information provision system according to claim 2,
The additional information display data includes relative coordinates of the position coordinates of the object in the real space with respect to the position of the mark image in the real space together with the additional information and the reference data.
The imaging device further includes a specifying unit that specifies the position coordinate of the object on the real space for displaying the additional information based on the calculated position coordinate in space of the mark image and the relative coordinate. And having
The additional information providing system in which the control unit superimposes the acquired additional information on the object image based on the position coordinates of the object in the real space specified by the specifying unit and displays the additional information on the image display area. . In the additional information provision system as described in any one of Claims 1 thru | or 3,
The additional information providing system, wherein the imaging apparatus further includes acquisition means for acquiring the additional information display data from a database via a network and storing the data in the storage means. In the additional information provision system according to claim 4,
The additional information providing system in which the acquisition unit acquires the additional information from the database based on current position information and an imaging direction of the imaging apparatus. In the additional information provision system as described in any one of Claims 1 thru | or 4,
Additional information further comprising: a reading unit that acquires the additional information from the recognized mark image and stores the additional information in the storage unit while the imaging device recognizes the mark image from the object image formed by imaging. Offer system. The additional information providing system according to claim 6,
The additional information providing system in which the acquisition unit acquires the additional information by reading the two-dimensional code and recognizing the read two-dimensional code. In the additional information provision system as described in any one of Claims 1 thru | or 7,
The additional information providing system, wherein the mark image is a rotationally asymmetric mark. Polarized light is incident from the back side of the object existing in the real space, and a part of the incident polarized light is based on the mark image data for recognizing the position coordinates of the object in the real space. An image pickup apparatus that picks up a mark image that has been rotated and formed into a latent image together with the object using the polarized light from the front surface of the object,
When imaging the object, a polarizing plate for visualizing the latent mark image based on the rotated polarized light and displaying the mark image superimposed on the object image on the display means;
Storage means for storing additional information display data including at least additional information for adding to the displayed object image and reference data for recognizing the mark image from the object image;
Recognizing means for analyzing an object image displayed on the display means based on the stored reference data and recognizing a mark image included in the object image;
Based on the recognized mark image, control means for superimposing the stored additional information on the image of the object and displaying it on the display means;
An imaging apparatus comprising:

Images