JP2010157035A - System, method, and program for displaying composite image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide highly accurate images or services superior in the added value thereof. <P>SOLUTION: A system for displaying a composite image for compositing and displaying another image at a prescribed position of an captured image includes: an imaging means for capturing an image of an information providing medium with the prescribed information recorded thereon by a changing material whose luminance varies with a lapse of time after predetermined light irradiation; a recognition information analyzing means for analyzing recognition information and luminance information in prescribed shapes included in the image imaged by the imaging means; a three-dimensional position acquisition means for acquiring a three-dimensional position from the recognition information acquired by the recognition information analyzing means; an image information extracting means for extracting an image corresponding to the recognition information among previously accumulated multiple images to be composited using a three-dimensional coordinate acquired by the three-dimensional position acquisition means; and an image compositing means for three-dimensionally compositing the image to be composited extracted by the image information extracting means based on real space of the captured image to display a composite image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a composite image display system, a composite image display method, and a composite image display program. In particular, the present invention relates to a composite image display system, a composite image display method, and a composite for providing a highly accurate image with excellent added value. The present invention relates to an image display program.

  Conventionally, for example, a card on which a pattern or the like is drawn is photographed, and a MIDI (Musical Instrument Digital Interface) signal for controlling a musical instrument is output based on the type of card and three-dimensional position information detected from the photographed camera image. In addition, a technique for outputting video is known (for example, see Patent Document 1).

  In addition, there is known a technique for outputting various images corresponding to the movement contents of a card by selecting an image using not only the information of the card to be photographed but also the speed information of the card itself (for example, Patent Documents). 2).

  Further, there is a conventional system tool that recognizes a graphic on a paper medium with a camera and displays a virtual graphic on a display of a personal computer (for example, see Non-Patent Document 1).

Further, a target image having identification information corresponding to a predetermined image pattern is acquired, the identification information is recognized from the target image, and the identification information recognized from a plurality of pre-registered processes is obtained. The corresponding predetermined process is executed, the target image is displayed in the predetermined display area, the position information of two or more predetermined positions in the displayed target image is acquired, and the two or more acquired positions An image processing technique for drawing an image corresponding to recognized identification information in a direction and position based on the position information is known (see, for example, Patent Document 3).
JP 2002-32076 A Japanese Patent Laid-Open No. 2003-271928 "http://www.hit.washington.edu/arttoolkit/", "ARTToolKit" JP 2000-82107 A

  However, in the above-described prior art, by moving a card or the like in a predetermined direction (parallel or rotation), it is possible to display an image to be synthesized with a motion, but synthesis is performed with the card or the like stopped. The image to be changed could not be changed. For this reason, for example, it is only possible to synthesize a motion picture produced in advance, and only a specific operation can be performed for each card type. Therefore, highly accurate images and videos with excellent added value have not been provided.

  Further, in the virtual graphic display system shown in Non-Patent Document 1 described above, once the virtual graphic is displayed, it is difficult to change the graphic. That is, the figure cannot be changed according to the external environmental change. Furthermore, in the prior art, since the image to be displayed is displayed based on the position of the card or the like including the identification information, there is a limitation in the display method, and for example, it is not possible to display a 3D video or the like. .

  The present invention has been made in view of the above-described problems, and provides a composite image display system, a composite image display method, and a composite image display program for providing a highly accurate image with excellent added value. For the purpose.

  In order to solve the above-described problems, the present invention employs means for solving the problems having the following characteristics.

  According to the first aspect of the present invention, in a composite image display system that combines and displays another image at a predetermined position of a photographed image, the luminance changes over time after irradiation with a preset light. Imaging means for imaging an information providing medium in which predetermined information is recorded by a change object, recognition information analysis means for analyzing recognition information and luminance information having a predetermined shape included in an image captured by the imaging means, and A plurality of synthesis target images stored in advance using a three-dimensional position acquisition unit that acquires a three-dimensional position from recognition information obtained by the recognition information analysis unit, and a three-dimensional coordinate obtained by the three-dimensional position acquisition unit. Image information extraction means for extracting an image corresponding to the recognition information, and a synthesis target image extracted by the image information extraction means on the basis of the real space of the photographed image. And body synthesis, and having an image synthesis means for displaying the synthesized image.

  According to the first aspect of the present invention, it is possible to provide a highly accurate image or service with excellent added value.

  The invention described in claim 2 is characterized in that the recognition information analyzing means is a luminance of the change object after being irradiated from at least one light source of ultraviolet light, infrared light, and visible light having a preset wavelength. It is characterized by acquiring information.

  According to the second aspect of the present invention, a target image can be provided to the user by irradiating light from a light source such as a light having a preset wavelength. Also, various composite images can be provided by changing the set wavelength.

  According to a third aspect of the present invention, the image information extracting unit is configured to combine the luminance information corresponding to each luminance information at a predetermined time according to a change in luminance information with time of the changed object when the recognition information is printed. The target image is sequentially extracted, and the image synthesis unit performs synthesis of the captured image and the synthesis target image every time the synthesis target image is extracted by the image information extraction unit.

  According to the third aspect of the present invention, when the luminance information is changed, another synthesis target image can be selected and synthesized and displayed. Thereby, a moving image etc. can also be provided.

  According to a fourth aspect of the present invention, there is provided a rotation direction detecting means for analyzing a direction of recognition information written on the information providing medium by the change object, and the image information extracting means is provided by the recognition information analyzing means. A target image to be synthesized is extracted based on the obtained luminance information and the rotation information obtained by the rotation direction detecting means.

  According to the fourth aspect of the present invention, not only the information providing medium but also the three-dimensional shape can be synthesized on the photographed real space.

  The invention described in claim 5 is characterized in that the recognition information is at least one of a character, a figure, a pattern, an image, and barcode information.

  According to the invention described in claim 5, it is possible to provide a variety of services with high added value using one or a plurality of pieces of recognition information.

  According to a sixth aspect of the present invention, the image synthesizing means is based on posture information including at least one of a position, an angle, and a direction of the information providing medium included in the captured image, and the luminance information. Display, non-display, size, position, and direction are controlled for the synthesis target image.

  According to the sixth aspect of the present invention, it is possible to provide more composite images using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value. Can be provided.

  The invention described in claim 7 is characterized in that the light source is a retroreflective observation light, and the information providing medium is a retroreflective sheet.

  According to the seventh aspect of the invention, more accurate information can be acquired.

  The invention described in claim 8 is a composite image display method in which another image is combined and displayed at a predetermined position of a photographed image, and the luminance changes over time after irradiation with a preset light. An imaging step for imaging an information providing medium in which predetermined information is recorded by a change object, and a recognition information analysis step for analyzing recognition information and luminance information consisting of a predetermined shape included in the image captured by the imaging step; , A three-dimensional position acquisition step for acquiring a three-dimensional position from the recognition information obtained by the recognition information analysis step, and a plurality of synthesis target images stored in advance using the three-dimensional coordinates obtained by the three-dimensional position acquisition step. An image information extraction step for extracting an image corresponding to the recognition information, and a synthesis target image extracted by the image information extraction step. And three-dimensional synthesized on the basis of the real space of the shadow image, and having an image combining step of displaying the synthesized image.

  According to invention of Claim 8, the highly accurate image or service excellent in added value property can be provided.

  In the invention described in claim 9, in the recognition information analyzing step, the luminance of the change object after being irradiated from at least one light source of ultraviolet light, infrared light, and visible light having a preset wavelength. It is characterized by acquiring information.

  According to the ninth aspect of the present invention, a target image can be provided to the user by irradiating light from a light source such as a light having a preset wavelength. Also, various composite images can be provided by changing the set wavelength.

  According to a tenth aspect of the present invention, in the image information extraction step, the synthesis corresponding to each luminance information at a predetermined time is accompanied by a change in luminance information with time of the changed object when the recognition information is printed. A target image is sequentially extracted, and the image synthesis step combines the captured image and the synthesis target image each time the synthesis target image is extracted by the image information extraction step.

  According to the tenth aspect of the present invention, when the luminance information is changed, another synthesis target image can be selected and synthesized and displayed. Thereby, a moving image etc. can also be provided.

  The invention described in claim 11 includes a rotation direction detecting step for analyzing a direction of recognition information written on the information providing medium by the change object, and the image information extracting step is performed by the recognition information analyzing step. A target image to be synthesized is extracted based on the obtained luminance information and the rotation information obtained by the rotation direction detection step.

  According to the eleventh aspect of the present invention, it is possible not only to display on the information providing medium but also to synthesize a three-dimensional shape on the photographed real space.

  The invention described in claim 12 is characterized in that the recognition information is at least one of a character, a figure, a pattern, an image, and barcode information.

  According to the twelfth aspect of the present invention, it is possible to provide a variety of services with high added value using one or a plurality of pieces of recognition information.

  According to a thirteenth aspect of the present invention, the image synthesizing step is based on posture information including at least one of a position, an angle, and a direction of the information providing medium included in the captured image, and the luminance information. Display, non-display, size, position, and direction are controlled for the synthesis target image.

  According to the thirteenth aspect of the invention, it is possible to provide more composite images using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value. Can be provided.

  The invention described in claim 14 is characterized in that the light source is a retroreflection observation light, and the information providing medium is a retroreflection sheet.

  According to the fourteenth aspect of the present invention, more accurate information can be acquired.

  The invention described in claim 15 is a composite image display program that causes a computer to execute processing relating to each means of the composite image display system according to any one of claims 1 to 7. .

  According to the invention of the fifteenth aspect, it is possible to provide a highly accurate image or service excellent in added value. Further, by installing the program, the composite image display process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  ADVANTAGE OF THE INVENTION According to this invention, the highly accurate image or service excellent in added value property can be provided.

<Outline of the present invention>
The present invention uses, for example, an imaging means such as a camera and a PC (Personal Computer) connected to the camera, a mobile terminal with a camera, etc., and the luminance changes with the passage of time after irradiation with a preset light. An information providing medium (for example, card-like paper or resin sheet) on which predetermined information is recorded by a change object is imaged, and an image (composite target image (such as a synthesis target image) corresponding to information included in the captured image is captured. (Including still images, moving images, character information, etc.) are combined with the captured image and displayed on a screen such as a display, etc. In the present invention, the three-dimensional information in the virtual space is converted into rotation information and three-dimensional coordinates. Provided is a virtual graphic display system that displays a real space image in correspondence.

  Specifically, at least three points, or a graphic that is the recognition information from an image obtained by capturing recognition information that consists of a graphic (for example, a two-dimensional barcode) that includes a pattern (for example, an L character) that can specify three points. The rotation direction and three-dimensional coordinates are acquired, and the three-dimensional information (including graphics and characters) in the virtual space selected from the information obtained by the figure is displayed on the real space image in correspondence with the rotation direction and the three-dimensional coordinates.

  In the present invention, the change used for writing information on the information providing medium is, for example, a light emitting / absorbing substance that emits or absorbs light by an external stimulus (ultraviolet light, infrared light, visible light, heat), or the environment. Substances that cause color changes (such as retroreflective substances) can be used, and luminance information of the change on the information providing medium is acquired from the captured image, and images corresponding to the acquired luminance information are stored in advance. Extracted from the image group, the extracted image and the captured image are combined and displayed on the screen. Note that the luminance information in the present invention does not mean only light emission (brightness) but also includes color information such as hue, brightness, and saturation.

  Specifically, in the present invention, when ultraviolet rays are used as an external stimulus, for example, by illuminating the information providing medium with a light (light source) containing ultraviolet rays on the PC side, the luminescent substance emits light and changes the figure. Can be made. Further, when the luminance information changes with the passage of time, the image, video, character information, etc. can be changed in accordance with the luminance. In addition, by mounting an ultraviolet sensor on the PC side, the virtual figure can be changed from the detection result obtained using the detection signal from the sensor. Furthermore, in the present invention, by using heat such as body temperature as an external stimulus and touching the luminescent material of the information providing medium with a finger, the luminescent material can emit light and change the figure.

  Hereinafter, embodiments in which a composite image display system, a composite image display method, and a composite image display program according to the present invention are suitably implemented will be described with reference to the drawings. In the following description, a paper card is used as the information providing medium. However, the present invention is not limited to this, and for example, resin or metal may be used, and the shape may be flat like a card. It may have not only a shape but also a curved shape such as a sphere or a cylinder or an uneven shape.

<Composite image display system: functional configuration example>
Here, an example of a mechanism configuration of the composite image display system in the present embodiment will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a mechanism configuration of a composite image display system in the present embodiment. A composite image display system 10 shown in FIG. 1 includes an imaging device (imaging means) 11, a composite image display device 12, a mobile terminal 13, an image management database 14, a composite image display device 12 or a mobile terminal 13, and an image. It is configured to have a wired or wireless communication network 15 connected to the management terminal 14 in a state where data can be transmitted and received. In FIG. 1, the imaging device 11 may be integrally formed in the composite image display device 12. Further, the configurations of the imaging device 11 and the composite image display device 12 in the present embodiment can be replaced with, for example, a portable terminal 13 with a camera function as shown in FIG.

  A camera function (imaging means) in the imaging device 11 such as a digital camera or the portable terminal 13 captures an image including predetermined recognition information 21 printed on a card (information providing medium) 20. In the present embodiment, one or both of the recognition information 21 is printed by a change object whose luminance changes with the passage of time after irradiation with light set in advance in the present embodiment.

  The recognition information 21 is not limited to the QR code as shown in FIG. 1, but for example, a JAN code, ITF code, NW-7, CODE39, CODE128, UPC, PDF417, CODE49, Data Matrix, Maxi Code, or the like. A code can be used, and further, character information or a pattern that indicates the rotation direction such as “A”, a frame such as a square or a star, and the like may be used. In other words, it may be a graphic (for example, a two-dimensional barcode) including at least three points or a pattern (for example, L characters) that can specify three points. As the print information such as the recognition information 21, for example, at least one of characters, figures, patterns, images, and barcode information can be used. Further, by writing the recognition information 21 as print information, the position information for obtaining luminance information, and the composition position, orientation, size, and the like can be adjusted when the composition target image is composed with the photographed image. .

  As a result, various high value-added services can be provided using one or a plurality of print information. Details of the print information applied in this embodiment will be described later.

  The composite image display device 12 analyzes the image taken from the imaging device 11 and acquires luminance information, character information, thermal information, and the like from the content of the recognition information 21 included in the image. Further, the synthesized image display device 12 extracts a synthesis target image based on the acquired various pieces of information, synthesizes the extracted image with a photographed image, and displays the synthesized image on a screen such as a display.

  At this time, the synthesis target image, the program necessary for performing the synthesized image display processing obtained by the present embodiment, various update setting information, and the like are stored in advance in the synthesized image display device 12 as image groups, data information, and the like. May be acquired from the image management database 14 or the like capable of transmitting and receiving data connected via the communication network 15. Further, the captured image is transmitted from the composite image display device 12 to the image management database 14, and the extracted composition target image and the captured image are combined on the image management database side, and the combined image is combined with the composite image display device 12 or the like. It can be transmitted to the mobile terminal 13.

  In addition, the mobile terminal 13 performs processing equivalent to that of the imaging device 11 and the composite image display device 12 described above. When the portable terminal 13 is connected to the image management database 14, it is connected wirelessly based on the address information of the image management database 14 set in advance, and the above-described luminance information, character information, thermal information, etc. Based on this, the image to be synthesized and the photographed image are also sent to the image management database 14 to obtain a synthesized image synthesized on the database side.

  The image management database 14 synthesizes various kinds of information such as a target image to be synthesized and other setting information with respect to inquiry information (luminance information, character information, thermal information) from the synthesized image display device 12 and the portable terminal 13. This is provided to the image display device 12 and the portable terminal 13.

  As shown in FIG. 1, in the present embodiment, position (orientation) information in the x, y, and z coordinates of the card-like card 20 obtained with respect to the shooting direction of the camera can be represented. In addition, the frame information can be printed not only with a bold marker, but also with characters, numbers, figures, patterns, facial photographs, etc. on the frame itself.

  That is, as shown in FIG. 1, in the position information of the card 20, the x coordinate indicates the position in the horizontal direction with respect to the camera, the y coordinate indicates the position in the vertical direction with respect to the camera, and the z coordinate indicates the distance to the camera. Show. The posture information indicates the degree of rotation based on the position of a predetermined card with respect to the above-described x-axis, y-axis, and z-axis, and represents the posture of the card itself. Thus, when the user moves the card, card information (card type, three-dimensional position information, posture information) is acquired from the camera that has photographed the card.

  Therefore, by reading and analyzing these pieces of information, it is possible to select a large number of compositing target images according to various conditions, thereby providing a service with high added value. In addition, the card | curd 20 is not limited to paper shape, You may have metallicity, such as resin property and aluminum.

  Furthermore, in the present embodiment, the brightness of the change object after being irradiated with light from at least one light source 22 among ultraviolet light, infrared light, and visible light having a wavelength set in advance with respect to the card 20. By acquiring information, a target image can be provided to the user. Also, various composite images can be provided by changing the set wavelength.

<Composite Image Display Device 12: Functional Configuration Example>
Next, a functional configuration example of the composite image display device 12 will be described with reference to the drawings. Note that the functional configuration of the composite image display device 12 can be replaced by the portable terminal 13 and can also have each function in the image management database 14.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the composite image display apparatus. 2 includes an input unit 31, an output unit 32, a storage unit 33, a captured image acquisition unit 34, a recognition information analysis unit 35, a rotation direction detection unit 36, and a three-dimensional position. The acquisition unit 37, the image information extraction unit 38, the image synthesis unit 39, the transmission / reception unit 40, and the control unit 41 are configured.

  The input means 31 inputs input such as start / end of various instructions such as a captured image acquisition instruction, luminance information acquisition instruction, print information analysis instruction, thermal information acquisition instruction, image information extraction instruction, and image composition instruction from a user or the like. Accept. For example, the input unit 31 includes a keyboard and a pointing device such as a mouse in the case of a PC, and each operation button group in the case of the mobile terminal 13. The input unit 31 also has a function of inputting an image including an imaging part of the card 20 (information providing medium) taken by an imaging unit such as a digital camera.

  The output unit 32 displays / outputs the content input by the input unit 31 and the content executed based on the input content. Note that the output unit 32 includes a display, a speaker, and the like. Further, the output unit 32 may have a function such as a printer. In this case, the display result of the composite image, the captured image, the luminance information, the composite target image information, the print information, the thermal information, the composite image, etc. indicate. Further, the output means 32 can print each output content described above on a print medium such as paper and provide it to the user.

  The input means 31 and the output means 32 may be integrated input / output means such as a touch panel, for example. In this case, a predetermined position using a user's finger or a pen-type input device is used. You can input by touching.

  Further, the storage means 33 is a photographed image obtained by the photographed image acquisition means 34, luminance information obtained by the recognition information analysis means 35, print information obtained by the rotation direction detection means 36, and heat obtained by the three-dimensional position acquisition means 37. Various data such as information, image information obtained by the image information extraction means 38, and a composite image obtained by the image composition means 39 are stored.

  Further, in the present embodiment, the storage unit 33 stores the composition target image to be combined with the photographed image in association with the luminance information and print information of the change object obtained from the image obtained by photographing the card 20 and the like. Further, the storage means 33 can read out various data stored as required.

  The captured image acquisition unit 34 acquires an image included in the card 20 captured by the imaging device 11 such as a camera.

  The recognition information analysis unit 35 analyzes the recognition information included in the captured image captured by the imaging device 11 as the imaging unit. Note that the recognition information analysis unit 35 obtains the coordinates of a graphic (for example, a two-dimensional bar code) including at least three points or a pattern (for example, L letters, squares, etc.) that can be specified in the image. Then, projective conversion is performed on the acquired recognition information.

  Moreover, the recognition information analysis means 35 acquires the luminance information of the changed object after being irradiated from at least one light source among ultraviolet light, infrared light, visible light, and retroreflective observation light having a preset wavelength. . Thereby, by irradiating light of a light (light source) having a preset wavelength, the composite image can be changed and provided to the user. Note that the recognition information analyzing unit 35 can also use the average luminance value of the region of the change object written in the frame from the coordinates as the luminance information. As a result, it is possible to acquire luminance information with less error.

  Further, the recognition information analysis unit 35 presets a basic luminance value for the changing object, and acquires a difference (change) value between the luminance information of the changing object and the basic luminance value obtained from the captured image as the luminance information. As the basic luminance value, for example, luminance information of a change object obtained from an image captured in a state where light from the light source described above is not irradiated can be used. Accordingly, it is possible to provide a composite image without depending on the surrounding environment where the information providing medium is photographed.

  The rotation direction detection means 36 detects the rotation direction of the recognition information printed by the change object. That is, the rotation direction detection unit 36 detects a rotation method with respect to a certain reference direction based on the positions of the three points included in the recognition information.

  Specifically, the rotation direction detection means 36 can perform character analysis using a character reading tool such as OCR (Optical Character Reader). The print information may be the recognition information 21 or may use characters, figures, characters, barcode information, or the like. Specific examples of various information to be printed will be described later.

  The three-dimensional position acquisition unit 37 calculates three-dimensional coordinates from the state detected by the rotation direction detection unit. Specifically, for the captured real space image, the position in the real space according to the position, direction, size, etc. of the three points included in the recognition information is used as three-dimensional information with the captured image as a reference. Output.

The image information extraction unit 38 includes one or more of the information of the recognition information obtained by the recognition information analysis unit 35, the rotation direction obtained by the rotation direction detection unit, and the three-dimensional coordinate obtained by the three-dimensional position acquisition unit 37. Using the information, at least one target image to be synthesized is extracted.
Note that what is extracted by the image information extraction unit 38 is not limited to the image itself, but may be characters, video, patterns, character information, or the like. Further, the image information extraction unit 38 displays display instruction information indicating how to display the extracted information in the real space. Note that the display instruction information specifically indicates how to display the displayed information, such as which position, what size, how much is displayed, etc. Various information such as position information, format information, display information, and time information is included.

  The image synthesizing unit 39 synthesizes the captured image and the synthesis target image every time the synthesis information image (including still images, moving images, character information, etc.) is extracted by the image information extraction unit 38. Specifically, the image composition unit 39 displays characters, images, and videos in accordance with the time corresponding to various information obtained by the image information extraction unit 38.

  Note that the image composition means 39 can be displayed as a three-dimensional display on a real space image actually taken regardless of the position of the information providing medium. That is, the image synthesizing unit 39 displays a virtual space three-dimensional image model or the like corresponding to the angle and position of the information providing medium to be photographed.

  At this time, when a plurality of images are obtained by the image information extraction means 38, the image composition means 39 may synthesize all of the obtained plurality of composition target images, or one image for the photographed image. The compositing target image may be sequentially changed over time or repeatedly displayed.

  In addition, the image information extraction unit 38 described above acquires luminance information over time of a changed object, for example, every elapse of a predetermined time, and a synthesis target image corresponding to each luminance information over time as the luminance information changes. By sequentially extracting the images, the image composition means 39 can perform composition processing on the photographed image every time a composition target image is input, and can sequentially display the composition images. Accordingly, the image can be changed without moving the information providing medium, and a moving image or the like can be provided to the user.

  The transmission / reception means 40 is configured to display a desired external image (such as a photographed image or a compositing target image) from an external device (for example, the image management database 14) that can be connected using the communication network 15 or the like, or a composite image display in the present invention It is an interface for acquiring an execution program or the like for realizing processing.

  The control means 41 controls the entire components of the composite image display device 12. Specifically, the control unit 41 acquires a captured image, acquires rotation information, luminance information, print information, thermal information, and the like based on an instruction from the input unit 31 by a user, for example, Each control of each process such as extraction and image composition is performed.

<About recognition information 21>
Here, a specific example of the recognition information 21 written on the card 20 in the present embodiment will be described with reference to the drawings.

  FIG. 3 is a diagram illustrating a specific example of recognition information. The recognition information shown in FIG. 3 is composed of a figure (for example, a two-dimensional barcode) including at least three points or a pattern (for example, an L character) that can specify three points. For example, as shown in FIGS. 3A to 3I, one or more types of patterns (lines are also included in one type of patterns) are given to the card 20 to extract a composite image corresponding to the patterns. can do.

  For example, three points 51 may be used as shown in FIG. 3A, or a star 52 may be used as shown in FIG. 3B, and rotation from a predetermined reference angle as shown in FIG. A character 53 that can recognize an angle may be used. As shown in FIG. 3D, the position is recognized using three points obtained by one corner point of two end points of an L-shaped character 54. Also good.

  Furthermore, as shown in FIG. 3E, by printing character information, numerical information, etc. on the recognition information itself, it is possible to read the character information, etc., and obtain a corresponding composition target image. At this time, as shown in FIG. 3E, the rotation direction can be acquired by an L-shape formed by the recognition information. Further, as shown in FIG. 3F, more information can be acquired by adding a two-dimensional barcode 56 such as a one-dimensional barcode 55 or QR code to a part or all of the recognition information. It is possible to acquire a variety of corresponding compositing target images.

  Further, a QR code 57 may be printed as shown in FIG. 3 (g), and characters are put in the DataMatrix code 58 shown in FIG. 3 (h) or the frame 59 shown in FIG. 3 (i). Also good. 3A to 3I described above may be printed using one or a plurality of colors. Thereby, since the amount of information obtained from the card 20 is increased by using color information in addition to the recognition information and luminance information, a large number can be easily set for various conditions in image composition and the like.

  In addition to changing the composition target image according to the change (difference) value of the luminance information of the change object obtained by the presence or absence of irradiation of light (ultraviolet light, infrared light, visible light, etc.) to the card 20, printing is performed as well. By extracting one or a plurality of corresponding compositing target images from a variety of compositing target images based on the information, and combining them with the captured images and displaying them, a service with higher added value can be provided.

  Further, the print information described above may be visible or invisible. That is, by making it invisible, it can be used as security information such as authentication. In the present embodiment, the visible change is, for example, a phosphor for CRT (ZnS: Ag, Al, ZnS: Cu, Al, Y2O2S: Eu) or a lamp as a fluorescent paint manufactured by Nichia Corporation. Phosphor ((SrCaBaMg) 5 (PO4) 3Cl: Eu, LaPO4: Ce, Tb, Y2O3: Eu,), PDP phosphor (BaMgAl10O17: Eu, Zn2SiO4: Mn, (Y, Gd) BO3: Eu), X The phosphors for lines (CaWO4, Gd2O2S: Tb), phosphorescent phosphors (Sr4Al14O25: Eu, Dy), and Maruoka Educational Fluorescent / Night / UV paints are used as they are.

  Further, black light color, aqua glow, fluorescent paint spray, magic lumino paint, ultraviolet reflection paint, sun light marker, etc. can be used.

  Furthermore, LUMILUX Long After-Pigments, LUMLUUX Effect-Pigments, LUMLUUX C Pigments, LUMLUUX Z Pigments, LUMLUUX R Pigments, etc. can be used as fluorescent pigments from Honeywell.

  Moreover, as a nocturnal pigment of Nemoto Special Chemical Co., Ltd., N nocturnal (luminova) G-300, GLL-300, GSS, BG-300, BGL-300, V-300, B-300 can be used. Kasei Optonics True-Blue series fluorescent pigments, TASCO CO. , LTD thermochromic pigments, Tokyo Liquid Color, Asahi Products' Thermo-Temperature Paint Thermax Clear Line, etc. can be used. Furthermore, fluorescent dyes such as fluorescein, eosin, rhodamine B, and thioflavine, azobenzene that changes from colorless to orange by ultraviolet light, and diallyl ethene that changes from colorless to red by ultraviolet light can be used. Furthermore, N, N-diethylethylenediamine copper complex (43 ° C. or less reddish purple, 43 ° C. or more dark purple) that changes color by heat, pyridinium N-phenoxide betaine (methanol: red, ethanol: purple, isobutyl alcohol) that changes color depending on the solvent : Blue), phenolphthalein that changes from colorless to basic and red. A quinone diimine that changes from yellow to red by an acid, and a material that turns blue when an acid is added can be used.

  Further, as the invisible change object, for example, titanium mica whose color changes according to an angle, a polymer multilayer film, or a retroreflective material can be used. Further, although the appearance is not known, the color may be visible depending on the angle, and these materials can also be used for the card 20. It is possible to use a paint whose luminance information hardly changes over time.

<About retroreflection>
In the present invention, more accurate information can be read by using a retroreflective sheet as the information providing medium. The contents will be described below. As an example, a method for reading information from a laminated material that is a retroreflective sheet will be described.

  The information reading method according to the present invention is characterized in that information recorded on a laminated material is read by irradiating the laminated material as an information providing medium with light and observing retroreflected light from the laminated material. This information reading method can be implemented by the apparatus described in FIG. In the composite image display system 10 shown in FIG. 1, the reading unit includes a light source 22 (white light source), a light detection unit (CCD camera) that is the imaging device 11, a composite image display device 12, and the like.

  The light emitted from the white light source 22 travels to the card 20 made of a retroreflective sheet. Light directed toward the card 20 enters the card 20. The retroreflected light from the card 20 goes to the CCD camera and is detected there. The detection signal detected by the CCD camera is sent to the composite image display device 12 as shown in FIG. The composite image display device 12 analyzes the information recorded on the card 20 based on this detection signal.

  In addition, in order to read information from the card | curd 20 which concerns on this embodiment mentioned above, it is necessary to observe from the substantially the same direction as the irradiation direction of light. Therefore, for example, a half mirror may be provided between the light source 22 and the card 20. Accordingly, the CCD camera can be installed on the opposite side of the surface of the half mirror from the position where the light source is disposed, and the light source 22 and the CCD camera interfere with each other by using the half mirror. The retroreflected light can be preferably observed. In the present embodiment, for example, a configuration in which the position of the light source 22 and the position of the CCD camera are replaced may be employed.

  Specific examples of the present invention will be described below, but the present invention is not limited to the following configurations.

<Example 1>
-Transparent resin Co., Ltd. Seiko Advance Co., Ltd. screen printing ink SG-410 (90 parts) and toluene (10 parts) were mixed to prepare a transparent paint.
-Preparation of plate for screen printing Using a nylon 150 mesh screen, a plate of 10 mm square QR type two-dimensional code (14 x 14 cells, code information: SHISEIDO) was prepared.
-Retroreflective material Retroreflective material (microsphere fixed layer + reflective layer) Open type retroreflective film (LIGHT FORCETM LFU-1200, manufactured by Maruhito Co., Ltd., reflective layer: aluminum vapor deposition, diameter of glass beads: about 40-50 μm , The burial rate of beads: about 50%).
-Printing method of information code Using the screen printing plate on the surface of the retroreflective material (the surface where the transparent microspheres are exposed), the thickness of the two-dimensional code (transparent resin layer) with a screen printer Was printed to be about 10 μm. The printed two-dimensional code was hardly visually recognized in appearance, and the design of the retroreflective material was not impaired.
-Reading analysis of code information Using the apparatus shown in FIG. 1, the code information was read from the laminated material prepared above. Here, a CCD camera code reader THIR-3000LP (manufactured by Tohken Co., Ltd.)) was used as the light detection means and the information code analysis means. As a result of reading and analyzing the code information, it was possible to read “SHISEIDO” actually recorded as code information.

<Composite Image Display System 10: Hardware Configuration>
Here, in the composite image display system 10 described above, an execution program (composite image display program) capable of causing a computer to execute each function is generated, and the execution program is installed in, for example, a general-purpose personal computer or server. Thus, the composite image display process and the like in the present invention can be realized.

  Here, a hardware configuration example of a computer capable of realizing the composite image display process in the present embodiment will be described with reference to the drawings. FIG. 4 is a diagram illustrating an example of a hardware configuration capable of realizing the composite image display process according to the present embodiment.

  4 includes an input device 61, an output device 62, a drive device 63, an auxiliary storage device 64, a memory device 65, a CPU (Central Processing Unit) 66 that performs various controls, and a network connection device. 67, which are connected to each other by a system bus B.

  The input device 61 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like. The input device 61 includes an input unit that inputs an image including an information providing medium photographed from an imaging unit such as a camera.

  The output device 62 has a display for displaying various windows and data necessary for operating the computer main body for performing processing according to the present invention, and displays the program execution progress, results, and the like by the control program of the CPU 66. can do.

  The input device 61 and the output device 62 may be integrated input / output means such as a touch panel. In this case, a predetermined position using a user's finger or a pen-type input device is used. You can input by touching.

  Here, the execution program installed in the computer main body in the present invention is provided by, for example, a portable recording medium 68 such as a USB (Universal Serial Bus) memory or a CD-ROM. The recording medium 68 on which the program is recorded can be set in the drive device 63, and the execution program included in the recording medium 68 is installed from the recording medium 68 to the auxiliary storage device 64 via the drive device 63.

  The auxiliary storage device 64 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, and the like, and can perform input / output as necessary.

  The memory device 65 stores an execution program read from the auxiliary storage device 64 by the CPU 66. The memory device 65 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 66 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 65. Thus, each process in the composite image display can be realized. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 64, and the execution result and the like can also be stored.

  The network connection device 67 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or an execution in the present invention The program itself can be provided to other terminals.

  With the hardware configuration as described above, the composite image display process according to the present invention can be executed. Further, by installing the program, the composite image display process according to the present invention can be easily realized by a general-purpose personal computer or the like.

<Composite image display processing procedure>
Next, a synthetic image display processing procedure in the present embodiment will be described. FIG. 5 is a flowchart illustrating an example of a composite image display processing procedure in the present embodiment.

  In the composite image display process shown in FIG. 5, first, an image including an information providing medium photographed by an imaging means such as a camera is acquired (S01), and recognition information included in the image photographed from the information providing medium is extracted (S02). ). Here, it is determined whether or not the recognition information has been extracted (S03). If the recognition information has not been extracted (NO in S03), the process returns to S01 and the subsequent process is performed.

  If recognition information is extracted in the process of S03 (YES in S03), projective transformation is performed on the extracted recognition information figure (S04), and whether or not the recognition information is unanalyzed. Is determined (S05). Here, when unanalyzed (YES in S05), the recognition information is analyzed. If the recognition information is analyzed after the processing of S05 or in the processing of S05 (NO in S05), the rotation direction of the recognition information is recognized (S07).

  Also, three-dimensional coordinates are calculated from the rotation information, position information, etc. (S08). Further, based on the calculated result, a model display of the composite image is performed (S09). Further, it is determined whether or not the process is to be terminated (S10). If the process is not to be terminated (NO in S10), the process returns to S01 to perform the subsequent process. When the process is to be ended (YES in S10), the entire composite image display process is ended.

  In the above-described processing, the current luminance information (basic luminance value) is acquired, for example, the average luminance of a predetermined color (for example, green) within the frame is acquired, and the content is stored. The model to be displayed may be changed according to the luminance information or the change in luminance information or the average luminance.

  By the processing described above, a highly accurate image with excellent added value can be provided. In the above-described processing, an input image or an image obtained by each processing can be input to the storage unit 33. Further, in each process, the output unit 32 or the like may be used to acquire an input image, an execution progress or result of each process, a case where an instruction is given to the user by the input unit 31, and the like.

<Example 2>
Next, as Example 2, an example of an image displayed in the present embodiment will be described with reference to the drawings. FIG. 6 is a diagram showing an example of a composite image displayed in the present invention.

  First, in the example shown in FIG. 6A, a frame 71 is printed on the card (information providing medium) 70. At this time, the composite images 72-1 and 72-2 acquired from the database or the like using the luminance information of the print information (frame, character information, etc.) obtained from the first photographed image is the coordinate information of the frame 71. Correspondingly displayed on the screen. In FIG. 6A, as an example, a flower pot (composite image) 72-1 and a seedling (composite image) 72-2 planted in the flower pot (composite image) 72-1 are displayed.

  Next, the luminous paint (Nemoto Special Chemicals Co., Ltd. Luminova G-300) was printed on the part shown in FIG.6 (b). Here, by irradiating a light source such as a black light, the luminance information of the paint on which the print information is printed changes, so that instead of the seedling (composite image) 72-2 shown in FIG. ) 72-3 could be displayed.

  As described above, in this embodiment, various types of images can be displayed by combining and displaying the type of the synthesis target image corresponding to the luminance information. In addition, by properly using a paint whose luminance changes with time, for example, a moving image in which a seedling grows, blooms, and then withers can be provided. In this embodiment, the orientation of the image and each displayed image is changed according to the position of the corner of the frame, the display method such as enlargement or reduction is changed, or the corresponding sound is displayed. You can also

  Moreover, FIG. 7 is a figure which shows an example of the three-dimensional display corresponding to the recognition information in this embodiment. By applying this embodiment, for example, as shown in FIG. 7A, a three-dimensional cube 81 is synthesized and displayed on the information providing medium 20 on the screen 80, and the recognition information is further displayed thereon. The character information 82 included in can be displayed in 3D. Further, as shown in FIG. 7B, the character information 82 obtained from the recognition information of the information providing medium 20 is three-dimensionally displayed on the screen 80 on the basis of the real space of the photographed image. I was able to. In the example of FIG. 7B, the characters “shiseido” as character information can be displayed on the screen 80 in accordance with the width of the screen on which the image is taken. Further, as shown in FIG. 7C, the character information 82 can be displayed on the surface of the three-dimensional shape 83 displayed on the information providing medium 20 on the screen 80. In the example of FIG. 7C, the characters “shiseido” as the character information 82 are displayed on the screen 80 by rotating around the earth as the three-dimensional shape 83.

  As described above, according to the present invention, at least three points or recognition information consisting of a figure (for example, a two-dimensional barcode) including a pattern (for example, an L character) that can specify three points is captured from the captured image. The rotation direction and three-dimensional coordinates of a certain figure are acquired, and the three-dimensional information (including figures and characters) in the virtual space selected from the information obtained by the figure is made to correspond to the rotation direction and the three-dimensional coordinates into a real space image. The predetermined color, size, and shape can be freely displayed at an arbitrary position.

<Other embodiments>
Here, in the above-described embodiment, it is possible to capture a change object (two-dimensional barcode, etc.) such as print information printed on an information providing medium such as a card, and analyze the print information from the captured image. However, the present invention is not limited to this. For example, the information providing medium is a display device such as a display, and an image obtained by photographing the above-described change object displayed on the display is displayed. By acquiring and analyzing a change object included in the acquired image, it is possible to extract a corresponding stereoscopic image and display a combined image obtained by combining the captured image with the captured image. In this embodiment, for example, a change object displayed on the screen by, for example, video display or slide display can be easily switched over time, and the photographing side described above includes the image including the change object displayed on the screen. By shooting in this manner, a stereoscopic image corresponding to the changed object can be synthesized and displayed as a moving image.

  Furthermore, in this embodiment, as described above, not only the information printed or displayed on the information providing medium is used, but also, for example, a shadow projected on the ground or a wall by illuminating is used as a change object, and the shadow is converted into a change object. Projecting with additional information such as a two-dimensional code, and analyzing the shadow from the image obtained by photographing the shadow, the corresponding stereoscopic image was extracted, and the image was synthesized with the photographed image A composite image can be displayed. Further, the image to be extracted may be changed depending on the shadow density or the like.

  As described above, according to the present invention, it is possible to provide a highly accurate image or service excellent in added value. Further, by irradiating light from a light source such as a light having a preset wavelength, a target image can be provided to the user. Also, various composite images can be provided by changing the set wavelength.

  Further, it is possible to provide a composite image without depending on the surrounding environment where the information providing medium is photographed. Further, when the luminance information changes, another image to be combined can be selected and combined and displayed. Thereby, a moving image etc. can also be provided. In addition, more combined images can be provided using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value can be provided. Furthermore, by using the thermal information, it is possible to provide a service according to the environment at that time. Further, even if the change object is invisible, a composite image can be provided by using thermal information.

  Note that the above-described apparatus can also be applied to a mobile terminal or the like. Therefore, by applying the present invention, for example, in combination with a gift card or the like, a card service in which a flower blooms when it hits the sun, an Internet authentication field, a food inspection field such as a food freshness analysis, It can also be applied to the field of health checkups that require color judgment such as urine tests.

  In other words, in the present invention, for example, a coating material that reacts to ultraviolet light or light source power (light intensity) is applied to the information providing medium, and the information providing medium is set in a portion exposed to external light such as sunlight. Is captured by a photographing device, and image data is acquired over time and input to the system. In addition, the images in the PC and the mobile terminal change realistically according to time and weather. Thereby, it is possible to obtain an image suitable for the season, climate, etc. even in a basement or a room inside the building where outside light does not reach, or in a place that is actually far away.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

It is a figure which shows an example of the mechanism structure of the synthesized image display system in this embodiment. It is a figure which shows an example of a function structure of a synthesized image display apparatus. It is a figure which shows the specific example of recognition information. It is a figure which shows an example of the hardware constitutions which can implement | achieve the composite image display process in this embodiment. It is a flowchart which shows an example of the composite image display process sequence in this embodiment. It is a figure which shows an example of the synthesized image displayed in this invention. It is a figure which shows an example of the three-dimensional display corresponding to the recognition information in this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Composite image display system 11 Imaging device 12 Composite image display device 13 Portable terminal 14 Image management database 15 Communication network 20,70 Card (information provision medium)
DESCRIPTION OF SYMBOLS 21 Recognition information 22 Light source 31 Input means 32 Output means 33 Accumulation means 34 Captured image acquisition means 35 Recognition information analysis means 36 Rotation direction detection means 37 Three-dimensional position acquisition means 38 Image information extraction means 39 Image composition means 40 Transmission / reception means 41 Control means 51 points 52 Stars 53, 54 Characters 55 One-dimensional barcode 56 Two-dimensional barcode 57 QR code 58 DataMatrix code 59 Frame 61 Input device 62 Output device 63 Drive device 64 Auxiliary storage device 65 Memory device 66 CPU
67 Network connection device 72 Composite image 80 Screen 81 Cube 82 Character information 83 3D shape

Claims (15)

In a composite image display system that combines and displays another image at a predetermined position of a photographed image,
Imaging means for imaging an information providing medium in which predetermined information is recorded by a change object whose luminance changes with the passage of time after irradiation of light set in advance;
Recognition information analysis means for analyzing recognition information and luminance information having a predetermined shape included in the image taken by the imaging means;
Three-dimensional position acquisition means for acquiring a three-dimensional position from the recognition information obtained by the recognition information analysis means;
Image information extraction means for extracting an image corresponding to the recognition information among a plurality of synthesis target images accumulated in advance using the three-dimensional coordinates obtained by the three-dimensional position acquisition means;
A composite image display system comprising: an image compositing means for three-dimensionally compositing the target image extracted by the image information extracting means with reference to the real space of the photographed image and displaying the composited image. The recognition information analyzing means includes
2. The composite image according to claim 1, wherein luminance information of the change object after being irradiated from at least one light source among ultraviolet light, infrared light, and visible light having a preset wavelength is acquired. Display system. The image information extraction means sequentially extracts a synthesis target image corresponding to each luminance information at a predetermined time according to a change in luminance information with time of the changed object when the recognition information is printed,
3. The composition according to claim 1, wherein the image synthesizing unit synthesizes the photographed image and the synthesis target image every time the synthesis target image is extracted by the image information extraction unit. Image display system. Rotation direction detection means for analyzing the orientation of the recognition information written on the information providing medium by the change object,
The image information extraction means includes
4. The target image to be synthesized is extracted based on luminance information obtained by the recognition information analysis unit and rotation information obtained by the rotation direction detection unit. 5. The described composite image display system. The recognition information is
The composite image display system according to any one of claims 1 to 4, wherein the composite image display system is at least one of a character, a figure, a pattern, an image, and barcode information. The image composition means includes
Based on the posture information including at least one of the position, angle, and orientation of the information providing medium included in the captured image, and the luminance information, display, non-display, size, 6. The composite image display system according to claim 1, wherein the position and direction are controlled.   The composite image display system according to claim 1, wherein the light source is a retroreflection observation light, and the information providing medium is a retroreflection sheet. In a combined image display method of combining and displaying another image at a predetermined position of a photographed image,
An imaging step of imaging an information providing medium in which predetermined information is recorded by a change object whose luminance changes with the passage of time after irradiation with light set in advance;
A recognition information analysis step for analyzing recognition information and luminance information having a predetermined shape included in the image captured by the imaging step;
A three-dimensional position acquisition step for acquiring a three-dimensional position from the recognition information obtained by the recognition information analysis step;
An image information extraction step for extracting an image corresponding to the recognition information among a plurality of synthesis target images stored in advance using the three-dimensional coordinates obtained by the three-dimensional position acquisition step;
A composite image display method comprising: an image composition step of three-dimensionally compositing the composition target image extracted in the image information extraction step with reference to a real space of the photographed image and displaying the composited image. The recognition information analysis step includes
9. The composite image according to claim 8, wherein luminance information of the change object after being irradiated from at least one light source of ultraviolet light, infrared light, and visible light having a preset wavelength is acquired. Display method. The image information extraction step sequentially extracts a synthesis target image corresponding to each luminance information for each predetermined time according to a change in luminance information over time of the changed object when the recognition information is printed,
The composition according to claim 8 or 9, wherein the image composition step performs composition of the captured image and the composition target image every time the composition target image is extracted by the image information extraction step. Image display method. A rotation direction detecting step of analyzing a direction of recognition information written on the information providing medium by the change object;
The image information extraction step includes:
The target image to be synthesized is extracted based on luminance information obtained by the recognition information analysis step and rotation information obtained by the rotation direction detection step. The composite image display method as described. The recognition information is
The composite image display method according to claim 8, wherein the composite image display method is at least one of a character, a figure, a pattern, an image, and barcode information. The image composition step includes
Based on the posture information including at least one of the position, angle, and orientation of the information providing medium included in the captured image, and the luminance information, display, non-display, size, The composite image display method according to claim 8, wherein the position and direction are controlled.   The composite image display method according to claim 8, wherein the light source is a retroreflection observation light, and the information providing medium is a retroreflection sheet.   A composite image display program for causing a computer to execute processing relating to each means of the composite image display system according to any one of claims 1 to 7.

Images