JP5258548B2 - Composite image display system, composite image display method, and composite image display program - Google Patents

Description

  The present invention relates to a composite image display system, a composite image display method, and a composite image display program. In particular, the present invention relates to a composite image display system, a composite image display method, and a composite for providing a highly accurate image with excellent added value. The present invention relates to an image display program.

  Conventionally, for example, a card on which a pattern or the like is drawn is photographed, and a MIDI (Musical Instrument Digital Interface) signal for controlling a musical instrument is output based on the type of card and three-dimensional position information detected from the photographed camera image. In addition, a technique for outputting video is known (for example, see Patent Document 1).

  In addition, there is known a technique for outputting various images corresponding to the movement contents of a card by selecting an image using not only the information of the card to be photographed but also the speed information of the card itself (for example, Patent Documents). 2).

Further, there is a conventional system tool that recognizes a graphic on a paper medium with a camera and displays a virtual graphic on a display of a personal computer (for example, see Non-Patent Document 1).
JP 2002-32076 A Japanese Patent Laid-Open No. 2003-271928 "http://www.hit.washington.edu/arttoolkit/", "ARTToolKit"

  However, in the above-described prior art, by moving a card or the like in a predetermined direction (parallel or rotation), it is possible to display an image to be synthesized with a motion, but synthesis is performed with the card or the like stopped. The image to be changed could not be changed. For this reason, for example, it is only possible to synthesize a motion picture produced in advance, and only a specific operation can be performed for each card type. Therefore, highly accurate images and videos with excellent added value have not been provided.

  Further, in the virtual graphic display system shown in Non-Patent Document 1 described above, once the virtual graphic is displayed, it is difficult to change the graphic. That is, the figure cannot be changed according to the external environmental change.

  The present invention has been made in view of the above-described problems, and provides a composite image display system, a composite image display method, and a composite image display program for providing a highly accurate image with excellent added value. For the purpose.

  In order to solve the above-described problems, the present invention employs means for solving the problems having the following characteristics.

According to the first aspect of the present invention, in a composite image display system that combines and displays another image at a predetermined position of a photographed image, the luminance changes over time after irradiation with a preset light. Imaging means for imaging an information providing medium in which predetermined information is recorded by the changing object, luminance information acquisition means for acquiring luminance information of the changing object included in the captured image captured by the imaging means, Using the luminance information obtained by the luminance information acquisition unit, the image information extraction unit that extracts an image corresponding to the luminance information from among a plurality of synthesis target images stored in advance, and the image information extraction unit the composited image and the captured image are synthesized and the synthesized image have a image synthesizing means for displaying said image information extracting means, with the change with time in luminance information of the change thereof, the predetermined time The synthesis target image corresponding to each luminance information is sequentially extracted, and the image synthesis unit synthesizes the captured image and the synthesis target image every time the synthesis target image is extracted by the image information extraction unit. It is characterized by performing .

According to the first aspect of the present invention, it is possible to provide a highly accurate image or service with excellent added value. In addition, moving images and the like can be provided by sequentially changing images to be combined.

  The invention described in claim 2 is characterized in that the luminance information acquisition means uses the average luminance value of the region of the change object included in the image as luminance information.

  According to the second aspect of the present invention, it is possible to acquire luminance information with few errors.

  In the invention described in claim 3, the luminance information acquisition means is a luminance of the change object after being irradiated from at least one light source of ultraviolet light, infrared light, and visible light having a preset wavelength. It is characterized by acquiring information.

  According to the third aspect of the present invention, a target image can be provided to the user by irradiating light from a light source such as a light having a preset wavelength. Also, various composite images can be provided by changing the set wavelength.

  In the invention described in claim 4, the luminance information acquisition unit presets a basic luminance value for the change object, and obtains a difference value between the luminance information of the change object obtained from the photographed image and the basic luminance value. It is obtained as luminance information.

  According to the fourth aspect of the present invention, a composite image can be provided without depending on the surrounding environment where the information providing medium is photographed.

According to a fifth aspect of the present invention, there is provided print information analysis means for analyzing print information written on the information providing medium by the change object, and the image information extraction means is obtained by the luminance information acquisition means. A target image to be synthesized is extracted based on luminance information and print information obtained by the print information analyzing means.

According to the invention described in claim 5, it is possible to provide a service with higher added value.

The invention described in claim 6 is characterized in that the print information is at least one of a character, a figure, a pattern, an image, and barcode information.

According to the invention described in claim 6, it is possible to provide a variety of services with high added value using one or a plurality of print information.

According to a seventh aspect of the present invention, the image synthesizing unit is based on posture information including at least one of a position, an angle, and a direction of the information providing medium included in the captured image, and the luminance information. Display, non-display, size, position, and direction are controlled for the synthesis target image.

According to the seventh aspect of the invention, it is possible to provide more composite images using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value. Can be provided.

The invention described in claim 8 has thermal information acquisition means for acquiring thermal information of the change object, and the image information extraction means is luminance information obtained by the luminance information acquisition means, and print information analysis means. The target image to be synthesized is extracted using at least one of the print information obtained by the above and the thermal information obtained by the thermal information acquisition means.

According to invention of Claim 8, the service according to the environment at that time etc. can be provided by using thermal information. Even if the paint is invisible, a composite image can be provided by using thermal information.

According to a ninth aspect of the present invention, in the composite image display method for combining and displaying another image at a predetermined position of the photographed image, the luminance changes with the passage of time after irradiation with the preset light. An imaging step of imaging an information providing medium in which predetermined information is recorded by the changing object, a luminance information acquiring step of acquiring luminance information of the changing object included in the captured image captured by the imaging step, Using the luminance information obtained in the luminance information acquisition step, an image information extraction step for extracting an image corresponding to the luminance information among a plurality of synthesis target images accumulated in advance, and the image information extraction step. the composited image and the captured image are synthesized and possess an image synthesizing step of displaying a composite image, the image information extraction step, a time of the change thereof In accordance with the change in the degree information, the compositing target images corresponding to each luminance information for each predetermined time are sequentially extracted, and the image compositing step is performed each time the compositing target image is extracted by the image information extracting step. The image is synthesized with the synthesis target image .

According to invention of Claim 9 , the highly accurate image or service excellent in added value property can be provided. In addition, moving images and the like can be provided by sequentially changing images to be combined.

The invention described in claim 10 is characterized in that the luminance information acquisition step uses an average luminance value of the region of the change object included in the image as luminance information.

According to the tenth aspect of the present invention, it is possible to acquire luminance information with little error.

In the invention described in claim 11 , in the luminance information acquisition step, the luminance of the change object after being irradiated from at least one light source of ultraviolet light, infrared light, and visible light having a preset wavelength. It is characterized by acquiring information.

According to the eleventh aspect of the present invention, a target image can be provided to the user by irradiating light from a light source such as a light having a preset wavelength. Also, various composite images can be provided by changing the set wavelength.

In the invention described in claim 12 , in the luminance information acquisition step, a basic luminance value for the change object is set in advance, and a difference value between the luminance information of the change object obtained from the captured image and the basic luminance value is obtained. It is obtained as luminance information.

According to the twelfth aspect of the present invention, a composite image can be provided without depending on the surrounding environment where the information providing medium is photographed.

The invention described in claim 13 has a print information analysis step of analyzing the print information written on the information providing medium by the change object, and the image information extraction step is obtained by the luminance information acquisition step. A target image to be synthesized is extracted based on luminance information and print information obtained by the print information analysis step.

According to the thirteenth aspect of the invention, a service with higher added value can be provided.

The invention described in claim 14 is characterized in that the print information is at least one of a character, a figure, a pattern, an image, and barcode information.

According to the fourteenth aspect of the present invention, it is possible to provide a variety of services with high added value using one or a plurality of print information.

According to a fifteenth aspect of the present invention, the image composition step is based on posture information including at least one of a position, an angle, and a direction of the information providing medium included in the captured image, and the luminance information. Display, non-display, size, position, and direction are controlled for the synthesis target image.

According to the fifteenth aspect of the present invention, it is possible to provide more composite images using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value. Can be provided.

The invention described in claim 16 includes a thermal information acquisition step of acquiring thermal information of the change object, and the image information extraction step includes the luminance information obtained by the luminance information acquisition step, and the thermal information acquisition step. The target image to be synthesized is extracted using a plurality of pieces of information among the thermal information obtained by the printing information and the printing information obtained by the printing information analysis step.

According to the invention described in claim 16 , by using the thermal information, it is possible to provide a service according to the environment at that time. Even if the paint is invisible, a composite image can be provided by using thermal information.

The invention described in claim 17 is a composite image display program for causing a computer to function as each means included in the composite image display system according to any one of claims 1 to 8 .

According to the invention of the seventeenth aspect , it is possible to provide a highly accurate image or service excellent in added value. Further, by installing the program, the composite image display process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  ADVANTAGE OF THE INVENTION According to this invention, the highly accurate image or service excellent in added value property can be provided.

<Outline of the present invention>
The present invention uses, for example, an imaging means such as a camera and a PC (Personal Computer) connected to the camera, a mobile terminal with a camera, etc., and the luminance changes with the passage of time after irradiation with a preset light. An information providing medium (for example, card-like paper or resin sheet) in which predetermined information is recorded by a change object is imaged, and an image (composite target image) such as a virtual figure corresponding to information included in the photographed image Is combined with the captured image and displayed on a screen such as a display.

  At this time, in the present invention, as a change used for writing information on the information providing medium, for example, a light emitting / absorbing substance that emits and absorbs an external stimulus (ultraviolet light, infrared light, visible light, heat), or the environment A substance that causes a color change (such as retroreflection) can be used to acquire luminance information of a change object on the information providing medium from a captured image, and an image (still image, moving image, (Including character information and the like) is extracted from a previously stored image group, and the extracted image and the captured image are combined and displayed on the screen. Note that the luminance information in the present invention does not mean only light emission (brightness) but also includes color information such as hue, brightness, and saturation.

  Specifically, in the present invention, when ultraviolet rays are used as an external stimulus, for example, by illuminating the information providing medium with a light (light source) containing ultraviolet rays on the PC side, the luminescent substance emits light and changes the figure. Can be made. Further, when the luminance information changes with the passage of time, the image, video, character information, etc. can be changed in accordance with the luminance. In addition, by mounting an ultraviolet sensor on the PC side, the virtual figure can be changed from the detection result obtained using the detection signal from the sensor. Furthermore, in the present invention, by using heat such as body temperature as an external stimulus and touching the luminescent material of the information providing medium with a finger, the luminescent material can emit light and change the figure.

  Hereinafter, embodiments in which a composite image display system, a composite image display method, and a composite image display program according to the present invention are suitably implemented will be described with reference to the drawings. In the following description, a paper card is used as the information providing medium. However, the present invention is not limited to this, and for example, resin or metal may be used, and the shape may be flat like a card. It may have not only a shape but also a curved shape such as a sphere or a cylinder or an uneven shape.

<Composite image display system: functional configuration example>
Here, an example of a mechanism configuration of the composite image display system in the present embodiment will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating an example of a mechanism configuration of a composite image display system in the present embodiment. A composite image display system 10 shown in FIG. 1 includes an imaging device (imaging means) 11, a composite image display device 12, a mobile terminal 13, an image management database 14, a composite image display device 12 or a mobile terminal 13, and an image. It is configured to have a wired or wireless communication network 15 connected to the management terminal 14 in a state where data can be transmitted and received. In FIG. 1, the imaging device 11 may be integrally formed in the composite image display device 12. Further, the configurations of the imaging device 11 and the composite image display device 12 in the present embodiment can be replaced with, for example, a portable terminal 13 with a camera function as shown in FIG.

  A camera function (imaging means) in the imaging device 11 such as a digital camera or the portable terminal 13 captures an image including character information 22 in a predetermined frame 21 printed on a card (information providing medium) 20. In the present embodiment, one or both of the frame 21 and the character information 22 are printed with a change in the present embodiment. The character information 22 is not limited to characters such as “A”, and the pattern may be barcode information or the like. Further, as the printing information such as the frame 21 and the character information 22, at least one of characters, figures, patterns, images, and barcode information can be used, for example. In addition, by describing the frame 21 as print information, position information for obtaining luminance information, and the composition position, orientation, size, and the like can be adjusted when the composition target image is composed with the captured image.

  As a result, various high value-added services can be provided using one or a plurality of print information. Details of the print information applied in this embodiment will be described later.

  The composite image display device 12 analyzes the image photographed from the imaging device 11, and acquires luminance information, character information, thermal information, and the like from the contents of the frame 21 and the character information 22 included in the image. Further, the synthesized image display device 12 extracts a synthesis target image based on the acquired various pieces of information, synthesizes the extracted image with a photographed image, and displays the synthesized image on a screen such as a display.

  At this time, the synthesis target image, the program necessary for performing the synthesized image display processing obtained by the present embodiment, various update setting information, and the like are stored in advance in the synthesized image display device 12 as image groups, data information, and the like. May be acquired from the image management database 14 or the like capable of transmitting and receiving data connected via the communication network 15. Further, the captured image is transmitted from the composite image display device 12 to the image management database 14, and the extracted composition target image and the captured image are combined on the image management database side, and the combined image is combined with the composite image display device 12 or the like. It can be transmitted to the mobile terminal 13.

  In addition, the mobile terminal 13 performs processing equivalent to that of the imaging device 11 and the composite image display device 12 described above. When the portable terminal 13 is connected to the image management database 14, it is connected wirelessly based on the address information of the image management database 14 set in advance, and the above-described luminance information, character information, thermal information, etc. Based on this, the image to be synthesized and the photographed image are also sent to the image management database 14 to obtain a synthesized image synthesized on the database side.

  The image management database 14 synthesizes various kinds of information such as a target image to be synthesized and other setting information with respect to inquiry information (luminance information, character information, thermal information) from the synthesized image display device 12 and the portable terminal 13. This is provided to the image display device 12 and the portable terminal 13.

  As shown in FIG. 1, in the present embodiment, position (orientation) information in the x, y, and z coordinates of the card-like card 20 obtained with respect to the shooting direction of the camera can be represented. In addition, the frame information can be printed not only with a bold marker, but also with characters, numbers, figures, patterns, facial photographs, etc. on the frame itself.

  That is, as shown in FIG. 1, in the position information of the card 20, the x coordinate indicates the position in the horizontal direction with respect to the camera, the y coordinate indicates the position in the vertical direction with respect to the camera, and the z coordinate indicates the distance to the camera. Show. The posture information indicates the degree of rotation based on the position of a predetermined card with respect to the above-described x-axis, y-axis, and z-axis, and represents the posture of the card itself. Thus, when the user moves the card, card information (card type, three-dimensional position information, posture information) is acquired from the camera that has photographed the card.

  Therefore, by reading and analyzing these pieces of information, it is possible to select a large number of compositing target images according to various conditions, thereby providing a service with high added value. In addition, the card | curd 20 is not limited to paper shape, You may have metallicity, such as resin property and aluminum.

  Further, in the present embodiment, the brightness of the change object after being irradiated with light from at least one light source 23 among ultraviolet light, infrared light, and visible light having a wavelength set in advance with respect to the card 20. By acquiring information, a target image can be provided to the user. Also, various composite images can be provided by changing the set wavelength.

<Composite Image Display Device 12: Functional Configuration Example>
Next, a functional configuration example of the composite image display device 12 will be described with reference to the drawings. Note that the functional configuration of the composite image display device 12 can be replaced by the portable terminal 13 and can also have each function in the image management database 14.

  FIG. 2 is a diagram illustrating an example of a functional configuration of the composite image display apparatus. 2 includes an input unit 31, an output unit 32, a storage unit 33, a captured image acquisition unit 34, a luminance information acquisition unit 35, a print information analysis unit 36, and a thermal information acquisition unit. Means 37, image information extraction means 38, image composition means 39, transmission / reception means 40, and control means 41 are provided.

  The input means 31 inputs input such as start / end of various instructions such as a captured image acquisition instruction, luminance information acquisition instruction, print information analysis instruction, thermal information acquisition instruction, image information extraction instruction, and image composition instruction from a user or the like. Accept. For example, the input unit 31 includes a keyboard and a pointing device such as a mouse in the case of a PC, and each operation button group in the case of the mobile terminal 13. The input unit 31 also has a function of inputting an image including an imaging part of the card 20 (information providing medium) taken by an imaging unit such as a digital camera.

  The output unit 32 displays / outputs the content input by the input unit 31 and the content executed based on the input content. Note that the output unit 32 includes a display, a speaker, and the like. Further, the output unit 32 may have a function such as a printer. In this case, the display result of the composite image, the captured image, the luminance information, the composite target image information, the print information, the thermal information, the composite image, etc. indicate. Further, the output means 32 can print each output content described above on a print medium such as paper and provide it to the user.

  The input means 31 and the output means 32 may be integrated input / output means such as a touch panel, for example. In this case, a predetermined position using a user's finger or a pen-type input device is used. You can input by touching.

  Further, the storage means 33 is a photographic image obtained by the photographic image acquisition means 34, luminance information obtained by the luminance information acquisition means 35, print information obtained by the print information analysis means 36, and thermal information obtained by the thermal information acquisition means 37. Various data such as image information obtained by the image information extracting means 38 and a synthesized image obtained by the image synthesizing means 39 are stored.

  Further, in the present embodiment, the storage unit 33 stores the composition target image to be combined with the photographed image in association with the luminance information and print information of the change object obtained from the image obtained by photographing the card 20 and the like. Further, the storage means 33 can read out various data stored as required.

  The captured image acquisition unit 34 acquires an image included in the card 20 captured by the imaging device 11 such as a camera.

  The luminance information acquisition unit 35 acquires the luminance information of the change object included in the captured image captured by the imaging device 11 as the imaging unit. Note that the luminance information acquisition unit 35 acquires, for example, the coordinates of the above-described frame included in the image, and uses the average luminance value of the region of the change object written in the frame from the acquired coordinates as the luminance information. As a result, it is possible to acquire luminance information with less error. Further, the luminance information acquisition unit 35 acquires luminance information of the changed object after being irradiated from at least one light source among ultraviolet light, infrared light, and visible light having a preset wavelength. Thereby, by irradiating light of a light (light source) having a preset wavelength, the composite image can be changed and provided to the user.

  Further, the luminance information acquisition unit 35 presets a basic luminance value for the change object, and acquires a difference (change) value between the luminance information of the change object obtained from the captured image and the basic luminance value as the luminance information. As the basic luminance value, for example, luminance information of a change object obtained from an image captured in a state where light from the light source described above is not irradiated can be used. Accordingly, it is possible to provide a composite image without depending on the surrounding environment where the information providing medium is photographed.

  The print information analysis unit 36 analyzes information printed by the change object. Specifically, the print information analyzing means 36 can perform character analysis using a character reading tool such as OCR (Optical Character Reader). The print information may be the character information 22 in the frame 21, or the frame 21 itself may directly use characters, figures, characters, barcode information, or the like. Specific examples of various information to be printed will be described later.

  The thermal information acquisition unit 37 acquires thermal information of a changed object that has been changed by a user or the like directly touching the changed object with a finger, a palm, or the surrounding environment (temperature or the like). In addition, as the heat information acquisition means 37, for example, a heat method can be acquired using a general heat detection sensor such as a thermosensor.

  The image information extraction unit 38 uses one or more information among the luminance information obtained by the luminance information acquisition unit 35, the print information obtained by the print information analysis unit 36, and the thermal information obtained by the thermal information acquisition unit 37. At least one of the target image or video to be synthesized, character information, etc. is extracted. Note that what is extracted by the image information extraction unit 38 is not limited to the image itself, but may be characters, video, patterns, character information, or the like.

  The image synthesizing unit 39 synthesizes the captured image and the synthesis target image every time the synthesis information image (including still images, moving images, character information, etc.) is extracted by the image information extraction unit 38. At this time, when a plurality of images are obtained by the image information extraction means 38, the image composition means 39 may synthesize all of the obtained plurality of composition target images, or one image for the photographed image. The compositing target image may be sequentially changed over time or repeatedly displayed.

  In addition, the image information extraction unit 38 described above acquires luminance information over time of a changed object, for example, every elapse of a predetermined time, and a synthesis target image corresponding to each luminance information over time as the luminance information changes. By sequentially extracting the images, the image composition means 39 can perform composition processing on the photographed image every time a composition target image is input, and can sequentially display the composition images. Accordingly, the image can be changed without moving the information providing medium, and a moving image or the like can be provided to the user.

  The transmission / reception means 40 is configured to display a desired external image (such as a photographed image or a compositing target image) from an external device (for example, the image management database 14) that can be connected using the communication network 15 or the like, or a composite image display according to the present invention. It is an interface for acquiring an execution program or the like for realizing processing.

  The control means 41 controls the entire components of the composite image display device 12. Specifically, the control unit 41 acquires, for example, a captured image, luminance information, print information, and thermal information, extraction of a synthesis target image, and image synthesis based on an instruction from the input unit 31 by a user or the like. Each control of each process is performed.

<About print information>
Here, a specific example of print information such as a frame 21 and character information 22 written on the card 20 in the present embodiment will be described with reference to the drawings.

  FIG. 3 is a diagram illustrating a specific example of print information. The print information shown in FIG. 3 is a composite image corresponding to a pattern by adding one or more types of patterns (lines are also included in one type of pattern) to the frame 51 as shown in FIG. Can be extracted. Note that by changing the thickness of the frame 51, a compositing target image corresponding to the thickness can be extracted.

  Further, as shown in FIG. 3B, the frame 52 itself prints character information, numerical information, and the like, whereby the character information and the like can be read and a corresponding composition target image can be acquired.

  Further, as shown in FIG. 3C, more information can be obtained by attaching a two-dimensional barcode 54-2 such as a one-dimensional barcode 54-1 or a QR code to part or all of the frame 53. It is possible to acquire various types of compositing target images. Therefore, as described above, the composition target image is changed and displayed in accordance with the change (difference) value of the luminance information of the change object obtained by the presence or absence of irradiation of light (ultraviolet light, infrared light, visible light, etc.) to the card 20. In addition to extracting one or a plurality of corresponding compositing target images from various types of compositing target images based on the print information and combining them with the photographed image, it is possible to provide a service with higher added value. Can do.

  Further, the print information described above may be visible or invisible. That is, by making it invisible, it can be used as security information such as authentication. In the present embodiment, the visible change is, for example, a phosphor for CRT (ZnS: Ag, Al, ZnS: Cu, Al, Y2O2S: Eu) or a lamp as a fluorescent paint manufactured by Nichia Corporation. Phosphor ((SrCaBaMg) 5 (PO4) 3Cl: Eu, LaPO4: Ce, Tb, Y2O3: Eu,), PDP phosphor (BaMgAl10O17: Eu, Zn2SiO4: Mn, (Y, Gd) BO3: Eu), X The phosphors for lines (CaWO4, Gd2O2S: Tb), phosphorescent phosphors (Sr4Al14O25: Eu, Dy), and Maruoka Educational Fluorescent / Night / UV paints are used as they are.

  Further, black light color, aqua glow, fluorescent paint spray, magic lumino paint, ultraviolet reflection paint, sun light marker, etc. can be used.

  Furthermore, LUMILUX Long After-Pigments, LUMILUX Effect-Pigments, LUMILUX C Pigments, LUMILUX Z Pigments, LUMILUX R Pigments, etc. can be used as fluorescent pigments from Honeywell.

  Moreover, as a nocturnal pigment of Nemoto Special Chemical Co., Ltd., N nocturnal (luminova) G-300, GLL-300, GSS, BG-300, BGL-300, V-300, B-300 can be used. Kasei Optonics True-Blue series fluorescent pigments, TASCO CO. , LTD thermochromic pigments, Tokyo Liquid Color, Asahi Products' Thermo-Temperature Paint Thermax Clear Line, etc. can be used. Furthermore, fluorescent dyes such as fluorescein, eosin, rhodamine B and thioflavine, azobenzene which changes from colorless to orange by ultraviolet light, and diallylethene which changes from colorless to red by ultraviolet light can be used. Furthermore, N, N-diethylethylenediamine copper complex (43 ° C. or less reddish purple, 43 ° C. or more dark purple) that changes color by heat, pyridinium N-phenoxide betaine (methanol: red, ethanol: purple, isobutyl alcohol) that changes color depending on the solvent : Blue), phenolphthalein that changes from colorless to basic and red. A quinone diimine that changes from yellow to red by an acid, and a material that turns blue when an acid is added can be used.

  Further, as the invisible change object, for example, titanium mica whose color changes according to an angle, a polymer multilayer film, or a retroreflective material can be used. Further, although the appearance is not known, the color may be visible depending on the angle, and these materials can also be used for the card 20. It is possible to use a paint whose luminance information hardly changes over time.

<About retroreflection>
In the present invention, more accurate information can be read by using a retroreflective sheet as the information providing medium. The contents will be described below. As an example, a method for reading information from a laminated material that is a retroreflective sheet will be described.

  The information reading method according to the present invention is characterized in that information recorded on a laminated material is read by irradiating the laminated material as an information providing medium with light and observing retroreflected light from the laminated material. This information reading method can be implemented by the apparatus described in FIG. In the composite image display system 10 shown in FIG. 1, the reading unit includes a light source 23 (white light source), a light detection unit (CCD camera) that is the imaging device 11, a composite image display device 12, and the like.

  The light emitted from the white light source 23 travels to the card 20 made of a retroreflective sheet. Light directed toward the card 20 enters the card 20. The retroreflected light from the card 20 goes to the CCD camera and is detected there. The detection signal detected by the CCD camera is sent to the composite image display device 12 as shown in FIG. The composite image display device 12 analyzes the information recorded on the card 20 based on this detection signal.

  In addition, in order to read information from the card | curd 20 which concerns on this embodiment mentioned above, it is necessary to observe from the substantially the same direction as the irradiation direction of light. Therefore, for example, a half mirror may be provided between the light source 23 and the card 20. Thereby, the CCD camera can be installed on the opposite side of the surface of the half mirror from the position where the light source is arranged, and the light source 23 and the CCD camera interfere with each other by using the half mirror. The retroreflected light can be preferably observed. In the present embodiment, for example, the position of the light source 23 and the position of the CCD camera may be replaced.

  Specific examples of the present invention will be described below, but the present invention is not limited to the following configurations.

<Example 1>
-Transparent resin Co., Ltd. Seiko Advance Co., Ltd. screen printing ink SG-410 (90 parts) and toluene (10 parts) were mixed to prepare a transparent paint.
-Preparation of plate for screen printing Using a nylon 150 mesh screen, a plate of 10 mm square QR type two-dimensional code (14 x 14 cells, code information: SHISEIDO) was prepared.
-Retroreflective material As a retroreflective material (microsphere fixed layer + reflective layer), an open-type retroreflective film (LIGHT FORCETM LFU-1200, manufactured by Maruhito Co., Ltd., reflective layer: aluminum vapor deposition, diameter of glass beads: about 40-50 μm , The burial rate of beads: about 50%).
-Printing method of information code Using the screen printing plate on the surface of the retroreflective material (the surface where the transparent microspheres are exposed), the thickness of the two-dimensional code (transparent resin layer) with a screen printer Was printed to be about 10 μm. The printed two-dimensional code was hardly visually recognized in appearance, and the design of the retroreflective material was not impaired.
-Reading analysis of code information Using the apparatus shown in Fig. 1, code information was read from the laminated material prepared above. Here, a CCD camera-attached code reader THIR-3000LP (manufactured by Tohken Co., Ltd.)) was used as the light detection means and the information code analysis means. As a result of reading and analyzing the code information, it was possible to read “SHISEIDO” actually recorded as code information.

<Composite Image Display System 10: Hardware Configuration>
Here, in the composite image display system 10 described above, an execution program (composite image display program) capable of causing a computer to execute each function is generated, and the execution program is installed in, for example, a general-purpose personal computer or server. Thus, the composite image display process and the like in the present invention can be realized.

  Here, a hardware configuration example of a computer capable of realizing the composite image display process in the present embodiment will be described with reference to the drawings. FIG. 4 is a diagram illustrating an example of a hardware configuration capable of realizing the composite image display process according to the present embodiment.

  4 includes an input device 61, an output device 62, a drive device 63, an auxiliary storage device 64, a memory device 65, a CPU (Central Processing Unit) 66 that performs various controls, and a network connection device. 67, which are connected to each other by a system bus B.

  The input device 61 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like. The input device 61 includes an input unit that inputs an image including an information providing medium photographed from an imaging unit such as a camera.

  The output device 62 has a display for displaying various windows and data necessary for operating the computer main body for performing processing according to the present invention, and displays the program execution progress, results, and the like by the control program of the CPU 66. can do.

  The input device 61 and the output device 62 may be integrated input / output means such as a touch panel. In this case, a predetermined position using a user's finger or a pen-type input device is used. You can input by touching.

  Here, the execution program installed in the computer main body in the present invention is provided by, for example, a portable recording medium 68 such as a USB (Universal Serial Bus) memory or a CD-ROM. The recording medium 68 on which the program is recorded can be set in the drive device 63, and the execution program included in the recording medium 68 is installed from the recording medium 68 to the auxiliary storage device 64 via the drive device 63.

  The auxiliary storage device 64 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, and the like, and can perform input / output as necessary.

  The memory device 65 stores an execution program read from the auxiliary storage device 64 by the CPU 66. The memory device 65 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 66 controls processing of the entire computer, such as various operations and input / output of data with each hardware component, based on a control program such as an OS (Operating System) and an execution program stored in the memory device 65. Thus, each process in the composite image display can be realized. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 64, and the execution result and the like can also be stored.

  The network connection device 67 obtains an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program or an execution in the present invention The program itself can be provided to other terminals.

  With the hardware configuration as described above, the composite image display process according to the present invention can be executed. Further, by installing the program, the composite image display process according to the present invention can be easily realized by a general-purpose personal computer or the like.

<Composite image display processing procedure>
Next, a synthetic image display processing procedure in the present embodiment will be described. FIG. 5 is a flowchart illustrating an example of a composite image display processing procedure in the present embodiment.

  In the composite image display process shown in FIG. 5, first, an image including an information providing medium photographed by an imaging means such as a camera is acquired (S01), the coordinates of the frame of the information providing medium are acquired (S02), and the current image is displayed. Luminance information (basic luminance value) is acquired (S03). In the process of S03, for example, the average luminance of a predetermined color (for example, green) within the frame is acquired, and the content is stored. Note that the average luminance may be set in advance in the above-described processes from S01 to S03.

  Next, for convenience of explaining the contents of the present invention, it is assumed that a light source such as a black light is irradiated on the information providing medium and a subsequent camera image is acquired. That is, the information providing medium irradiated with light of a preset wavelength from the light source is photographed by the camera, and an image from the camera is acquired (S04).

  Also, the coordinates of the frame included in the image are acquired (S05), and the luminance information after the light from the light source is irradiated on the acquired frame and the print information printed in the frame is acquired (S06). Note that the luminance information in S06 may be the luminance information of the coating area after the light from the light source is irradiated as described above, and is a difference value between the luminance information and the basic luminance value acquired in S03 described above. May be. By acquiring the difference value, the target image can be acquired without depending on the shooting environment of the information providing medium.

  If necessary, print information is analyzed by OCR or the like (S07), and thermal information is acquired by thermography or the like (S08).

  Here, it is determined whether or not to synthesize an image using one or a plurality of pieces of information among preset luminance information, print information, and heat information (S09). If the image is not synthesized (NO in S09), the process returns to S04 and continues. Further, when compositing images (YES in S09), a compositing target image is extracted using at least one of luminance difference, print information, and heat information (S10). Further, the extracted synthesis target image is synthesized with the photographed image (S11), and is displayed corresponding to the coordinate information of the frame from which the synthesized image was acquired (S12).

  Further, it is determined whether or not to end the process (S13). When the process ends (NO in S13), the process returns to S04 and continues. When the process is to be ended (YES in S13), the entire composite image display process is ended.

  By the processing described above, a highly accurate image with excellent added value can be provided. In the above-described processing, an input image or an image obtained by each processing can be input to the storage unit 33. Further, in each process, the output unit 32 or the like may be used to acquire an input image, an execution progress or result of each process, a case where an instruction is given to the user by the input unit 31, and the like.

<Example 2>
Next, as Example 2, an example of an image displayed in the present embodiment will be described with reference to the drawings. FIG. 6 is a diagram showing an example of a composite image displayed in the present invention.

  First, in the example shown in FIG. 6A, a frame 71 is printed on the card (information providing medium) 70. At this time, the composite images 72-1 and 72-2 acquired from the database or the like using the luminance information of the print information (frame, character information, etc.) obtained from the first photographed image is the coordinate information of the frame 71. Correspondingly displayed on the screen. In FIG. 6A, as an example, a flower pot (composite image) 72-1 and a seedling (composite image) 72-2 planted in the flower pot (composite image) 72-1 are displayed.

  Next, the luminous paint (Nemoto Special Chemicals Co., Ltd. Luminova G-300) was printed on the part shown in FIG.6 (b). Here, by irradiating a light source such as a black light, the luminance information of the paint on which the print information is printed changes, so that instead of the seedling (composite image) 72-2 shown in FIG. ) 72-3 could be displayed.

  As described above, in this embodiment, various types of images can be displayed by combining and displaying the type of the synthesis target image corresponding to the luminance information. In addition, by properly using a paint whose luminance changes with time, for example, a moving image in which a seedling grows, blooms, and then withers can be provided. In this embodiment, the orientation of the image and each displayed image is changed according to the position of the corner of the frame, the display method such as enlargement or reduction is changed, or the corresponding sound is displayed. You can also.

<Other embodiments>
Here, in the above-described embodiment, it is possible to capture a change object (two-dimensional barcode, etc.) such as print information printed on an information providing medium such as a card, and analyze the print information from the captured image. However, the present invention is not limited to this. For example, the information providing medium is a display device such as a display, and an image obtained by photographing the above-described change object displayed on the display is displayed. By acquiring and analyzing a change object included in the acquired image, it is possible to extract a corresponding stereoscopic image and display a combined image obtained by combining the captured image with the captured image. In this embodiment, for example, a change object displayed on the screen by, for example, video display or slide display can be easily switched over time, and the photographing side described above includes the image including the change object displayed on the screen. By shooting in this manner, a stereoscopic image corresponding to the changed object can be synthesized and displayed as a moving image.

  Furthermore, in this embodiment, as described above, not only the information printed or displayed on the information providing medium is used, but also, for example, a shadow projected on the ground or a wall by illuminating is used as a change object, and the shadow is converted into a change object. Projecting with additional information such as a two-dimensional code, and analyzing the shadow from the image obtained by photographing the shadow, the corresponding stereoscopic image was extracted, and the image was synthesized with the photographed image A composite image can be displayed. Further, the image to be extracted may be changed depending on the shadow density or the like.

  As described above, according to the present invention, it is possible to provide a highly accurate image or service excellent in added value. Further, by irradiating light from a light source such as a light having a preset wavelength, a target image can be provided to the user. Also, various composite images can be provided by changing the set wavelength.

  Further, it is possible to provide a composite image without depending on the surrounding environment where the information providing medium is photographed. Further, when the luminance information changes, another image to be combined can be selected and combined and displayed. Thereby, a moving image etc. can also be provided. In addition, more combined images can be provided using posture information including at least one of the position, angle, and orientation of the information providing medium, and a service with higher added value can be provided. Furthermore, by using the thermal information, it is possible to provide a service according to the environment at that time. Even if the paint is invisible, a composite image can be provided by using thermal information.

  Note that the above-described apparatus can also be applied to a mobile terminal or the like. Therefore, by applying the present invention, for example, by combining with a gift card or the like, a card service in which a flower blooms when hitting the card against the sun, an Internet authentication field, a food inspection field such as food freshness analysis, It can also be applied to the field of health checkups that require color judgment such as urine tests.

  In other words, in the present invention, for example, a coating material that reacts to ultraviolet light or light source power (light intensity) is applied to the information providing medium, and the information providing medium is set in a portion exposed to external light such as sunlight. Is captured by a photographing device, and image data is acquired over time and input to the system. In addition, the images in the PC and the mobile terminal change realistically according to time and weather. Thereby, it is possible to obtain an image suitable for the season, climate, etc. even in a basement or a room inside the building where outside light does not reach, or in a place that is actually far away.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

It is a figure which shows an example of the mechanism structure of the synthesized image display system in this embodiment. It is a figure which shows an example of a function structure of a synthesized image display apparatus. It is a figure which shows the specific example of printing information. It is a figure which shows an example of the hardware constitutions which can implement | achieve the composite image display process in this embodiment. It is a flowchart which shows an example of the composite image display process sequence in this embodiment. It is a figure which shows an example of the synthesized image displayed in this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Composite image display system 11 Imaging device 12 Composite image display device 13 Portable terminal 14 Image management database 15 Communication network 20,70 Card (information provision medium)
21, 51, 52, 53, 71 Frame 22 Character information 23 Light source 31 Input means 32 Output means 33 Storage means 34 Captured image acquisition means 35 Luminance information acquisition means 36 Print information analysis means 37 Thermal information acquisition means 38 Image information extraction means 39 Image composition means 40 Transmission / reception means 41 Control means 54 Bar code information 61 Input device 62 Output device 63 Drive device 64 Auxiliary storage device 65 Memory device 66 CPU
67 Network connection device 72 Composite image

Claims (17)

In a composite image display system that combines and displays another image at a predetermined position of a photographed image,
Imaging means for imaging an information providing medium in which predetermined information is recorded by a change object whose luminance changes with the passage of time after irradiation of light set in advance;
Luminance information acquisition means for acquiring luminance information of the change object included in the captured image captured by the imaging means;
Image information extraction means for extracting an image corresponding to the luminance information among a plurality of synthesis target images accumulated in advance using the luminance information obtained by the luminance information acquisition means;
The synthesized composited image extracted by the image information extracting means and the photographed image, possess an image synthesis means for displaying a composite image,
The image information extraction means sequentially extracts a synthesis target image corresponding to each luminance information for each predetermined time with a change in luminance information over time of the change object,
The composite image display system , wherein the image composition unit performs composition of the photographed image and the composition target image every time the composition target image is extracted by the image information extraction unit . The luminance information acquisition means includes
The composite image display system according to claim 1, wherein the luminance information is an average luminance value of the region of the change object included in the image. The luminance information acquisition means includes
The brightness information of the change object after being irradiated from at least one light source among ultraviolet light, infrared light, and visible light having a preset wavelength is acquired. Composite image display system. The luminance information acquisition means includes
4. The basic brightness value for the change object is set in advance, and a difference value between the brightness information of the change object and the basic brightness value obtained from the captured image is acquired as brightness information. The composite image display system according to claim 1. Print information analysis means for analyzing the print information recorded on the information providing medium by the change object,
The image information extraction means includes
The luminance information obtained by the luminance information obtaining unit, wherein based on the printing information obtained by the print information analysis unit, to any one of claims 1 to 4, characterized in that extracting the target image to be combined The described composite image display system. The print information is
6. The composite image display system according to claim 5 , wherein the composite image display system is at least one of a character, a figure, a pattern, an image, and barcode information. The image composition means includes
Based on the posture information including at least one of the position, angle, and orientation of the information providing medium included in the captured image, and the luminance information, display, non-display, size, position, the composite image display system according to any one of claims 1 to 6, characterized in that to control the direction. Heat information acquisition means for acquiring heat information of the change object,
The image information extraction means includes
The target image to be synthesized is extracted using at least one of the luminance information obtained by the luminance information obtaining unit, the printing information obtained by the printing information analyzing unit, and the thermal information obtained by the thermal information obtaining unit. The composite image display system according to claim 5 , wherein: In a combined image display method of combining and displaying another image at a predetermined position of a photographed image,
An imaging step of imaging an information providing medium in which predetermined information is recorded by a change object whose luminance changes with the passage of time after irradiation with light set in advance;
Luminance information acquisition step for acquiring luminance information of the change object included in the captured image captured by the imaging step;
An image information extraction step for extracting an image corresponding to the luminance information among a plurality of synthesis target images accumulated in advance using the luminance information obtained by the luminance information acquisition step;
The image information extraction compositing target image extracted in step and said captured image synthesizing, possess an image synthesizing step of displaying a composite image,
The image information extraction step sequentially extracts a synthesis target image corresponding to each luminance information for each predetermined time with a change in luminance information over time of the change object,
The combined image display method characterized in that the image combining step combines the captured image and the combination target image every time the combination target image is extracted by the image information extraction step . The luminance information acquisition step includes:
The composite image display method according to claim 9 , wherein the luminance information is an average luminance value of the region of the change object included in the image. The luminance information acquisition step includes:
UV light of a wavelength that is set in advance, infrared light, and among the visible light, according to claim 9 or 10, characterized in that to obtain the luminance information of the change of after being irradiated from at least one light source Composite image display method. The luminance information acquisition step includes:
The basic luminance value set in advance with respect to said change thereof, any of claims 9 to 11, characterized in that to obtain a difference value between the luminance information and the basic luminance value of the change obtained from the captured image as luminance information The synthetic image display method according to claim 1. A printing information analysis step of analyzing printing information written on the information providing medium by the change object;
The image information extraction step includes:
The luminance information obtained by the luminance information acquisition step, wherein on the basis of the print information analysis printing information obtained by step, extracting the target image to be combined to any one of claims 9 to 12, characterized in The composite image display method as described. The print information is
The composite image display method according to claim 13 , wherein the composite image display method is at least one of a character, a figure, a pattern, an image, and barcode information. The image composition step includes
Based on the posture information including at least one of the position, angle, and orientation of the information providing medium included in the captured image, and the luminance information, display, non-display, size, The composite image display method according to claim 9 , wherein the position and direction are controlled. A thermal information acquisition step of acquiring thermal information of the change object,
The image information extraction step includes:
Extracting a target image to be synthesized using a plurality of pieces of information among the luminance information obtained by the luminance information obtaining step, the thermal information obtained by the thermal information obtaining step, and the printing information obtained by the printing information analyzing step. The composite image display method according to claim 13 . A composite image display program for causing a computer to function as each unit included in the composite image display system according to any one of claims 1 to 8 .

Images